How does this work?

[DUOLabs333]

I'm looking to do something similar (communicate over a shared block device), but I couldn't figure out a way to synchronize reads/writes without using an auxiliary communication channel (like an existing TCP connection).

[fiddyschmitt]

Hi,

Could you please tell me a bit more about your setup? Your OS, shared device, programs you're trying to bridge.

[DUOLabs333]

Oh, I meant that I wanted to write my own program to do this (so I wanted to understand how you got around the synchronization issue), not that yours was broken on my machine.

In any case, I'm communicating between a macOS host and a Linux guest running on QEMU. The block device is a ramdisk shared between the two machines.

[fiddyschmitt]

Oh cool. Yes synchronization took the most effort. In the end, the key to it was:

1. The reader and writer have to open the shared file in a way that's tolerant of other processes having it open (See here).
2. Using a preallocated size (eg. 10 MB) for the shared file was problematic. It's better for the writer to start with an empty file and append to it. That made the reader much more reliable when it came to using PeekChar to sense when data was available.
3. Recycling the file took a lot of effort, but in the end it worked reliably by having the writer and reader do a special dance to coordinate the file purge.

I'm sure .NET is also doing a lot of things under the hood for reading and writing the shared file.

If there's a particular use case for your macOS / Linux setup that File Tunnel doesn't support, let me know and I might be able to help.

Cheers!
Fidel

[DUOLabs333]

I guess the main issue would be being able to multiplex multiple connections over the same file --- I saw an issue about the same problem, but I don't know if there's been any work on that.

[DUOLabs333]

Also, in my case, I have access to two shared files, but not folders. Will that be enough?

[fiddyschmitt]

I guess the main issue would be being able to multiplex multiple connections over the same file --- I saw an issue about the same problem, but I don't know if there's been any work on that.

If it's for the same destination, the program already supports multiplexing multiple connections using the same files. You just need to use --tcp-listen 0.0.0.0:5000 and any computer can connect to the tunnel.

For multiple destinations, I'm currently making an update to support SSH-style args:
eg.
-L 5000:192.168.1.20:3389 -L 5001:192.168.1.21:3389
Creates two local listeners on ports 5000 & 5001, and sends them onto 192.168.1.20 and 21.

[fiddyschmitt]

Also, in my case, I have access to two shared files, but not folders. Will that be enough?

Yes the program only accepts files to read & write from. ie. it doesn't create random files in a directory for each connection.

[DUOLabs333]

When you say a file gets "purged", do you mean it gets deleted or resized? In either case, is it possible for File-Tunnel to work on preallocated files of a certain length without changing its length?

[fiddyschmitt]

Initially I purged files by deleting - it worked fine but it was slow. I then tried the Truncate operation - that worked well for SMB and NFS but isn't supported by RDP. In the end I settled on resizing (done here). It works well and is much faster.

I did implement preallocated files at one point. When the end of the file was reached, the writer would start writing from byte 0. This worked fine 99% of the time, but occasionally the reader would read from the file, only to get content from the previous pass (even though the writer had explicitly cleared it). It was as though the reader was caching, despite args telling it not to.

I guess preallocated files could be made reliable, by signaling when commands are "ready", and when commands have been "processed". This could be done using flag bytes as is done for purging here. But it would likely come with a big impact to performance.

[kim0]

@fiddyschmitt since you mentioned you'd like to support -L forwarding similar to ssh. It would be awesome to allow adding more port forwards dynamically while the tunnel is running. ssh does this by reading stdin, and if it sees \n~C it allows the user to enter a port forward like -L 123:10.1.2.3:456 and that gets forwarded without affecting running tunnels :)

[fiddyschmitt]

Love it! I'll try to implement that too when I get a chance.

I'm also considering adding a SOCKS Proxy feature soon - that would also provide dynamic port forwarding.

[kim0]

Amazing. While socks is awesome in its own right, it still requires that the client tool supports socks proxies, and needs me to configure it to use it. So being able to add some forwards manually dynamically still has value. Good job so far!

[fiddyschmitt]

Okay, summary:

1. File Tunnel v2.2.0 now supports multiple forwards using SSH syntax: -L 5000:192.168.1.20:3389 -L 5001:192.168.1.21:3389.
2. It also supports remote tunneling using -R, as described here.
3. SOCKS Proxy Servers are easy to set up in both Windows and Linux, so I won't implement SOCKS in File Tunnel for now.
4. I'd still like to add the ability to add new tunnels at runtime, by reading stdin as @kim0 suggested. Tracking that over here.

[fiddyschmitt]

Back to you @DUOLabs333. Did you want to discuss anything else or are you happy for this issue to be closed?

[DUOLabs333]

No, I guess not --- it would be nice if preallocated files were supported (I'm not sure how well QEMU supports shrinking and growing drives), but it seems that caching can interfere with that use case (and could probably be its own issue).

[fiddyschmitt]

Cool. Thanks for the suggestion.
Tracking it here:
Add ability to use a preallocated file

[fiddyschmitt]

I use VirtualBox and use a Shared Folder between the host and guest.

• In a Windows guest, I can mount the shared folder as a drive letter (after Guest Additions are installed).

• In a Linux guest, I can mount the shared folder using: mount -t vboxsf shared_folder_name /media/host

In both cases, File Tunnel works fine over the shared folder, including the file resize it does when it reaches 10 MB.

I'm sure QEMU would handle it just as well, over virtiofs or 9p.

[DUOLabs333]

Oh, I was planning to share two files with the guest as drives directly --- I found that 9p (macOS hosts don't have support for virtiofs) introduces too much overhead, and is much slower than sharing block devices directly (I'm hoping for a ~8 Gbps link speed).

[fiddyschmitt]

Interesting - what is the mechanism QEMU uses to use a host file as a guest drive?
And what does the guest do to mount it?

[DUOLabs333]

You just pass a file in with -drive. The guest should then be able to use the drive like any other file.

[fiddyschmitt]

The file that gets passed in using -drive, isn't it created with qemu-img create and therefore already has content (img format, partitions, file system)?

[DUOLabs333]

No, you can just create another file, and just pass that in without formatting it or mounting it.

Example:
dd if=/dev/zero test (I forgot the exact syntax)

-drive if=none,file=./test,format=raw,index=3,media=disk,id=drive3,cache=writethrough
-device virtio-blk-pci,addr=0x0.0x7,backend_defaults=on,bus=pcie,drive=drive3

[fiddyschmitt]

How does it appear to the guest OS, and how do you interact with it in the guest?

[DUOLabs333]

See my edit. The guest can just use this as a normal file.

[fiddyschmitt]

Very interesting.

There's a few of things to investigate:

1. When the file is in use by QEMU, can another program write to the file?
2. If the host writes to the file, can the guest read the file and see what was written?
3. If the guest writes to the file, can the host read from the file and see what was written?

[DUOLabs333]

Yes to all three --- as a test, I allocated a file, shared it, and wrote "hi" to it on the guest. I tried to read from the file on the host and it worked. I tried the same test in reverse (write on host, read on guest), and it also worked.

[fiddyschmitt]

So good!!

Okay I'll reimplement preallocated files.

[DUOLabs333]

Haha... no --- I'm aiming for high-throughput for my networked GPU driver, and I found that in almost all cases, the network is the bottleneck. I found that 5 Gbps (achieved using a tap device) is almost good enough, so I inferred that a higher link speed will make it even faster.

[fiddyschmitt]

Woah!! Very cool!

[fiddyschmitt]

To improve throughput, try giving File Tunnel a higher value, such as --read-duration 1000.

This increases bandwidth but it comes at the cost of increased latency.

[DUOLabs333]

I switched back to using singular drives (and using the version with preallocation support), but it seems that File-Tunnel has trouble with it --- on the host, I get Receive file (disk5) is not established, and on the guest, I get SendPump: Invalid argument. These are both referring to the same file. As a quick check, I made sure that I can echo back and forth through the file.

[fiddyschmitt]

Thanks. I've just uploaded a new version, which will provide us more detail about why SendPump is returning 'Invalid argument'

https://github.com/fiddyschmitt/File-Tunnel/releases/tag/v2.2.0_2024-08-16_2315

[DUOLabs333]

Never mind --- it turns out I was using the stable version by accident on the guest, so it likely tries to wipe the file; however, since it is a drive, QEMU doesn't allow that.

Using the proper version gets past that error, but now I just get "offline" on both sides. On the host, I still have the additional message Receive file (disk5) is not established.

[fiddyschmitt]

Could you please post both ft commands you're running?

[DUOLabs333]

Host: ./ft-osx-arm64_* --write /dev/disk4 --read /dev/disk5
Guest: ./ft-linux-arm64_2024-08-17_1225 --read /dev/disk/by-id/virtio-conn-write --write /dev/disk/by-id/virtio-conn-read

[fiddyschmitt]

Thanks, so virtio-conn-write on the host is being mounted as /dev/disk4 in the guest?

[DUOLabs333]

Yes.

[DUOLabs333]

If I switch to using regular files on the host, I no longer get the "not established" message, but both sides are still offline.

[fiddyschmitt]

Hmm try deleting and recreating the files. The stable and experimental builds use the files quite differently, and the existing content could be interfering.

[DUOLabs333]

I zeroed out the files before starting again --- same problem.

[fiddyschmitt]

So the experimental build does not work on the 9p shared files, nor the --drive files?

[DUOLabs333]

Yes, it does not work.

[fiddyschmitt]

Okay I'll set up QEMU on my machine tomorrow. Busy all today

[fiddyschmitt]

Hey,

On my setup (Debian 12 Host, Debian 12 Guest) both virtiofs and virtio-9p shared folders work fine, using stable & preallocated versions.

I created the VM using Virtual Machine Manager, and it launched this process:

/usr/bin/qemu-system-x86_64 -name guest=debian11,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-10-debian11/master-key.aes"} -machine pc-q35-7.2,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -accel kvm -cpu host,migratable=on -m 2048 -object {"qom-type":"memory-backend-memfd","id":"pc.ram","share":true,"x-use-canonical-path-for-ramblock-id":false,"size":2147483648} -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 52c1ed32-7d37-49f8-9dbb-3fc518fe99c3 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=34,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device {"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"} -device {"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"} -device {"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"} -device {"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"} -device {"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"} -device {"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"} -device {"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"} -device {"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"} -device {"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"} -device {"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"} -device {"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"} -device {"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"} -device {"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"} -device {"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"} -device {"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/debian11.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","driver":"qcow2","file":"libvirt-2-storage","backing":null} -device {"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":2} -blockdev {"driver":"file","filename":"/home/smith/iso/clonezilla-live-3.1.2-22-amd64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"} -device {"driver":"ide-cd","bus":"ide.0","drive":"libvirt-1-format","id":"sata0-0-0","bootindex":1} -chardev socket,id=chr-vu-fs0,path=/var/lib/libvirt/qemu/domain-10-debian11/fs0-fs.sock -device {"driver":"vhost-user-fs-pci","id":"fs0","chardev":"chr-vu-fs0","tag":"sharedfolder1","bus":"pci.7","addr":"0x0"} -fsdev local,security_model=mapped,id=fsdev-fs1,path=/home/smith/for_debian_11_vm -device {"driver":"virtio-9p-pci","id":"fs1","fsdev":"fsdev-fs1","mount_tag":"sharedfolder2","bus":"pci.8","addr":"0x0"} -netdev {"type":"tap","fd":"35","vhost":true,"vhostfd":"37","id":"hostnet0"} -device {"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:67:a9:1c","bus":"pci.1","addr":"0x0"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev socket,id=charchannel0,fd=33,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -chardev spicevmc,id=charchannel1,name=vdagent -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"com.redhat.spice.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"spice"} -spice port=5900,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on -device {"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"} -device {"driver":"ich9-intel-hda","id":"sound0","bus":"pcie.0","addr":"0x1b"} -device {"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"} -chardev spicevmc,id=charredir0,name=usbredir -device {"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"} -chardev spicevmc,id=charredir1,name=usbredir -device {"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"} -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"} -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device {"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on1

[DUOLabs333]

Interesting --- it's the same setup I have (except for the virtiofs --- it's not available on macOS).

[DUOLabs333]

I think the reason the --drive option isn't working well with FT is because since both sides are drives, C# probably can't get the right size (if you try to seek to the end of the block device in C, you get a size of 0, regardless of how big the actual drive is).

UPDATE: Yeah, that's the problem. There's no good way to fix this other than using OS-specific methods.

[fiddyschmitt]

Interesting! I'll take a look tonight.

[DUOLabs333]

I switched back to using 9p (it turns out that trying to use drives directly is a recipe for disaster), and it's interesting that the host does notice when the guest attaches/detaches, but the guest can not detect the host.

[fiddyschmitt]

Interesting. 9p had high latency for me (800 ms) whereas virtiofs was fine (5ms). I'll get bandwidth measurements tonight

…

On Thu, 22 Aug 2024, 06:50 DUO Labs, ***@***.***> wrote: I switched back to using 9p (it turns out that trying to use drives directly is a recipe for disaster), and it's interesting that the host does notice when the guest attaches/detaches, but the guest can not detect the host. — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADVA3TCZ5QXKW5T6T4DGQOLZST4QJAVCNFSM6AAAAABMGRRNA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBSHE4DSMZWGY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[DUOLabs333]

Sorry I haven't been responding for a while --- I was writing an alternative POC of the same idea (using files to share data). Now that I have a prototype working (though it's slower than plain TCP), I think I know why FT hasn't been working well with macOS <-> Linux --- it was on Linux's side, or more precisely, it was due to my setup. Since I mounted the files as drives in Linux, if a change is made on the drives by QEMU, Linux is not notified that something changed. Therefore, we have to manually notify the OS to check the file/drive for updates with posix_fadvise.

[fiddyschmitt]

Hi @DUOLabs333, I just released v2.2.3 which adjusts how Linux guests read the shared file. If able, could you please give it a try with your setup which wasn't getting sync? Thanks

[DUOLabs333]

Cool! I'll look at this again.

[DUOLabs333]

Running FileTunnel on Linux gives me SendPump: Invalid argument --- I'm guessing it's because C# can not open block devices.

[YourSandwich]

I don't have that issue. What command are you exactly running?

Running FileTunnel on Linux gives me SendPump: Invalid argument --- I'm guessing it's because C# can not open block devices.

I have just packaged ft for ArchLinux https://aur.archlinux.org/packages/ft-bin

[DUOLabs333]

@YourSandwich I'm running ./ft-linux-arm64 --read /dev/disk/by-id/virtio-conn-write --write /dev/disk/by-id/virtio-conn-read.

[YourSandwich]

Ok, yeah, it cannot read block devices.