Refine(IB): ib connect using 1byte probe msg instead of empty payload.

[SimonCqk]

Fix #320

We observed that RDMA connections consistently fail between different VMs. However, RDMA communication works when performed on the same host (loopback through the same physical NIC). After adding extensive debug logging, we traced the problem to the connection setup phase. It appears that the transport layer in these specific virtualized environments silently drops empty (0-byte) RDMA packets, in the current 3FS implementation, the IBConnect class sends a 0-byte event to notify the peer as part of the connection handshake. This 0-byte packet is the one being dropped, and as a test, we modified the implementation to send a 1-byte event instead of a 0-byte one.
With this change, the RDMA connection is established successfully across different VMs, and the system works as expected in both native IB bare-metal environment and cloud-vendor virtualized RDMA environment.

[SimonCqk]

@SF-Zhou hi, please take a look when you have a chance, many thanks!

[Copilot]

Pull Request Overview

This PR fixes RDMA connection failures in virtualized environments by replacing empty (0-byte) packets with 1-byte probe messages during the InfiniBand connection handshake. The issue was traced to certain virtualized RDMA environments silently dropping empty packets, preventing successful connection establishment between VMs.

• Modified IBConnect to send 1-byte probe messages instead of empty payloads during connection setup
• Updated IBSocket message handling to properly consume and discard the 1-byte connection probe messages
• Added comments explaining the rationale for using 1-byte messages in virtualized environments

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/common/net/ib/IBConnect.cc	Modified connection logic to send 1-byte probe message and updated comments
src/common/net/ib/IBSocket.cc	Added logic to consume and discard connection probe messages without passing to application

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[SF-Zhou]

After our discussion, the revised plan itself is not problematic, but compatibility issues need to be considered. If the server is upgraded separately, the client will receive a message of length 1, which breaks compatibility. Sending an immediate message could be considered to resolve this issue without breaking compatibility.

[SimonCqk]

After our discussion, the revised plan itself is not problematic, but compatibility issues need to be considered. If the server is upgraded separately, the client will receive a message of length 1, which breaks compatibility. Sending an immediate message could be considered to resolve this issue without breaking compatibility.

Exactly, I'll push a commit to address this shortly.