MlxRing hostfile uses the nodes' LAN addresses even when a Thunderbolt link is available (pipeline over Wi-Fi: 8.7 vs 25.4 tok/s)

[zhast]

Summary

For an MlxRing (TCP) instance, the hostfile written for MLX_HOSTFILE lists each node's LAN address. On a cluster whose LAN is Wi-Fi and whose fast path is Thunderbolt, that routes every per-token hop and synchronisation over Wi-Fi even though the topology already knows the Thunderbolt link-local addresses (they are used for RDMA edges).

Measurement

Four Mac Studio M3 Ultra, Thunderbolt 5 mesh, LAN on Wi-Fi. GLM-5.3-Flash (110 GB), Pipeline ×4, 400-token generations:

backend	one request	16 concurrent
MlxJaccl (RDMA over Thunderbolt)	25.4 tok/s	69.1 tok/s
MlxRing (TCP, hostfile = LAN addresses)	8.7 tok/s	32.8 tok/s

Round-trip latency from one node to its peers during the run:

LAN  (10.x.x.x, en1 = Wi-Fi)          4.9 – 54 ms   (avg 7.5 – 30 ms)
Thunderbolt bridge (169.254.x.x)      0.38 – 0.95 ms

The runner's hostfile:

rank 1 hostfile: .../hosts_<id>_1.json hosts: ["10.x.x.x:63673","0.0.0.0:63673","10.x.x.x:63673","198.51.100.1:0"]

A pipeline makes only a few synchronising round trips per token (the hand-offs plus the per-step task-agreement all-gather), but at 10–30 ms each that is more than the ~37 ms of compute per token, hence the 3× gap. The same comparison on a 27B model in August (when the TCP path evidently ran over a wired link) showed 12.3 vs 12.4 tok/s for the pipeline, so this is about which interface the ring is given, not about TCP itself.

Where

utils_mlx.py mlx_distributed_init writes hosts_by_node[bound_node_id] from MlxRingInstance.hosts_by_node, which the master fills in at placement from the nodes' advertised addresses. The topology edges for the same node pairs carry sinkMultiaddr entries with the 169.254.x.x Thunderbolt addresses, so the information to prefer them is already present.

Suggested fix

When building hosts_by_node for an MlxRing placement, prefer an address on an interface that the topology shows as a direct (Thunderbolt) link to the peer, falling back to the advertised LAN address. That would make the "ordinary TCP" backend run at wired speeds wherever RDMA does, and make ring-vs-RDMA comparisons meaningful.

I have not attempted the change because my cluster is currently degraded (a separate issue, see #2294); happy to help test.

[maximilliangrand]

I traced this on current main (21a54c5ea0230a3bec1e1a786d200126c7e34ec6) and exercised get_mlx_ring_hosts_by_node with a four-node topology containing both Wi-Fi and Thunderbolt socket edges.

There is already a per-peer preference in the placement path: get_mlx_ring_hosts_by_node calls find_ip_prioritised(..., ring=True), which gives thunderbolt priority 0 and wifi priority 3. With the matching node_network interface metadata present, all eight neighbor entries select the Thunderbolt addresses, independent of whether the LAN or Thunderbolt edges were inserted first. Self-binding and non-neighbor placeholders also remain correct.

The same test with missing network metadata can select LAN instead: both candidate IPs become unknown, and the tie follows topology edge order. This is a reproducible failure condition, but I have not established that it is the condition on your cluster. A 169.254 prefix alone is not sufficient evidence of Thunderbolt, so I would avoid a subnet-based preference.

Could you share a small, anonymized placement-time example for one affected source/peer pair: the socket edges between them, the peer's nodeNetwork.interfaces entries (names, address/type mapping), and the selected hostfile entry, plus the EXO revision? Consistent address substitutions are enough; a full state dump is unnecessary. That should distinguish missing/mismatched interface metadata, missing TCP reachability edges, and an instance created before the Thunderbolt path was discovered.

I also found #2254, which adds measured-latency selection. I'd like to isolate the remaining cause here before proposing overlapping code. I can prepare a targeted regression and fix once the failing state is identified; I do not currently have a multi-Mac cluster to substantiate an inference-speed claim.