Tensor-parallel (MlxJaccl) output corrupts on prompts >~4k tokens — 2× M3 Ultra 512GB, EXO 1.0.71. Is there a newer build with the JACCL/prefill fixes?

[nachogala]

Setup

• 2× Mac Studio M3 Ultra 512GB, direct Thunderbolt 5 cable (80 Gb/s link), RDMA enabled on both (rdma_ctl enabled), macOS 26.5.2
• EXO 1.0.71 (EXO-latest.dmg from assets.exolabs.net — same as the latest GitHub release)
• Model: GLM-5.2-DQ4plus-q8 (433GB, mlx-community), local weights, Tensor sharding, MlxJaccl, both nodes
• Cluster forms cleanly: 2 nodes, 2 connections, jaccl devices [[null,"rdma_en3"],["rdma_en3",null]]

Symptom

Needle-in-haystack test at temperature 0 (N sentences "Section i has internal code C(3i+1)", ask for the code of section N/2):

prompt tokens	result
~370	correct
~2,100	correct
~2,650	correct
~4,500	garbage ("0.0.00…")
~7,300	garbage
~15,000	garbage

Short prompts and decode are always clean and fast. Above ~4k prompt tokens the output degrades to token salad, deterministically.

We see the same corruption pattern (slightly lower threshold) running bare mlx.launch --backend jaccl + mlx_lm.server outside EXO, so this looks like the upstream MLX JACCL issue rather than EXO's own code — filed with full repro as ml-explore/mlx#4342, likely same root as ml-explore/mlx#3149 (consecutive send/recv with different shapes produces wrong data).

Question

The public 1.0.71 build predates several main-branch changes that look relevant: "Misc fixes: upstream JACCL all_sum" (#1952), "MLX P/D" (#1993), and the libp2p→zenoh migration (#2132). Recent press coverage of 4-node clusters also mentions patched builds beyond the public DMG.

• Does current main (or an internal build) fix long-prompt corruption over JACCL tensor parallelism?
• If so, is a new DMG release planned, or is building from source the recommended path today?

Happy to run any diagnostic or test builds on this 2-node setup — it reproduces in minutes.

[nachogala]

Update: we also built and tested main@June (zenoh transport) with mlx 0.32.0.dev20260708 on the same 2-node setup, and updated both machines to macOS 26.6.2 — the corruption above ~2.6k prompt tokens is unchanged and byte-identical in every combination (Tensor and Pipeline sharding, fast-synch on/off, 1.0.71 release and source). So it's not the OS driver and not fixed by the post-1.0.71 commits we could see.

Two extra notes from running main on 2 nodes, in case useful:

• Cluster formation on main needed workarounds on our LAN: discovery announce is multicast-only (one of our nodes doesn't receive multicast — we added unicast Hello to static peers via env) and the discovered peer is never dialed by zenoh (we added connect/endpoints injection). Election clocks also prevent merging clusters with divergent event logs.
• Short-prompt/long-generation works great — 19 tok/s decode on a 433GB GLM-5.2 q8 across 2× M3 Ultra. It's specifically multi-chunk prefill that corrupts.

Since press coverage mentioned patched builds beyond the public DMG: does your internal build address long-prompt corruption over JACCL? Glad to test anything — repro is ~2 min here.

[goransh-walia]

This is a known pattern that's worth having documented clearly in the issue tracker.