fix(discovery): multicast discovery never forms a cluster on macOS

[zhast]

fix(discovery): multicast discovery never forms a cluster on macOS

Summary

On macOS, nodes never discover each other and every node reports a topology of
one. Three independent defects combine to cause it: two restrict which
interfaces are announced on, and a third stops announcing altogether after the
first discovery. This PR fixes all three.

Verified on 4× Mac Studio (M3 Ultra, macOS 26.6.2) connected by both wifi and a
full Thunderbolt 5 mesh.

Defect 1 — AddrInUse silently drops interfaces

rust/networking/src/discovery.rs, in the netwatcher callback:

match sock.join_multicast_v6(&GROUP, *iface_idx) {
    Ok(()) => ifaces.lock().push(/* ... */),
    Err(e) if e.kind() != io::ErrorKind::AddrInUse => { warn!(/* ... */) }
    _ => {}          // AddrInUse lands here — interface never registered
}

Multicast membership is held per-socket, so on macOS the second and every
subsequent join_multicast_v6 for the same group returns AddrInUse. The
existing comment ("skip AddrInUse - just means we've already joined the mv6")
shows the intent was to tolerate it, but because the guard excludes AddrInUse
the value falls through to _ => {} and the interface is never pushed onto
ifaces.

Since announce() iterates ifaces, only the first interface to join
successfully ever receives a Hello.

Defect 2 — the multicast egress interface is never set

announce() sends to SocketAddrV6::new(GROUP, port, 0, iface_idx), relying on
the scope id to select the outgoing interface. That is not sufficient for IPv6
multicast: IPV6_MULTICAST_IF must be set on the socket before each send.
Without it every datagram leaves via the default multicast interface no matter
which scoped address it was sent to. The socket only ever calls
set_multicast_loop_v6.

Defect 3 — a blocking connect_peer silences discovery permanently

rust/networking/src/lib.rs. Discovery::next() is the only driver of
announce() — it announces on each 1s tick and returns as soon as a peer is
discovered. The task that consumes it awaits connect_peer inline:

let discovered = discovery.next().await;   // announces while polling
// ...
runtime.connect_peer(&discovered.zid.into(), &[locator]).await;   // blocks here

connect_peer can block for a long time — indefinitely, when a peer is
unreachable or its handshake stalls. While it is blocked the task never returns
to next(), so the node stops announcing entirely, and never resumes.

This is the defect that actually prevents the cluster forming. Fixing defects 1

---

Full findings, including the GLM-5.3-Flash / glm5_next bring-up and the out-of-tree patches, are in PR-DRAFT.md and patches/ on this branch.

🤖 Generated with Claude Code

[zhast]

Follow-up, to close out the open question this PR left and to save anyone chasing a non-bug.

The glm5_next / GLM-5.3-Flash output-quality issue is not an exo bug, and not an mlx-vlm bug.

I verified mlx-vlm 0.6.17's glm5_next component-by-component against the official reference in transformers 5.16.1 (modeling_glm5_next.py), feeding both implementations identical real weights from the checkpoint:

Component	Coverage	vs reference
HyperConnection (sinkhorn; fused Metal kernel and pure-MLX path)	all layers	exact (1 bf16 ULP)
Linear attention / KDA (fused + unfused input projection)	34 of 45 layers	corr 0.999974
DSA sparse attention + k-pool indexer	11 of 45 layers	corr 0.999995
MoE router (noaux_tc, norm_topk_prob, routed_scaling_factor)	42 layers	matches

I also checked the checkpoint's own transformations against the real upstream weights, pulling individual tensors out of the 640 GB FP8 original with HTTP range requests (~42 MB total rather than the whole repo). The community quantiser had replaced upstream's kv_b_proj with an MLA-absorbed embed_q/unembed_out pair of its own construction — that absorption is correct (corr 0.99997 against the true kv_b_proj).

What is actually wrong is the quantisation itself: the 2-bit routed experts measure cosine 0.922 against the true upstream weights (~38% orthogonal noise per expert matrix, compounded over 42 MoE layers). The 8-bit tensors in the same checkpoint measure 0.99997. That is enough to destroy coherence while leaving activation statistics perfectly healthy — every layer's mean/std/absmax looks normal end-to-end, with no NaN or inf, which is what made it awkward to diagnose.

So: nothing for exo to fix here, and patches/ in this PR is only about loading the architecture, not about output quality.

Separately, the second finding I listed above — pipeline_auto_parallel replacing inner.layers after construction, so a model that cached layer indices in __init__ silently builds masks from the wrong cache — is a real exo bug and is now filed on its own as #2287 with a regression test.

🤖 Generated with Claude Code

[zhast]

Retracting my previous comment — it was wrong.

I claimed the GLM-5.3-Flash output problem was the community 2-bit quantisation and not exo. That conclusion does not hold, and I'd rather correct it here than leave it standing.

I have since run the identical checkpoint single-process (no pipeline sharding, no distributed group, same mlx-vlm code path):

prompt: What is the capital of France? Answer in one word.
top-1 next token: 'The'  (logit +16.75)
greedy: 'The user asks: "What is the capital of France?'

That is coherent. The same weights under exo's 3-way pipeline return '\nfor\n-\n ... }} \nnal'. I also sampled the 2-bit routed experts against the true upstream FP8 weights across layers 4/10/20/33/44 — cosine is a consistent 0.922, which is evidently sufficient. So the checkpoint is fine and the defect is in the distributed path.

Where I went wrong: the component-level comparisons in my previous comment are all still valid (hyper-connection exact, linear attention corr 0.999974, DSA corr 0.999995, MLA absorption corr 0.99997 — those numbers stand). But I also checked that the pipeline was internally consistent — each rank's received tensor matched the previous rank's sent tensor, activations stayed finite and well-scaled across all 45 layers — and wrongly read that as correct. Consistent is not correct, and "everything I tested is fine" is not "everything is fine."

Useful narrowing for anyone hitting this: exo's very first generated token already differs from the single-process argmax, so the divergence is in the sharded forward pass itself, not in the post-prefill c.trim(2) / SSM-snapshot rollback and not in decode. KV-cache quantisation is off by default (KV_BITS = None), so that is not a factor either.

Still bisecting; I'll report the actual root cause rather than another guess. Apologies for the noise.

🤖 Generated with Claude Code

[zhast]

Root cause found, and it is neither exo nor the quantisation. Filing here to close out the two incorrect calls I made above.

It is the fused sinkhorn Metal kernel in mlx-vlm's deepseek_v4/hyper_connection.py — filed as Blaizzy/mlx-vlm#2139.

That kernel reads mixes, scale and base as float32, type-punning them via *(const device float4*). HyperConnection.__init__ allocates all three as float32, so that holds for a freshly constructed module — but load_weights() replaces them with the checkpoint's dtype, which is bfloat16 in every GLM-5.3-Flash and DeepSeek-V4 release. The kernel then reinterprets bfloat16 bytes as float32.

Measured on GLM-5.3-Flash layer 0 against a float64 reference of the same sinkhorn: _hc_ops matches to 0.00000 relative error, the kernel is 1.98 off. Casting the three inputs makes them agree exactly (max_abs 0.999160 -> 0.000000).

Why it survived so much scrutiny, which may be useful to others:

• Nothing raises. Sinkhorn renormalises the corrupted logits back onto the doubly-stochastic manifold — column sums come out exactly 1.0 — so the residual stream stays finite and well-scaled through all 45 layers, no NaN, no inf. Every "is anything blowing up" diagnostic looks clean.
• nn.Module.training defaults to True, so a freshly constructed HyperConnection takes the _hc_ops path. My first kernel-vs-reference test never called .eval() and therefore never exercised the kernel — it compared ops against ops and reported an exact match. That false negative is what sent me down the quantisation path.
• Only .eval() + Metal hits it, i.e. precisely the inference path and nothing else.

After the fix, on 3 nodes over JACCL/RDMA:

'391 divided by 17. 17 x 23 = 391 (17 x 20 = 340, 17 x 3 = 51, 340 + 51 = 391).
 So the answer is 23.'   -> 23

Throughput improves as well (4.09 tok/s with the fixed kernel vs 2.39 forcing _hc_ops).

So: the component numbers in my first comment stand, the "it's the 2-bit quant" conclusion was wrong, and patches/ in this PR remains purely about loading the architecture. #2287 (stale layer indices after pipeline sharding) is still a genuine exo bug and is unaffected by any of this.

🤖 Generated with Claude Code