diag(MODEL-MM-QWEN4-EXP): the MoE expert selections DIFFER, and a third of the boundaries are exact bf16 ties (#2552)

[localai-org-maint-bot]

The selections DIFFER, and it is NOT a defect. At num_experts = 512 and
num_experts_per_tok = 10 the router's top-k boundary is an EXACT bf16 tie at
79 of 240 prefill token-slots (32.9%) on the CPU control, and inside ONE
representable bf16 step at 55.8%. At a tie the selection carries no information:
the lowest-index tie-break decides it, and any perturbation upstream reshuffles
it. The flip rate is the tie rate — 75 of 240 token-slots flip on the
VT_GDN_CHUNKED=0 arm, 78 on the production CUDA arm.

And #2552's own headline number is not a flip, which is the half the issue
got wrong. At decoder layer 0 with the Gated DeltaNet source removed — the arm
its table was taken on — both arms select the same experts for all five
tokens, and the first flip is at layer 1. The 7.269e-05 residue there
decomposes onto the expert GEMM instead.

Why a value tap could never have said this

VT_Q4EXP_LAYER_FP reports rel(sum|x|). A discrete selection has bimodal
error rather than a tolerance: a token's top-k set either changes, and that
token's MoE output moves by an O(1) amount, or it does not and the residue is
rounding. Averaging both into one relative difference destroys exactly the bit
that decides which happened.

So this lands VT_MOE_SEL_FP=<calls>, a tap on the SELECTED EXPERT IDS on the
reference arm of the shared sparse-MoE block — the arm a stacked keep-quant
checkpoint takes and the only arm any qwen4_exp checkpoint reaches. Per token:
the ids sorted, since the assertion between two arms is set equality and
vt::MoeCombine is order-invariant over the k slots; the smallest selected and
largest rejected router logit with their raw bf16 bit patterns; and the margin
counted in bf16 ULPS under the sign-magnitude total order.

The margin is in logit space, not probability space, and that is the design.
The softmax denominator is itself a device-order f32 reduction and differs
between the arms, while the bf16 logits are what the selection is a function of.
A probability gap reads as "small" for one representable step and for fifty
alike; ulps = 0 says the two logits are the SAME bf16 value. Per call: an
FNV-1a hash of every token's sorted list, so comparing two arms at one layer is
one string comparison, plus four sum|x| axes decomposing the block.

The measurement

thor:gpu0 sm_110, rc job a5bf074b-0f1a-490d-ab51-5d561857ef9e, one binary
b3d5d97c…, three arms, the released unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ1_S
(shard 1 88a14208…) verified inside the lease. CMAKE rc=0, BUILD rc=0 wall=921s. Full result in
the evidence file, with
the raw comparator output and all 384 digest lines per arm committed beside it so
every number is checkable rather than transcribed.

The instrument asserts it ran: every arm printed 48 prefill calls and a final
lines=576, the derivable total (48x5 prefill + 336x1 decode), and the
comparator re-derives it as COUNTED-PROPERTY … agree=True on both sides.

Two independent instruments agree bit for bit on the MoE block's input. This
tap's x axis reads the tensor VT_Q4EXP_LAYER_FP calls L00 mhc.mix, from a
different tree, and all three arms reproduce PREFILLDIV's values exactly
(3613.82031 / 3615.62777 / 3613.74301, and both rel values).

Layer 0, VT_GDN_CHUNKED=0, selections equal, rel against the CPU control:

axis	what it is	rel	vs the input
x	the block input	2.139e-05	1.0x
logit	the bf16 router logits	2.378e-05	1.1x
exp	the assembled per-slot expert outputs	1.421e-04	6.6x
shr	the shared expert	4.310e-05	2.0x

The router GEMM is not the amplifier. The keep-quant grouped expert GEMM is.

Why it is not a defect

vLLM routes this model on bf16 logits at 512 experts too, so upstream sits on
the same knife edge. Read at cdefd9d499, a FORWARD REFERENCE 1566 commits past
the pin 5559679229, which carries no vllm/models/qwen4_exp/ at all:
Qwen4ExpSparseMoeBlock (vllm/models/qwen4_exp/nvidia/model.py:160) inherits
Qwen3Next's gate untouched; that gate is a plain ReplicatedLinear with no
params_dtype (vllm/model_executor/models/qwen3_next.py:170-176); the runner
applies it as a plain F.linear
(fused_moe/runner/moe_runner.py:897-902); and the f32 widening happens inside
ops.topk_softmax (fused_moe/router/fused_topk_router.py:26-41). Same width,
same widening point, same lowest-index tie-break as ours.

vLLM does own a fp32-capable gate — GateLinear
(fused_moe/router/gate_linear.py:18-33), a PluggableLayer with three of five
tiers emitting fp32 logits — and Qwen3Next does not opt into it, so it is
unreachable for this architecture. That is recorded as a thing to mirror IF
upstream flips this model onto it. Widening ours first would move this model AWAY
from the oracle.

The expert GEMM's arm-to-arm difference is a documented re-association, and the
k-quant secondary oracle has the same one. cuda_quant_dot.cu:2158-2170 states
it: the integer core is bit-identical and only the per-block float scale sum is
reassociated, warp tree against CPU sequential; the activation quantizer is a
bit-exact port either way. llama.cpp @ 237ad9b96 splits identically —
ggml-cuda/mmvq.cu:566-641 finishes with warp_reduce_sum while
ggml-cpu/arch/x86/quants.c:841-852 sums sequentially.

So neither arm is wrong, and #2552 closes as answered rather than fixed.

What it says about a gate

PREFILLDIV observed that token agreement is not monotone in numerical distance
and proposed no CPU-vs-CUDA token gate. This gives the mechanism: a third of the
routing decisions carry zero margin, so the emitted sequence is a function of
tie-break order rather than accuracy. No CPU-vs-CUDA token-exactness gate is
well posed for qwen4_exp. The selection-set agreement rate and the tie-rate
histogram ARE well posed and are now measurable. A gate against vLLM stays the
right target and stays OWED: vLLM's GGUF support is an out-of-tree plugin and
every safetensors arm of this model exceeds the largest fleet box.

Gates, read literally

test_qwen4_exp_moe gates the printed ids against MoeReference, that suite's
independent double-precision reimplementation of the lane-pinned upstream, and
gates ulps against the reference's own bf16 logit bit patterns with the
sign-magnitude ordering re-derived test-side so a sign defect in the tap has
nothing agreeing with it. The case asserts its own capture through a sentinel
written into the redirected descriptor, because a redirect that caught nothing
would otherwise read as "the tap printed nothing" — the polarity the OFF arm
asserts.

The second ctest entry re-runs the same binary with the switch set, since the
budget is a read-once static. It is deliberately unfiltered: a -tc pattern
matching nothing exits 0 having run nothing and doctest reports
assertions: 0 … SUCCESS!, a skip wearing a pass this row has paid for. Running
the whole binary cannot fail that way and buys a second property free — every
value case re-runs with the tap on, so a tap that perturbed the block's output
would red them.

Three mutations, each with an applied-proof count 1 to 0 and each built at
rc=0 before it was run, because a mutation whose build failed reads as a
passing test:

mutation	rc	assertions
M1 delete the tap's call inside MoeBlock, keeping the symbol referenced so -Wunused-function cannot stand in for the test	1	120 of 121, 1 failed
M2 hash and print the SELECTION ORDER instead of the sorted set	1	167 of 172, 5 failed
M3 drop the sign-magnitude total order from the bf16 ulp count	1	158 of 172, 14 failed

Tree restored byte-for-byte after each. Green on the final merge result: build
rc=0, 121 | 121 passed switch unset, 172 | 172 passed switch set, both rc=0,
and the same two arms ran green inside the lease on thor (TAP GATE OFF rc=0,
TAP GATE ON rc=0). scripts/agent-preflight.sh rc=0.

What this is NOT

n=1: one prompt, one artifact, one box, one repetition, greedy, UD-IQ1_S,
num_reqs = 1. Not a token gate — no oracle decoded this prompt and the CPU
arm is a CONTROL, not an oracle, so nothing here says which arm is closer to
correct. No speed number. The "vLLM sits on the same knife edge" conclusion is
a SOURCE READ, not a measurement: no vLLM process was run on this checkpoint.

The decode columns are VOID and are published only to say so. The arms were
measured on a99b9c69a, before #2550 landed, so the CUDA arm answered
11751 271 271 271 271 271 0 0 and from step 1 the two arms ran different token
sequences — the 336-of-336 decode flip count is a different-input artifact.
#2550 has since landed at bb78d1ee8 and this branch merges it; the prefill
result does not depend on it, because step 0 consumes the prompt rather than a
sampled token. A post-#2550 decode comparison is now well posed and is named as
the cheap next measurement rather than claimed.

docs/USAGE.md is deliberately untouched. Its checkpoint row still says no
qwen4_exp step reaches a CUDA queue, which is false — but #2554 already carries
that correction, and two pull requests writing the same enormous table cell is
the lock shape AGENTS.md warns about.

Refs #2547.
Refs #2550.
Closes #2552.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [claude-code]