diag(MODEL-MM-QWEN4-EXP): the MoE expert selections DIFFER, and a third of the boundaries are exact bf16 ties (#2552)

.agents/specs/qwen4-exp-flash-next.md

@@ -8678,6 +8678,173 @@ product-behaviour change in this wave, because the measurement says there is no
 defect at the first divergence to change.
 
 
+## Wave MOEDIV — is the MoE prefill divergence a SELECTION FLIP? (#2552)
+
+### The question, and why the existing instrument cannot answer it
+
+Wave PREFILLDIV ([#2547](https://github.com/mudler/vllm.cpp/issues/2547),
+[#2554](https://github.com/mudler/vllm.cpp/pull/2554)) isolated TWO sources of
+CPU/CUDA prefill divergence at decoder layer 0, the only layer whose input is
+bit-identical on both CUDA arms. The first is the chunked Gated DeltaNet
+prefill, and it is not a defect: it mirrors vLLM's vendored Flash-Linear-Attention
+precision map at all five anchors. The second is the MoE block, which turns a
+`2.1e-05` input difference into a `7.269e-05` output difference and which
+`VT_GDN_CHUNKED=0` moves by only 1.7x, against 332x on the Gated DeltaNet tap.
+[#2552](https://github.com/mudler/vllm.cpp/issues/2552) owns that second source.
+
+`VT_Q4EXP_LAYER_FP` taps VALUES. **A value tap cannot separate an
+expert-selection flip from re-association inside the expert GEMM**, because a
+discrete selection has bimodal error and not a tolerance: one token's top-k set
+either changes, and that token's MoE output changes by an O(1) amount, or it
+does not change at all and the residue is rounding. Averaging both into one
+`rel(sum|x|)` destroys exactly the bit that decides which of the two happened.
+
+### What this wave lands
+
+`VT_MOE_SEL_FP=<calls>` — a **selected-expert-id tap** on the reference arm of
+the shared sparse-MoE block (`MoeBlock`, `qwen3_5.cpp`), which is the arm
+`RunQwen4ExpMoeBlock` reaches on a stacked keep-quant checkpoint. For the first
+`<calls>` block invocations of the process it prints, per token:
+
+* the selected expert ids **SORTED**, because the assertion between two arms is
+  SET equality and the selection ORDER is not part of it (`vt::MoeCombine` sums
+  over the k slots and is order-invariant given the weights);
+* `lo`, the smallest selected router logit, and `hi`, the largest rejected one,
+  each with its raw bf16 bit pattern;
+* `margin = lo - hi` in **LOGIT** space rather than probability space, because
+  the softmax denominator is itself a device-order f32 reduction and differs
+  between the arms, while the bf16 logits are the values the selection is
+  actually a function of;
+* `ulps`, the number of representable **bf16** steps between `hi` and `lo`
+  under the sign-magnitude total order. `ulps=0` means the two logits are the
+  SAME bf16 value and the boundary was decided by the lowest-index tie-break —
+  a knife edge that one ulp of re-association anywhere upstream will flip.
+  `ulps` is the honest margin for a bimodal error: a probability difference
+  reads as "small" for a gap of one representable step and for a gap of fifty.
+
+and, per call, one digest line carrying `sel` (an FNV-1a hash over every token's
+sorted id list, so **selection-set equality between two arms is one string
+comparison per layer**), the minimum `ulps` over the call's tokens, and four
+`sum|x|` axes that decompose the block's own output: `x` (the block input),
+`logit` (the router logits), `exp` (the assembled per-slot expert outputs before
+the combine) and `shr` (the shared expert). With the selection sets equal, those
+four say WHICH of the block's GEMMs carries the residue; with them unequal, the
+per-token lines say which token flipped and how near the tie was.
+
+### The counted property
+
+`lines=` on every digest line is the running total of value lines the tap has
+printed. **An instrument that never ran and two arms whose taps agreed look the
+same in a diff**, and this row has already paid for that twice (a job that
+searched `$BLD/vllm-server` while ninja links `examples/vllm-server`, and a
+doctest case whose name contained a comma so `-tc` split it). The measuring job
+asserts `lines == calls * T` and refuses to report a comparison otherwise.
+
+### Scope, and what this wave does NOT do
+
+It is a diagnostic tap and one measurement. It changes no math on any arm: the
+tap reads host buffers the reference path had already downloaded (`h`, `logits`,
+`ids`, `expert_out`) plus one guarded download of the shared-expert output, and
+it is inert with the environment variable unset. It lands on the reference arm
+only. The three fused CUDA arms (`MoeBlockFusedCuda`, `MoeBlockFusedMarlinCuda`,
+`MoeBlockBf16Cuda`) keep their ids on device and are NOT tapped, which is
+recorded rather than hidden: no `qwen4_exp` checkpoint reaches them, and adding
+a device readback to a capturable path to instrument a model that cannot enter
+it would be dead code by construction.
+
+### Gates
+
+* `ctest -R test_qwen4_exp_moe` at rc 0, with the tap's ids and `ulps` gated
+  against `MoeReference`, the suite's independent double-precision oracle
+  reimplementation, on both the bf16 and the keep-quant arm.
+* The reachability mutation deletes the tap's production call site inside
+  `MoeBlock` and the tap case must RED.
+* `scripts/agent-preflight.sh --fail-on-skip` at rc 0.
+
+### Evidence required
+
+One `rc` lease on `thor:gpu0`, one binary, the released
+`unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S artifact (shard 1 sha256
+`88a1420825a9304063e882ada29d438263617f51ac8923d438d927496693bafd`), the same
+prompt and server flags PREFILLDIV used, and three arms: `--device cpu` as the
+control, `--device cuda`, and `--device cuda` with `VT_GDN_CHUNKED=0` so the
+MoE residue is read with the first source removed. The result is the per-layer
+`sel` hash comparison plus the `ulps` distribution.
+
+### Stop conditions
+
+Stop and report `PENDING` if the artifact cannot be staged; do not substitute
+another checkpoint. Stop if the answer is that the CUDA arm mirrors vLLM and our
+CPU arm is the outlier, exactly as PREFILLDIV concluded for the Gated DeltaNet:
+say so with the upstream anchors and close #2552 as answered rather than fixed.
+**Do not force a fix.** Changing a CUDA arm to agree with our CPU arm, when the
+CUDA arm is the one that mirrors the oracle, moves this model AWAY from vLLM.
+
+### Outcome
+
+**MEASURED. The selections DIFFER, and it is NOT a defect.** Full result in
+[the evidence file](../../docs/bench-evidence/qwen4exp-moe-selection-20260902.md);
+`thor:gpu0` `sm_110`, job `a5bf074b-0f1a-490d-ab51-5d561857ef9e`, one binary
+`b3d5d97c86ffa75f6de9e70e7a036f495962c39c8d76d8ed182b9a44c83beacf`, three arms,
+the released UD-IQ1_S artifact verified inside the lease.
+
+**The number that answers the row.** At `E = 512` and `top_k = 10` the router's
+top-k boundary is an **EXACT bf16 tie at 79 of 240 prefill token-slots (32.9%)**
+on the CPU control, and inside one representable bf16 step at 55.8%. At a tie
+the selection carries no information: the lowest-index tie-break decides it. The
+flip rate IS the tie rate — 75 of 240 slots flip on the `VT_GDN_CHUNKED=0` arm
+(31.3%), 78 on the production arm.
+
+**#2552's own headline number is NOT a flip, and this is the half the issue got
+wrong.** At layer 0 with the Gated DeltaNet source removed — the arm its table
+was taken on — both arms select the same experts for all five tokens. The
+`7.269e-05` residue decomposes onto the expert GEMM: `x` 2.139e-05, `logit`
+2.378e-05 (the router GEMM does not amplify), `exp` **1.421e-04** (6.6x), `shr`
+4.310e-05. The keep-quant grouped GEMM is the amplifier, and
+`cuda_quant_dot.cu:2158-2170` already says what it is — the integer core is
+bit-identical and only the per-block float scale sum is reassociated, which
+llama.cpp's own CPU/CUDA split does too.
+
+**Why it is not a defect.** vLLM routes this model on bf16 logits at 512 experts
+as well: `Qwen4ExpSparseMoeBlock` inherits Qwen3Next's plain `ReplicatedLinear`
+gate with no `params_dtype`, `moe_runner.py:897-902` runs it as a plain
+`F.linear`, and the f32 widening happens inside `topk_softmax` — exactly our
+polarity, our widening point and our lowest-index tie-break. Upstream sits on
+the same knife edge. Read at `cdefd9d499`, a FORWARD REFERENCE 1566 commits past
+the pin, which carries no `qwen4_exp` at all. **This is a source read, not a
+measurement: no vLLM process was run on this checkpoint.**
+
+**One thing to WATCH rather than fix.** vLLM owns a fp32-capable router gate,
+`GateLinear` (`gate_linear.py:18-33`), a `PluggableLayer` with three of five
+tiers emitting fp32 logits. Qwen3Next does not opt in, so it is unreachable for
+this architecture today. If upstream ever flips `qwen4_exp` onto it we mirror
+that; widening ours first would move this model AWAY from the oracle, which is
+what the stop condition above forbids.
+
+**The gate question, settled with a mechanism.** PREFILLDIV observed that token
+agreement is not monotone in numerical distance and proposed no CPU-vs-CUDA
+token gate. This says WHY: a third of the routing decisions carry zero margin,
+so the emitted sequence is a function of tie-break order rather than of
+accuracy. **No CPU-vs-CUDA token-exactness gate is well posed for `qwen4_exp`.**
+The selection-set agreement rate and the tie-rate histogram ARE well posed and
+are now measurable; a gate against vLLM stays the right target and stays OWED,
+because vLLM's GGUF support is an out-of-tree plugin and every safetensors arm
+of this model exceeds the largest fleet box.
+
+**What this run could NOT prove, and one thing it found by accident.** The
+decode half is VOID: measured on `origin/main` `a99b9c69a`, the CUDA arm answers
+`11751 271 271 271 271 271 0 0`, not the fluent sequence PREFILLDIV recorded,
+because that wave measured a tree carrying
+[#2550](https://github.com/mudler/vllm.cpp/pull/2550)'s decode fix and this one
+did not. From step 1 the two arms therefore run different token sequences, so
+the comparator's 336-of-336 decode flip count is a different-input artifact and
+is published only to say so. **#2550 landed at `bb78d1ee8` while this wave was
+writing up**, which is why this branch carries a merge of it; the numbers above
+were taken BEFORE that merge and are not re-derived here. A decode-phase
+selection comparison is now well posed and is NOT claimed by this wave — the
+prefill result stands on its own, and re-running the tap on a post-#2550 tree is
+the cheap next measurement.
+
 ## Now
 
 `ACTIVE`. **THE COUNT IS THE TABLE, AND THIS SENTENCE NO LONGER RESTATES IT.**