diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index c6db8b964..3875aca20 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -3110,6 +3110,240 @@ raised. The predicate was replaced by the op-table probe described above, non-expert tower is what makes this model fit, and moving 94.6758 GiB of experts to the device does not remove it. +### W9c-3b — the KV binding reads the engine's pages where they ARE (GPU, medium) + +Issue: [#2480](https://github.com/mudler/vllm.cpp/issues/2480). +Claim: `CLAIM-GLM53-FLASH-W9C3B`. Base `3cd467643`. + +**O46's mechanism is FALSE, and the correction is the point of this wave.** O46 +reads the crashing legs' log order — one device-arm announcement, then one +fallback warning, then SIGSEGV — as evidence that "the fault is in the MIXED +residency state the per-layer fit guard creates". Both of those lines are +`static bool said` once-flags (`glm5_next_moe.cpp`, `AnnounceDeviceArmOnce` and +`WarnDeviceFallbackOnce`). Their order says only that at least one layer staged +and at least one LATER layer did not. It is not evidence about WHERE the process +died, and the process died somewhere else. + +**WHERE IT DIED, read off four files rather than guessed:** + +1. `GPUModelRunner::initialize_kv_cache` resolves + `kv_cache_backend_resident_ = !platforms::GetPlatform(dev.type).is_cpu() && (VT_DEVICE_KV_CACHE != "0")` + (`src/vllm/v1/worker/gpu/runner.cpp:1119-1122`). On `--device cuda` it is + **true**, and its own comment says the predicate is deliberately "has a + device", not "is CUDA". +2. `GPUModelRunner::CacheBuffer` then allocates every paged cache and every + recurrent state with `vt::Alloc(device_, ...)` (`runner.cpp:575-593`). On + CUDA that is `cudaMalloc`. The `host_data_` vector is the CPU-queue arm and + is not taken. +3. A `cudaMalloc` pointer is **not host-dereferenceable, on GB10 included**. + `src/vt/cuda/cuda_backend.cu:354-391` says so and holds it with a + `static_assert` that CUDA keeps the inherited + `DeviceMemoryIsHostAddressable() == false`, naming #844 and #1435 as the + SIGSEGVs that came from reading the WIDE predicate (`UnifiedMemory()`) + instead. +4. `glm5_next_kv.cpp` reads and writes those pages with **plain host loops**: + `ReadElem`/`WriteElem` are `static_cast(kv.data)[i]` and + `ReadTensorElem`/`WriteTensorElem` are `t.Ptr()[i]`. `LoadCaches` and + `StoreCaches` are their only callers, and + `ForwardGlm5NextForConditionalGeneration` is the only caller of those two. + +`LoadCaches` returns before touching any page on a fresh sequence +(`if (b.cached_len <= 0) return;` — "A FRESH SEQUENCE READS NOTHING"), so on +step 1 the FIRST host access to device memory is in `StoreCaches`, **after the +whole forward has returned**. That is after the announcement, after the fallback +warning, on a step that emitted no token, with no message — which is the +recorded signature exactly, and it is why the wall of a crashing leg sits at +roughly one forward step past its load. + +**THE DEFECT IS OLDER THAN THE ARM THAT EXPOSED IT.** Nothing in W9c-3a's diff +touches a KV page. `origin/main` never reached this because +`Glm5NextHostForward` refused a non-CPU queue several thousand instructions +earlier; W9c-3a removed that refusal, and a hole that had been unreachable +became reachable on the first `--device cuda` step that got past it. The three +hypotheses O46 eliminated are all inside `MoeExpertsKeepQuant` and are all +correctly eliminated. They were aimed at the wrong subsystem. + +**AND NOTHING GATED IT.** `ResolveKvBinding`, `LoadCaches` and `StoreCaches` +have **zero** call sites in `tests/` across the whole tree. The engine binding's +read/write path — the one W5b-2c landed and the one this crash is in — has no +unit coverage of its own; it is exercised only incidentally, through +`ModelRegistry::Forward` on a CPU queue, where every page IS host memory and the +defect cannot appear. + +#### Scope + +`src/vllm/model_executor/models/glm5_next_kv.cpp` only. Every span this file +reads or writes goes through `vt::Backend::Copy` instead of being dereferenced. +`Backend::Copy` is direction-agnostic on CUDA (`cudaMemcpyAsync` with +`cudaMemcpyDefault`, `cuda_backend.cu:116-118`), so ONE code path is correct +whether the pages are device memory or host memory. That is what keeps this file +from re-deriving the runner's residency policy: it never asks where the pages +are, and `VT_DEVICE_KV_CACHE=0` therefore needs no second branch here. + +On a CPU queue there is no backend to ask and the direct `memcpy` path is kept, +so every `--device cpu` run is byte-for-byte unchanged. + +Every span this file addresses is already contiguous, which is what makes the +change a substitution rather than a redesign: one paged row is `head_size` +elements at `PagedRowOffset(...)`, and a recurrent state is `conv_elems` or +`rec_elems` elements at `slot * elems`. + +#### Not in scope + +The other ten host arms O43 lists. The engine's residency policy. A +`ModelInfo` flag that would let a model ask for host-resident caches — that is a +shared-seam change with more than one consumer to survey, and it is not needed +to make this model correct. + +#### Gates + +* `test_glm5_next_forward` — a new case drives `ResolveKvBinding`, + `StoreCaches` and `LoadCaches` over the file's existing three-group + `Topology` fixture, with the pages re-pointed at a backend whose allocations + are NOT host-readable. RED before the change and GREEN after. +* The whole `glm5_next` suite set unchanged, by hand, with counts. +* Sibling inertness: `GlmMoeDsaForCausalLM`, DeepSeek-V4, dots3-note, Kimi. +* `dgx:gpu0`, the real 101.2535 GiB `UD-Q2_K_XL` artifact, four interleaved + legs on two binaries — see `## Evidence required` below. + +#### Evidence required + +The hermetic gate cannot prove the diagnosis, only the property. The diagnosis +is proved on the box, by a leg that changes NOTHING but where the pages live: + +**RUN 1 IS TAKEN and legs A and B are SETTLED** (O50): B died with SIGSEGV at +`StoreCaches` with the whole call chain named, and A survived at rc=0 in 1756 s +with one variable moved. **Legs C and D did not run**, because the job's identity +guard hashed `vllm-cli` alone and refused a pair whose `libvllm.so` genuinely +differed (O51). **RUN 2 IS NOW ALSO TAKEN** (O53 below); it carried the repaired +predicate and these legs: + +| leg | binary | device | env | the question it answers | +|---|---|---|---|---| +| D | fixed | cpu | -- | the OPERAND, from this binary on this box; taken FIRST | +| C | fixed | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1` | **the merge criterion**: rc=0 and byte-identical to D | +| E | fixed | cuda | + `VT_ASYNC_DEVICE_MIRROR=0` | is the stale decode input id the second-token cause? | +| F | fixed | cuda | `--max-tokens 1` | prefill against decode, separated by construction | +| G | base | cuda | `VT_DEVICE_KV_CACHE=0` | run 1's leg A, repeated WITHIN run 2 | + +C is the fix. D is the operand every other leg is byte-compared against, taken +from the same binary on the same box, because the ` Paris.` in the records came +from a different build and is not a byte operand. E, F and G separate the +second-token question O52 opens -- and none of them is a discriminator unless it +RUNS: the obvious one, unsetting `VT_GLM5_NEXT_DEVICE_EXPERTS`, emits no token, +because without it the forward refuses a non-CPU queue by name. + +**No speed number is admissible from any of them** (O47: 89% generation spread +across three identical legs, cause unknown), and none is claimed. + +**C-against-D byte-identity also answers a SECOND question, and that is why the +legs run two tokens rather than one.** The async runner can hand a forward +`ModelForwardInput::device_token_ids` and leave `token_ids` deliberately stale +for decode rows (`v1/worker/gpu/runner.cpp:2374-2414, 2748`); this model reads +`token_ids` and ignores the device field. Whether that path ENGAGES here is not +established -- it needs `async_input_combine_`, and this row has never observed a +decode step on a device queue -- so it is not filed as a defect. It is named +here because a two-token leg contains exactly one decode step, so a leg C whose +stdout matches leg D byte-for-byte has measured it, and a leg C that emits +` Paris` followed by the wrong second token has found it. + +#### O53 -- RUN 2 IS TAKEN, and the criterion FAILED ON A CLAUSE I MIS-SPECIFIED + +`glm53-kvres/submit2.log`, `dgx:gpu0`, finished 2026-09-01T21:56:05Z, 12068 s +wall, five legs interleaved on ONE box from TWO separate build directories. The +tested bytes are the PR's bytes: `glm5_next_kv.cpp` hashes +`cb34a6d198476864...` at PR head `868b49bdd` and `37d337a7538fcefb...` at base +`3cd467643`, matching the `fix`/`base` digests the job recorded. + +| leg | binary | device | env | rc | stdout | bytes | +|---|---|---|---|---|---|---| +| D | fixed | cpu | -- | 0 | ` Paris.` | `2050 6172 6973 2e0a` | +| C | fixed | cuda | `…DEVICE_EXPERTS=1` | 0 | ` Paris Paris` | `2050 6172 6973 2050 6172 6973 0a` | +| E | fixed | cuda | + `VT_ASYNC_DEVICE_MIRROR=0` | 0 | ` Paris.` | `2050 6172 6973 2e0a` | +| F | fixed | cuda | `--max-tokens 1` | 0 | ` Paris` | `2050 6172 6973 0a` | +| G | base | cuda | `VT_DEVICE_KV_CACHE=0` | 0 | ` Paris Paris` | `2050 6172 6973 2050 6172 6973 0a` | + +Binary identity held this time on the repaired predicate: linked-set digests +`e6762476d2b97ded...` (base) against `f643a2f1bb540f49...` (fixed), with +`fix_sentinel=no` / `fix_sentinel=yes`. + +**THE CRASH IS FIXED.** Run 1 leg B died `rc=139` with `StoreCaches` at frame #0 +(`+840`); leg C is the same workload on the same box at `rc=0`. + +**THE STATED CRITERION -- "C byte-identical to D" -- FAILED, and the fault is in +the criterion.** C and D differ by TWO variables, not one: where the KV pages +live AND whether the async device mirror is engaged. A `StoreCaches` fix can +only be held to the first. The single-variable operand is **leg G**, the BASE +binary on host pages, and: + + C (fixed, device KV) == G (base, host KV) byte-for-byte, 13 bytes + +So the fix reproduces the already-working path exactly. It converts a SIGSEGV +into the same bytes, and it changes nothing else. + +This is recorded as a mis-specified criterion rather than a widened one. The +clause as written is still red, and nothing was deleted to make it green. + +**THE SECOND TOKEN IS A SEPARATE, PRE-EXISTING DEFECT, AND RUN 2 CONVICTS IT.** +Leg G is a base build and diverges identically, so the divergence predates this +fix and is neither caused nor cured by it. Leg F shows prefill is clean +(` Paris` matches D's first token). Leg E flips ONE variable, +`VT_ASYNC_DEVICE_MIRROR=0`, and the divergence disappears into byte-identity +with D. That is exactly the pre-registered read-key above, and it promotes +[#2544](https://github.com/mudler/vllm.cpp/issues/2544) -- which names +`Glm5NextForConditionalGeneration` on a grep and calls itself "a candidate, not +a conviction" -- to a measured conviction for this model. O52 is answered and +its ownership moves to #2544. + +**What run 2 does NOT establish, stated rather than glossed:** leg A as +originally specified (FIXED binary with `VT_DEVICE_KV_CACHE=0`) was never run; +what exists is the BASE binary on host pages, twice, rc=0 both times, which +serves the falsification purpose but is not the literal leg. Legs B and C were +never in the same job -- B is run 1's binary, C is run 2's, source-anchored to +the same `3cd467643` but not interleaved with each other, so **leg C is n=1**. +No speed number is admissible or claimed from any leg; the walls are durations +of which ~85% is GGUF load. + +#### Mutations, with what each one killed + +Every mutation is applied to PRODUCT code and rebuilt, because a mutation the +compiler rejects and a mutation the binary never contained both read as a pass. + +| # | mutation | result | +|---|---|---| +| M1 | `PageIo::Write` always takes the direct `memcpy` arm | KILLED, 1 case / 5637 assertions | +| M2 | `PageIo::Read` always takes the direct `memcpy` arm | KILLED, 1 case / 2818 assertions | +| M3 | `PageIo` drops its CPU early return, so a CPU queue bounces too | **SURVIVED** | +| M4 | delete the production `StoreCaches` call in `glm5_next_registry.cpp` | KILLED, 2 cases / 3 assertions -- the REACHABILITY mutation | + +**M3 SURVIVED and that is a finding rather than a gap to paper over.** It has +to survive: the CPU backend's `Copy` IS `std::memcpy`, so the bytes are +identical whichever arm runs, and the case's counter watches the SHADOW backend, +which a CPU queue never reaches under either version. The early return is +therefore not load-bearing for correctness. It is there so a build with no CPU +backend registered does not `Fail` on the host path, and so the `--device cpu` +instruction stream stays the one that was already measured. The case says so in +its own comment instead of asserting something it cannot see. + +M4 is the reachability answer: the production call site is +`ForwardGlm5NextForConditionalGeneration`, `ModelRegistry::Forward` is the entry +point, and deleting the `StoreCaches` line reds the two W5b-2c cases that read +the engine's pages back on a second step. The two W9c-3b cases do NOT red under +M4, because they call the three functions directly -- they localise the defect +and they are not the reachability proof, which is why both are kept. + +#### Stop conditions + +* If leg A dies too, the diagnosis is wrong: report it, keep O46 open, and do + not ship the change on a hermetic gate alone. **DISCHARGED: A survived at rc=0 + and B's backtrace names `StoreCaches` -- O50.** +* If leg C emits a token but does not match leg D byte-for-byte, the CRASH is + fixed and the SECOND-TOKEN question (O52) is open. Those are two results, and + the first is not reported without the second beside it. +* If a fifth arm of this model turns out to host-dereference engine device + memory, that is a wider residency question than this wave, and it goes back as + `NEEDS_DECISION` rather than being absorbed here. + ## Tests to port `tests/models/` in transformers `v5.16.1` is the upstream suite. What is @@ -5353,11 +5587,21 @@ Debts this row carries, each visible rather than waived: `139` is SIGSEGV. **THREE of three CUDA legs crashed and three of three CPU legs emitted ` Paris.`**, interleaved, from one binary on one box -- so this is - a property of the arm and not of the box's mood. Each crashing leg logged + a property of the run and not of the box's mood. Each crashing leg logged exactly one device-arm announcement followed by exactly one fallback warning - before dying -- so the fault is in the MIXED residency state the per-layer fit - guard creates, not in the device arm itself, which the CUDA unit gate exercises - cleanly at NMSE 3.833e-15 against the host arm. + before dying. + + **THE SENTENCE THAT USED TO FOLLOW IS FALSIFIED, and O49 carries the + replacement.** It read "so the fault is in the MIXED residency state the + per-layer fit guard creates". Both of those log lines are `static bool said` + once-flags, so their order says only that at least one layer staged and at + least one LATER layer did not; it is not a location. The fault is in + `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after + the whole forward has returned -- a defect older than this arm, which only + became reachable when the non-CPU refusal above it was removed. Read O49 + before spending anything on the mixed-residency story. The device arm itself is + unimplicated either way: the CUDA unit gate exercises it cleanly at NMSE + 3.833e-15 against the host arm. **This is a REGRESSION and it is named as one.** On the base tree the same command produced a clean refusal at 1066 s (the operand table above, leg C). @@ -5441,6 +5685,161 @@ Debts this row carries, each visible rather than waived: without checking -- and because the ordering that caused it is this row's own script putting the sibling suites after the legs. +- **O49 -- O46's MECHANISM IS FALSIFIED, and the crash is a KV-page residency + defect that is OLDER than the arm which exposed it.** O46 infers "the fault is + in the MIXED residency state the per-layer fit guard creates" from the log + order of one device-arm announcement followed by one fallback warning. Both of + those lines are `static bool said` once-flags + (`glm5_next_moe.cpp`, `AnnounceDeviceArmOnce`, `WarnDeviceFallbackOnce`), so + the order carries only "at least one layer staged, at least one LATER layer did + not". It is not a location. + + The location is `StoreCaches`. On `--device cuda` the runner sets + `kv_cache_backend_resident_` true (`runner.cpp:1119-1122`) and allocates every + paged cache and every recurrent state with `vt::Alloc` — `cudaMalloc` + (`runner.cpp:575-593`) — while `glm5_next_kv.cpp` reads and writes those pages + with plain host loops. A `cudaMalloc` pointer is not host-dereferenceable on + GB10 (`cuda_backend.cu:354-391`, held by a `static_assert`, naming #844 and + #1435 as the same fault twice before). `LoadCaches` returns before touching a + page on a fresh sequence, so the FIRST host access to device memory on step 1 + is in `StoreCaches`, after the whole forward has returned — which is exactly + after the announcement, after the fallback warning, on a step that emitted no + token, with no message. + + **AND THE MIXED STATE CANNOT BE IT, which is a stronger statement than "the + fault is elsewhere".** The two arms of `MoeExpertsKeepQuant` share no mutable + state across layers except `OwnedTensor::d_dev`, which is per tensor. + `DeviceBanksFit` returns true whenever `need == 0`, so a layer whose three + banks are resident ALWAYS takes the device arm and never the host one; a layer + that falls back is one whose banks were never staged, and its host views + (`src.gate_exps.View()`) aim at bytes staging did not touch, because + `AdoptDeviceBytesAsHost` returns at `!DeviceMemoryIsHostAddressable()` on CUDA + (O46's own hypothesis 2, correctly killed). There is no ordering of staged and + unstaged layers that makes one arm read the other's memory. + + W9c-3a did not introduce it. `origin/main` refused a non-CPU queue before + `StoreCaches` could run; removing that refusal made a pre-existing hole + reachable. O46's three eliminated hypotheses are all inside + `MoeExpertsKeepQuant` and are all correctly eliminated — they were aimed at the + wrong subsystem, because the ordering that pointed them there was an artifact + of two once-flags. + + **NOTHING GATED THE PATH IT IS IN.** `ResolveKvBinding`, `LoadCaches` and + `StoreCaches` have ZERO call sites anywhere in `tests/`. The engine binding + W5b-2c landed is reached only incidentally, through `ModelRegistry::Forward` on + a CPU queue, where every page IS host memory and this defect cannot appear. + W9c-3b ([#2480](https://github.com/mudler/vllm.cpp/issues/2480)) owns the fix + and the first direct gate over those three functions. + + What this costs the next reader is the general lesson, and it is O44's shape + once more: an ordering in a log is evidence about ORDER and not about place, + and a once-flag makes it evidence about even less than that. + + **OWED, and small: the staging is PER ROW.** `PageIo` copies one paged row per + `Backend::Copy` and synchronises on each, so a step that hydrates a full 8192 + token prefix pays about 180,000 round trips across 11 DSA layers and two + caches. Every row inside one block is contiguous with the next, so coalescing + a block at a time would cut that to about 5,600. It is not done here because + this forward is a host reference at roughly 85 s per step and no speed number + is admissible from this row anyway (O47); it is written down so the next reader + does not have to rediscover the shape of the cost. + +- **O50 -- THE DIAGNOSIS IS CONFIRMED ON `dgx:gpu0`, BY A NAMED FRAME AND BY A + ONE-VARIABLE FALSIFICATION TEST.** Run 1 of the W9c-3b job, `dgx:gpu0` (GB10, + `sm_121a`), against the published 101.24 GiB UD-Q2_K_XL artifact, base + `3cd467643`: + + | leg | binary | device | env | rc | wall | stdout | + |---|---|---|---|---:|---:|---| + | B | base | cuda | default | **139** | -- | 0 bytes | + | A | base | cuda | `VT_DEVICE_KV_CACHE=0` | **0** | 1756 s | ` Paris Paris` | + + Leg B's `gdb` backtrace names the frame O49 predicted, and it is the whole + chain rather than a symbol on its own: + + ```text + Thread 4 "vllm-cli" received signal SIGSEGV, Segmentation fault. + #0 vllm::glm5_next::StoreCaches(...) + #1 vllm::(anonymous namespace)::ForwardGlm5NextForConditionalGeneration(...) + #2 vllm::ModelRegistry::Forward(...) + #3 vllm::v1::GPUModelRunner::execute_model(...) + ``` + + Leg A moved the engine's KV pages into host memory and changed nothing else -- + same binary, same box, minutes later -- and it SURVIVED. A predicted frame and + a one-variable falsification that fails to falsify are two independent + confirmations, and O46's mixed-residency mechanism is retired rather than + merely doubted. + +- **O51 -- THE JOB'S OWN IDENTITY GUARD REFUSED A PAIR THAT WAS GENUINELY + DIFFERENT, and the reason is that it hashed the wrong artifact.** Run 1 built + base and fixed into ONE directory and compared `sha256sum vllm-cli` across the + two halves. They matched, the guard declared "the two halves are the SAME + BINARY" and refused to report legs C and D, so **the fix went unmeasured on + hardware**. The `.so` hashes the same script RECORDED show the halves were not + the same at all: + + | artifact | base | fixed | + |---|---|---| + | `examples/vllm-cli` | `bbc55a61...` | `bbc55a61...` | + | `libvllm.so.0.0.3` | `bfd0a6e7...` | **`66412203...`** | + + `vllm-cli` is a thin ABI client (AGENTS.md §"Shared seams": examples never + include internal headers), so this change cannot alter it and its hash is + identical BY CONSTRUCTION. The model code is in `libvllm.so`. **Separate build + directories do not fix this** -- `vllm-cli` would hash identically across them + too -- so the repair is the PREDICATE: digest the executable AND every `.so` + beside it, which is exactly the breadth the same script's identity SCAN + already used two blocks earlier. Run 2 does both, and adds a source-level + sentinel (a string present only in the patched file) checked in the linked + set, so "is this the fixed tree" is answered by content rather than by a build + artifact's timestamp. + + The guard firing was correct behaviour and is kept. A gate that refuses to + report is the right failure mode; a gate that refuses for a reason that is not + the one it names is the defect. + +- **O52 -- LEG A EMITTED ` Paris Paris`, ITS FIRST TOKEN AGREES WITH THE CPU ARM + AND ITS SECOND DOES NOT, so PREFILL is clean and the DECODE step is not.** + `PARIS_A=NO` in the run-1 log is a string match against ` Paris.` and is an + artifact of that flag, not the finding. The finding is the divergence under it, + and it is recorded here rather than folded into a summary as noise. + + **The discriminator that suggests itself does not work, and this is why.** + Running leg A's configuration with `VT_GLM5_NEXT_DEVICE_EXPERTS` unset does not + run the model with the expert arm off: `Glm5NextHostForward` refuses a non-CPU + queue BY NAME when the flag is unset, so the leg emits no token at all. An + engine-fatal refusal that prints nothing is indistinguishable at the file level + from the crash it would be mistaken for -- the "instrument whose failure looks + like a result" shape this row has already paid for. + + **THE CANDIDATE WITH A MECHANISM, and it is one this spec already flagged as + unestablished.** `GPUModelRunner::async_device_mirror()` is DEFAULT ON for an + integrated CUDA GPU (`runner.cpp:4437-4459`; `AsyncDeviceMirrorEnvDefault` is + "on unless the value is `0`", and GB10 satisfies `is_integrated_gpu()`). On + that path the combine patches the DEVICE input ids and leaves the host + `token_ids` "deliberately stale for decode rows" -- the runner's own words + (`runner.cpp:2374-2414`, assigned at `:2748`). **This model references + `device_token_ids` ZERO times** in `glm5_next_registry.cpp` and + `glm5_next_forward.cpp`; it embeds `input.token_ids`. On a decode step that is + a stale id, so the model never sees the token it just emitted -- which is + precisely what re-emitting ` Paris` looks like, and precisely why prefill, + which has no decode row, is unaffected. + + It is a candidate and not a conclusion. Run 2's leg E is `VT_ASYNC_DEVICE_MIRROR=0`, + the documented rollback: a second token of `.` implicates that path and + exonerates the routed-expert arm, and a second token of ` Paris` rules it out. + Leg F (`--max-tokens 1`) separates prefill from decode by construction, and + leg G repeats run 1's leg A inside run 2 so the comparison is within-job. + + **A bimodal reading is required here.** The routed-expert arm's unit gate is + NMSE 3.833e-15, and the prefill argmax margin measured on `thor` was 1.279 + (` Paris` 16.427 over ` one`). A perturbation far too small to move token 1 + can still flip a near-tie at token 2, because an argmax is a discrete + selection and its error is bimodal, not proportional. So "token 1 matched" + does NOT by itself clear the arm, and whatever run 2 returns, the top-5 and + the MARGIN are what settle it rather than the token string. + ## Now `ACTIVE`, 2026-09-01. **THE KERNEL THAT BLOCKED THIS MODEL'S DEVICE ARM LANDED @@ -5459,11 +5858,14 @@ expiry. **AND THEN IT SEGFAULTED, so the arm is OPT-IN and defaults OFF.** Driven end to end on the 101.24 GiB artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) emitting no token, against three CPU legs that all emitted ` Paris.`, -interleaved on one binary -- once the expert banks stopped fitting -and the arm fell back to the host mid-model. The base tree produced a clean -refusal on the same command, so this was a REGRESSION and the default is -restored byte-for-byte; O46 carries the four legs, the three hypotheses already -eliminated, and the one that is untested. O47 records the other half of that +interleaved on one binary. The base tree produced a clean refusal on the same +command, so this was a REGRESSION and the default is restored byte-for-byte; +O46 carries the four legs and the three hypotheses eliminated inside the MoE +arm. **O46's own MECHANISM is falsified and O49 replaces it**: the process dies +in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the +forward has already returned, which is a defect W9c-3a exposed rather than +introduced. W9c-3b ([#2480](https://github.com/mudler/vllm.cpp/issues/2480)) +owns the fix and the four `dgx:gpu0` legs that decide it. O47 records the other half of that job: THREE identical cpu legs spread 69% on wall and 89% on generation, so NO speed number was available on any axis and none is claimed. The n=2 mechanism this row briefly asserted -- page-cache eviction moving generation -- was diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 9bde20f86..03a451b15 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -278,7 +278,7 @@ portable/reference path. In normal operation leave them unset. | `VT_VULKAN_RMSNORM` | auto | Which `vt::RmsNorm` SPIR-V module runs: `wide` forces the 1024-invocation subgroup-reducing one, `base` forces the portable 128-invocation one, unset lets the device capability decide (1024 invocations on the X axis plus compute subgroup BASIC and ARITHMETIC). The wide module exists because `RmsNorm` dispatches ONE WORKGROUP PER ROW and a batch-1 decode step has exactly one row: on Qwen3.6-27B that put 128 invocations on a 5120-wide row, four warps of one SM, with the rest of the GPU idle. MEASURED on GB10 by the two-length GPU-timestamp diff: **0.0611 -> 0.0123 ms/call, 7.88 -> 1.59 ms/token**, and paired decode **241.9 -> 235.6 ms** median TPOT. The tell that it was OCCUPANCY and not the reduction is that the SAME shader costs 0.066 ms/call during PREFILL, where 32 rows give it 32 workgroups and 32x the data. It exists for the same-binary A/B and so the unit gate can exercise the fallback on hardware that would always pick the wide arm. Vulkan-only | | `VT_VULKAN_MATMUL_NCOLS` | 4 | Output columns each lane of the portable scalar GEMM computes, in the `[K,N]` (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes `128*NCOLS` CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the `vt_matmul_vec` tactic, because in `[K,N]` the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, `m=1 k=5120 n=248320`, 2.54 GB moved per token. MEASURED on GB10, 27B decode, `ms/call` medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, **4 = 11.54**, 8 = 12.81, with 4 winning **6 of 6** interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only | | `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only | -| `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. The fault appears only in the MIXED residency state the per-layer fit guard creates: each leg logged the device arm engaging for one layer and then `DeviceBanksFit` declining a later one -- 94.6758 GiB of expert banks do not fit beside the KV pool and the GGUF page cache on a 119 GiB box -- and died after that. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | +| `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. **The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere**: the two log lines that suggested it are once-flags, and the process dies in `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see `.agents/specs/glm5-next-flash.md` O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | ## Diagnostic diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7ef7009e4..338ec381d 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -150,7 +150,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) and the runner now ALLOCATES all 167 of them ([#2068](https://github.com/mudler/vllm.cpp/issues/2068)), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | -| `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, [#2348](https://github.com/mudler/vllm.cpp/issues/2348); W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242); KV-cache spec + MoE W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223); the forward W5b-2b, [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). MEASURED on `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)), which writes the forward the `multi_kv` guard at the top of the same function was waiting for ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223)): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of [#2260](https://github.com/mudler/vllm.cpp/issues/2260) the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly, once the 94.6758 GiB of expert banks stopped fitting and the arm fell back to the host mid-model (spec O46). The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | +| `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, [#2348](https://github.com/mudler/vllm.cpp/issues/2348); W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242); KV-cache spec + MoE W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223); the forward W5b-2b, [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). MEASURED on `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)), which writes the forward the `multi_kv` guard at the top of the same function was waiting for ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223)): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of [#2260](https://github.com/mudler/vllm.cpp/issues/2260) the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly (spec O46). **The cause is now diagnosed and it is NOT the mixed residency O46 inferred**: the forward dies in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the forward has already returned, which is a defect older than this arm and unreachable while the non-CPU refusal above it stood -- spec O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | | `GlmMoeDsaForCausalLM` | zai-org/GLM-5.3 (753.33B, DSA sparse MLA MoE), HF revision `935644c05e76fc198714f4cca449fd8b970ff6d7` — **REGISTERED AND VALIDATING; IT LOADS NOTHING AND FORWARDS NOTHING** (W2, [#2214](https://github.com/mudler/vllm.cpp/issues/2214)). The architecture resolves, its config parses from a `config.json` and from a `glm-dsa` GGUF header through one validator, and the `glm-dsa` row of the GGUF dispatch table is reached from `LoadedEngine::FromModelDir`. Both `load_weights` arms refuse by name: safetensors permanently (703.74 GiB across 141 shards, no streaming loader, no MoE-expert block-fp8 rung) and GGUF until W7. The forward refuses and names all seven missing primitives — the indexer KV side cache, the expert-streaming seam, sparse prefill, the per-layer indexer schedule reuse, the `IQ4_XS` keep-quant `vec_dot`, the loader and the fp32 router GEMM. The one staged GGUF arm states no `glm-dsa.attention.indexer.types` and is refused rather than resolved off llama.cpp's hardcoded table | **NONE, and none is reachable on this fleet.** vLLM implements this architecture at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98` (`registry.py:117`) and cannot fit it on any device this project reaches, so no wave may promise a token-exact number against it | no run, no number | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | diff --git a/src/vllm/model_executor/models/glm5_next_kv.cpp b/src/vllm/model_executor/models/glm5_next_kv.cpp index dd98fa1b1..96bf3e1f8 100644 --- a/src/vllm/model_executor/models/glm5_next_kv.cpp +++ b/src/vllm/model_executor/models/glm5_next_kv.cpp @@ -4,6 +4,7 @@ #include "vllm/model_executor/models/glm5_next_kv.h" #include +#include // std::memcpy, the CPU-queue arm of PageIo #include #include #include // std::pair, in the per-cache geometry loop @@ -12,6 +13,7 @@ #include "vllm/model_executor/models/glm5_next_attn.h" // IndexerRoleFor #include "vllm/v1/attention/backend.h" // CommonAttentionMetadata #include "vllm/v1/attention/backends/gdn_attn.h" // GDNAttentionMetadata +#include "vt/backend.h" // vt::Backend, vt::TryGetBackend — W9c-3b (#2480) #include "vt/dtype.h" namespace vllm::glm5_next { @@ -64,33 +66,111 @@ std::string ChannelSummary(const MultiKvCacheIndex& mk) { return s; } -// Read one element of a paged buffer. `kAuto` fp8 means the dtype IS the -// storage type; anything else is refused before this is reached. -float ReadElem(const PagedKvCache& kv, int64_t i) { - if (kv.dtype == vt::DType::kF32) - return static_cast(kv.data)[i]; - return vt::BF16ToF32(static_cast(kv.data)[i]); +// Read one element out of a HOST-side span of `dt` storage. `kAuto` fp8 means +// the dtype IS the storage type; anything else is refused before this is +// reached. +// +// W9c-3b (#2480): these four take a raw host pointer rather than the +// `PagedKvCache` or the `vt::Tensor`, because the buffer they address is no +// longer necessarily the engine's own. On a device queue it is the staging span +// `PageIo` below filled, and giving these functions the cache would let a +// future edit reach past the staging and dereference the engine's pointer again +// -- which is the whole defect. +float ReadSpanElem(const void* host, vt::DType dt, int64_t i) { + if (dt == vt::DType::kF32) return static_cast(host)[i]; + return vt::BF16ToF32(static_cast(host)[i]); } -void WriteElem(const PagedKvCache& kv, int64_t i, float v) { - if (kv.dtype == vt::DType::kF32) { - static_cast(kv.data)[i] = v; +void WriteSpanElem(void* host, vt::DType dt, int64_t i, float v) { + if (dt == vt::DType::kF32) { + static_cast(host)[i] = v; return; } - static_cast(kv.data)[i] = vt::F32ToBF16(v); + static_cast(host)[i] = vt::F32ToBF16(v); } -float ReadTensorElem(const vt::Tensor& t, int64_t i) { - if (t.dtype == vt::DType::kF32) return t.Ptr()[i]; - return vt::BF16ToF32(t.Ptr()[i]); -} +// ─── W9c-3b (#2480): THE ENGINE'S PAGES ARE NOT ALWAYS HOST MEMORY ─────────── +// +// EVERY reader and writer in this file used to dereference `PagedKvCache::data` +// and `GdnStateCache`'s tensors directly, and on `--device cuda` those are +// `cudaMalloc` pointers. `GPUModelRunner::initialize_kv_cache` resolves +// `kv_cache_backend_resident_` from `!platform.is_cpu()` +// (`v1/worker/gpu/runner.cpp:1119-1122`) and `CacheBuffer` then allocates every +// paged cache and every recurrent state through `vt::Alloc(device, ...)` +// (`:575-593`); a `cudaMalloc` pointer is NOT host-dereferenceable, on GB10 +// included, and `src/vt/cuda/cuda_backend.cu:354-391` holds CUDA to +// `DeviceMemoryIsHostAddressable() == false` with a `static_assert` naming #844 +// and #1435 as the two SIGSEGVs that came from believing otherwise. +// +// So this was a host store into device memory, and it is why `--device cuda` +// died with SIGSEGV after a forward that had already completed: `LoadCaches` +// returns before touching a page on a fresh sequence, so the FIRST such access +// on step 1 is in `StoreCaches`, after the whole stack has run and before any +// token is emitted. Spec `## Owed` O49 carries the bisect and falsifies O46's +// mixed-residency reading of the same evidence. +// +// COPY THE SPAN, DO NOT ASK WHERE IT LIVES. `vt::Backend::Copy` is +// direction-agnostic on CUDA (`cudaMemcpyAsync` with `cudaMemcpyDefault`, +// `cuda_backend.cu:116-118`), so one path is correct whether the pages are +// device memory or host memory. That is deliberate and it is what keeps this +// file from re-deriving the runner's residency policy: `VT_DEVICE_KV_CACHE=0` +// moves the pages back to the host and needs no branch here. +// +// A CPU QUEUE KEEPS THE DIRECT PATH, byte-for-byte. There is no backend to ask +// and the host IS the device, so every `--device cpu` run copies exactly the +// bytes it copied before, through `std::memcpy` rather than through a backend. +class PageIo { + public: + explicit PageIo(vt::Queue& q) : q_(q) { + if (q.device.type == vt::DeviceType::kCPU) return; + b_ = vt::TryGetBackend(q.device); + if (b_ == nullptr) { + Fail("this step arrived on a non-CPU queue whose backend is not " + "registered in this build, so the engine's KV pages cannot be " + "staged to the host. Every reader and writer of a cache on this " + "model's forward is a host loop, and the runner allocates those " + "pages on the queue's device."); + } + } -void WriteTensorElem(const vt::Tensor& t, int64_t i, float v) { - if (t.dtype == vt::DType::kF32) { - t.Ptr()[i] = v; - return; + // `bytes` from `base + byte_off` into the host buffer `dst`. + void Read(const void* base, size_t byte_off, size_t bytes, void* dst) { + const uint8_t* src = static_cast(base) + byte_off; + if (b_ == nullptr) { + std::memcpy(dst, src, bytes); + return; + } + b_->Copy(q_, dst, src, bytes); + // SYNCHRONISE ON EVERY SPAN, not once at the end. `Copy` is asynchronous on + // this queue and the staging buffer is REUSED by the next span, so a + // deferred wait would read one row while the driver is still writing the + // one before it -- and would hand the driver a pageable source that the + // next iteration has already overwritten. + b_->Synchronize(q_); } - t.Ptr()[i] = vt::F32ToBF16(v); + + // `bytes` from the host buffer `src` into `base + byte_off`. + void Write(void* base, size_t byte_off, size_t bytes, const void* src) { + uint8_t* dst = static_cast(base) + byte_off; + if (b_ == nullptr) { + std::memcpy(dst, src, bytes); + return; + } + b_->Copy(q_, dst, src, bytes); + b_->Synchronize(q_); + } + + private: + vt::Queue& q_; + vt::Backend* b_ = nullptr; +}; + +// The host staging buffer for one span, grown to fit and never shrunk. One per +// `LoadCaches` / `StoreCaches` call, so the largest span a step touches is the +// whole cost. +uint8_t* Staging(std::vector* buf, size_t bytes) { + if (buf->size() < bytes) buf->resize(bytes); + return buf->data(); } // Every storage type this row can address. A quantized or fp8 page is refused @@ -532,6 +612,12 @@ void LoadCaches(const Glm5NextParams& p, const KvBinding& b, const int64_t conv_elems = kd.conv_dim() * kd.conv_kernel_size; const int64_t rec_elems = kd.num_heads * kd.head_dim * kd.head_dim; + // W9c-3b (#2480): every span below is STAGED through the backend rather than + // dereferenced, because on a device queue these pointers are the runner's + // `vt::Alloc` allocations. See `PageIo`. + PageIo io(input.queue); + std::vector span; + for (int64_t l = 0; l < L; ++l) { const LayerKvBinding& lb = b.layers[static_cast(l)]; LayerCache& c = (*out)[static_cast(l)]; @@ -542,27 +628,39 @@ void LoadCaches(const Glm5NextParams& p, const KvBinding& b, c.kda.assign(1, glm5_next_kda::Glm5NextKdaCache{}); glm5_next_kda::Glm5NextKdaCache& kc = c.kda[0]; kc.conv_state.resize(static_cast(conv_elems)); + const size_t conv_elt = vt::SizeOf(gs.conv_state.dtype); const int64_t cbase = b.state_slot * conv_elems; + uint8_t* cbuf = Staging(&span, static_cast(conv_elems) * conv_elt); + io.Read(gs.conv_state.data, static_cast(cbase) * conv_elt, + static_cast(conv_elems) * conv_elt, cbuf); for (int64_t i = 0; i < conv_elems; ++i) kc.conv_state[static_cast(i)] = - ReadTensorElem(gs.conv_state, cbase + i); + ReadSpanElem(cbuf, gs.conv_state.dtype, i); kc.recurrent_state.resize(static_cast(rec_elems)); + const size_t ssm_elt = vt::SizeOf(gs.ssm_state.dtype); const int64_t rbase = b.state_slot * rec_elems; + uint8_t* rbuf = Staging(&span, static_cast(rec_elems) * ssm_elt); + io.Read(gs.ssm_state.data, static_cast(rbase) * ssm_elt, + static_cast(rec_elems) * ssm_elt, rbuf); for (int64_t i = 0; i < rec_elems; ++i) kc.recurrent_state[static_cast(i)] = - ReadTensorElem(gs.ssm_state, rbase + i); + ReadSpanElem(rbuf, gs.ssm_state.dtype, i); continue; } c.dsa.cached_len = b.cached_len; const PagedKvCache& lat = attn_kv[static_cast(lb.latent)]; const std::vector& lblocks = b.group_blocks[static_cast(lb.latent_group)]; + const size_t lat_elt = vt::SizeOf(lat.dtype); c.dsa.k_pass.resize(static_cast(b.cached_len * latent_row)); for (int64_t t = 0; t < b.cached_len; ++t) { const int64_t off = PagedRowOffset(lblocks, b.block_size, latent_row, t); + uint8_t* row = Staging(&span, static_cast(latent_row) * lat_elt); + io.Read(lat.data, static_cast(off) * lat_elt, + static_cast(latent_row) * lat_elt, row); for (int64_t i = 0; i < latent_row; ++i) c.dsa.k_pass[static_cast(t * latent_row + i)] = - ReadElem(lat, off + i); + ReadSpanElem(row, lat.dtype, i); } // A `shared` layer never appends to its side cache and never validates it // (`glm5_next_attn.cpp:353-366`), so its stored rows do not exist and @@ -571,12 +669,16 @@ void LoadCaches(const Glm5NextParams& p, const KvBinding& b, const PagedKvCache& ix = attn_kv[static_cast(lb.indexer)]; const std::vector& iblocks = b.group_blocks[static_cast(lb.indexer_group)]; + const size_t ix_elt = vt::SizeOf(ix.dtype); c.dsa.indexer_packed.resize(static_cast(b.cached_len * indexer_row)); for (int64_t t = 0; t < b.cached_len; ++t) { const int64_t off = PagedRowOffset(iblocks, b.block_size, indexer_row, t); + uint8_t* row = Staging(&span, static_cast(indexer_row) * ix_elt); + io.Read(ix.data, static_cast(off) * ix_elt, + static_cast(indexer_row) * ix_elt, row); for (int64_t i = 0; i < indexer_row; ++i) c.dsa.indexer_packed[static_cast(t * indexer_row + i)] = - ReadElem(ix, off + i); + ReadSpanElem(row, ix.dtype, i); } } } @@ -597,6 +699,12 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, const glm5_next_kda::Glm5NextKdaDims kd = KdaDimsFrom(p); const int64_t conv_elems = kd.conv_dim() * kd.conv_kernel_size; const int64_t rec_elems = kd.num_heads * kd.head_dim * kd.head_dim; + // W9c-3b (#2480): the row is BUILT in the staging buffer and then copied + // through the backend. Writing straight into `lat.data` was a host store into + // a `cudaMalloc` allocation on every `--device cuda` step, and it is where the + // three SIGSEGV legs of O46 died. See `PageIo` and spec `## Owed` O49. + PageIo io(input.queue); + std::vector span; for (int64_t l = 0; l < L; ++l) { const LayerKvBinding& lb = b.layers[static_cast(l)]; @@ -623,14 +731,22 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, std::to_string(rec_elems) + "."); } const GdnStateCache& gs = gdn[static_cast(lb.recurrent)]; + const size_t conv_elt = vt::SizeOf(gs.conv_state.dtype); const int64_t cbase = b.state_slot * conv_elems; + uint8_t* cbuf = Staging(&span, static_cast(conv_elems) * conv_elt); for (int64_t i = 0; i < conv_elems; ++i) - WriteTensorElem(gs.conv_state, cbase + i, - kc.conv_state[static_cast(i)]); + WriteSpanElem(cbuf, gs.conv_state.dtype, i, + kc.conv_state[static_cast(i)]); + io.Write(gs.conv_state.data, static_cast(cbase) * conv_elt, + static_cast(conv_elems) * conv_elt, cbuf); + const size_t ssm_elt = vt::SizeOf(gs.ssm_state.dtype); const int64_t rbase = b.state_slot * rec_elems; + uint8_t* rbuf = Staging(&span, static_cast(rec_elems) * ssm_elt); for (int64_t i = 0; i < rec_elems; ++i) - WriteTensorElem(gs.ssm_state, rbase + i, - kc.recurrent_state[static_cast(i)]); + WriteSpanElem(rbuf, gs.ssm_state.dtype, i, + kc.recurrent_state[static_cast(i)]); + io.Write(gs.ssm_state.data, static_cast(rbase) * ssm_elt, + static_cast(rec_elems) * ssm_elt, rbuf); continue; } if (c.dsa.cached_len != total) { @@ -646,11 +762,15 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, const PagedKvCache& lat = attn_kv[static_cast(lb.latent)]; const std::vector& lblocks = b.group_blocks[static_cast(lb.latent_group)]; + const size_t lat_elt = vt::SizeOf(lat.dtype); for (int64_t t = b.cached_len; t < total; ++t) { const int64_t off = PagedRowOffset(lblocks, b.block_size, latent_row, t); + uint8_t* row = Staging(&span, static_cast(latent_row) * lat_elt); for (int64_t i = 0; i < latent_row; ++i) - WriteElem(lat, off + i, - c.dsa.k_pass[static_cast(t * latent_row + i)]); + WriteSpanElem(row, lat.dtype, i, + c.dsa.k_pass[static_cast(t * latent_row + i)]); + io.Write(lat.data, static_cast(off) * lat_elt, + static_cast(latent_row) * lat_elt, row); } if (!lb.has_own_indexer) { if (!c.dsa.indexer_packed.empty()) { @@ -671,11 +791,15 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, const PagedKvCache& ix = attn_kv[static_cast(lb.indexer)]; const std::vector& iblocks = b.group_blocks[static_cast(lb.indexer_group)]; + const size_t ix_elt = vt::SizeOf(ix.dtype); for (int64_t t = b.cached_len; t < total; ++t) { const int64_t off = PagedRowOffset(iblocks, b.block_size, indexer_row, t); + uint8_t* row = Staging(&span, static_cast(indexer_row) * ix_elt); for (int64_t i = 0; i < indexer_row; ++i) - WriteElem(ix, off + i, - c.dsa.indexer_packed[static_cast(t * indexer_row + i)]); + WriteSpanElem(row, ix.dtype, i, + c.dsa.indexer_packed[static_cast(t * indexer_row + i)]); + io.Write(ix.data, static_cast(off) * ix_elt, + static_cast(indexer_row) * ix_elt, row); } } } diff --git a/tests/vllm/models/test_glm5_next_forward.cpp b/tests/vllm/models/test_glm5_next_forward.cpp index 4ef3ffb52..5d254335b 100644 --- a/tests/vllm/models/test_glm5_next_forward.cpp +++ b/tests/vllm/models/test_glm5_next_forward.cpp @@ -62,6 +62,7 @@ #include #include #include +#include // W9c-3b: the shadow backend's memcpy/memset #include #include #include @@ -71,6 +72,7 @@ #include "support/glm5_next_gguf_fixture.h" #include "vllm/model_executor/models/glm5_next_bridge.h" #include "vllm/model_executor/models/glm5_next_forward.h" +#include "vllm/model_executor/models/glm5_next_kv.h" // W9c-3b (#2480) #include "vllm/model_executor/models/glm5_next_layer.h" #include "vllm/model_executor/models/glm5_next_loader.h" #include "vllm/model_executor/models/glm5_next_moe.h" @@ -1430,3 +1432,367 @@ TEST_CASE("glm5_next W5b-2c: ModelRegistry::Forward NARROWS its refusal, not dro REQUIRE(kimi.factory != nullptr); CHECK_FALSE(kimi.factory->consumes_multi_kv); } + +// ─── W9c-3b (#2480): THE ENGINE'S PAGES ARE NOT ALWAYS HOST MEMORY ─────────── +// +// WHAT WENT WRONG AND WHY NOTHING SAW IT. Every case above hands this model a +// CPU queue, and on a CPU queue `GPUModelRunner::CacheBuffer` keeps its pages in +// a `std::vector` (`v1/worker/gpu/runner.cpp:575-593`). On any other +// queue `kv_cache_backend_resident_` is true (`:1119-1122`) and every paged +// cache and every recurrent state is a `vt::Alloc` allocation -- `cudaMalloc` on +// CUDA, which is NOT host-dereferenceable even on GB10 +// (`src/vt/cuda/cuda_backend.cu:354-391`, held by a `static_assert` that names +// #844 and #1435 as the same fault twice already). +// +// `glm5_next_kv.cpp` read and wrote those pages with plain host loops, so a +// `--device cuda` step was a host store into device memory. `LoadCaches` returns +// before touching a page on a fresh sequence, so the first such access on step 1 +// is inside `StoreCaches`, AFTER the whole forward has returned -- which is why +// the three legs in spec O46 died with SIGSEGV having emitted no token, and why +// the last two lines on their stderr were the MoE arm's two once-flags. O49 +// carries the bisect. +// +// WHAT THIS CASE MEASURES, AND WHY IT IS NOT A CRASH TEST. A test binary cannot +// hold a `cudaMalloc` pointer, and a SIGSEGV is not an assertion. `ShadowBackend` +// is the next-strongest thing and is deterministic on every platform: the +// pointer `Alloc` hands back is a DECOY filled with a poison pattern, and the +// real storage lives in a side block only `Copy` can reach. Code that +// dereferences the pointer therefore reads poison and writes where nothing will +// ever look, and code that goes through the backend is correct -- which is +// exactly the distinction the defect is, with the fault turned into a value. +namespace { + +class ShadowBackend final : public vt::Backend { + public: + void* Alloc(size_t bytes) override { + const size_t n = bytes == 0 ? 1 : bytes; + auto block = std::make_unique(); + block->decoy.assign(n, kPoison); + block->shadow.assign(n, 0); + void* p = block->decoy.data(); + blocks_.push_back(std::move(block)); + ++allocs; + return p; + } + void Free(void*) override {} + void Memset(vt::Queue&, void* p, int v, size_t bytes) override { + uint8_t* dst = Translate(p, bytes); + std::memset(dst, v, bytes); + } + void Copy(vt::Queue&, void* dst, const void* src, size_t bytes) override { + ++copies; + uint8_t* d = Translate(dst, bytes); + const uint8_t* s = Translate(const_cast(src), bytes); + std::memcpy(d, s, bytes); + } + vt::Queue CreateQueue() override { + return vt::Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + } + void DestroyQueue(vt::Queue&) override {} + // The GB10 CUDA backend's own two answers (`cuda_backend.cu:113` and the + // inherited default): one physical RAM, and still not host-dereferenceable. + bool UnifiedMemory() const override { return true; } + bool DeviceMemoryIsHostAddressable() const override { return false; } + + // Is `p` a byte a HOST loop would have had to fault on? Used by the case to + // read the shadow without going through `Copy` twice. + const uint8_t* ShadowOf(const void* p, size_t bytes) const { + for (const std::unique_ptr& b : blocks_) { + const uint8_t* base = b->decoy.data(); + const auto* q = static_cast(p); + if (q >= base && q + bytes <= base + b->decoy.size()) + return b->shadow.data() + (q - base); + } + return nullptr; + } + + static constexpr uint8_t kPoison = 0xDD; + int allocs = 0; + int copies = 0; + + private: + struct Block { + std::vector decoy; + std::vector shadow; + }; + // A pointer into one of our decoys resolves to the SAME offset in its shadow; + // anything else is an ordinary host buffer and is used as it is. + uint8_t* Translate(void* p, size_t bytes) { + for (const std::unique_ptr& b : blocks_) { + uint8_t* base = b->decoy.data(); + auto* q = static_cast(p); + if (q >= base && q + bytes <= base + b->decoy.size()) + return b->shadow.data() + (q - base); + } + return static_cast(p); + } + std::vector> blocks_; +}; + +ShadowBackend& Shadow() { + static ShadowBackend b; + return b; +} + +struct ShadowRegistrar { + ShadowRegistrar() { + vt::RegisterBackend(vt::Device{vt::DeviceType::kXPU, 0}, &Shadow()); + } +}; +const ShadowRegistrar kShadowRegistrar; + +// A `Topology` whose every page and every recurrent state has been re-homed +// onto the shadow backend, with the sizes and the block permutation unchanged. +// The `vt::Tensor` device tags move with them, because a state that says kCPU +// while its bytes are on a device is the lie this whole case is about. +struct ShadowTopology { + Topology t; + ShadowTopology() { + for (size_t i = 0; i < t.attn_kv.size(); ++i) { + const size_t n = t.attn_bytes[i].size(); + t.attn_kv[i].data = Shadow().Alloc(n); + } + for (size_t j = 0; j < t.gdn.size(); ++j) { + t.gdn[j].conv_state.data = Shadow().Alloc(t.conv_bytes[j].size()); + t.gdn[j].conv_state.device = vt::Device{vt::DeviceType::kXPU, 0}; + t.gdn[j].ssm_state.data = Shadow().Alloc(t.ssm_bytes[j].size()); + t.gdn[j].ssm_state.device = vt::Device{vt::DeviceType::kXPU, 0}; + t.gdn[j].states = {t.gdn[j].conv_state, t.gdn[j].ssm_state}; + } + // NOT `Publish()`: that re-points `attn_kv[i].data` back at the host + // vectors, which would silently undo this whole fixture. + t.mk.layer_names = &t.names; + t.mk.group_ids = &t.group_ids; + t.mk.layer_indices = &t.layer_indices; + t.mk.payload_kinds = &t.payload_kinds; + t.mk.payload_slots = &t.payload_slots; + t.mk.group_block_tables = &t.group_bt; + t.mk.group_block_table_cols = &t.group_cols; + } +}; + +// Read `n` f32 values out of the SHADOW at element offset `first`. Deliberately +// not `Copy`: a case that read the storage the same way the code under test does +// would pass whenever the two agreed, including when both were the decoy. +std::vector ShadowFloats(const void* base, int64_t first, int64_t n) { + const auto* p = static_cast(base) + + static_cast(first) * sizeof(float); + const uint8_t* s = Shadow().ShadowOf(p, static_cast(n) * sizeof(float)); + REQUIRE(s != nullptr); + std::vector out(static_cast(n)); + std::memcpy(out.data(), s, out.size() * sizeof(float)); + return out; +} + +// A deterministic, non-constant pattern. Constant fill would pass against a +// zeroed shadow for the zero value and against poison for nothing, so the values +// are spread and none of them is 0. +float Pattern(int64_t tag, int64_t i) { + return 1.0F + static_cast(tag) * 0.125F + static_cast(i) * 0.03125F; +} + +} // namespace + +TEST_CASE("glm5_next W9c-3b: the KV binding COPIES the engine's pages instead " + "of dereferencing them") { + TempFile f(BuildFixture()); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + std::unique_ptr model = LoadThroughRegistry(g); + REQUIRE(model != nullptr); + const vllm::Glm5NextParams& p = Weights(model).params; + REQUIRE(p.num_hidden_layers == kLayers); + + ShadowTopology pages; + const int64_t latent_row = Topology::LatentRow(); + const int64_t indexer_row = Topology::IndexerRow(); + const int64_t conv_elems = Topology::ConvElems(); + const int64_t rec_elems = Topology::RecElems(); + constexpr int64_t kNewTokens = 2; + + Step s1({1, 2}); + s1.queue = vt::Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + s1.Bind(pages.t); + const vllm::ModelForwardInput in1 = s1.Get(); + const gn::KvBinding b1 = gn::ResolveKvBinding(p, in1); + REQUIRE(b1.cached_len == 0); + REQUIRE(b1.new_tokens == kNewTokens); + + // A fresh sequence reads NOTHING, so this call must not touch a page at all. + std::vector caches; + gn::LoadCaches(p, b1, in1, &caches); + REQUIRE(caches.size() == static_cast(kLayers)); + + // Fill the states the forward would have produced. The VALUES are the point: + // they have to arrive in the shadow, at the offsets the block permutation + // puts them at, or the write went to the decoy. + for (int64_t l = 0; l < kLayers; ++l) { + gn::LayerCache& c = caches[static_cast(l)]; + if (l == Topology::kDsaLayer) { + c.dsa.cached_len = kNewTokens; + c.dsa.k_pass.resize(static_cast(kNewTokens * latent_row)); + for (size_t i = 0; i < c.dsa.k_pass.size(); ++i) + c.dsa.k_pass[i] = Pattern(l, static_cast(i)); + c.dsa.indexer_packed.resize(static_cast(kNewTokens * indexer_row)); + for (size_t i = 0; i < c.dsa.indexer_packed.size(); ++i) + c.dsa.indexer_packed[i] = Pattern(l + 64, static_cast(i)); + continue; + } + c.kda.assign(1, vllm::glm5_next_kda::Glm5NextKdaCache{}); + c.kda[0].conv_state.resize(static_cast(conv_elems)); + for (size_t i = 0; i < c.kda[0].conv_state.size(); ++i) + c.kda[0].conv_state[i] = Pattern(l + 128, static_cast(i)); + c.kda[0].recurrent_state.resize(static_cast(rec_elems)); + for (size_t i = 0; i < c.kda[0].recurrent_state.size(); ++i) + c.kda[0].recurrent_state[i] = Pattern(l + 192, static_cast(i)); + } + + const int copies_before = Shadow().copies; + gn::StoreCaches(p, b1, caches, in1); + // The write went through the backend at all. Necessary, never sufficient -- + // the value checks below are what say it went to the right place. + CHECK(Shadow().copies > copies_before); + + // (1) THE PAGED ROWS. Read out of the SHADOW, at the flat slot the gathered + // block table maps each logical position to, so an implementation that + // addressed page `p` at block `p` lands on the wrong row rather than passing. + const gn::LayerKvBinding& lb = + b1.layers[static_cast(Topology::kDsaLayer)]; + const void* lat = in1.attn_kv[static_cast(lb.latent)].data; + const void* ix = in1.attn_kv[static_cast(lb.indexer)].data; + for (int64_t t = 0; t < kNewTokens; ++t) { + const std::vector got = + ShadowFloats(lat, Topology::Slot(t) * latent_row, latent_row); + for (int64_t i = 0; i < latent_row; ++i) { + CHECK(got[static_cast(i)] == + doctest::Approx(Pattern(Topology::kDsaLayer, t * latent_row + i))); + } + const std::vector gix = + ShadowFloats(ix, Topology::Slot(t) * indexer_row, indexer_row); + for (int64_t i = 0; i < indexer_row; ++i) { + CHECK(gix[static_cast(i)] == + doctest::Approx(Pattern(Topology::kDsaLayer + 64, t * indexer_row + i))); + } + } + + // (2) THE RECURRENT STATES, which are the FIRST thing `StoreCaches` writes on + // this model and therefore the byte the three `dgx:gpu0` legs died on. + for (int64_t l = 0; l < kLayers; ++l) { + if (l == Topology::kDsaLayer) continue; + const gn::LayerKvBinding& rb = b1.layers[static_cast(l)]; + const vllm::GdnStateCache& gs = + in1.gdn_state[static_cast(rb.recurrent)]; + const std::vector conv = ShadowFloats(gs.conv_state.data, 0, conv_elems); + for (int64_t i = 0; i < conv_elems; ++i) { + CHECK(conv[static_cast(i)] == doctest::Approx(Pattern(l + 128, i))); + } + const std::vector rec = ShadowFloats(gs.ssm_state.data, 0, rec_elems); + for (int64_t i = 0; i < rec_elems; ++i) { + CHECK(rec[static_cast(i)] == doctest::Approx(Pattern(l + 192, i))); + } + } + + // (3) THE READ DIRECTION, on a second step that has history. The decoy still + // holds nothing but poison, so a `LoadCaches` that dereferenced the page would + // hydrate every state from 0xDDDDDDDD instead of from what step 1 stored. + Step s2({3}, {}, kNewTokens); + s2.queue = vt::Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + s2.Bind(pages.t); + const vllm::ModelForwardInput in2 = s2.Get(); + const gn::KvBinding b2 = gn::ResolveKvBinding(p, in2); + REQUIRE(b2.cached_len == kNewTokens); + std::vector back; + gn::LoadCaches(p, b2, in2, &back); + REQUIRE(back.size() == static_cast(kLayers)); + + const gn::LayerCache& dsa = back[static_cast(Topology::kDsaLayer)]; + REQUIRE(dsa.dsa.k_pass.size() == + static_cast(kNewTokens * latent_row)); + for (size_t i = 0; i < dsa.dsa.k_pass.size(); ++i) { + CHECK(dsa.dsa.k_pass[i] == + doctest::Approx(Pattern(Topology::kDsaLayer, static_cast(i)))); + } + REQUIRE(dsa.dsa.indexer_packed.size() == + static_cast(kNewTokens * indexer_row)); + for (size_t i = 0; i < dsa.dsa.indexer_packed.size(); ++i) { + CHECK(dsa.dsa.indexer_packed[i] == + doctest::Approx(Pattern(Topology::kDsaLayer + 64, + static_cast(i)))); + } + for (int64_t l = 0; l < kLayers; ++l) { + if (l == Topology::kDsaLayer) continue; + const gn::LayerCache& c = back[static_cast(l)]; + REQUIRE(c.kda.size() == 1); + REQUIRE(c.kda[0].conv_state.size() == static_cast(conv_elems)); + for (size_t i = 0; i < c.kda[0].conv_state.size(); ++i) { + CHECK(c.kda[0].conv_state[i] == + doctest::Approx(Pattern(l + 128, static_cast(i)))); + } + REQUIRE(c.kda[0].recurrent_state.size() == static_cast(rec_elems)); + for (size_t i = 0; i < c.kda[0].recurrent_state.size(); ++i) { + CHECK(c.kda[0].recurrent_state[i] == + doctest::Approx(Pattern(l + 192, static_cast(i)))); + } + } +} + +TEST_CASE("glm5_next W9c-3b: a CPU queue writes the pages IN PLACE, and the " + "shadow backend is not involved") { + // The `--device cpu` arm, which is the one this wave must not move. The bytes + // land in the topology's OWN host vectors, at the slot the block permutation + // maps each position to, and no other backend is touched on the way. + // + // WHAT THIS CASE CANNOT SEE, stated because a mutation proved it. Deleting + // `PageIo`'s `if (q.device.type == kCPU) return;` -- so that a CPU queue + // resolves `vt::GetBackend(kCPU)` and bounces through it as well -- SURVIVES + // this case and the whole suite. It has to: the CPU backend's `Copy` IS + // `std::memcpy`, so the bytes are identical either way, and `Shadow().copies` + // below counts the SHADOW backend, which a CPU queue never reaches under + // either version. The early return is therefore not load-bearing for + // correctness. It is there so that a build with no CPU backend registered + // does not `Fail` on the host path, and so the `--device cpu` instruction + // stream is the one that was already measured. Spec `## Owed` O49 records the + // survival rather than leaving it for the next reader to rediscover. + TempFile f(BuildFixture()); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + std::unique_ptr model = LoadThroughRegistry(g); + const vllm::Glm5NextParams& p = Weights(model).params; + + Topology host; + Step s({1, 2}); + s.Bind(host); + const vllm::ModelForwardInput in = s.Get(); + const gn::KvBinding b = gn::ResolveKvBinding(p, in); + std::vector caches; + gn::LoadCaches(p, b, in, &caches); + const int64_t latent_row = Topology::LatentRow(); + gn::LayerCache& c = caches[static_cast(Topology::kDsaLayer)]; + c.dsa.cached_len = 2; + c.dsa.k_pass.assign(static_cast(2 * latent_row), 0.0F); + for (size_t i = 0; i < c.dsa.k_pass.size(); ++i) + c.dsa.k_pass[i] = Pattern(7, static_cast(i)); + c.dsa.indexer_packed.assign( + static_cast(2 * Topology::IndexerRow()), 0.5F); + for (int64_t l = 0; l < kLayers; ++l) { + if (l == Topology::kDsaLayer) continue; + gn::LayerCache& k = caches[static_cast(l)]; + k.kda.assign(1, vllm::glm5_next_kda::Glm5NextKdaCache{}); + k.kda[0].conv_state.assign(static_cast(Topology::ConvElems()), 0.25F); + k.kda[0].recurrent_state.assign(static_cast(Topology::RecElems()), 0.75F); + } + const int copies_before = Shadow().copies; + gn::StoreCaches(p, b, caches, in); + CHECK(Shadow().copies == copies_before); + + // And the bytes landed in the topology's OWN host vectors, at the permuted + // slot, so "no backend" did not become "no write". + const gn::LayerKvBinding& lb = + b.layers[static_cast(Topology::kDsaLayer)]; + const auto* lat = static_cast( + in.attn_kv[static_cast(lb.latent)].data); + for (int64_t t = 0; t < 2; ++t) { + for (int64_t i = 0; i < latent_row; ++i) { + CHECK(lat[Topology::Slot(t) * latent_row + i] == + doctest::Approx(Pattern(7, t * latent_row + i))); + } + } +}