From 17718f5b9d9d2d725402c04a20dd80966cda8576 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 13:57:46 +0000 Subject: [PATCH 01/10] spec(MODEL-MM-GLM53-FLASH): W9c-3b, because O46's mechanism is an artifact of two once-flags O46 reads the crashing legs' log order -- one device-arm announcement, then one fallback warning, then SIGSEGV -- as evidence that the fault is in the mixed residency state the per-layer fit guard creates. Both of those lines are `static bool said` once-flags, so their order says only that at least one layer staged and at least one later layer did not. It is not a location, and the process died somewhere else. It died in `StoreCaches`. On `--device cuda` the runner sets `kv_cache_backend_resident_` from `!platform.is_cpu()` and allocates every paged cache and every recurrent state with `vt::Alloc` -- `cudaMalloc` -- while `glm5_next_kv.cpp` reads and writes those pages with plain host loops. A `cudaMalloc` pointer is not host-dereferenceable on GB10 either, which `cuda_backend.cu` holds with a `static_assert` naming #844 and #1435 as the same fault twice before. `LoadCaches` returns before touching a page on a fresh sequence, so the first host access to device memory on step 1 is after the whole forward has returned: after the announcement, after the fallback warning, on a step that emitted no token. W9c-3a did not introduce it. `origin/main` refused a non-CPU queue several thousand instructions earlier, and removing that refusal made a pre-existing hole reachable. O46's three eliminated hypotheses are all inside `MoeExpertsKeepQuant` and are all correctly eliminated; the ordering that pointed them there was the artifact. This commit is the wave block and O49 only. The change it scopes is the next commit, and the on-box legs that discriminate the diagnosis are named here before either. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464, #2410 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 163 +++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index c6db8b964..0c326a682 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -3110,6 +3110,130 @@ raised. The predicate was replaced by the op-table probe described above, non-expert tower is what makes this model fit, and moving 94.6758 GiB of experts to the device does not remove it. +### W9c-3b — the KV binding reads the engine's pages where they ARE (GPU, medium) + +Issue: [#2480](https://github.com/mudler/vllm.cpp/issues/2480). +Claim: `CLAIM-GLM53-FLASH-W9C3B`. Base `3cd467643`. + +**O46's mechanism is FALSE, and the correction is the point of this wave.** O46 +reads the crashing legs' log order — one device-arm announcement, then one +fallback warning, then SIGSEGV — as evidence that "the fault is in the MIXED +residency state the per-layer fit guard creates". Both of those lines are +`static bool said` once-flags (`glm5_next_moe.cpp`, `AnnounceDeviceArmOnce` and +`WarnDeviceFallbackOnce`). Their order says only that at least one layer staged +and at least one LATER layer did not. It is not evidence about WHERE the process +died, and the process died somewhere else. + +**WHERE IT DIED, read off four files rather than guessed:** + +1. `GPUModelRunner::initialize_kv_cache` resolves + `kv_cache_backend_resident_ = !platforms::GetPlatform(dev.type).is_cpu() && (VT_DEVICE_KV_CACHE != "0")` + (`src/vllm/v1/worker/gpu/runner.cpp:1119-1122`). On `--device cuda` it is + **true**, and its own comment says the predicate is deliberately "has a + device", not "is CUDA". +2. `GPUModelRunner::CacheBuffer` then allocates every paged cache and every + recurrent state with `vt::Alloc(device_, ...)` (`runner.cpp:575-593`). On + CUDA that is `cudaMalloc`. The `host_data_` vector is the CPU-queue arm and + is not taken. +3. A `cudaMalloc` pointer is **not host-dereferenceable, on GB10 included**. + `src/vt/cuda/cuda_backend.cu:354-391` says so and holds it with a + `static_assert` that CUDA keeps the inherited + `DeviceMemoryIsHostAddressable() == false`, naming #844 and #1435 as the + SIGSEGVs that came from reading the WIDE predicate (`UnifiedMemory()`) + instead. +4. `glm5_next_kv.cpp` reads and writes those pages with **plain host loops**: + `ReadElem`/`WriteElem` are `static_cast(kv.data)[i]` and + `ReadTensorElem`/`WriteTensorElem` are `t.Ptr()[i]`. `LoadCaches` and + `StoreCaches` are their only callers, and + `ForwardGlm5NextForConditionalGeneration` is the only caller of those two. + +`LoadCaches` returns before touching any page on a fresh sequence +(`if (b.cached_len <= 0) return;` — "A FRESH SEQUENCE READS NOTHING"), so on +step 1 the FIRST host access to device memory is in `StoreCaches`, **after the +whole forward has returned**. That is after the announcement, after the fallback +warning, on a step that emitted no token, with no message — which is the +recorded signature exactly, and it is why the wall of a crashing leg sits at +roughly one forward step past its load. + +**THE DEFECT IS OLDER THAN THE ARM THAT EXPOSED IT.** Nothing in W9c-3a's diff +touches a KV page. `origin/main` never reached this because +`Glm5NextHostForward` refused a non-CPU queue several thousand instructions +earlier; W9c-3a removed that refusal, and a hole that had been unreachable +became reachable on the first `--device cuda` step that got past it. The three +hypotheses O46 eliminated are all inside `MoeExpertsKeepQuant` and are all +correctly eliminated. They were aimed at the wrong subsystem. + +**AND NOTHING GATED IT.** `ResolveKvBinding`, `LoadCaches` and `StoreCaches` +have **zero** call sites in `tests/` across the whole tree. The engine binding's +read/write path — the one W5b-2c landed and the one this crash is in — has no +unit coverage of its own; it is exercised only incidentally, through +`ModelRegistry::Forward` on a CPU queue, where every page IS host memory and the +defect cannot appear. + +#### Scope + +`src/vllm/model_executor/models/glm5_next_kv.cpp` only. Every span this file +reads or writes goes through `vt::Backend::Copy` instead of being dereferenced. +`Backend::Copy` is direction-agnostic on CUDA (`cudaMemcpyAsync` with +`cudaMemcpyDefault`, `cuda_backend.cu:116-118`), so ONE code path is correct +whether the pages are device memory or host memory. That is what keeps this file +from re-deriving the runner's residency policy: it never asks where the pages +are, and `VT_DEVICE_KV_CACHE=0` therefore needs no second branch here. + +On a CPU queue there is no backend to ask and the direct `memcpy` path is kept, +so every `--device cpu` run is byte-for-byte unchanged. + +Every span this file addresses is already contiguous, which is what makes the +change a substitution rather than a redesign: one paged row is `head_size` +elements at `PagedRowOffset(...)`, and a recurrent state is `conv_elems` or +`rec_elems` elements at `slot * elems`. + +#### Not in scope + +The other ten host arms O43 lists. The engine's residency policy. A +`ModelInfo` flag that would let a model ask for host-resident caches — that is a +shared-seam change with more than one consumer to survey, and it is not needed +to make this model correct. + +#### Gates + +* `test_glm5_next_forward` — a new case drives `ResolveKvBinding`, + `StoreCaches` and `LoadCaches` over the file's existing three-group + `Topology` fixture, with the pages re-pointed at a backend whose allocations + are NOT host-readable. RED before the change and GREEN after. +* The whole `glm5_next` suite set unchanged, by hand, with counts. +* Sibling inertness: `GlmMoeDsaForCausalLM`, DeepSeek-V4, dots3-note, Kimi. +* `dgx:gpu0`, the real 101.2535 GiB `UD-Q2_K_XL` artifact, four interleaved + legs on two binaries — see `## Evidence required` below. + +#### Evidence required + +The hermetic gate cannot prove the diagnosis, only the property. The diagnosis +is proved on the box, by a leg that changes NOTHING but where the pages live: + +| leg | binary | device | env | expected if the diagnosis holds | +|---|---|---|---|---| +| A | base `3cd467643` | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1 VT_DEVICE_KV_CACHE=0` | ` Paris.`, rc=0 | +| B | base `3cd467643` | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1` | SIGSEGV, rc=139 (the reproduction) | +| C | fixed | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1` | ` Paris.`, rc=0 | +| D | fixed | cpu | — | ` Paris.`, rc=0, byte-identical to C | + +A alone is the discriminator: it moves the pages to host memory and changes +nothing else, so an A that survives while B dies puts the fault in the page +residency and nowhere else. C is the fix. D is the control that says the fix did +not change the host arm. + +**No speed number is admissible from any of them** (O47: 89% generation spread +across three identical legs, cause unknown), and none is claimed. + +#### Stop conditions + +* If leg A dies too, the diagnosis is wrong: report it, keep O46 open, and do + not ship the change on a hermetic gate alone. +* If a fifth arm of this model turns out to host-dereference engine device + memory, that is a wider residency question than this wave, and it goes back as + `NEEDS_DECISION` rather than being absorbed here. + ## Tests to port `tests/models/` in transformers `v5.16.1` is the upstream suite. What is @@ -5441,6 +5565,45 @@ Debts this row carries, each visible rather than waived: without checking -- and because the ordering that caused it is this row's own script putting the sibling suites after the legs. +- **O49 -- O46's MECHANISM IS FALSIFIED, and the crash is a KV-page residency + defect that is OLDER than the arm which exposed it.** O46 infers "the fault is + in the MIXED residency state the per-layer fit guard creates" from the log + order of one device-arm announcement followed by one fallback warning. Both of + those lines are `static bool said` once-flags + (`glm5_next_moe.cpp`, `AnnounceDeviceArmOnce`, `WarnDeviceFallbackOnce`), so + the order carries only "at least one layer staged, at least one LATER layer did + not". It is not a location. + + The location is `StoreCaches`. On `--device cuda` the runner sets + `kv_cache_backend_resident_` true (`runner.cpp:1119-1122`) and allocates every + paged cache and every recurrent state with `vt::Alloc` — `cudaMalloc` + (`runner.cpp:575-593`) — while `glm5_next_kv.cpp` reads and writes those pages + with plain host loops. A `cudaMalloc` pointer is not host-dereferenceable on + GB10 (`cuda_backend.cu:354-391`, held by a `static_assert`, naming #844 and + #1435 as the same fault twice before). `LoadCaches` returns before touching a + page on a fresh sequence, so the FIRST host access to device memory on step 1 + is in `StoreCaches`, after the whole forward has returned — which is exactly + after the announcement, after the fallback warning, on a step that emitted no + token, with no message. + + W9c-3a did not introduce it. `origin/main` refused a non-CPU queue before + `StoreCaches` could run; removing that refusal made a pre-existing hole + reachable. O46's three eliminated hypotheses are all inside + `MoeExpertsKeepQuant` and are all correctly eliminated — they were aimed at the + wrong subsystem, because the ordering that pointed them there was an artifact + of two once-flags. + + **NOTHING GATED THE PATH IT IS IN.** `ResolveKvBinding`, `LoadCaches` and + `StoreCaches` have ZERO call sites anywhere in `tests/`. The engine binding + W5b-2c landed is reached only incidentally, through `ModelRegistry::Forward` on + a CPU queue, where every page IS host memory and this defect cannot appear. + W9c-3b ([#2480](https://github.com/mudler/vllm.cpp/issues/2480)) owns the fix + and the first direct gate over those three functions. + + What this costs the next reader is the general lesson, and it is O44's shape + once more: an ordering in a log is evidence about ORDER and not about place, + and a once-flag makes it evidence about even less than that. + ## Now `ACTIVE`, 2026-09-01. **THE KERNEL THAT BLOCKED THIS MODEL'S DEVICE ARM LANDED From b5e4cbf06813a3c22d5a84198074e75ac9e07605 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 14:00:06 +0000 Subject: [PATCH 02/10] fix(MODEL-MM-GLM53-FLASH): copy the engine's KV pages, because on a device queue they are not host memory `--device cuda` died with SIGSEGV on the real 101.24 GiB artifact, three legs of three, emitting no token. The fault is a host store into `cudaMalloc` memory in `StoreCaches`, and it is older than the device-expert arm that exposed it. On any non-CPU queue the runner sets `kv_cache_backend_resident_` from `!platform.is_cpu()` and allocates every paged cache and every recurrent state through `vt::Alloc` (`v1/worker/gpu/runner.cpp:575-593`, `:1119-1122`). On CUDA that is `cudaMalloc`, and `cuda_backend.cu:354-391` holds CUDA to `DeviceMemoryIsHostAddressable() == false` with a `static_assert` naming #844 and #1435 as the two SIGSEGVs that came from believing the wide predicate instead. This file read and wrote those pages with plain host loops. Every span it addresses is contiguous already -- one paged row is `head_size` elements at `PagedRowOffset`, and a recurrent state is a whole slot -- so the change is a substitution rather than a redesign: build or read the span in a host staging buffer and move it with `vt::Backend::Copy`. `Copy` is direction-agnostic on CUDA (`cudaMemcpyDefault`), so ONE path is correct whether the pages are device memory or host memory, and this file therefore never has to re-derive the runner's residency policy or read `VT_DEVICE_KV_CACHE`. A CPU queue has no backend to ask and keeps the direct `memcpy`, so every `--device cpu` run is byte-for-byte unchanged. `ResolveKvBinding`, `LoadCaches` and `StoreCaches` had ZERO call sites anywhere in `tests/` before this commit, which is how a host-only page reader landed in the engine binding at all. The new cases drive all three over this suite's own three-group topology with the pages re-homed onto a backend whose allocations are not host-readable: the pointer `Alloc` returns is a decoy full of poison and the real storage sits in a side block only `Copy` can reach, so dereferencing it is a wrong VALUE instead of a fault a test binary cannot survive. The second case pins the other direction -- a CPU queue must reach `Copy` zero times and must still write the bytes in place. The opt-in default stays OFF until the four `dgx:gpu0` legs the spec names have run, because a fix that has not been driven on the artifact that produced the crash is a hypothesis. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .../model_executor/models/glm5_next_kv.cpp | 186 +++++++-- tests/vllm/models/test_glm5_next_forward.cpp | 354 ++++++++++++++++++ 2 files changed, 509 insertions(+), 31 deletions(-) diff --git a/src/vllm/model_executor/models/glm5_next_kv.cpp b/src/vllm/model_executor/models/glm5_next_kv.cpp index dd98fa1b1..96bf3e1f8 100644 --- a/src/vllm/model_executor/models/glm5_next_kv.cpp +++ b/src/vllm/model_executor/models/glm5_next_kv.cpp @@ -4,6 +4,7 @@ #include "vllm/model_executor/models/glm5_next_kv.h" #include +#include // std::memcpy, the CPU-queue arm of PageIo #include #include #include // std::pair, in the per-cache geometry loop @@ -12,6 +13,7 @@ #include "vllm/model_executor/models/glm5_next_attn.h" // IndexerRoleFor #include "vllm/v1/attention/backend.h" // CommonAttentionMetadata #include "vllm/v1/attention/backends/gdn_attn.h" // GDNAttentionMetadata +#include "vt/backend.h" // vt::Backend, vt::TryGetBackend — W9c-3b (#2480) #include "vt/dtype.h" namespace vllm::glm5_next { @@ -64,33 +66,111 @@ std::string ChannelSummary(const MultiKvCacheIndex& mk) { return s; } -// Read one element of a paged buffer. `kAuto` fp8 means the dtype IS the -// storage type; anything else is refused before this is reached. -float ReadElem(const PagedKvCache& kv, int64_t i) { - if (kv.dtype == vt::DType::kF32) - return static_cast(kv.data)[i]; - return vt::BF16ToF32(static_cast(kv.data)[i]); +// Read one element out of a HOST-side span of `dt` storage. `kAuto` fp8 means +// the dtype IS the storage type; anything else is refused before this is +// reached. +// +// W9c-3b (#2480): these four take a raw host pointer rather than the +// `PagedKvCache` or the `vt::Tensor`, because the buffer they address is no +// longer necessarily the engine's own. On a device queue it is the staging span +// `PageIo` below filled, and giving these functions the cache would let a +// future edit reach past the staging and dereference the engine's pointer again +// -- which is the whole defect. +float ReadSpanElem(const void* host, vt::DType dt, int64_t i) { + if (dt == vt::DType::kF32) return static_cast(host)[i]; + return vt::BF16ToF32(static_cast(host)[i]); } -void WriteElem(const PagedKvCache& kv, int64_t i, float v) { - if (kv.dtype == vt::DType::kF32) { - static_cast(kv.data)[i] = v; +void WriteSpanElem(void* host, vt::DType dt, int64_t i, float v) { + if (dt == vt::DType::kF32) { + static_cast(host)[i] = v; return; } - static_cast(kv.data)[i] = vt::F32ToBF16(v); + static_cast(host)[i] = vt::F32ToBF16(v); } -float ReadTensorElem(const vt::Tensor& t, int64_t i) { - if (t.dtype == vt::DType::kF32) return t.Ptr()[i]; - return vt::BF16ToF32(t.Ptr()[i]); -} +// ─── W9c-3b (#2480): THE ENGINE'S PAGES ARE NOT ALWAYS HOST MEMORY ─────────── +// +// EVERY reader and writer in this file used to dereference `PagedKvCache::data` +// and `GdnStateCache`'s tensors directly, and on `--device cuda` those are +// `cudaMalloc` pointers. `GPUModelRunner::initialize_kv_cache` resolves +// `kv_cache_backend_resident_` from `!platform.is_cpu()` +// (`v1/worker/gpu/runner.cpp:1119-1122`) and `CacheBuffer` then allocates every +// paged cache and every recurrent state through `vt::Alloc(device, ...)` +// (`:575-593`); a `cudaMalloc` pointer is NOT host-dereferenceable, on GB10 +// included, and `src/vt/cuda/cuda_backend.cu:354-391` holds CUDA to +// `DeviceMemoryIsHostAddressable() == false` with a `static_assert` naming #844 +// and #1435 as the two SIGSEGVs that came from believing otherwise. +// +// So this was a host store into device memory, and it is why `--device cuda` +// died with SIGSEGV after a forward that had already completed: `LoadCaches` +// returns before touching a page on a fresh sequence, so the FIRST such access +// on step 1 is in `StoreCaches`, after the whole stack has run and before any +// token is emitted. Spec `## Owed` O49 carries the bisect and falsifies O46's +// mixed-residency reading of the same evidence. +// +// COPY THE SPAN, DO NOT ASK WHERE IT LIVES. `vt::Backend::Copy` is +// direction-agnostic on CUDA (`cudaMemcpyAsync` with `cudaMemcpyDefault`, +// `cuda_backend.cu:116-118`), so one path is correct whether the pages are +// device memory or host memory. That is deliberate and it is what keeps this +// file from re-deriving the runner's residency policy: `VT_DEVICE_KV_CACHE=0` +// moves the pages back to the host and needs no branch here. +// +// A CPU QUEUE KEEPS THE DIRECT PATH, byte-for-byte. There is no backend to ask +// and the host IS the device, so every `--device cpu` run copies exactly the +// bytes it copied before, through `std::memcpy` rather than through a backend. +class PageIo { + public: + explicit PageIo(vt::Queue& q) : q_(q) { + if (q.device.type == vt::DeviceType::kCPU) return; + b_ = vt::TryGetBackend(q.device); + if (b_ == nullptr) { + Fail("this step arrived on a non-CPU queue whose backend is not " + "registered in this build, so the engine's KV pages cannot be " + "staged to the host. Every reader and writer of a cache on this " + "model's forward is a host loop, and the runner allocates those " + "pages on the queue's device."); + } + } -void WriteTensorElem(const vt::Tensor& t, int64_t i, float v) { - if (t.dtype == vt::DType::kF32) { - t.Ptr()[i] = v; - return; + // `bytes` from `base + byte_off` into the host buffer `dst`. + void Read(const void* base, size_t byte_off, size_t bytes, void* dst) { + const uint8_t* src = static_cast(base) + byte_off; + if (b_ == nullptr) { + std::memcpy(dst, src, bytes); + return; + } + b_->Copy(q_, dst, src, bytes); + // SYNCHRONISE ON EVERY SPAN, not once at the end. `Copy` is asynchronous on + // this queue and the staging buffer is REUSED by the next span, so a + // deferred wait would read one row while the driver is still writing the + // one before it -- and would hand the driver a pageable source that the + // next iteration has already overwritten. + b_->Synchronize(q_); } - t.Ptr()[i] = vt::F32ToBF16(v); + + // `bytes` from the host buffer `src` into `base + byte_off`. + void Write(void* base, size_t byte_off, size_t bytes, const void* src) { + uint8_t* dst = static_cast(base) + byte_off; + if (b_ == nullptr) { + std::memcpy(dst, src, bytes); + return; + } + b_->Copy(q_, dst, src, bytes); + b_->Synchronize(q_); + } + + private: + vt::Queue& q_; + vt::Backend* b_ = nullptr; +}; + +// The host staging buffer for one span, grown to fit and never shrunk. One per +// `LoadCaches` / `StoreCaches` call, so the largest span a step touches is the +// whole cost. +uint8_t* Staging(std::vector* buf, size_t bytes) { + if (buf->size() < bytes) buf->resize(bytes); + return buf->data(); } // Every storage type this row can address. A quantized or fp8 page is refused @@ -532,6 +612,12 @@ void LoadCaches(const Glm5NextParams& p, const KvBinding& b, const int64_t conv_elems = kd.conv_dim() * kd.conv_kernel_size; const int64_t rec_elems = kd.num_heads * kd.head_dim * kd.head_dim; + // W9c-3b (#2480): every span below is STAGED through the backend rather than + // dereferenced, because on a device queue these pointers are the runner's + // `vt::Alloc` allocations. See `PageIo`. + PageIo io(input.queue); + std::vector span; + for (int64_t l = 0; l < L; ++l) { const LayerKvBinding& lb = b.layers[static_cast(l)]; LayerCache& c = (*out)[static_cast(l)]; @@ -542,27 +628,39 @@ void LoadCaches(const Glm5NextParams& p, const KvBinding& b, c.kda.assign(1, glm5_next_kda::Glm5NextKdaCache{}); glm5_next_kda::Glm5NextKdaCache& kc = c.kda[0]; kc.conv_state.resize(static_cast(conv_elems)); + const size_t conv_elt = vt::SizeOf(gs.conv_state.dtype); const int64_t cbase = b.state_slot * conv_elems; + uint8_t* cbuf = Staging(&span, static_cast(conv_elems) * conv_elt); + io.Read(gs.conv_state.data, static_cast(cbase) * conv_elt, + static_cast(conv_elems) * conv_elt, cbuf); for (int64_t i = 0; i < conv_elems; ++i) kc.conv_state[static_cast(i)] = - ReadTensorElem(gs.conv_state, cbase + i); + ReadSpanElem(cbuf, gs.conv_state.dtype, i); kc.recurrent_state.resize(static_cast(rec_elems)); + const size_t ssm_elt = vt::SizeOf(gs.ssm_state.dtype); const int64_t rbase = b.state_slot * rec_elems; + uint8_t* rbuf = Staging(&span, static_cast(rec_elems) * ssm_elt); + io.Read(gs.ssm_state.data, static_cast(rbase) * ssm_elt, + static_cast(rec_elems) * ssm_elt, rbuf); for (int64_t i = 0; i < rec_elems; ++i) kc.recurrent_state[static_cast(i)] = - ReadTensorElem(gs.ssm_state, rbase + i); + ReadSpanElem(rbuf, gs.ssm_state.dtype, i); continue; } c.dsa.cached_len = b.cached_len; const PagedKvCache& lat = attn_kv[static_cast(lb.latent)]; const std::vector& lblocks = b.group_blocks[static_cast(lb.latent_group)]; + const size_t lat_elt = vt::SizeOf(lat.dtype); c.dsa.k_pass.resize(static_cast(b.cached_len * latent_row)); for (int64_t t = 0; t < b.cached_len; ++t) { const int64_t off = PagedRowOffset(lblocks, b.block_size, latent_row, t); + uint8_t* row = Staging(&span, static_cast(latent_row) * lat_elt); + io.Read(lat.data, static_cast(off) * lat_elt, + static_cast(latent_row) * lat_elt, row); for (int64_t i = 0; i < latent_row; ++i) c.dsa.k_pass[static_cast(t * latent_row + i)] = - ReadElem(lat, off + i); + ReadSpanElem(row, lat.dtype, i); } // A `shared` layer never appends to its side cache and never validates it // (`glm5_next_attn.cpp:353-366`), so its stored rows do not exist and @@ -571,12 +669,16 @@ void LoadCaches(const Glm5NextParams& p, const KvBinding& b, const PagedKvCache& ix = attn_kv[static_cast(lb.indexer)]; const std::vector& iblocks = b.group_blocks[static_cast(lb.indexer_group)]; + const size_t ix_elt = vt::SizeOf(ix.dtype); c.dsa.indexer_packed.resize(static_cast(b.cached_len * indexer_row)); for (int64_t t = 0; t < b.cached_len; ++t) { const int64_t off = PagedRowOffset(iblocks, b.block_size, indexer_row, t); + uint8_t* row = Staging(&span, static_cast(indexer_row) * ix_elt); + io.Read(ix.data, static_cast(off) * ix_elt, + static_cast(indexer_row) * ix_elt, row); for (int64_t i = 0; i < indexer_row; ++i) c.dsa.indexer_packed[static_cast(t * indexer_row + i)] = - ReadElem(ix, off + i); + ReadSpanElem(row, ix.dtype, i); } } } @@ -597,6 +699,12 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, const glm5_next_kda::Glm5NextKdaDims kd = KdaDimsFrom(p); const int64_t conv_elems = kd.conv_dim() * kd.conv_kernel_size; const int64_t rec_elems = kd.num_heads * kd.head_dim * kd.head_dim; + // W9c-3b (#2480): the row is BUILT in the staging buffer and then copied + // through the backend. Writing straight into `lat.data` was a host store into + // a `cudaMalloc` allocation on every `--device cuda` step, and it is where the + // three SIGSEGV legs of O46 died. See `PageIo` and spec `## Owed` O49. + PageIo io(input.queue); + std::vector span; for (int64_t l = 0; l < L; ++l) { const LayerKvBinding& lb = b.layers[static_cast(l)]; @@ -623,14 +731,22 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, std::to_string(rec_elems) + "."); } const GdnStateCache& gs = gdn[static_cast(lb.recurrent)]; + const size_t conv_elt = vt::SizeOf(gs.conv_state.dtype); const int64_t cbase = b.state_slot * conv_elems; + uint8_t* cbuf = Staging(&span, static_cast(conv_elems) * conv_elt); for (int64_t i = 0; i < conv_elems; ++i) - WriteTensorElem(gs.conv_state, cbase + i, - kc.conv_state[static_cast(i)]); + WriteSpanElem(cbuf, gs.conv_state.dtype, i, + kc.conv_state[static_cast(i)]); + io.Write(gs.conv_state.data, static_cast(cbase) * conv_elt, + static_cast(conv_elems) * conv_elt, cbuf); + const size_t ssm_elt = vt::SizeOf(gs.ssm_state.dtype); const int64_t rbase = b.state_slot * rec_elems; + uint8_t* rbuf = Staging(&span, static_cast(rec_elems) * ssm_elt); for (int64_t i = 0; i < rec_elems; ++i) - WriteTensorElem(gs.ssm_state, rbase + i, - kc.recurrent_state[static_cast(i)]); + WriteSpanElem(rbuf, gs.ssm_state.dtype, i, + kc.recurrent_state[static_cast(i)]); + io.Write(gs.ssm_state.data, static_cast(rbase) * ssm_elt, + static_cast(rec_elems) * ssm_elt, rbuf); continue; } if (c.dsa.cached_len != total) { @@ -646,11 +762,15 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, const PagedKvCache& lat = attn_kv[static_cast(lb.latent)]; const std::vector& lblocks = b.group_blocks[static_cast(lb.latent_group)]; + const size_t lat_elt = vt::SizeOf(lat.dtype); for (int64_t t = b.cached_len; t < total; ++t) { const int64_t off = PagedRowOffset(lblocks, b.block_size, latent_row, t); + uint8_t* row = Staging(&span, static_cast(latent_row) * lat_elt); for (int64_t i = 0; i < latent_row; ++i) - WriteElem(lat, off + i, - c.dsa.k_pass[static_cast(t * latent_row + i)]); + WriteSpanElem(row, lat.dtype, i, + c.dsa.k_pass[static_cast(t * latent_row + i)]); + io.Write(lat.data, static_cast(off) * lat_elt, + static_cast(latent_row) * lat_elt, row); } if (!lb.has_own_indexer) { if (!c.dsa.indexer_packed.empty()) { @@ -671,11 +791,15 @@ void StoreCaches(const Glm5NextParams& p, const KvBinding& b, const PagedKvCache& ix = attn_kv[static_cast(lb.indexer)]; const std::vector& iblocks = b.group_blocks[static_cast(lb.indexer_group)]; + const size_t ix_elt = vt::SizeOf(ix.dtype); for (int64_t t = b.cached_len; t < total; ++t) { const int64_t off = PagedRowOffset(iblocks, b.block_size, indexer_row, t); + uint8_t* row = Staging(&span, static_cast(indexer_row) * ix_elt); for (int64_t i = 0; i < indexer_row; ++i) - WriteElem(ix, off + i, - c.dsa.indexer_packed[static_cast(t * indexer_row + i)]); + WriteSpanElem(row, ix.dtype, i, + c.dsa.indexer_packed[static_cast(t * indexer_row + i)]); + io.Write(ix.data, static_cast(off) * ix_elt, + static_cast(indexer_row) * ix_elt, row); } } } diff --git a/tests/vllm/models/test_glm5_next_forward.cpp b/tests/vllm/models/test_glm5_next_forward.cpp index 4ef3ffb52..cff62468f 100644 --- a/tests/vllm/models/test_glm5_next_forward.cpp +++ b/tests/vllm/models/test_glm5_next_forward.cpp @@ -62,6 +62,7 @@ #include #include #include +#include // W9c-3b: the shadow backend's memcpy/memset #include #include #include @@ -71,6 +72,7 @@ #include "support/glm5_next_gguf_fixture.h" #include "vllm/model_executor/models/glm5_next_bridge.h" #include "vllm/model_executor/models/glm5_next_forward.h" +#include "vllm/model_executor/models/glm5_next_kv.h" // W9c-3b (#2480) #include "vllm/model_executor/models/glm5_next_layer.h" #include "vllm/model_executor/models/glm5_next_loader.h" #include "vllm/model_executor/models/glm5_next_moe.h" @@ -1430,3 +1432,355 @@ TEST_CASE("glm5_next W5b-2c: ModelRegistry::Forward NARROWS its refusal, not dro REQUIRE(kimi.factory != nullptr); CHECK_FALSE(kimi.factory->consumes_multi_kv); } + +// ─── W9c-3b (#2480): THE ENGINE'S PAGES ARE NOT ALWAYS HOST MEMORY ─────────── +// +// WHAT WENT WRONG AND WHY NOTHING SAW IT. Every case above hands this model a +// CPU queue, and on a CPU queue `GPUModelRunner::CacheBuffer` keeps its pages in +// a `std::vector` (`v1/worker/gpu/runner.cpp:575-593`). On any other +// queue `kv_cache_backend_resident_` is true (`:1119-1122`) and every paged +// cache and every recurrent state is a `vt::Alloc` allocation -- `cudaMalloc` on +// CUDA, which is NOT host-dereferenceable even on GB10 +// (`src/vt/cuda/cuda_backend.cu:354-391`, held by a `static_assert` that names +// #844 and #1435 as the same fault twice already). +// +// `glm5_next_kv.cpp` read and wrote those pages with plain host loops, so a +// `--device cuda` step was a host store into device memory. `LoadCaches` returns +// before touching a page on a fresh sequence, so the first such access on step 1 +// is inside `StoreCaches`, AFTER the whole forward has returned -- which is why +// the three legs in spec O46 died with SIGSEGV having emitted no token, and why +// the last two lines on their stderr were the MoE arm's two once-flags. O49 +// carries the bisect. +// +// WHAT THIS CASE MEASURES, AND WHY IT IS NOT A CRASH TEST. A test binary cannot +// hold a `cudaMalloc` pointer, and a SIGSEGV is not an assertion. `ShadowBackend` +// is the next-strongest thing and is deterministic on every platform: the +// pointer `Alloc` hands back is a DECOY filled with a poison pattern, and the +// real storage lives in a side block only `Copy` can reach. Code that +// dereferences the pointer therefore reads poison and writes where nothing will +// ever look, and code that goes through the backend is correct -- which is +// exactly the distinction the defect is, with the fault turned into a value. +namespace { + +class ShadowBackend final : public vt::Backend { + public: + void* Alloc(size_t bytes) override { + const size_t n = bytes == 0 ? 1 : bytes; + auto block = std::make_unique(); + block->decoy.assign(n, kPoison); + block->shadow.assign(n, 0); + void* p = block->decoy.data(); + blocks_.push_back(std::move(block)); + ++allocs; + return p; + } + void Free(void*) override {} + void Memset(vt::Queue&, void* p, int v, size_t bytes) override { + uint8_t* dst = Translate(p, bytes); + std::memset(dst, v, bytes); + } + void Copy(vt::Queue&, void* dst, const void* src, size_t bytes) override { + ++copies; + uint8_t* d = Translate(dst, bytes); + const uint8_t* s = Translate(const_cast(src), bytes); + std::memcpy(d, s, bytes); + } + vt::Queue CreateQueue() override { + return vt::Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + } + void DestroyQueue(vt::Queue&) override {} + // The GB10 CUDA backend's own two answers (`cuda_backend.cu:113` and the + // inherited default): one physical RAM, and still not host-dereferenceable. + bool UnifiedMemory() const override { return true; } + bool DeviceMemoryIsHostAddressable() const override { return false; } + + // Is `p` a byte a HOST loop would have had to fault on? Used by the case to + // read the shadow without going through `Copy` twice. + const uint8_t* ShadowOf(const void* p, size_t bytes) const { + for (const std::unique_ptr& b : blocks_) { + const uint8_t* base = b->decoy.data(); + const auto* q = static_cast(p); + if (q >= base && q + bytes <= base + b->decoy.size()) + return b->shadow.data() + (q - base); + } + return nullptr; + } + + static constexpr uint8_t kPoison = 0xDD; + int allocs = 0; + int copies = 0; + + private: + struct Block { + std::vector decoy; + std::vector shadow; + }; + // A pointer into one of our decoys resolves to the SAME offset in its shadow; + // anything else is an ordinary host buffer and is used as it is. + uint8_t* Translate(void* p, size_t bytes) { + for (const std::unique_ptr& b : blocks_) { + uint8_t* base = b->decoy.data(); + auto* q = static_cast(p); + if (q >= base && q + bytes <= base + b->decoy.size()) + return b->shadow.data() + (q - base); + } + return static_cast(p); + } + std::vector> blocks_; +}; + +ShadowBackend& Shadow() { + static ShadowBackend b; + return b; +} + +struct ShadowRegistrar { + ShadowRegistrar() { + vt::RegisterBackend(vt::Device{vt::DeviceType::kXPU, 0}, &Shadow()); + } +}; +const ShadowRegistrar kShadowRegistrar; + +// A `Topology` whose every page and every recurrent state has been re-homed +// onto the shadow backend, with the sizes and the block permutation unchanged. +// The `vt::Tensor` device tags move with them, because a state that says kCPU +// while its bytes are on a device is the lie this whole case is about. +struct ShadowTopology { + Topology t; + ShadowTopology() { + for (size_t i = 0; i < t.attn_kv.size(); ++i) { + const size_t n = t.attn_bytes[i].size(); + t.attn_kv[i].data = Shadow().Alloc(n); + } + for (size_t j = 0; j < t.gdn.size(); ++j) { + t.gdn[j].conv_state.data = Shadow().Alloc(t.conv_bytes[j].size()); + t.gdn[j].conv_state.device = vt::Device{vt::DeviceType::kXPU, 0}; + t.gdn[j].ssm_state.data = Shadow().Alloc(t.ssm_bytes[j].size()); + t.gdn[j].ssm_state.device = vt::Device{vt::DeviceType::kXPU, 0}; + t.gdn[j].states = {t.gdn[j].conv_state, t.gdn[j].ssm_state}; + } + // NOT `Publish()`: that re-points `attn_kv[i].data` back at the host + // vectors, which would silently undo this whole fixture. + t.mk.layer_names = &t.names; + t.mk.group_ids = &t.group_ids; + t.mk.layer_indices = &t.layer_indices; + t.mk.payload_kinds = &t.payload_kinds; + t.mk.payload_slots = &t.payload_slots; + t.mk.group_block_tables = &t.group_bt; + t.mk.group_block_table_cols = &t.group_cols; + } +}; + +// Read `n` f32 values out of the SHADOW at element offset `first`. Deliberately +// not `Copy`: a case that read the storage the same way the code under test does +// would pass whenever the two agreed, including when both were the decoy. +std::vector ShadowFloats(const void* base, int64_t first, int64_t n) { + const auto* p = static_cast(base) + + static_cast(first) * sizeof(float); + const uint8_t* s = Shadow().ShadowOf(p, static_cast(n) * sizeof(float)); + REQUIRE(s != nullptr); + std::vector out(static_cast(n)); + std::memcpy(out.data(), s, out.size() * sizeof(float)); + return out; +} + +// A deterministic, non-constant pattern. Constant fill would pass against a +// zeroed shadow for the zero value and against poison for nothing, so the values +// are spread and none of them is 0. +float Pattern(int64_t tag, int64_t i) { + return 1.0F + static_cast(tag) * 0.125F + static_cast(i) * 0.03125F; +} + +} // namespace + +TEST_CASE("glm5_next W9c-3b: the KV binding COPIES the engine's pages instead " + "of dereferencing them") { + TempFile f(BuildFixture()); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + std::unique_ptr model = LoadThroughRegistry(g); + REQUIRE(model != nullptr); + const vllm::Glm5NextParams& p = Weights(model).params; + REQUIRE(p.num_hidden_layers == kLayers); + + ShadowTopology pages; + const int64_t latent_row = Topology::LatentRow(); + const int64_t indexer_row = Topology::IndexerRow(); + const int64_t conv_elems = Topology::ConvElems(); + const int64_t rec_elems = Topology::RecElems(); + constexpr int64_t kNewTokens = 2; + + Step s1({1, 2}); + s1.queue = vt::Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + s1.Bind(pages.t); + const vllm::ModelForwardInput in1 = s1.Get(); + const gn::KvBinding b1 = gn::ResolveKvBinding(p, in1); + REQUIRE(b1.cached_len == 0); + REQUIRE(b1.new_tokens == kNewTokens); + + // A fresh sequence reads NOTHING, so this call must not touch a page at all. + std::vector caches; + gn::LoadCaches(p, b1, in1, &caches); + REQUIRE(caches.size() == static_cast(kLayers)); + + // Fill the states the forward would have produced. The VALUES are the point: + // they have to arrive in the shadow, at the offsets the block permutation + // puts them at, or the write went to the decoy. + for (int64_t l = 0; l < kLayers; ++l) { + gn::LayerCache& c = caches[static_cast(l)]; + if (l == Topology::kDsaLayer) { + c.dsa.cached_len = kNewTokens; + c.dsa.k_pass.resize(static_cast(kNewTokens * latent_row)); + for (size_t i = 0; i < c.dsa.k_pass.size(); ++i) + c.dsa.k_pass[i] = Pattern(l, static_cast(i)); + c.dsa.indexer_packed.resize(static_cast(kNewTokens * indexer_row)); + for (size_t i = 0; i < c.dsa.indexer_packed.size(); ++i) + c.dsa.indexer_packed[i] = Pattern(l + 64, static_cast(i)); + continue; + } + c.kda.assign(1, vllm::glm5_next_kda::Glm5NextKdaCache{}); + c.kda[0].conv_state.resize(static_cast(conv_elems)); + for (size_t i = 0; i < c.kda[0].conv_state.size(); ++i) + c.kda[0].conv_state[i] = Pattern(l + 128, static_cast(i)); + c.kda[0].recurrent_state.resize(static_cast(rec_elems)); + for (size_t i = 0; i < c.kda[0].recurrent_state.size(); ++i) + c.kda[0].recurrent_state[i] = Pattern(l + 192, static_cast(i)); + } + + const int copies_before = Shadow().copies; + gn::StoreCaches(p, b1, caches, in1); + // The write went through the backend at all. Necessary, never sufficient -- + // the value checks below are what say it went to the right place. + CHECK(Shadow().copies > copies_before); + + // (1) THE PAGED ROWS. Read out of the SHADOW, at the flat slot the gathered + // block table maps each logical position to, so an implementation that + // addressed page `p` at block `p` lands on the wrong row rather than passing. + const gn::LayerKvBinding& lb = + b1.layers[static_cast(Topology::kDsaLayer)]; + const void* lat = in1.attn_kv[static_cast(lb.latent)].data; + const void* ix = in1.attn_kv[static_cast(lb.indexer)].data; + for (int64_t t = 0; t < kNewTokens; ++t) { + const std::vector got = + ShadowFloats(lat, Topology::Slot(t) * latent_row, latent_row); + for (int64_t i = 0; i < latent_row; ++i) { + CHECK(got[static_cast(i)] == + doctest::Approx(Pattern(Topology::kDsaLayer, t * latent_row + i))); + } + const std::vector gix = + ShadowFloats(ix, Topology::Slot(t) * indexer_row, indexer_row); + for (int64_t i = 0; i < indexer_row; ++i) { + CHECK(gix[static_cast(i)] == + doctest::Approx(Pattern(Topology::kDsaLayer + 64, t * indexer_row + i))); + } + } + + // (2) THE RECURRENT STATES, which are the FIRST thing `StoreCaches` writes on + // this model and therefore the byte the three `dgx:gpu0` legs died on. + for (int64_t l = 0; l < kLayers; ++l) { + if (l == Topology::kDsaLayer) continue; + const gn::LayerKvBinding& rb = b1.layers[static_cast(l)]; + const vllm::GdnStateCache& gs = + in1.gdn_state[static_cast(rb.recurrent)]; + const std::vector conv = ShadowFloats(gs.conv_state.data, 0, conv_elems); + for (int64_t i = 0; i < conv_elems; ++i) { + CHECK(conv[static_cast(i)] == doctest::Approx(Pattern(l + 128, i))); + } + const std::vector rec = ShadowFloats(gs.ssm_state.data, 0, rec_elems); + for (int64_t i = 0; i < rec_elems; ++i) { + CHECK(rec[static_cast(i)] == doctest::Approx(Pattern(l + 192, i))); + } + } + + // (3) THE READ DIRECTION, on a second step that has history. The decoy still + // holds nothing but poison, so a `LoadCaches` that dereferenced the page would + // hydrate every state from 0xDDDDDDDD instead of from what step 1 stored. + Step s2({3}, {}, kNewTokens); + s2.queue = vt::Queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + s2.Bind(pages.t); + const vllm::ModelForwardInput in2 = s2.Get(); + const gn::KvBinding b2 = gn::ResolveKvBinding(p, in2); + REQUIRE(b2.cached_len == kNewTokens); + std::vector back; + gn::LoadCaches(p, b2, in2, &back); + REQUIRE(back.size() == static_cast(kLayers)); + + const gn::LayerCache& dsa = back[static_cast(Topology::kDsaLayer)]; + REQUIRE(dsa.dsa.k_pass.size() == + static_cast(kNewTokens * latent_row)); + for (size_t i = 0; i < dsa.dsa.k_pass.size(); ++i) { + CHECK(dsa.dsa.k_pass[i] == + doctest::Approx(Pattern(Topology::kDsaLayer, static_cast(i)))); + } + REQUIRE(dsa.dsa.indexer_packed.size() == + static_cast(kNewTokens * indexer_row)); + for (size_t i = 0; i < dsa.dsa.indexer_packed.size(); ++i) { + CHECK(dsa.dsa.indexer_packed[i] == + doctest::Approx(Pattern(Topology::kDsaLayer + 64, + static_cast(i)))); + } + for (int64_t l = 0; l < kLayers; ++l) { + if (l == Topology::kDsaLayer) continue; + const gn::LayerCache& c = back[static_cast(l)]; + REQUIRE(c.kda.size() == 1); + REQUIRE(c.kda[0].conv_state.size() == static_cast(conv_elems)); + for (size_t i = 0; i < c.kda[0].conv_state.size(); ++i) { + CHECK(c.kda[0].conv_state[i] == + doctest::Approx(Pattern(l + 128, static_cast(i)))); + } + REQUIRE(c.kda[0].recurrent_state.size() == static_cast(rec_elems)); + for (size_t i = 0; i < c.kda[0].recurrent_state.size(); ++i) { + CHECK(c.kda[0].recurrent_state[i] == + doctest::Approx(Pattern(l + 192, static_cast(i)))); + } + } +} + +TEST_CASE("glm5_next W9c-3b: a CPU queue keeps the DIRECT path, byte-for-byte") { + // The other half of the discriminator. The bounce must be selected by the + // QUEUE and by nothing else, so on a CPU queue the pages are written in place + // and `ShadowBackend::Copy` is never reached -- which is also what keeps every + // `--device cpu` run unchanged by this wave. + TempFile f(BuildFixture()); + const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); + std::unique_ptr model = LoadThroughRegistry(g); + const vllm::Glm5NextParams& p = Weights(model).params; + + Topology host; + Step s({1, 2}); + s.Bind(host); + const vllm::ModelForwardInput in = s.Get(); + const gn::KvBinding b = gn::ResolveKvBinding(p, in); + std::vector caches; + gn::LoadCaches(p, b, in, &caches); + const int64_t latent_row = Topology::LatentRow(); + gn::LayerCache& c = caches[static_cast(Topology::kDsaLayer)]; + c.dsa.cached_len = 2; + c.dsa.k_pass.assign(static_cast(2 * latent_row), 0.0F); + for (size_t i = 0; i < c.dsa.k_pass.size(); ++i) + c.dsa.k_pass[i] = Pattern(7, static_cast(i)); + c.dsa.indexer_packed.assign( + static_cast(2 * Topology::IndexerRow()), 0.5F); + for (int64_t l = 0; l < kLayers; ++l) { + if (l == Topology::kDsaLayer) continue; + gn::LayerCache& k = caches[static_cast(l)]; + k.kda.assign(1, vllm::glm5_next_kda::Glm5NextKdaCache{}); + k.kda[0].conv_state.assign(static_cast(Topology::ConvElems()), 0.25F); + k.kda[0].recurrent_state.assign(static_cast(Topology::RecElems()), 0.75F); + } + const int copies_before = Shadow().copies; + gn::StoreCaches(p, b, caches, in); + CHECK(Shadow().copies == copies_before); + + // And the bytes landed in the topology's OWN host vectors, at the permuted + // slot, so "no backend" did not become "no write". + const gn::LayerKvBinding& lb = + b.layers[static_cast(Topology::kDsaLayer)]; + const auto* lat = static_cast( + in.attn_kv[static_cast(lb.latent)].data); + for (int64_t t = 0; t < 2; ++t) { + for (int64_t i = 0; i < latent_row; ++i) { + CHECK(lat[Topology::Slot(t) * latent_row + i] == + doctest::Approx(Pattern(7, t * latent_row + i))); + } + } +} From 2e350862d9e74a2022e06671149d99a516d1743a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 14:06:53 +0000 Subject: [PATCH 03/10] spec(MODEL-MM-GLM53-FLASH): retract O46's mechanism where it is written, not only where it is replaced O49 says the mixed-residency reading of the three SIGSEGV legs is wrong. O46 and the `## Now` block still asserted it in their own words, and a record that contradicts itself is read by whoever opens it first rather than by whoever opens it last. O46 keeps its measured table and its three eliminated hypotheses, which are sound; the one sentence that turned a log ORDER into a fault LOCATION is retracted in place with the reason, and points at O49. `## Now` loses the same clause. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 0c326a682..9b4da9c66 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -5477,11 +5477,21 @@ Debts this row carries, each visible rather than waived: `139` is SIGSEGV. **THREE of three CUDA legs crashed and three of three CPU legs emitted ` Paris.`**, interleaved, from one binary on one box -- so this is - a property of the arm and not of the box's mood. Each crashing leg logged + a property of the run and not of the box's mood. Each crashing leg logged exactly one device-arm announcement followed by exactly one fallback warning - before dying -- so the fault is in the MIXED residency state the per-layer fit - guard creates, not in the device arm itself, which the CUDA unit gate exercises - cleanly at NMSE 3.833e-15 against the host arm. + before dying. + + **THE SENTENCE THAT USED TO FOLLOW IS FALSIFIED, and O49 carries the + replacement.** It read "so the fault is in the MIXED residency state the + per-layer fit guard creates". Both of those log lines are `static bool said` + once-flags, so their order says only that at least one layer staged and at + least one LATER layer did not; it is not a location. The fault is in + `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after + the whole forward has returned -- a defect older than this arm, which only + became reachable when the non-CPU refusal above it was removed. Read O49 + before spending anything on the mixed-residency story. The device arm itself is + unimplicated either way: the CUDA unit gate exercises it cleanly at NMSE + 3.833e-15 against the host arm. **This is a REGRESSION and it is named as one.** On the base tree the same command produced a clean refusal at 1066 s (the operand table above, leg C). @@ -5622,11 +5632,14 @@ expiry. **AND THEN IT SEGFAULTED, so the arm is OPT-IN and defaults OFF.** Driven end to end on the 101.24 GiB artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) emitting no token, against three CPU legs that all emitted ` Paris.`, -interleaved on one binary -- once the expert banks stopped fitting -and the arm fell back to the host mid-model. The base tree produced a clean -refusal on the same command, so this was a REGRESSION and the default is -restored byte-for-byte; O46 carries the four legs, the three hypotheses already -eliminated, and the one that is untested. O47 records the other half of that +interleaved on one binary. The base tree produced a clean refusal on the same +command, so this was a REGRESSION and the default is restored byte-for-byte; +O46 carries the four legs and the three hypotheses eliminated inside the MoE +arm. **O46's own MECHANISM is falsified and O49 replaces it**: the process dies +in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the +forward has already returned, which is a defect W9c-3a exposed rather than +introduced. W9c-3b ([#2480](https://github.com/mudler/vllm.cpp/issues/2480)) +owns the fix and the four `dgx:gpu0` legs that decide it. O47 records the other half of that job: THREE identical cpu legs spread 69% on wall and 89% on generation, so NO speed number was available on any axis and none is claimed. The n=2 mechanism this row briefly asserted -- page-cache eviction moving generation -- was From cd9f5023eed1805f17e61f262327d661968fd0bf Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 14:11:34 +0000 Subject: [PATCH 04/10] test(MODEL-MM-GLM53-FLASH): say what the CPU-queue case can see, because a mutation proved it sees less Dropping `PageIo`'s CPU early return -- so that a CPU queue resolves the CPU backend and bounces through it too -- SURVIVES the whole suite. It has to. The CPU backend's `Copy` IS `std::memcpy`, so the bytes are identical either way, and the case's counter watches the SHADOW backend, which a CPU queue never reaches under either version. So the early return is not load-bearing for correctness, and the case no longer claims it is. It is there so a build with no CPU backend registered does not `Fail` on the host path, and so the `--device cpu` instruction stream stays the one that was already measured. The case now asserts what it can see -- the bytes land in the topology's own host vectors at the permuted slot -- and names the surviving mutation in its own comment. The spec gains the mutation table with what each one killed, M3's survival included, and the note that the staging is per ROW rather than per block: about 180,000 round trips to hydrate a full 8192-token prefix where block coalescing would be about 5,600. Not done here, because this forward is a host reference at roughly 85 s per step and no speed number is admissible from this row, but written down rather than left to be rediscovered. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 37 ++++++++++++++++++++ tests/vllm/models/test_glm5_next_forward.cpp | 22 +++++++++--- 2 files changed, 54 insertions(+), 5 deletions(-) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 9b4da9c66..752f835b1 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -3226,6 +3226,34 @@ not change the host arm. **No speed number is admissible from any of them** (O47: 89% generation spread across three identical legs, cause unknown), and none is claimed. +#### Mutations, with what each one killed + +Every mutation is applied to PRODUCT code and rebuilt, because a mutation the +compiler rejects and a mutation the binary never contained both read as a pass. + +| # | mutation | result | +|---|---|---| +| M1 | `PageIo::Write` always takes the direct `memcpy` arm | KILLED, 1 case / 5637 assertions | +| M2 | `PageIo::Read` always takes the direct `memcpy` arm | KILLED, 1 case / 2818 assertions | +| M3 | `PageIo` drops its CPU early return, so a CPU queue bounces too | **SURVIVED** | +| M4 | delete the production `StoreCaches` call in `glm5_next_registry.cpp` | KILLED, 2 cases / 3 assertions -- the REACHABILITY mutation | + +**M3 SURVIVED and that is a finding rather than a gap to paper over.** It has +to survive: the CPU backend's `Copy` IS `std::memcpy`, so the bytes are +identical whichever arm runs, and the case's counter watches the SHADOW backend, +which a CPU queue never reaches under either version. The early return is +therefore not load-bearing for correctness. It is there so a build with no CPU +backend registered does not `Fail` on the host path, and so the `--device cpu` +instruction stream stays the one that was already measured. The case says so in +its own comment instead of asserting something it cannot see. + +M4 is the reachability answer: the production call site is +`ForwardGlm5NextForConditionalGeneration`, `ModelRegistry::Forward` is the entry +point, and deleting the `StoreCaches` line reds the two W5b-2c cases that read +the engine's pages back on a second step. The two W9c-3b cases do NOT red under +M4, because they call the three functions directly -- they localise the defect +and they are not the reachability proof, which is why both are kept. + #### Stop conditions * If leg A dies too, the diagnosis is wrong: report it, keep O46 open, and do @@ -5614,6 +5642,15 @@ Debts this row carries, each visible rather than waived: once more: an ordering in a log is evidence about ORDER and not about place, and a once-flag makes it evidence about even less than that. + **OWED, and small: the staging is PER ROW.** `PageIo` copies one paged row per + `Backend::Copy` and synchronises on each, so a step that hydrates a full 8192 + token prefix pays about 180,000 round trips across 11 DSA layers and two + caches. Every row inside one block is contiguous with the next, so coalescing + a block at a time would cut that to about 5,600. It is not done here because + this forward is a host reference at roughly 85 s per step and no speed number + is admissible from this row anyway (O47); it is written down so the next reader + does not have to rediscover the shape of the cost. + ## Now `ACTIVE`, 2026-09-01. **THE KERNEL THAT BLOCKED THIS MODEL'S DEVICE ARM LANDED diff --git a/tests/vllm/models/test_glm5_next_forward.cpp b/tests/vllm/models/test_glm5_next_forward.cpp index cff62468f..5d254335b 100644 --- a/tests/vllm/models/test_glm5_next_forward.cpp +++ b/tests/vllm/models/test_glm5_next_forward.cpp @@ -1735,11 +1735,23 @@ TEST_CASE("glm5_next W9c-3b: the KV binding COPIES the engine's pages instead " } } -TEST_CASE("glm5_next W9c-3b: a CPU queue keeps the DIRECT path, byte-for-byte") { - // The other half of the discriminator. The bounce must be selected by the - // QUEUE and by nothing else, so on a CPU queue the pages are written in place - // and `ShadowBackend::Copy` is never reached -- which is also what keeps every - // `--device cpu` run unchanged by this wave. +TEST_CASE("glm5_next W9c-3b: a CPU queue writes the pages IN PLACE, and the " + "shadow backend is not involved") { + // The `--device cpu` arm, which is the one this wave must not move. The bytes + // land in the topology's OWN host vectors, at the slot the block permutation + // maps each position to, and no other backend is touched on the way. + // + // WHAT THIS CASE CANNOT SEE, stated because a mutation proved it. Deleting + // `PageIo`'s `if (q.device.type == kCPU) return;` -- so that a CPU queue + // resolves `vt::GetBackend(kCPU)` and bounces through it as well -- SURVIVES + // this case and the whole suite. It has to: the CPU backend's `Copy` IS + // `std::memcpy`, so the bytes are identical either way, and `Shadow().copies` + // below counts the SHADOW backend, which a CPU queue never reaches under + // either version. The early return is therefore not load-bearing for + // correctness. It is there so that a build with no CPU backend registered + // does not `Fail` on the host path, and so the `--device cpu` instruction + // stream is the one that was already measured. Spec `## Owed` O49 records the + // survival rather than leaving it for the next reader to rediscover. TempFile f(BuildFixture()); const vllm::GgufFile g = vllm::GgufFile::Open(f.path()); std::unique_ptr model = LoadThroughRegistry(g); From 76a79d3524fc2d062fd20dc4a48623687dc9b5c4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 14:15:53 +0000 Subject: [PATCH 05/10] docs(MODEL-MM-GLM53-FLASH): the two public surfaces repeated O46's mechanism, and it is false `docs/ENVIRONMENT.md`'s row for `VT_GLM5_NEXT_DEVICE_EXPERTS` and `docs/FEATURES.md`'s GLM-5.3-Flash row each told a user the `--device cuda` crash "appears only in the MIXED residency state the per-layer fit guard creates". A record correction that leaves the same claim in the product's own documentation has corrected nothing a user reads. Both now say what was measured -- three of three legs die, the knob stays off -- and then name the actual fault site, `StoreCaches` host-storing into the runner's `cudaMalloc` KV pages after the forward has already returned, with the issue that owns the fix. The default and the warning are unchanged: this is not a claim that the knob is safe. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/ENVIRONMENT.md | 2 +- docs/FEATURES.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 9bde20f86..03a451b15 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -278,7 +278,7 @@ portable/reference path. In normal operation leave them unset. | `VT_VULKAN_RMSNORM` | auto | Which `vt::RmsNorm` SPIR-V module runs: `wide` forces the 1024-invocation subgroup-reducing one, `base` forces the portable 128-invocation one, unset lets the device capability decide (1024 invocations on the X axis plus compute subgroup BASIC and ARITHMETIC). The wide module exists because `RmsNorm` dispatches ONE WORKGROUP PER ROW and a batch-1 decode step has exactly one row: on Qwen3.6-27B that put 128 invocations on a 5120-wide row, four warps of one SM, with the rest of the GPU idle. MEASURED on GB10 by the two-length GPU-timestamp diff: **0.0611 -> 0.0123 ms/call, 7.88 -> 1.59 ms/token**, and paired decode **241.9 -> 235.6 ms** median TPOT. The tell that it was OCCUPANCY and not the reduction is that the SAME shader costs 0.066 ms/call during PREFILL, where 32 rows give it 32 workgroups and 32x the data. It exists for the same-binary A/B and so the unit gate can exercise the fallback on hardware that would always pick the wide arm. Vulkan-only | | `VT_VULKAN_MATMUL_NCOLS` | 4 | Output columns each lane of the portable scalar GEMM computes, in the `[K,N]` (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes `128*NCOLS` CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the `vt_matmul_vec` tactic, because in `[K,N]` the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, `m=1 k=5120 n=248320`, 2.54 GB moved per token. MEASURED on GB10, 27B decode, `ms/call` medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, **4 = 11.54**, 8 = 12.81, with 4 winning **6 of 6** interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only | | `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only | -| `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. The fault appears only in the MIXED residency state the per-layer fit guard creates: each leg logged the device arm engaging for one layer and then `DeviceBanksFit` declining a later one -- 94.6758 GiB of expert banks do not fit beside the KV pool and the GGUF page cache on a 119 GiB box -- and died after that. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | +| `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. **The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere**: the two log lines that suggested it are once-flags, and the process dies in `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see `.agents/specs/glm5-next-flash.md` O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | ## Diagnostic diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7ef7009e4..338ec381d 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -150,7 +150,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) and the runner now ALLOCATES all 167 of them ([#2068](https://github.com/mudler/vllm.cpp/issues/2068)), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was | | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | -| `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, [#2348](https://github.com/mudler/vllm.cpp/issues/2348); W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242); KV-cache spec + MoE W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223); the forward W5b-2b, [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). MEASURED on `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)), which writes the forward the `multi_kv` guard at the top of the same function was waiting for ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223)): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of [#2260](https://github.com/mudler/vllm.cpp/issues/2260) the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly, once the 94.6758 GiB of expert banks stopped fitting and the arm fell back to the host mid-model (spec O46). The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | +| `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, [#2348](https://github.com/mudler/vllm.cpp/issues/2348); W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242); KV-cache spec + MoE W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223); the forward W5b-2b, [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). MEASURED on `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)), which writes the forward the `multi_kv` guard at the top of the same function was waiting for ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223)): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of [#2260](https://github.com/mudler/vllm.cpp/issues/2260) the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host — which was measured on GB10 to SEGFAULT, not merely to be slow, whenever the tensors came from the ordinary CUDA device allocator — and the fused MoE seam no longer throws — **and W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then tried to spend that discharge and FAILED, so `--device cuda` is still not a path to use.** The routed-expert device arm exists and is CUDA-gated at the unit level (NMSE 3.833e-15 against the host arm on `dgx:gpu0`), but driven end to end on the 101.24 GiB artifact BOTH `--device cuda` legs died with **SIGSEGV** (rc=139) emitting no token, reproducibly (spec O46). **The cause is now diagnosed and it is NOT the mixed residency O46 inferred**: the forward dies in `StoreCaches`, host-storing into the runner's `cudaMalloc` KV pages after the forward has already returned, which is a defect older than this arm and unreachable while the non-CPU refusal above it stood -- spec O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The split is therefore **OPT-IN and defaults OFF** (`VT_GLM5_NEXT_DEVICE_EXPERTS=1`, for debugging that crash and not for serving), and the default behaviour of `--device cuda` is the refusal it always was. **Use `--device cpu`**, which emits ` Paris.` on that artifact. State that precisely, because it is what is measured and no more -- **no token has yet come out of this model on a GPU, and none is claimed**; the end-to-end `--device cuda` leg on the 101.24 GiB artifact is queued on `dgx:gpu0` and an untaken device gate is PENDING, never a pass. What IS gated, on x86_64: the forward admits a CUDA-typed queue instead of throwing, and the routed-expert arm's device path -- residency, operand construction, arm selection, the fit guard -- runs and agrees bit-for-bit with the host arm on a CPU-backed `Dev`. The forward SPLITS its queue -- it interposes a CPU queue for the host-reference arms and hands the caller's device to exactly ONE consumer, the routed-expert keep-quant GEMM, whose banks `dense_attn::ResidentWeight` uploads once per model and keeps in their blocks. **READ THAT AS ONE ARM OF ELEVEN.** The KDA recurrence, the DSA k-pool indexer, the eager MLA attention, both mHC sites, the router, the combine, the dense and shared MLPs, the embedding gather and the chunked `lm_head` ALL STILL RUN ON THE HOST, and the row's spec records that as O43 with [#2410](https://github.com/mudler/vllm.cpp/issues/2410) owning the rest; the remaining port is priced at 2,500-3,500 lines off the two siblings that carry a device arm (`kimi_linear_device.cpp` 2,539, `nemotron_h_device.cpp` 2,144). A device that is neither CPU nor CUDA is still refused by name, and so is a CUDA queue in a build with no CUDA backend. A one-line stderr announcement names the device the expert GEMM ran on, because the two arms compute the same block and no logit can say which ran. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists | | `GlmMoeDsaForCausalLM` | zai-org/GLM-5.3 (753.33B, DSA sparse MLA MoE), HF revision `935644c05e76fc198714f4cca449fd8b970ff6d7` — **REGISTERED AND VALIDATING; IT LOADS NOTHING AND FORWARDS NOTHING** (W2, [#2214](https://github.com/mudler/vllm.cpp/issues/2214)). The architecture resolves, its config parses from a `config.json` and from a `glm-dsa` GGUF header through one validator, and the `glm-dsa` row of the GGUF dispatch table is reached from `LoadedEngine::FromModelDir`. Both `load_weights` arms refuse by name: safetensors permanently (703.74 GiB across 141 shards, no streaming loader, no MoE-expert block-fp8 rung) and GGUF until W7. The forward refuses and names all seven missing primitives — the indexer KV side cache, the expert-streaming seam, sparse prefill, the per-layer indexer schedule reuse, the `IQ4_XS` keep-quant `vec_dot`, the loader and the fp32 router GEMM. The one staged GGUF arm states no `glm-dsa.attention.indexer.types` and is refused rather than resolved off llama.cpp's hardcoded table | **NONE, and none is reachable on this fleet.** vLLM implements this architecture at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98` (`registry.py:117`) and cannot fit it on any device this project reaches, so no wave may promise a token-exact number against it | no run, no number | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | From f8b2689d780ed6c44e832769c48e5afacc95edc4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 14:18:27 +0000 Subject: [PATCH 06/10] spec(MODEL-MM-GLM53-FLASH): say why the mixed residency state CANNOT be the fault, not only that it is not O49 named a different fault site. That leaves the reader free to think both stories are live and this one merely arrived first, so the argument is completed where it is made. The two arms of `MoeExpertsKeepQuant` share no mutable state across layers except `OwnedTensor::d_dev`, which is per tensor. `DeviceBanksFit` returns true whenever nothing needs uploading, so a resident layer always takes the device arm and never the host one, and a layer that falls back is one whose banks were never staged -- its host views aim at bytes staging did not touch, because `AdoptDeviceBytesAsHost` returns early on CUDA. That last clause is O46's own hypothesis 2, which it killed correctly and then did not carry forward into the mechanism it kept. There is no interleaving of staged and unstaged layers that lets one arm read the other's memory. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 752f835b1..e7e9cdad5 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -5624,6 +5624,17 @@ Debts this row carries, each visible rather than waived: after the announcement, after the fallback warning, on a step that emitted no token, with no message. + **AND THE MIXED STATE CANNOT BE IT, which is a stronger statement than "the + fault is elsewhere".** The two arms of `MoeExpertsKeepQuant` share no mutable + state across layers except `OwnedTensor::d_dev`, which is per tensor. + `DeviceBanksFit` returns true whenever `need == 0`, so a layer whose three + banks are resident ALWAYS takes the device arm and never the host one; a layer + that falls back is one whose banks were never staged, and its host views + (`src.gate_exps.View()`) aim at bytes staging did not touch, because + `AdoptDeviceBytesAsHost` returns at `!DeviceMemoryIsHostAddressable()` on CUDA + (O46's own hypothesis 2, correctly killed). There is no ordering of staged and + unstaged layers that makes one arm read the other's memory. + W9c-3a did not introduce it. `origin/main` refused a non-CPU queue before `StoreCaches` could run; removing that refusal made a pre-existing hole reachable. O46's three eliminated hypotheses are all inside From 91a071dccd781d6dffab8814312e943345d0797a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 14:22:29 +0000 Subject: [PATCH 07/10] spec(MODEL-MM-GLM53-FLASH): say what else the C-against-D comparison measures The async runner can hand a forward device-resident input ids and leave `token_ids` stale for decode rows. This model reads `token_ids` and ignores the device field. Whether that path engages here is NOT established -- it needs `async_input_combine_`, and this row has never observed a decode step on a device queue -- so it is not filed as a defect and no fix is written for it. It is named beside the evidence table because the legs already answer it: two tokens is exactly one decode step, so a leg C whose stdout matches leg D byte-for-byte has measured it, and a leg C that emits ` Paris` and then the wrong second token has found it. Writing that down is what stops the next reader from either assuming it is fine or spending a lease to ask separately. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index e7e9cdad5..01072b713 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -3226,6 +3226,17 @@ not change the host arm. **No speed number is admissible from any of them** (O47: 89% generation spread across three identical legs, cause unknown), and none is claimed. +**C-against-D byte-identity also answers a SECOND question, and that is why the +legs run two tokens rather than one.** The async runner can hand a forward +`ModelForwardInput::device_token_ids` and leave `token_ids` deliberately stale +for decode rows (`v1/worker/gpu/runner.cpp:2374-2414, 2748`); this model reads +`token_ids` and ignores the device field. Whether that path ENGAGES here is not +established -- it needs `async_input_combine_`, and this row has never observed a +decode step on a device queue -- so it is not filed as a defect. It is named +here because a two-token leg contains exactly one decode step, so a leg C whose +stdout matches leg D byte-for-byte has measured it, and a leg C that emits +` Paris` followed by the wrong second token has found it. + #### Mutations, with what each one killed Every mutation is applied to PRODUCT code and rebuilt, because a mutation the From 0f0df02d9dfca9c74209029eb5b8e8506c6724df Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 17:19:20 +0000 Subject: [PATCH 08/10] spec(MODEL-MM-GLM53-FLASH): the diagnosis is confirmed on GB10, and the job's own guard hid the fix Run 1 on `dgx:gpu0` settles the cause two independent ways. Leg B died with SIGSEGV and its backtrace names the predicted frame and the whole chain -- `StoreCaches` under `ForwardGlm5NextForConditionalGeneration` under `ModelRegistry::Forward` under `execute_model`. Leg A moved the KV pages into host memory with `VT_DEVICE_KV_CACHE=0`, changed nothing else, and survived at rc=0 in 1756 s on the same binary minutes later. A predicted frame plus a one-variable falsification that fails to falsify retires O46's mechanism rather than merely doubting it. Legs C and D never ran, and the job refused them itself: it compared `sha256sum vllm-cli` across the two halves, they matched, and it declared them the same binary. They were not. The `.so` hashes the same script recorded differ -- `bfd0a6e7` base against `66412203` fixed -- and `vllm-cli` is a thin ABI client this change cannot alter, so its hash is identical by construction. Separate build directories would not have fixed that either. The repair is the predicate: digest the executable and every `.so` beside it, which is the breadth the same script's identity scan already used two blocks earlier. Run 2 does both and adds a source-level sentinel checked in the linked set. O52 opens the second-token question rather than closing it. Leg A's first token agrees with the CPU arm and its second does not, so prefill is clean and decode is not. The candidate with a mechanism is the runner's async device mirror, default ON for an integrated CUDA GPU, which leaves the host `token_ids` stale for decode rows while this model references `device_token_ids` zero times. That is a candidate, and the entry says so. It also records why the obvious discriminator cannot be run: with `VT_GLM5_NEXT_DEVICE_EXPERTS` unset the forward refuses a non-CPU queue by name and emits no token at all. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464, #2410 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 96 ++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 01072b713..c5f5d9e5f 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -5673,6 +5673,102 @@ Debts this row carries, each visible rather than waived: is admissible from this row anyway (O47); it is written down so the next reader does not have to rediscover the shape of the cost. +- **O50 -- THE DIAGNOSIS IS CONFIRMED ON `dgx:gpu0`, BY A NAMED FRAME AND BY A + ONE-VARIABLE FALSIFICATION TEST.** Run 1 of the W9c-3b job, `dgx:gpu0` (GB10, + `sm_121a`), against the published 101.24 GiB UD-Q2_K_XL artifact, base + `3cd467643`: + + | leg | binary | device | env | rc | wall | stdout | + |---|---|---|---|---:|---:|---| + | B | base | cuda | default | **139** | -- | 0 bytes | + | A | base | cuda | `VT_DEVICE_KV_CACHE=0` | **0** | 1756 s | ` Paris Paris` | + + Leg B's `gdb` backtrace names the frame O49 predicted, and it is the whole + chain rather than a symbol on its own: + + ```text + Thread 4 "vllm-cli" received signal SIGSEGV, Segmentation fault. + #0 vllm::glm5_next::StoreCaches(...) + #1 vllm::(anonymous namespace)::ForwardGlm5NextForConditionalGeneration(...) + #2 vllm::ModelRegistry::Forward(...) + #3 vllm::v1::GPUModelRunner::execute_model(...) + ``` + + Leg A moved the engine's KV pages into host memory and changed nothing else -- + same binary, same box, minutes later -- and it SURVIVED. A predicted frame and + a one-variable falsification that fails to falsify are two independent + confirmations, and O46's mixed-residency mechanism is retired rather than + merely doubted. + +- **O51 -- THE JOB'S OWN IDENTITY GUARD REFUSED A PAIR THAT WAS GENUINELY + DIFFERENT, and the reason is that it hashed the wrong artifact.** Run 1 built + base and fixed into ONE directory and compared `sha256sum vllm-cli` across the + two halves. They matched, the guard declared "the two halves are the SAME + BINARY" and refused to report legs C and D, so **the fix went unmeasured on + hardware**. The `.so` hashes the same script RECORDED show the halves were not + the same at all: + + | artifact | base | fixed | + |---|---|---| + | `examples/vllm-cli` | `bbc55a61...` | `bbc55a61...` | + | `libvllm.so.0.0.3` | `bfd0a6e7...` | **`66412203...`** | + + `vllm-cli` is a thin ABI client (AGENTS.md §"Shared seams": examples never + include internal headers), so this change cannot alter it and its hash is + identical BY CONSTRUCTION. The model code is in `libvllm.so`. **Separate build + directories do not fix this** -- `vllm-cli` would hash identically across them + too -- so the repair is the PREDICATE: digest the executable AND every `.so` + beside it, which is exactly the breadth the same script's identity SCAN + already used two blocks earlier. Run 2 does both, and adds a source-level + sentinel (a string present only in the patched file) checked in the linked + set, so "is this the fixed tree" is answered by content rather than by a build + artifact's timestamp. + + The guard firing was correct behaviour and is kept. A gate that refuses to + report is the right failure mode; a gate that refuses for a reason that is not + the one it names is the defect. + +- **O52 -- LEG A EMITTED ` Paris Paris`, ITS FIRST TOKEN AGREES WITH THE CPU ARM + AND ITS SECOND DOES NOT, so PREFILL is clean and the DECODE step is not.** + `PARIS_A=NO` in the run-1 log is a string match against ` Paris.` and is an + artifact of that flag, not the finding. The finding is the divergence under it, + and it is recorded here rather than folded into a summary as noise. + + **The discriminator that suggests itself does not work, and this is why.** + Running leg A's configuration with `VT_GLM5_NEXT_DEVICE_EXPERTS` unset does not + run the model with the expert arm off: `Glm5NextHostForward` refuses a non-CPU + queue BY NAME when the flag is unset, so the leg emits no token at all. An + engine-fatal refusal that prints nothing is indistinguishable at the file level + from the crash it would be mistaken for -- the "instrument whose failure looks + like a result" shape this row has already paid for. + + **THE CANDIDATE WITH A MECHANISM, and it is one this spec already flagged as + unestablished.** `GPUModelRunner::async_device_mirror()` is DEFAULT ON for an + integrated CUDA GPU (`runner.cpp:4437-4459`; `AsyncDeviceMirrorEnvDefault` is + "on unless the value is `0`", and GB10 satisfies `is_integrated_gpu()`). On + that path the combine patches the DEVICE input ids and leaves the host + `token_ids` "deliberately stale for decode rows" -- the runner's own words + (`runner.cpp:2374-2414`, assigned at `:2748`). **This model references + `device_token_ids` ZERO times** in `glm5_next_registry.cpp` and + `glm5_next_forward.cpp`; it embeds `input.token_ids`. On a decode step that is + a stale id, so the model never sees the token it just emitted -- which is + precisely what re-emitting ` Paris` looks like, and precisely why prefill, + which has no decode row, is unaffected. + + It is a candidate and not a conclusion. Run 2's leg E is `VT_ASYNC_DEVICE_MIRROR=0`, + the documented rollback: a second token of `.` implicates that path and + exonerates the routed-expert arm, and a second token of ` Paris` rules it out. + Leg F (`--max-tokens 1`) separates prefill from decode by construction, and + leg G repeats run 1's leg A inside run 2 so the comparison is within-job. + + **A bimodal reading is required here.** The routed-expert arm's unit gate is + NMSE 3.833e-15, and the prefill argmax margin measured on `thor` was 1.279 + (` Paris` 16.427 over ` one`). A perturbation far too small to move token 1 + can still flip a near-tie at token 2, because an argmax is a discrete + selection and its error is bimodal, not proportional. So "token 1 matched" + does NOT by itself clear the arm, and whatever run 2 returns, the top-5 and + the MARGIN are what settle it rather than the token string. + ## Now `ACTIVE`, 2026-09-01. **THE KERNEL THAT BLOCKED THIS MODEL'S DEVICE ARM LANDED From 868b49bdd89a4742fe7cd8d68d624e99c399f8e2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 1 Sep 2026 17:20:56 +0000 Subject: [PATCH 09/10] spec(MODEL-MM-GLM53-FLASH): re-cut the evidence table around what run 1 settled and what run 2 must answer Legs A and B are taken and are recorded as taken, so the table no longer asks for them. What replaces them is the set run 2 needs: D first, because every byte comparison needs an operand from the same binary on the same box and the ` Paris.` in the records came from a different build; C, which is the merge criterion; and E, F and G, which separate the second token. The stop conditions gain the one this row was missing. A leg C that emits a token but does not match D is TWO results -- the crash is fixed and the second token is wrong -- and the first must not be reported without the second beside it. FOLLOWING_AGENTS_PROTOCOL Refs: #2480, #2464 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 35 ++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index c5f5d9e5f..396a7ce7c 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -3211,17 +3211,26 @@ to make this model correct. The hermetic gate cannot prove the diagnosis, only the property. The diagnosis is proved on the box, by a leg that changes NOTHING but where the pages live: -| leg | binary | device | env | expected if the diagnosis holds | -|---|---|---|---|---| -| A | base `3cd467643` | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1 VT_DEVICE_KV_CACHE=0` | ` Paris.`, rc=0 | -| B | base `3cd467643` | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1` | SIGSEGV, rc=139 (the reproduction) | -| C | fixed | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1` | ` Paris.`, rc=0 | -| D | fixed | cpu | — | ` Paris.`, rc=0, byte-identical to C | +**RUN 1 IS TAKEN and legs A and B are SETTLED** (O50): B died with SIGSEGV at +`StoreCaches` with the whole call chain named, and A survived at rc=0 in 1756 s +with one variable moved. **Legs C and D did not run**, because the job's identity +guard hashed `vllm-cli` alone and refused a pair whose `libvllm.so` genuinely +differed (O51). Run 2 carries the repaired predicate and these legs: -A alone is the discriminator: it moves the pages to host memory and changes -nothing else, so an A that survives while B dies puts the fault in the page -residency and nowhere else. C is the fix. D is the control that says the fix did -not change the host arm. +| leg | binary | device | env | the question it answers | +|---|---|---|---|---| +| D | fixed | cpu | -- | the OPERAND, from this binary on this box; taken FIRST | +| C | fixed | cuda | `VT_GLM5_NEXT_DEVICE_EXPERTS=1` | **the merge criterion**: rc=0 and byte-identical to D | +| E | fixed | cuda | + `VT_ASYNC_DEVICE_MIRROR=0` | is the stale decode input id the second-token cause? | +| F | fixed | cuda | `--max-tokens 1` | prefill against decode, separated by construction | +| G | base | cuda | `VT_DEVICE_KV_CACHE=0` | run 1's leg A, repeated WITHIN run 2 | + +C is the fix. D is the operand every other leg is byte-compared against, taken +from the same binary on the same box, because the ` Paris.` in the records came +from a different build and is not a byte operand. E, F and G separate the +second-token question O52 opens -- and none of them is a discriminator unless it +RUNS: the obvious one, unsetting `VT_GLM5_NEXT_DEVICE_EXPERTS`, emits no token, +because without it the forward refuses a non-CPU queue by name. **No speed number is admissible from any of them** (O47: 89% generation spread across three identical legs, cause unknown), and none is claimed. @@ -3268,7 +3277,11 @@ and they are not the reachability proof, which is why both are kept. #### Stop conditions * If leg A dies too, the diagnosis is wrong: report it, keep O46 open, and do - not ship the change on a hermetic gate alone. + not ship the change on a hermetic gate alone. **DISCHARGED: A survived at rc=0 + and B's backtrace names `StoreCaches` -- O50.** +* If leg C emits a token but does not match leg D byte-for-byte, the CRASH is + fixed and the SECOND-TOKEN question (O52) is open. Those are two results, and + the first is not reported without the second beside it. * If a fifth arm of this model turns out to host-dereference engine device memory, that is a wider residency question than this wave, and it goes back as `NEEDS_DECISION` rather than being absorbed here. From 154382be1f2e079d77357503075b3fb5ceb6ce94 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 11:27:02 +0000 Subject: [PATCH 10/10] spec(MODEL-MM-GLM53-FLASH): record run 2, which fixed the crash and failed a criterion I mis-specified Run 2 completed on dgx:gpu0 at 2026-09-01T21:56:05Z and its results were never written back here; this spec still described those legs in the future tense. The crash is fixed: run 1 leg B died rc=139 with StoreCaches at frame #0, and leg C is the same workload on the same box at rc=0, on a pair whose linked-set digests genuinely differ. The stated criterion "C byte-identical to D" failed, and the fault is in the criterion rather than the fix. C and D differ by two variables, where the KV pages live and whether the async device mirror is engaged, and a StoreCaches fix can only be held to the first. Against the single-variable operand, leg G on the base binary and host pages, leg C is byte-identical. The clause stays red and nothing was deleted to make it green. The second token is a separate defect that predates this fix. Leg G is a base build and diverges identically; leg E flips VT_ASYNC_DEVICE_MIRROR alone and the divergence disappears into byte-identity with the CPU operand. That is the pre-registered read-key, and it promotes #2544 from a grep-based candidate to a measured conviction for this model. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/glm5-next-flash.md | 60 +++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 396a7ce7c..3875aca20 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -3215,7 +3215,8 @@ is proved on the box, by a leg that changes NOTHING but where the pages live: `StoreCaches` with the whole call chain named, and A survived at rc=0 in 1756 s with one variable moved. **Legs C and D did not run**, because the job's identity guard hashed `vllm-cli` alone and refused a pair whose `libvllm.so` genuinely -differed (O51). Run 2 carries the repaired predicate and these legs: +differed (O51). **RUN 2 IS NOW ALSO TAKEN** (O53 below); it carried the repaired +predicate and these legs: | leg | binary | device | env | the question it answers | |---|---|---|---|---| @@ -3246,6 +3247,63 @@ here because a two-token leg contains exactly one decode step, so a leg C whose stdout matches leg D byte-for-byte has measured it, and a leg C that emits ` Paris` followed by the wrong second token has found it. +#### O53 -- RUN 2 IS TAKEN, and the criterion FAILED ON A CLAUSE I MIS-SPECIFIED + +`glm53-kvres/submit2.log`, `dgx:gpu0`, finished 2026-09-01T21:56:05Z, 12068 s +wall, five legs interleaved on ONE box from TWO separate build directories. The +tested bytes are the PR's bytes: `glm5_next_kv.cpp` hashes +`cb34a6d198476864...` at PR head `868b49bdd` and `37d337a7538fcefb...` at base +`3cd467643`, matching the `fix`/`base` digests the job recorded. + +| leg | binary | device | env | rc | stdout | bytes | +|---|---|---|---|---|---|---| +| D | fixed | cpu | -- | 0 | ` Paris.` | `2050 6172 6973 2e0a` | +| C | fixed | cuda | `…DEVICE_EXPERTS=1` | 0 | ` Paris Paris` | `2050 6172 6973 2050 6172 6973 0a` | +| E | fixed | cuda | + `VT_ASYNC_DEVICE_MIRROR=0` | 0 | ` Paris.` | `2050 6172 6973 2e0a` | +| F | fixed | cuda | `--max-tokens 1` | 0 | ` Paris` | `2050 6172 6973 0a` | +| G | base | cuda | `VT_DEVICE_KV_CACHE=0` | 0 | ` Paris Paris` | `2050 6172 6973 2050 6172 6973 0a` | + +Binary identity held this time on the repaired predicate: linked-set digests +`e6762476d2b97ded...` (base) against `f643a2f1bb540f49...` (fixed), with +`fix_sentinel=no` / `fix_sentinel=yes`. + +**THE CRASH IS FIXED.** Run 1 leg B died `rc=139` with `StoreCaches` at frame #0 +(`+840`); leg C is the same workload on the same box at `rc=0`. + +**THE STATED CRITERION -- "C byte-identical to D" -- FAILED, and the fault is in +the criterion.** C and D differ by TWO variables, not one: where the KV pages +live AND whether the async device mirror is engaged. A `StoreCaches` fix can +only be held to the first. The single-variable operand is **leg G**, the BASE +binary on host pages, and: + + C (fixed, device KV) == G (base, host KV) byte-for-byte, 13 bytes + +So the fix reproduces the already-working path exactly. It converts a SIGSEGV +into the same bytes, and it changes nothing else. + +This is recorded as a mis-specified criterion rather than a widened one. The +clause as written is still red, and nothing was deleted to make it green. + +**THE SECOND TOKEN IS A SEPARATE, PRE-EXISTING DEFECT, AND RUN 2 CONVICTS IT.** +Leg G is a base build and diverges identically, so the divergence predates this +fix and is neither caused nor cured by it. Leg F shows prefill is clean +(` Paris` matches D's first token). Leg E flips ONE variable, +`VT_ASYNC_DEVICE_MIRROR=0`, and the divergence disappears into byte-identity +with D. That is exactly the pre-registered read-key above, and it promotes +[#2544](https://github.com/mudler/vllm.cpp/issues/2544) -- which names +`Glm5NextForConditionalGeneration` on a grep and calls itself "a candidate, not +a conviction" -- to a measured conviction for this model. O52 is answered and +its ownership moves to #2544. + +**What run 2 does NOT establish, stated rather than glossed:** leg A as +originally specified (FIXED binary with `VT_DEVICE_KV_CACHE=0`) was never run; +what exists is the BASE binary on host pages, twice, rc=0 both times, which +serves the falsification purpose but is not the literal leg. Legs B and C were +never in the same job -- B is run 1's binary, C is run 2's, source-anchored to +the same `3cd467643` but not interleaved with each other, so **leg C is n=1**. +No speed number is admissible or claimed from any leg; the walls are durations +of which ~85% is GGUF load. + #### Mutations, with what each one killed Every mutation is applied to PRODUCT code and rebuilt, because a mutation the