From 3d051bb0d22810cf4c4b4ebaf2b38ace2f8be809 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 09:05:44 +0000 Subject: [PATCH 1/4] spec(BACKEND-ROCM-IQ-EXPERT-RESIDENCY): route a placed routed-expert tower's residency through the device that will run it GLM-5.3 `UD-IQ1_S` refuses at load on `strix:gpu0` because `LoadStackedExperts` asks the ENGINE device whether an IQ1_S tower may keep its blocks, for a tower the installed hybrid-placement plan has already decided will execute on the CPU and whose bytes never reach the GPU. `DeviceKeepQuantSupported` serves four k-quants on ROCm, so the tower expands and the loader refuses by name. The spec traces the whole chain rather than the next hop: GLM-5.3 never uploads an expert tower to any device, its unplaced MoE forward cannot run on a non-host-addressable board at all, and the CPU already has a `vec_dot` for every encoding this checkpoint's experts use. Two edits close it -- the routing device for a placed tower, and the fit check crediting the plan it installed 220 lines earlier (#2517). Neither an i-quant ROCm kernel (#1940) nor the host-slot lane (#2515) is on the path to a token here, and the spec says why for each. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/rocm-placed-expert-residency.md | 282 ++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 .agents/specs/rocm-placed-expert-residency.md diff --git a/.agents/specs/rocm-placed-expert-residency.md b/.agents/specs/rocm-placed-expert-residency.md new file mode 100644 index 000000000..7ce7eec1c --- /dev/null +++ b/.agents/specs/rocm-placed-expert-residency.md @@ -0,0 +1,282 @@ +# BACKEND-ROCM-IQ-EXPERT-RESIDENCY — route a placed routed-expert tower's residency through the device that will RUN it + +Issue: [#2516](https://github.com/mudler/vllm.cpp/issues/2516) (the routing), +[#2517](https://github.com/mudler/vllm.cpp/issues/2517) (the fit check that +ignores the plan it just installed). +Related, and deliberately NOT repaired here: +[#2515](https://github.com/mudler/vllm.cpp/issues/2515) (the host-slot lane +cannot serve on `gfx1151`; this row's answer does not need it), +[#2518](https://github.com/mudler/vllm.cpp/issues/2518) (the managed ceiling; +measured below to be unable to decide this load), +[#1940](https://github.com/mudler/vllm.cpp/issues/1940) (ROCm i-quant +`vec_dot`; measured below to be unnecessary for this load), +[#2505](https://github.com/mudler/vllm.cpp/issues/2505) (no `--device rocm`). + +Base SHA: `35116605bb294e78f1be103cd040c378c6d00e20`. + +## Scope + +Make GLM-5.3 (`GlmMoeDsaForCausalLM`) `UD-IQ1_S` load and generate on +`strix:gpu0` (`gfx1151`, ROCm 7.2.4) with its routed experts placed on the +host, by fixing the two predicates that stop a HYBRID-PLACED load from +reaching a forward. It does not port a kernel, does not move a platform +predicate, and claims no performance number. + +IN SCOPE: + +- `GgufLoadPolicy::Route` / `PeekRoute`: for a `kStackedExpertWeight`, resolve + the residency against the device the installed `MoePlacementPlan` says will + COMPUTE that tower, instead of always against the engine device. +- `GgufStagedWeightFootprint` / `CheckDeviceWeightFit`: exclude the + routed-expert tensors of layers the installed plan places off the engine + device, and say so in the refusal. +- `tests/vllm/model_executor/test_gguf_device_fit.cpp`'s IQ case, which is red + on every ROCm build for a reason the case does not state (#2516). + +OUT OF SCOPE: + +- ROCm i-quant `vec_dot` kernels (#1940). Measured unnecessary: a tower placed + on the CPU is executed by `vt::cpu`'s `vec_dot`, and + `vt::cpu::HasQuantDotKernel` already answers true for all six encodings this + checkpoint's experts use (`cpu_quant_dot.cpp:995,996,1000,1001,1004,1005`). + The ROCm gap is what makes an UNPLACED tower expand, and this checkpoint has + no unplaced-tower arm on this board for an unrelated reason (see W2). +- `RocmPlatform::residency_policy().device_memory_total_bytes` (#2518). Left at + the `hipMemGetInfo` probe. See W4 for why it cannot decide this load. +- The host-slot expert-streaming lane (#2515). Not repaired and not needed: see + W2. +- `--fit`'s own resolver. It sizes a placement from bytes and cannot know that + an UNPLACED tower expands on this device, so its default answer on this board + is a placement that still refuses. Recorded under `## Owed`. + +## W1. The chain, measured rather than assumed + +Five facts, each read off the tree at the base SHA or off the board: + +1. **GLM-5.3's expert towers are never uploaded to any device.** + `LoadStackedExperts` (`glm_moe_dsa_loader.cpp:381-421`) takes no `Dev` and + no queue; it returns an mmap-borrowed, unprefaulted `OwnedTensor`. The + registry's `prepare` hook is a no-op (`glm_moe_dsa_registry.cpp:88-93`), and + the forward reaches a tower only through `GlmExpertSlice` + (`glm_moe_dsa_forward.cpp:293,294,297-300`), never through + `dense_attn::ResidentWeight` — which is the one place `d.b.Alloc` + + `d.b.Copy` live (`dense_attn_block.h:250-268`). The file says so itself at + `glm_moe_dsa_forward.cpp:94-113`. + +2. **On ROCm the unplaced MoE forward cannot run at all.** + `GlmResidentExpertSlice` throws unless the platform is CPU or host-addressable + (`glm_moe_dsa_forward.cpp:112-122`), and `expert_stream::ExpertSlice` admits + the streaming lane on the same predicate (`expert_stream_seam.cpp:431`). + `host_memory_is_device_addressable()` is FALSE on `gfx1151` + (#2515, re-measured in W4), so BOTH arms are closed. GLM-5.3's routed + experts on this board can only run on a CPU `Dev` handed to `MoeBlock` by + `RunMoePlaced` — which the model already composes, unconditionally placeable + (`glm_moe_dsa_forward.cpp:471-475`). + +3. **The loader asks the wrong device about a placed tower.** The one policy is + `GgufLoadPolicy::FromEnv(source.device)` (`glm_moe_dsa_registry.cpp:80`), and + `Route` hands `policy.device` straight to `RouteGgufTensor` + (`gguf_keep_quant.cpp:401-408`). `DeviceKeepQuantSupported` serves exactly + {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm (`gguf_keep_quant.cpp:135-147`), so every + IQ1_S / IQ2_XXS / IQ3_XXS / IQ4_XS / Q2_K / Q3_K tower routes to + `kExpandBf16` and `LoadStackedExperts` refuses by name + (`glm_moe_dsa_loader.cpp:392-400`) — for a tower the plan has already + decided will execute on the CPU and whose bytes will never touch the GPU. + +4. **The fit check ignores the plan installed 220 lines above it.** + `InstallMoePlacementPlan` runs at `model_loader.cpp:2672`, + `CheckDeviceWeightFit` at `:2895`. Neither `GgufStagedWeightFootprint` nor + `CheckDeviceWeightFit` takes any representation of a resolved placement, so + the 187.3125 GiB of towers is charged to the device pool whatever the plan + says. This is #2517, and on this board it is not cosmetic: it is the line + between the placed load and a forward. + +5. **The CPU can execute every encoding this checkpoint's experts use.** + `vt::cpu::HasQuantDotKernel` (`cpu_quant_traits.cpp:165-170`) accepts 17 + block dtypes including all six here. So a tower routed against `kCPU` keeps + its blocks and is read in place out of the mapping. + +The chain therefore closes with two edits and no new kernel. + +## W2. Design + +**One question, asked of the right device.** `GgufLoadPolicy::device` is +documented as "the device the ENGINE resolved for this load", and the field's +own comment states that EVERY device-dependent decision reads it. Hybrid +placement (#2023/#2314) introduced a second, narrower answer for exactly one +class of tensor: `ActiveMoePlacementPlan().DeviceForLayer(l)` is the device a +layer's routed experts are COMPUTED on. A residency decision for a +`kStackedExpertWeight` is a question about that device, not about the engine. +This is the same defect as #1136 (the bound and the policy naming different +devices) and #2406 (a host-ISA repack applied to a weight staged elsewhere), +one seam further along. + +`GgufLoadPolicy::Route` and `PeekRoute` resolve through ONE new private helper +so the two entry points cannot answer differently — the property +`GgufExpertTowersReachSlotLane` depends on, since it peeks the same tensors the +loader routes. + +**Inert by construction, not by a flag.** The override applies only when + +``` +role == kStackedExpertWeight + && ActiveMoePlacementPlan().PlacesAnything() + && the tensor names a block index + && that block's device != the plan's engine device +``` + +`PlacesAnything()` is false on every load that configured no placement, which is +every CUDA and CPU load in the tree's gates today, so those loads route +byte-for-byte as they did. The last term is what stops a default-constructed +global (whose `engine_device_` is `kCPU`) from silently answering `kCPU` on a +CUDA load: the answer is only ever ADOPTED when the plan actively moves that +tensor somewhere else. + +**The block index comes from the name**, because that is the only key the loader +has. `MoePlacementPlan::DeviceForRoutedExpertTensor(name)` parses the +`blk..` prefix llama.cpp's GGUF export writes — the same spelling +`RoutedExpertTensorNames` and `LlmFfnExpsBlockRegex` already compose — and +answers the engine device for a name that carries none. + +**The fit check credits the same plan.** `GgufStagedWeightFootprint` gains a +`const MoePlacementPlan*` (null = today's behaviour, byte for byte) and moves a +placed routed-expert tensor into two new fields, `placed_tensor_count` / +`placed_bytes`, exactly as `StreamedExpertLane` already moves a streamed one. +The justification is the placement seam's own: `ResidentWeight` aliases host +bytes on a CPU `Dev`, so a layer whose block only ever runs there is never +staged to the accelerator (`moe_placement_seam.h`, the "ACROSS" comment). The +refusal appends a NOTE naming the placed count and bytes, so an operator can +never again read two contradictory numbers in consecutive lines. + +**Why not make the lane serve instead.** #2515 is right and this row does not +contest it: `HostMemoryIsDeviceAddressable` is `integrated && +pageable_memory_access`, `pageableMemoryAccess` is 0 on `gfx1151`, and the slot +arena is a pageable `std::vector`. Forcing the lane on would delete a +correct refusal. The device-side slot store that would fix it is +`ENG-EXPERT-STREAM-DEVICE` W2 and a different row. + +**Why not widen ROCm keep-quant.** Writing IQ `vec_dot` in +`rocm_grouped_gemm.hip` (#1940) would keep an UNPLACED tower compressed — and +W1 fact 2 says an unplaced tower cannot be executed on this board at all, +because `GlmResidentExpertSlice` refuses a non-host-addressable device before +any kernel is asked for. So #1940 would not produce a token here. It stays owed +for the discrete-ROCm arm and for performance. + +## W3. Tests + +Red-first, and each named test fails for its intended reason before the change. + +**`tests/vllm/test_gguf_keep_quant.cpp`** (new cases): + +- An IQ1_S stacked-expert tower under a policy with `device = kROCM` and an + installed plan that places its layer on `kCPU` routes `kKeepQuant`. RED + before (`kExpandBf16`). +- The SAME tensor with NO plan installed still routes `kExpandBf16` on + `kROCM` — the inertness pin, green before and after. +- A tensor of a layer the plan does NOT place still routes against the engine + device while a placed sibling does not, in one assertion pair, so the + override is proved per-layer rather than per-load. +- `Route` and `PeekRoute` agree on every case above. +- `kMatmulWeight` and `kEmbeddingTable` are UNMOVED by an installed plan: the + override is scoped to the one role whose compute the plan actually moves. + +**`tests/vllm/model_executor/test_device_placement.cpp`**: +`DeviceForRoutedExpertTensor` on `blk.7.ffn_gate_exps.weight`, on a name with +no block index, and on an out-of-range block. + +**`tests/vllm/model_executor/test_gguf_device_fit.cpp`**: + +- A footprint with a plan that places the trailing N layers reports + `placed_bytes` equal to those towers and drops them from + `lower_bound_bytes`; with a null plan it is byte-identical to today. +- `CheckDeviceWeightFit` REFUSES without the plan and PASSES with it, on one + synthetic file whose numbers straddle the budget. RED before (the parameter + does not exist). +- The #2516 case is re-pointed from `CurrentPlatform().device_type()` to the + two devices it is actually about: `kCPU` routes IQ towers `kKeepQuant`, and + `kROCM` routes them `kExpandBf16`. Platform-independent, and it PINS the + ROCm gap #1940 owns instead of failing over it. + +**Reachability (`.agents/reachability.md`).** The production entry point is +`vllm_engine_load` → `ModelLoader::FromModelDir`'s GGUF branch. The mutation +deletes the placement term at BOTH production call sites (the `Route` helper +and the `CheckDeviceWeightFit` argument), one at a time, and requires the +focused suites to go red; a by-hand construction of the policy is not the +proof and is not offered as one. + +## W4. Gates + +- `scripts/agent-preflight.sh --fail-on-skip` on the branch. +- Hand-run, with case AND assertion counts read together: + `test_gguf_keep_quant`, `test_gguf_device_fit`, `test_gguf_device_fit_reach`, + `test_device_placement`, `test_glm_moe_dsa_gguf_load`, `test_expert_stream_wiring`. +- On `strix:gpu0` inside an `rc` lease: the same suites on a `gfx1151` HIP + build, plus the product legs below. +- CUDA/CPU inertness: every suite above on a non-HIP build, and the assertion + that a load with no placement installed routes and prices byte-identically. + +**The product bar.** `vllm-cli --model --device auto` with +`VT_CPU_MOE=1` on `strix:gpu0`, against +`/workspace/ckpt/GLM-5.3-UD-IQ1_S` (six shards, first shard, never the +directory; shard 1 derived by `scripts/glm-dsa-write-indexer-types.py`). +Reported with it: the emitted text verbatim, the resident footprint, whether +the towers stayed compressed, and `GetReferenceTierHits()` / the +`[vt reference-tier]` distinct count — because #2505 means no `--device` value +names ROCm and configuration alone cannot separate a GPU run from a host +fallback. + +**No speed number, either way.** `docs/ROCM.md:60-61` disqualifies a +performance result from a run with a non-zero reference-tier count, and this +model's MLA/DSA arm is eight ops short on ROCm +(`.agents/specs/rocm-glm53-dsa.md` W1.3), every one of which lands on that +tier. + +## W5. Risks + +- **A fourth predicate.** Three predicates on this row have each looked like + the last one (#2507, #2515, #2516). W1 traces the whole chain from load to + `MoeBlock` rather than the next hop, but the MLA/DSA arm is unexercised on + this board and may refuse for its own reasons. The stop condition below says + what happens then. +- **A global read inside a routing function.** `Route` becomes dependent on + process state. Argued for rather than against: the plan IS the resolved + answer to "which device runs this tensor", it is installed before any weight + I/O by construction (`model_loader.cpp:2668-2672`), and the alternative — + threading a plan pointer through twelve `FromEnv` call sites — widens the + change without making any of them able to answer differently. The inertness + term keeps a never-installed global from ever being consulted. +- **`quant_repack` is still resolved from the ENGINE device**, so a tower + placed on the CPU is not i8mm-repacked. That is a performance gap, not a + correctness one (the CPU quant GEMM reads unrepacked blocks), and it is + listed as owed rather than fixed, because moving it means moving a flag + `FromEnv` resolves once for the whole file. + +## W6. Stop conditions + +Stop and report `NEEDS_DECISION` rather than widening scope if the placed load +reaches a forward and then refuses inside the MLA/DSA arm: porting those eight +ops is `.agents/specs/rocm-glm53-dsa.md`'s recorded campaign, not this row. + +Stop rather than forcing the streaming lane on, or lowering +`host_memory_is_device_addressable()`'s bar, if the placed route turns out to +need it. #2515 records why that deletes a correct refusal. + +## Owed + +- **ROCm i-quant `vec_dot`** (#1940), for a discrete ROCm board and for the + performance arm. Unnecessary for this row's result, for the reason in W2. +- **The ROCm MLA/DSA arm**, eight ops (`.agents/specs/rocm-glm53-dsa.md`). + Every one of them runs on the reference tier in this row's result, which is + why no speed number is quoted. +- **`--fit`'s resolver is not residency-aware.** It sizes a placement from + bytes alone, so on a device whose keep-quant set does not cover the file's + expert encodings it can resolve a placement that leaves towers unplaced — + and those towers then expand and refuse. On this board the operator has to + say `cpu_moe`. Needs its own issue before anyone changes the resolver. +- **`quant_repack` for a placed tower** (W5). +- **The device budget** (#2518) and **the host-slot lane** (#2515), both + untouched and both re-measured here. + +## Now + +`ACTIVE`. From 2e127b4d7f36287715c91df484c0b43232d3a6be Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 09:21:37 +0000 Subject: [PATCH 2/4] fix(BACKEND-ROCM): ask the device that will RUN a placed expert tower whether it may keep its blocks GLM-5.3 `UD-IQ1_S` refused at load on `strix:gpu0` (`gfx1151`) with `blk.3.ffn_gate_exps.weight routed to an EXPAND residency`. The tower is IQ1_S, `DeviceKeepQuantSupported` serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, so it expanded and `LoadStackedExperts` refused by name -- for a tower the installed hybrid-placement plan had already sent to the CPU, whose `vec_dot` table covers every encoding this checkpoint's experts use, and whose bytes never reach the GPU at all: that model reads a tower only through `GlmExpertSlice`, never through `ResidentWeight`. `GgufLoadPolicy::device` is the ENGINE's device, and its own field comment says every device-dependent decision reads it. Hybrid placement (#2023, #2314) introduced a second, narrower answer for exactly one class of tensor: `RunMoePlaced` hands `MoeBlock` the PLACEMENT device. A keep-quant decision asks whether the device that will execute this GEMM has a `vec_dot` for the encoding, so for a placed routed-expert tower it is a question about that device. `Route` and `PeekRoute` now resolve it through one shared `ComputeDeviceFor`, so the lane predicate and the loader cannot disagree about a file. This is #1136 and #2406 one seam further along, and it is inert in four terms -- the last, that the plan must actively move THIS tensor off its own engine device, is what stops a never-installed global (whose engine device is `kCPU`) from routing a CUDA load's experts as though they were on the host. The fit check now credits the plan installed 220 lines above it (#2517). One load printed "56 layers run their routed experts on cpu ... to bring a 216433205760 B footprint under a 68719476736 B budget" and then refused needing 216433205760 B: two lines of one load contradicting each other, because nothing in `CheckDeviceWeightFit` carried any representation of a resolved placement. A placed routed-expert tensor moves into `placed_tensor_count` / `placed_bytes` and leaves the bound, exactly as a streamed one already did, and the refusal says what the placement took out rather than leaving it to be inferred. `test_gguf_device_fit`'s IQ case asked `CurrentPlatform()`, which made it RED on every ROCm build for a reason the case did not state (#2516). It now pins both devices it is actually about: `kCPU` keeps the blocks, `kROCM` expands them, so the gap #1940 owns is asserted instead of stumbled over. Local: `test_gguf_device_fit` 24 cases / 182 assertions, 0 failed. Deleting the placement term from `ComputeDeviceFor` kills 1 case / 1 assertion; deleting it from the footprint kills 3 cases / 8 assertions. Refs #2516, #2517. No token is claimed here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .../vllm/model_executor/device_placement.h | 19 ++ .../model_loader/gguf_device_fit.h | 33 ++- .../model_loader/gguf_keep_quant.h | 18 ++ src/vllm/entrypoints/model_loader.cpp | 19 +- src/vllm/model_executor/device_placement.cpp | 40 ++++ .../model_loader/gguf_device_fit.cpp | 57 +++++- .../model_loader/gguf_keep_quant.cpp | 56 +++++- .../model_executor/test_device_placement.cpp | 52 +++++ .../model_executor/test_gguf_device_fit.cpp | 189 +++++++++++++++++- tests/vllm/test_gguf_keep_quant.cpp | 162 +++++++++++++++ 10 files changed, 633 insertions(+), 12 deletions(-) diff --git a/include/vllm/model_executor/device_placement.h b/include/vllm/model_executor/device_placement.h index 4f2e5d2d3..ea09ffe87 100644 --- a/include/vllm/model_executor/device_placement.h +++ b/include/vllm/model_executor/device_placement.h @@ -171,6 +171,19 @@ class MoePlacementPlan { // gets the inert answer rather than an exception on the decode path. vt::DeviceType DeviceForLayer(int64_t l) const; + // The device THIS ROUTED-EXPERT TENSOR's block runs on, keyed by the only + // thing a weight loader has: the name. + // + // WHY THE LOADER NEEDS THIS. `RunMoePlaced` hands the placed device to the + // forward, but the RESIDENCY of a routed-expert tower is decided far earlier, + // by `GgufLoadPolicy::Route`, and that decision is a question about the device + // that will EXECUTE the tower — whether it has a `vec_dot` for the encoding. + // Asking the engine device about a tower the plan has already sent elsewhere + // is how a checkpoint whose experts every placement device can execute is + // refused at load. A name with no `blk..` prefix, and a layer this plan was + // not resolved against, both answer the engine device. + vt::DeviceType DeviceForRoutedExpertTensor(const std::string& name) const; + // True when at least one layer runs its experts somewhere other than the // engine device. The forward reads this to decide whether any of the placement // machinery runs at all, so an unplaced model pays nothing. @@ -212,6 +225,12 @@ class MoePlacementPlan { // of them. std::vector RoutedExpertTensorNames(int64_t layer); +// The decoder-block index a llama.cpp GGUF tensor name carries (`blk..…`), +// or -1 when it carries none. The inverse of `RoutedExpertTensorNames`' own +// prefix, exposed so a caller resolves the layer with the same rule the plan was +// built with instead of a private copy of it. +int64_t GgufBlockIndexFromTensorName(const std::string& name); + // ── The placement device's queue, and the active plan ──────────────────────── // The queue a placed group runs on. One per device type, created on FIRST USE and diff --git a/include/vllm/model_executor/model_loader/gguf_device_fit.h b/include/vllm/model_executor/model_loader/gguf_device_fit.h index f791c49bf..706f581f0 100644 --- a/include/vllm/model_executor/model_loader/gguf_device_fit.h +++ b/include/vllm/model_executor/model_loader/gguf_device_fit.h @@ -35,6 +35,7 @@ #include #include "vllm/model_executor/model_loader/gguf_keep_quant.h" +#include "vllm/model_executor/device_placement.h" #include "vllm/model_executor/model_loader/gguf_reader.h" namespace vllm { @@ -131,6 +132,15 @@ struct GgufStagedFootprint { // The arena term actually added (0 when the lane is off, and 0 when it is on // but nothing matched). size_t arena_bytes = 0; + // #2517. How many routed-expert tensors the installed hybrid PLACEMENT took + // out of the sum, and how many bytes they would otherwise have contributed. + // Both 0 with no plan, and 0 with a plan that places nothing. + // + // Reported rather than merely subtracted, for the same reason the streamed + // pair is: a figure that shrank by 137 GiB without saying which tensors left + // it is a number an operator cannot check. + size_t placed_tensor_count = 0; + size_t placed_bytes = 0; }; // The per-expert SLICE size of a stacked expert tower, which is the slot size @@ -254,10 +264,28 @@ inline bool GgufPolicyForcesFullExpand(const GgufLoadPolicy& policy) { // available on that device) where the min-based bound under-counted by // roughly 4x and the load reached an uncaught `hipMalloc: out of memory` // instead of this refusal. +// +// `placement` (#2517) is the resolved hybrid-placement plan, or null for no +// plan. A routed-expert tensor whose layer the plan places away from the plan's +// own engine device is moved into `placed_*` and leaves the bound, because a +// layer whose MoE block only ever runs on the placement device is never staged +// to the accelerator -- `RunMoePlaced` hands that device to the block and +// `ResidentWeight` aliases host bytes on a CPU `Dev` rather than uploading them +// (`moe_placement_seam.h`). NULL, or a plan that places nothing, reproduces the +// pre-#2517 answer byte for byte, which is every load in this tree that +// configured no placement. +// +// WHY THIS IS NOT OPTIONAL BOOKKEEPING. `InstallMoePlacementPlan` runs 220 lines +// ABOVE the one `CheckDeviceWeightFit` call site (`model_loader.cpp:2672` and +// `:2895`), and before this parameter existed the refusal quoted the UN-reduced +// footprint on the line after the plan announced it had reduced it. On +// `strix:gpu0` that contradiction was the whole distance between a placed +// GLM-5.3 load and a forward. GgufStagedFootprint GgufStagedWeightFootprint( const GgufFile& gguf, size_t model_dtype_bytes = 2, const StreamedExpertLane& lane = {}, - bool policy_forces_full_expand = false); + bool policy_forces_full_expand = false, + const MoePlacementPlan* placement = nullptr); // The budget to compare a footprint against, in bytes, or 0 for UNKNOWN. // @@ -383,6 +411,7 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, size_t budget_bytes, size_t model_dtype_bytes = 2, const StreamedExpertLane& lane = {}, - bool policy_forces_full_expand = false); + bool policy_forces_full_expand = false, + const MoePlacementPlan* placement = nullptr); } // namespace vllm diff --git a/include/vllm/model_executor/model_loader/gguf_keep_quant.h b/include/vllm/model_executor/model_loader/gguf_keep_quant.h index 831cdb4ed..96ce9a08a 100644 --- a/include/vllm/model_executor/model_loader/gguf_keep_quant.h +++ b/include/vllm/model_executor/model_loader/gguf_keep_quant.h @@ -381,6 +381,24 @@ struct GgufLoadPolicy { // Route one tensor and notify `audit`. This is the ONLY entry point the // loader uses, so every routed tensor is observable. GgufResidency Route(const GgufTensorInfo& tensor, GgufTensorRole role) const; + + // The device that will EXECUTE this tensor, which is what a residency decision + // is about and is not always `device`. + // + // It differs for exactly one role. A routed-expert tower whose layer the + // installed `MoePlacementPlan` places away from the engine is computed on the + // PLACEMENT device (`RunMoePlaced` hands that device to the MoE block), so + // asking the engine whether its `vec_dot` covers the encoding refuses a + // checkpoint the placement device can execute perfectly well. Every other + // role, and every load with no placement installed, answers `device` + // unchanged -- see the implementation for the four inertness terms and why the + // last one is required rather than defensive. + // + // Public because `PeekRoute` must resolve the device THE SAME WAY `Route` + // does; a second spelling is how a bound and a forward come to disagree about + // one file. + vt::DeviceType ComputeDeviceFor(const std::string& name, + GgufTensorRole role) const; }; // A policy copy with the KEEP-QUANT residency disabled. diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 06fce92f0..ffbf2488e 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -2892,12 +2892,29 @@ std::unique_ptr LoadedEngine::FromModelDir( // defaults) should run. Using the narrower predicate here is what makes // this refusal reachable on ROCm without moving any of the other one's // consumers; the row's spec records why that flag stays untouched. + // #2517: THE PLAN INSTALLED 220 LINES ABOVE, credited here. + // + // `InstallMoePlacementPlan` runs at the top of this branch, before any + // weight I/O, and announces what it placed. Until this argument existed + // the refusal on the very next line quoted the UN-reduced footprint: one + // load printed "56 layers run their routed experts on cpu ... to bring a + // 216433205760 B footprint under a 68719476736 B budget" and then refused + // needing 216433205760 B. Two lines of one load contradicting each other, + // and on `strix:gpu0` the contradiction was the whole distance between a + // placed GLM-5.3 load and a forward. + // + // The GLOBAL rather than a local, because `SetActiveMoePlacementPlan` is + // where the plan lives and the forward reads the same object; a copy taken + // here could be the one the seam never sees. It is installed + // unconditionally, including when nothing is placed, so this is never a + // stale plan from a previous load in the same process. const DeviceWeightFit fit = CheckDeviceWeightFit( gguf, vt::DeviceTypeName(target.device_type()), target.allocates_bounded_device_memory(), DeviceWeightBudgetBytes( target.residency_policy().device_memory_total_bytes), - /*model_dtype_bytes=*/2, lane, policy_forces_full_expand); + /*model_dtype_bytes=*/2, lane, policy_forces_full_expand, + &vllm::ActiveMoePlacementPlan()); if (fit.refuse) throw std::runtime_error(fit.message); } // QUANT-QWEN38-27B-GGUF-ARM (#821): refuse a qwen3_5-family GGUF carrying diff --git a/src/vllm/model_executor/device_placement.cpp b/src/vllm/model_executor/device_placement.cpp index 9aba4cd87..5db8c0318 100644 --- a/src/vllm/model_executor/device_placement.cpp +++ b/src/vllm/model_executor/device_placement.cpp @@ -142,6 +142,46 @@ vt::DeviceType MoePlacementPlan::DeviceForLayer(int64_t l) const { return per_layer_[static_cast(l)]; } +int64_t GgufBlockIndexFromTensorName(const std::string& name) { + // `blk..` is llama.cpp's GGUF spelling for a decoder block, and it is the + // ONLY key a loader has: `GgufTensorInfo` carries a name, never a layer. + // `RoutedExpertTensorNames` composes the same prefix and + // `LlmFfnExpsBlockRegex` matches it, so this is the inverse of two functions + // that already exist rather than a new naming convention. + // + // ANCHORED AT THE START, deliberately. `regex_search`'s unanchored semantics + // are right for an operator's `-ot` pattern and wrong here: a name that merely + // CONTAINS `blk.3.` is not a block-3 tensor, and answering as though it were + // would move a weight no plan claimed. + static constexpr char kPrefix[] = "blk."; + const size_t plen = sizeof(kPrefix) - 1; + if (name.size() <= plen || name.compare(0, plen, kPrefix) != 0) return -1; + size_t i = plen; + int64_t idx = 0; + bool any = false; + while (i < name.size() && name[i] >= '0' && name[i] <= '9') { + // A count that cannot be a layer index is not a layer index. Refusing + // instead of wrapping keeps a malformed name out of `per_layer_`'s range + // rather than aliasing it onto a real layer. + if (idx > (INT64_MAX - 9) / 10) return -1; + idx = idx * 10 + (name[i] - '0'); + any = true; + ++i; + } + if (!any || i >= name.size() || name[i] != '.') return -1; + return idx; +} + +vt::DeviceType MoePlacementPlan::DeviceForRoutedExpertTensor( + const std::string& name) const { + const int64_t layer = GgufBlockIndexFromTensorName(name); + // A name carrying no block index belongs to no layer, so no placement claims + // it. The engine's own device is the inert answer, which is `DeviceForLayer`'s + // own out-of-range behaviour rather than a second convention. + if (layer < 0) return engine_device_; + return DeviceForLayer(layer); +} + std::string MoePlacementPlan::Describe() const { if (placed_ == 0) return ""; std::string out = std::to_string(placed_); diff --git a/src/vllm/model_executor/model_loader/gguf_device_fit.cpp b/src/vllm/model_executor/model_loader/gguf_device_fit.cpp index a15736897..bb6c3ba5d 100644 --- a/src/vllm/model_executor/model_loader/gguf_device_fit.cpp +++ b/src/vllm/model_executor/model_loader/gguf_device_fit.cpp @@ -131,12 +131,47 @@ bool GgufExpertTowersReachSlotLane(const GgufFile& gguf, return matched; } +namespace { + +// #2517. Is this tensor a ROUTED-EXPERT tower whose layer the installed plan +// runs somewhere other than the engine device? +// +// The name test is `_exps.weight`, which is the class the plan's own +// `RoutedExpertTensorNames` composes and the class `RunMoePlaced` moves. The +// layer term goes through `DeviceForRoutedExpertTensor`, so the bound and the +// forward resolve the placement with ONE rule rather than two. +// +// The engine-device comparison is against the PLAN's engine device, not the +// caller's: a plan resolved for a different device than the one being priced is +// a caller bug, and answering "placed" for a tensor that will in fact stage here +// would DELETE a correct refusal -- the unsafe direction #1378 names. +bool PlacedAwayFromEngine(const MoePlacementPlan* plan, + const std::string& name) { + if (plan == nullptr || !plan->PlacesAnything()) return false; + if (!NameHasSuffix(name, "_exps.weight")) return false; + return plan->DeviceForRoutedExpertTensor(name) != plan->engine_device(); +} + +} // namespace + GgufStagedFootprint GgufStagedWeightFootprint(const GgufFile& gguf, size_t model_dtype_bytes, const StreamedExpertLane& lane, - bool policy_forces_full_expand) { + bool policy_forces_full_expand, + const MoePlacementPlan* placement) { GgufStagedFootprint out; for (const GgufTensorInfo& t : gguf.Tensors()) { + // BEFORE the lane test, because a placed tower does not reach the lane + // either: `RunMoePlaced` hands the block a CPU `Dev`, and the expert-stream + // seam's own admission test (`expert_stream_seam.cpp:431`) is about the + // ENGINE platform. Counting one tensor in both classes would report the same + // bytes twice in a message whose whole job is to account for them once. + if (PlacedAwayFromEngine(placement, t.name)) { + ++out.placed_tensor_count; + out.placed_bytes += + StagedBytes(t, model_dtype_bytes, policy_forces_full_expand); + continue; + } // W0d: a tensor the slot lane serves is never staged, so it contributes // nothing to the bound and cannot be the largest single allocation either. // It is still COUNTED, in its own two fields, so the caller can say what was @@ -216,7 +251,8 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, size_t budget_bytes, size_t model_dtype_bytes, const StreamedExpertLane& lane, - bool policy_forces_full_expand) { + bool policy_forces_full_expand, + const MoePlacementPlan* placement) { DeviceWeightFit fit; fit.budget_bytes = budget_bytes; // A platform that does not stage weights reads them where they already are, so @@ -229,7 +265,7 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, if (budget_bytes == 0) return fit; const GgufStagedFootprint fp = GgufStagedWeightFootprint( - gguf, model_dtype_bytes, lane, policy_forces_full_expand); + gguf, model_dtype_bytes, lane, policy_forces_full_expand, placement); fit.needed_bytes = fp.lower_bound_bytes; if (fp.lower_bound_bytes <= budget_bytes) return fit; @@ -282,6 +318,21 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, // correction is appended rather than left to be read as a stale claim. The // lane-OFF message is untouched, byte for byte, because every CPU and discrete // user reads that one and it is still true for them. + // #2517. The plan announced a reduction on the line above this refusal and + // the refusal used to quote the un-reduced figure, which is two contradictory + // numbers in consecutive lines of one load. Whatever the verdict, say what the + // placement took out of it. + if (fp.placed_tensor_count > 0) { + fit.message += + " NOTE: an installed hybrid MoE placement runs " + + std::to_string(fp.placed_tensor_count) + + " routed-expert tensors (" + std::to_string(fp.placed_bytes) + + " bytes, " + Gib(fp.placed_bytes) + ") on another device, so they are " + "ALREADY EXCLUDED from the figure above rather than being a reduction " + "still available. What does not fit is the remainder. Placing more " + "layers (\"cpu_moe\", \"n_cpu_moe\", VT_CPU_MOE=1) is the knob that " + "shrinks it further."; + } if (fp.streamed_tensor_count > 0) { fit.message += " NOTE: the expert-stream lane IS active for this load, so the " diff --git a/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp b/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp index d2c5e7146..65cc5db45 100644 --- a/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp +++ b/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp @@ -7,6 +7,7 @@ #include #include "vllm/config/weight_residency.h" +#include "vllm/model_executor/device_placement.h" #include "vt/ops.h" #include "vt/quant.h" @@ -398,11 +399,52 @@ GgufLoadPolicy GgufLoadPolicy::FromEnv(vt::DeviceType dev) { return p; } +// THE DEVICE A RESIDENCY DECISION IS ACTUALLY ABOUT (#2516). +// +// `GgufLoadPolicy::device` is the ENGINE's resolved device, and its own field +// comment says every device-dependent decision in the struct reads it. Hybrid +// placement (#2023, #2314) introduced a SECOND, narrower answer for exactly one +// class of tensor: `ActiveMoePlacementPlan().DeviceForLayer(l)` is the device a +// layer's routed experts are COMPUTED on, and `RunMoePlaced` hands that device +// to `MoeBlock`. A keep-quant decision asks "does the device that will execute +// this GEMM have a `vec_dot` for this encoding", so for a PLACED routed-expert +// tower it is a question about the placement device, not about the engine. +// +// Measured, on GLM-5.3 `UD-IQ1_S` on `strix:gpu0`: `DeviceKeepQuantSupported` +// serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, so all 228 IQ1_S/IQ3_XXS/IQ2_XXS/ +// IQ4_XS/Q2_K/Q3_K towers routed `kExpandBf16` and `LoadStackedExperts` refused +// the load by name -- for towers whose bytes never reach the GPU at all (that +// model reads a tower only through `GlmExpertSlice`, never through +// `ResidentWeight`) and which the installed plan had already sent to the CPU, +// whose `vec_dot` table covers every one of those six encodings. +// +// THIS IS #1136 AND #2406 ONE SEAM FURTHER ALONG. Both were the same shape: a +// residency decision resolved against a device other than the one that would +// read the bytes. The repair is the same shape too -- ask the device that will +// run it. +// +// INERT BY CONSTRUCTION, in four terms, and the last one is the load-bearing +// one. The plan is a process-global that a load with no placement never writes, +// and a DEFAULT-CONSTRUCTED plan's `engine_device()` is `kCPU`. Returning its +// answer unconditionally would therefore route a CUDA load's experts as though +// they were on the host. So the placed answer is ADOPTED only when the plan +// actively moves THIS tensor somewhere other than the device the plan itself was +// resolved for; every other path returns the policy's own device unchanged. +vt::DeviceType GgufLoadPolicy::ComputeDeviceFor(const std::string& name, + GgufTensorRole role) const { + if (role != GgufTensorRole::kStackedExpertWeight) return device; + const MoePlacementPlan& plan = ActiveMoePlacementPlan(); + if (!plan.PlacesAnything()) return device; + const vt::DeviceType placed = plan.DeviceForRoutedExpertTensor(name); + if (placed == plan.engine_device()) return device; + return placed; +} + GgufResidency GgufLoadPolicy::Route(const GgufTensorInfo& tensor, GgufTensorRole role) const { - const GgufResidency r = - RouteGgufTensor(keep_quant, keep_f16, nvfp4_fp4, cpu_ref, role, - tensor.ggml_type, tensor.shape, device); + const GgufResidency r = RouteGgufTensor( + keep_quant, keep_f16, nvfp4_fp4, cpu_ref, role, tensor.ggml_type, + tensor.shape, ComputeDeviceFor(tensor.name, role)); if (audit) audit(tensor.name, role, r); return r; } @@ -415,9 +457,15 @@ GgufLoadPolicy NoKeepQuant(const GgufLoadPolicy& policy) { GgufResidency PeekRoute(const GgufLoadPolicy& policy, const GgufTensorInfo& tensor, GgufTensorRole role) { + // THROUGH THE SAME RESOLVER `Route` USES, not a copy of the expression. This + // function exists so a caller can ask what the loader WILL do without + // notifying the audit, and `GgufExpertTowersReachSlotLane` is one such caller + // -- it peeks the very tensors the loader then routes. Two spellings of the + // device term is exactly how the bound and the forward come to disagree about + // one file, which is the defect #1378 records. return RouteGgufTensor(policy.keep_quant, policy.keep_f16, policy.nvfp4_fp4, policy.cpu_ref, role, tensor.ggml_type, tensor.shape, - policy.device); + policy.ComputeDeviceFor(tensor.name, role)); } } // namespace vllm diff --git a/tests/vllm/model_executor/test_device_placement.cpp b/tests/vllm/model_executor/test_device_placement.cpp index adacef1b5..1adc23de6 100644 --- a/tests/vllm/model_executor/test_device_placement.cpp +++ b/tests/vllm/model_executor/test_device_placement.cpp @@ -439,3 +439,55 @@ TEST_CASE("placement seam: an UNPLACEABLE arm REFUSES a real placement, and stay vllm::ResetActiveMoePlacementPlanForTesting(); vt::DestroyQueue(q); } + +// ── BACKEND-ROCM-IQ-EXPERT-RESIDENCY (#2516) ──────────────────────────────── +// The name -> layer inverse. It exists because a weight LOADER has a tensor +// name and no layer index, and the residency of a routed-expert tower is +// decided there. +TEST_CASE("GgufBlockIndexFromTensorName reads llama.cpp's blk.. prefix") { + CHECK(vllm::GgufBlockIndexFromTensorName("blk.0.ffn_gate_exps.weight") == 0); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.7.ffn_up_exps.weight") == 7); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.77.ffn_down_exps.weight") == 77); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.3.attn_q.weight") == 3); + + // ANCHORED. A name that merely CONTAINS the prefix is not that block's + // tensor, and answering as though it were would move a weight no plan + // claimed. This is the one place this rule differs from an operator's `-ot` + // pattern, which is deliberately `regex_search`. + CHECK(vllm::GgufBlockIndexFromTensorName("x.blk.3.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("token_embd.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("output_norm.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.3") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.x.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("") == -1); + // An index no `per_layer_` can hold is refused rather than wrapped onto a + // real layer. + CHECK(vllm::GgufBlockIndexFromTensorName( + "blk.99999999999999999999999.ffn_gate_exps.weight") == -1); +} + +TEST_CASE("MoePlacementPlan::DeviceForRoutedExpertTensor answers by name") { + std::vector ov; + for (int64_t l = 2; l < 4; ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + const vllm::MoePlacementPlan plan = vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, vt::DeviceType::kCUDA), + /*num_hidden_layers=*/4); + REQUIRE(plan.PlacesAnything()); + REQUIRE(plan.placed_layer_count() == 2); + + CHECK(plan.DeviceForRoutedExpertTensor("blk.2.ffn_gate_exps.weight") == + vt::DeviceType::kCPU); + CHECK(plan.DeviceForRoutedExpertTensor("blk.3.ffn_down_exps.weight") == + vt::DeviceType::kCPU); + CHECK(plan.DeviceForRoutedExpertTensor("blk.1.ffn_gate_exps.weight") == + vt::DeviceType::kCUDA); + // A name with no block index, and a block beyond what the plan was resolved + // against, both take the inert answer rather than throwing on a load path. + CHECK(plan.DeviceForRoutedExpertTensor("token_embd.weight") == + vt::DeviceType::kCUDA); + CHECK(plan.DeviceForRoutedExpertTensor("blk.9.ffn_gate_exps.weight") == + vt::DeviceType::kCUDA); +} diff --git a/tests/vllm/model_executor/test_gguf_device_fit.cpp b/tests/vllm/model_executor/test_gguf_device_fit.cpp index 2757ee8cd..2d3d89a9c 100644 --- a/tests/vllm/model_executor/test_gguf_device_fit.cpp +++ b/tests/vllm/model_executor/test_gguf_device_fit.cpp @@ -19,6 +19,7 @@ #include "support/test_env.h" #include "vllm/gguf_builder.h" +#include "vllm/model_executor/device_placement.h" #include "vllm/model_executor/model_loader/gguf_device_fit.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" @@ -700,12 +701,24 @@ TEST_CASE( // And the residency itself, stated directly in the same role, so a reader // does not have to infer it from the predicate above. + // + // #2516: NAMED DEVICES, not `CurrentPlatform()`. This case is about the CPU + // and CUDA `vec_dot` kernels #2247 added, and asking the running platform made + // it RED on every ROCm build for a reason the case did not state -- + // `DeviceKeepQuantSupported` serves exactly {Q8_0, Q4_K, Q5_K, Q6_K} there + // (#1940 owns the gap). Both answers are pinned instead, because the ROCm one + // is a real property of this tree and a case that merely skipped it would stop + // noticing when #1940 lands. for (const vllm::GgufTensorInfo& t : gguf.Tensors()) { CAPTURE(t.name); CHECK(vllm::RouteGgufTensor(true, false, false, false, vllm::GgufTensorRole::kStackedExpertWeight, - t.ggml_type, - t.shape, vllm::platforms::CurrentPlatform().device_type()) == vllm::GgufResidency::kKeepQuant); + t.ggml_type, t.shape, vt::DeviceType::kCPU) == + vllm::GgufResidency::kKeepQuant); + CHECK(vllm::RouteGgufTensor(true, false, false, false, + vllm::GgufTensorRole::kStackedExpertWeight, + t.ggml_type, t.shape, vt::DeviceType::kROCM) == + vllm::GgufResidency::kExpandBf16); } } @@ -775,3 +788,175 @@ TEST_CASE( CHECK(on.streamed_bytes == kF16TowerStaged); CHECK(on.lower_bound_bytes == 256); } + +// ── #2517: the fit check credits the plan installed 220 lines above it ─────── +// +// Measured on `strix:gpu0`, three consecutive lines of ONE load: +// "device placement INSTALLED: 56 layers run their routed experts on cpu" +// "--fit placed 56 layer(s) (147798884352 B) to bring a 216433205760 B +// footprint under a 68719476736 B budget" +// "cannot serve this GGUF: staging its weights needs at least 216433205760 +// bytes" +// The refusal quoted the UN-reduced figure. Nothing in `CheckDeviceWeightFit` +// carried any representation of a resolved placement, so there was no path by +// which the placed bytes could be subtracted. +namespace { + +// Three layers, one routed-expert tower each, plus one non-expert tensor. +// tower: Q8_0 [E=4, N=2, K=32] = 256 elems = 8 blocks * 34 = 272 B on disk, +// 512 B expanded to bf16, so `min` takes 272. +// embd: F32 8 elems = 32 B on disk, 16 B expanded, so `min` takes 16. +// Whole-file bound with no plan: 3*272 + 16 = 832. +constexpr size_t kPlaceTowerStaged = 272; +constexpr size_t kPlaceEmbdStaged = 16; +constexpr size_t kPlaceLayers = 3; +constexpr size_t kPlaceWholeBound = + kPlaceLayers * kPlaceTowerStaged + kPlaceEmbdStaged; + +std::string BuildGgufWithPerLayerExpertTowers() { + GgufModelBuilder b; + b.AddKv(StrKv("general.architecture", "qwen35moe")); + for (size_t l = 0; l < kPlaceLayers; ++l) { + b.AddTensor("blk." + std::to_string(l) + ".ffn_gate_exps.weight", + {32, 2, 4}, /*ggml_type=*/8, [] { + std::string s; + for (int i = 0; i < 8; ++i) s += Q8Block(); + return s; + }()); + } + b.AddTensor("token_embd.weight", {4, 2}, /*ggml_type=*/0, + std::string(32, '\1')); + return b.Build(); +} + +// Built through the production resolver, not by hand, so a pass cannot come +// from a per-layer vector this test invented. +vllm::MoePlacementPlan PlanPlacingTrailing(vt::DeviceType engine, + int64_t first_placed) { + std::vector ov; + for (int64_t l = first_placed; l < static_cast(kPlaceLayers); ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + return vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, engine), + static_cast(kPlaceLayers)); +} + +} // namespace + +TEST_CASE("#2517: a placed routed-expert tower leaves the staged footprint") { + TempFile f(BuildGgufWithPerLayerExpertTowers()); + const vllm::GgufFile gguf = vllm::GgufFile::Open(f.path()); + + // NULL PLAN: byte-for-byte the pre-#2517 answer, which is what every load in + // this tree that configured no placement gets. + const vllm::GgufStagedFootprint bare = vllm::GgufStagedWeightFootprint(gguf); + CHECK(bare.lower_bound_bytes == kPlaceWholeBound); + CHECK(bare.placed_tensor_count == 0); + CHECK(bare.placed_bytes == 0); + CHECK(bare.tensor_count == kPlaceLayers + 1); + + // A plan that PLACES NOTHING is the same answer again, through the other + // door: `cpu_moe` on a CPU engine resolves to exactly this. + const vllm::MoePlacementPlan trivial = + PlanPlacingTrailing(vt::DeviceType::kCPU, /*first_placed=*/0); + REQUIRE_FALSE(trivial.PlacesAnything()); + const vllm::GgufStagedFootprint inert = vllm::GgufStagedWeightFootprint( + gguf, 2, {}, false, &trivial); + CHECK(inert.lower_bound_bytes == kPlaceWholeBound); + CHECK(inert.placed_tensor_count == 0); + + // TWO OF THREE LAYERS PLACED. The two towers move into `placed_*` and leave + // the bound; the third tower and the vocabulary table stay. + const vllm::MoePlacementPlan plan = + PlanPlacingTrailing(vt::DeviceType::kCUDA, /*first_placed=*/1); + REQUIRE(plan.PlacesAnything()); + const vllm::GgufStagedFootprint placed = vllm::GgufStagedWeightFootprint( + gguf, 2, {}, false, &plan); + CHECK(placed.placed_tensor_count == 2); + CHECK(placed.placed_bytes == 2 * kPlaceTowerStaged); + CHECK(placed.lower_bound_bytes == kPlaceTowerStaged + kPlaceEmbdStaged); + CHECK(placed.tensor_count == 2); + // The bytes are ACCOUNTED, not lost: what left plus what stayed is what was + // there. A subtraction that merely shrank the figure would pass the line + // above and fail this one. + CHECK(placed.lower_bound_bytes + placed.placed_bytes == kPlaceWholeBound); +} + +TEST_CASE("#2517: CheckDeviceWeightFit refuses without the plan and passes " + "with it") { + TempFile f(BuildGgufWithPerLayerExpertTowers()); + const vllm::GgufFile gguf = vllm::GgufFile::Open(f.path()); + // A budget that straddles the two figures: 832 does not fit, 288 does. One + // file, one budget, one difference. + constexpr size_t kBudget = 500; + REQUIRE(kPlaceWholeBound > kBudget); + + const vllm::DeviceWeightFit no_plan = vllm::CheckDeviceWeightFit( + gguf, "cuda", /*needs_weight_staging=*/true, kBudget, 2, {}, false, + nullptr); + CHECK(no_plan.refuse); + CHECK(no_plan.needed_bytes == kPlaceWholeBound); + + const vllm::MoePlacementPlan plan = + PlanPlacingTrailing(vt::DeviceType::kCUDA, /*first_placed=*/1); + const vllm::DeviceWeightFit with_plan = vllm::CheckDeviceWeightFit( + gguf, "cuda", /*needs_weight_staging=*/true, kBudget, 2, {}, false, + &plan); + CHECK_FALSE(with_plan.refuse); + CHECK(with_plan.needed_bytes == kPlaceTowerStaged + kPlaceEmbdStaged); + + // A plan that does not place enough still refuses -- and then SAYS what the + // placement already took out, so the operator can never again read two + // contradictory numbers in consecutive lines of one load. + const vllm::MoePlacementPlan partial = + PlanPlacingTrailing(vt::DeviceType::kCUDA, /*first_placed=*/2); + const vllm::DeviceWeightFit still = vllm::CheckDeviceWeightFit( + gguf, "cuda", /*needs_weight_staging=*/true, /*budget_bytes=*/200, 2, {}, + false, &partial); + CHECK(still.refuse); + CHECK(still.message.find("ALREADY EXCLUDED") != std::string::npos); + CHECK(still.message.find(std::to_string(kPlaceTowerStaged)) != + std::string::npos); +} + +TEST_CASE( + "#2516: PeekRoute follows the placement, so the lane predicate and the " + "loader agree about a placed tower") { + // `GgufExpertTowersReachSlotLane` is the PRODUCTION consumer of `PeekRoute`, + // and it asks about the same tensors, in the same role, that + // `LoadStackedExperts` then routes. This case enters through that predicate + // rather than through `Route` directly, so it measures the pair rather than + // the function. + TempFile f(BuildGgufWithIQuantExpertTowers()); + const vllm::GgufFile gguf = vllm::GgufFile::Open(f.path()); + vllm::GgufLoadPolicy rocm = PolicyWith(true, false, false, false); + rocm.device = vt::DeviceType::kROCM; + + // NO PLAN: ROCm has no IQ `vec_dot` (#1940), so the towers expand and the + // predicate is false. This is the state #2516 reports and the inertness pin + // for every ROCm load that configures no placement. + vllm::ResetActiveMoePlacementPlanForTesting(); + CHECK_FALSE( + vllm::GgufExpertTowersReachSlotLane(gguf, "_exps.weight", rocm)); + + // A PLAN over both layers, placing them on the CPU whose `vec_dot` table + // covers IQ2_XXS and IQ4_XS: the same file, the same policy, the same + // predicate, and now true. + { + std::vector ov; + for (int64_t l = 0; l < 2; ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + const vllm::MoePlacementPlan plan = vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, vt::DeviceType::kROCM), + /*num_hidden_layers=*/2); + REQUIRE(plan.PlacesAnything()); + vllm::SetActiveMoePlacementPlan(plan); + CHECK(vllm::GgufExpertTowersReachSlotLane(gguf, "_exps.weight", rocm)); + } + vllm::ResetActiveMoePlacementPlanForTesting(); + + // ...and it is false again once the plan is gone, so the override belongs to + // the plan and is not a latch some earlier case set. + CHECK_FALSE( + vllm::GgufExpertTowersReachSlotLane(gguf, "_exps.weight", rocm)); +} diff --git a/tests/vllm/test_gguf_keep_quant.cpp b/tests/vllm/test_gguf_keep_quant.cpp index a8ee0fd58..c7f1e7448 100644 --- a/tests/vllm/test_gguf_keep_quant.cpp +++ b/tests/vllm/test_gguf_keep_quant.cpp @@ -35,6 +35,7 @@ #include "gguf_builder.h" #include "vllm/config/weight_residency.h" +#include "vllm/model_executor/device_placement.h" #include "vllm/model_executor/model_loader/gguf_dequant.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" @@ -2520,3 +2521,164 @@ TEST_CASE("gguf residency: the policy carries its device, and Route uses it") { CHECK(keep_metal.Route(t, vllm::GgufTensorRole::kEmbeddingTable) == vllm::GgufResidency::kExpandBf16); } + +// ── BACKEND-ROCM-IQ-EXPERT-RESIDENCY (#2516) ──────────────────────────────── +// +// A residency decision is a question about the device that will EXECUTE the +// tensor. Hybrid placement (#2023/#2314) made that device differ from the engine +// for exactly one role, and the loader kept asking the engine. +// +// The file this exists for: GLM-5.3 `UD-IQ1_S` on `strix:gpu0`. Its routed +// experts are IQ1_S/IQ3_XXS/IQ2_XXS/IQ4_XS/Q2_K/Q3_K, +// `DeviceKeepQuantSupported` serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, and the +// towers are never uploaded to any device — so every one of them expanded and +// `LoadStackedExperts` refused a load whose experts the CPU can execute. +namespace { + +// A routed-expert tower as the file stores it: [E, N, K] with a K that is a +// whole number of 256-element super-blocks, which is what `KeepQuantKDim` reads. +vllm::GgufTensorInfo IqTower(const std::string& name) { + vllm::GgufTensorInfo t; + t.name = name; + t.ggml_type = 19u; // IQ1_S + t.shape = {4, 2, 256}; + return t; +} + +// The plan the loader installs: `cpu_moe` over `layers` layers on an engine that +// is NOT the CPU, built through the SAME resolver production uses so a pass here +// cannot come from a hand-made per-layer vector. +vllm::MoePlacementPlan CpuMoePlan(vt::DeviceType engine, int64_t layers, + int64_t first_placed) { + std::vector ov; + for (int64_t l = first_placed; l < layers; ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + return vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, engine), layers); +} + +// RAII, because the plan is a process global and a case that leaked one would +// change the answer of every case that ran after it — including the inertness +// pins, which would then pass for the wrong reason. +struct ScopedPlan { + explicit ScopedPlan(const vllm::MoePlacementPlan& p) { + vllm::SetActiveMoePlacementPlan(p); + } + ~ScopedPlan() { vllm::ResetActiveMoePlacementPlanForTesting(); } +}; + +} // namespace + +TEST_CASE( + "#2516: a routed-expert tower PLACED on the CPU keeps its blocks on a ROCm " + "engine") { + vllm::ResetActiveMoePlacementPlanForTesting(); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + const vllm::GgufTensorInfo t = IqTower("blk.3.ffn_gate_exps.weight"); + + // NO PLAN INSTALLED — the inertness pin, and the state every load in this + // tree that configured no placement is in. IQ1_S has no ROCm `vec_dot` + // (#1940), so the tower expands, exactly as before this row. + CHECK(rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); + + // WITH the plan installed, the same tensor under the same policy keeps its + // blocks, because the device that will run it is the CPU and + // `vt::cpu::HasQuantDotKernel(kIQ1_S)` is true. + { + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/0)); + CHECK(rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kKeepQuant); + } + + // ...and the plan going away restores the old answer, so the override is the + // plan's and not a latch. + CHECK(rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); +} + +TEST_CASE("#2516: the override is PER LAYER, not per load") { + // Layers 4..7 placed, 0..3 not. One assertion pair on one file under one + // policy: a change that keyed the override on "a plan exists" rather than on + // THIS tensor's layer passes the case above and fails this one. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/4)); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + + CHECK(rocm.Route(IqTower("blk.5.ffn_up_exps.weight"), + vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kKeepQuant); + CHECK(rocm.Route(IqTower("blk.1.ffn_up_exps.weight"), + vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); +} + +TEST_CASE("#2516: only the STACKED-EXPERT role moves; the plan places nothing " + "else") { + // The plan moves a routed-expert BLOCK. An attention weight and the vocabulary + // table of a placed layer's block stay on the engine and must keep answering + // the engine's own capability — widening the override to every role would + // keep a Q8_0 attention weight quantized for a device that will execute it. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/0)); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + + vllm::GgufTensorInfo w; + w.name = "blk.3.attn_q.weight"; + w.ggml_type = 19u; // IQ1_S: kept on the CPU, expanded on ROCm + w.shape = {2, 256}; + CHECK(rocm.ComputeDeviceFor(w.name, vllm::GgufTensorRole::kMatmulWeight) == + vt::DeviceType::kROCM); + CHECK(rocm.Route(w, vllm::GgufTensorRole::kMatmulWeight) == + vllm::GgufResidency::kExpandBf16); + CHECK(rocm.ComputeDeviceFor("blk.3.ffn_gate_exps.weight", + vllm::GgufTensorRole::kStackedExpertWeight) == + vt::DeviceType::kCPU); +} + +TEST_CASE("#2516: PeekRoute and Route resolve the device the same way") { + // `GgufExpertTowersReachSlotLane` peeks the very tensors the loader routes. + // Two spellings of the device term is how a bound and a forward come to + // disagree about one file (#1378), so this pins that they cannot. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/4)); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + for (const char* name : {"blk.5.ffn_gate_exps.weight", + "blk.1.ffn_gate_exps.weight", + "blk.5.attn_q.weight"}) { + CAPTURE(name); + const vllm::GgufTensorInfo t = IqTower(name); + CHECK(vllm::PeekRoute(rocm, t, + vllm::GgufTensorRole::kStackedExpertWeight) == + rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight)); + } +} + +TEST_CASE("#2516: a plan that places NOTHING never overrides the engine") { + // `cpu_moe` on a CPU engine resolves to a plan over every layer that places + // nothing (`DevicePlacement::IsTrivial`). A default-constructed plan's engine + // device is also `kCPU`. Either one, adopted unconditionally, would route a + // ROCm load's experts as though they were on the host — which is the same + // silent wrong answer in the opposite direction. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kCPU, /*layers=*/8, + /*first_placed=*/0)); + CHECK_FALSE(vllm::ActiveMoePlacementPlan().PlacesAnything()); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + CHECK(rocm.ComputeDeviceFor("blk.3.ffn_gate_exps.weight", + vllm::GgufTensorRole::kStackedExpertWeight) == + vt::DeviceType::kROCM); + CHECK(rocm.Route(IqTower("blk.3.ffn_gate_exps.weight"), + vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); +} From 98016d129a28595753a28c747be71ec7c2477a00 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 10:02:16 +0000 Subject: [PATCH 3/4] record(BACKEND-ROCM-IQ-EXPERT-RESIDENCY): the outcome, and the blocker that is now next The two predicates this row names are sequential rather than alternative, which the hardware settled: on the base build the fit check refuses with all 78 layers already placed, and suppressing that refusal only buys one more stage before the loader refuses the first IQ1_S tower. With both fixed the same command loads -- 1809 tensors, 11.620 GiB paged in, the engine sizes its KV cache and the scheduler starts -- and all 228 towers stayed compressed, which is proved by the load completing at all because the loader throws by name on any tower that does not. No token came out and none is claimed. The first forward throws in the MLA block: `fused_nr` asks `vt::OpRegistered(kFusedNormRope, ...)`, ROCm registers no such op, and `OpRegistered` deliberately cannot see the reference tier -- so the split A-projection path is taken and refuses a block-quantized `kv_a_proj_with_mqa`. Its own comment says that path needs `VT_MLA_FUSED_NORM_ROPE=0`; this run had it unset. Filed as #2564, with #2565 for the `--fit` resolver gap the same run exposed. Both are recorded under `## Owed` rather than started, which is this spec's own stop condition. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/rocm-placed-expert-residency.md | 64 +++++++++++++++++-- 1 file changed, 58 insertions(+), 6 deletions(-) diff --git a/.agents/specs/rocm-placed-expert-residency.md b/.agents/specs/rocm-placed-expert-residency.md index 7ce7eec1c..7524b27e6 100644 --- a/.agents/specs/rocm-placed-expert-residency.md +++ b/.agents/specs/rocm-placed-expert-residency.md @@ -268,15 +268,67 @@ need it. #2515 records why that deletes a correct refusal. - **The ROCm MLA/DSA arm**, eight ops (`.agents/specs/rocm-glm53-dsa.md`). Every one of them runs on the reference tier in this row's result, which is why no speed number is quoted. -- **`--fit`'s resolver is not residency-aware.** It sizes a placement from - bytes alone, so on a device whose keep-quant set does not cover the file's - expert encodings it can resolve a placement that leaves towers unplaced — - and those towers then expand and refuse. On this board the operator has to - say `cpu_moe`. Needs its own issue before anyone changes the resolver. +- **`--fit`'s resolver is not residency-aware**, filed as + [#2565](https://github.com/mudler/vllm.cpp/issues/2565) and CONFIRMED on + hardware: with no placement configured it places the trailing 56 of 78, the + fit check then passes (this row's own repair), and the load refuses on + `blk.3`, one of the 22 it left on the device. On this board the operator has + to say `cpu_moe`. +- **The MLA `fused_nr` availability probe**, + [#2564](https://github.com/mudler/vllm.cpp/issues/2564) — the blocker that + now stands between this checkpoint and a token on `gfx1151`. Owner + `BACKEND-ROCM`. - **`quant_repack` for a placed tower** (W5). - **The device budget** (#2518) and **the host-slot lane** (#2515), both untouched and both re-measured here. ## Now -`ACTIVE`. +`ACTIVE` — implemented, gated, and measured on `strix:gpu0`; open as #2562 for a +fresh review. The row's own scope is complete and the model's remaining blocker +is named and filed (#2564), not left to be discovered. + +## Outcome + +**What was measured, on the real artifact.** Both predicates this row names are +real, and they are SEQUENTIAL rather than alternative: on the base build with +`VT_CPU_MOE=1` the fit check refuses quoting the un-reduced 216433205760 B with +all 78 layers already placed (#2517), and with that refusal suppressed the load +gets one stage further and refuses on +`blk.3.ffn_gate_exps.weight routed to an EXPAND residency` (#2516). With both +fixed, the same command LOADS: 1809 tensors, `[vt load] weights 1372.153 s`, +`gguf prefault spans=589 paged_in=11.620 GiB in 1037.930 s (11.5 MiB/s)`, the +engine auto-fits `max_model_len` from 1048576 to 8192 against 256 blocks of 32 +tokens, and the scheduler starts. **All 228 routed-expert towers stayed +compressed**, which is not an inference: `LoadStackedExperts` throws by name on +any tower that expands, so a load that completes is the proof. + +**NO TOKEN CAME OUT, and none is claimed.** The first forward throws inside the +MLA block. `fused_nr` (`mla_attention.cpp:550-551`) has a +`vt::OpRegistered(kFusedNormRope, ...)` term, ROCm registers no +`kFusedNormRope`, and `OpRegistered` is deliberately a native-only probe that +cannot see the reference tier — so the split A-projection path is taken and +refuses this checkpoint's block-quantized `kv_a_proj_with_mqa`. The comment at +`:628-631` states that path is reachable "only with VT_MLA_FUSED_NORM_ROPE=0"; +this run had it unset and is the counterexample. Filed as +[#2564](https://github.com/mudler/vllm.cpp/issues/2564). That is exactly this +spec's W6 stop condition, and the row stops there rather than widening into the +MLA arm. + +**What was rejected, with the reason measured rather than assumed.** Repairing +the host-slot lane: `pageableMemoryAccess = 0` was read off this board, so +`HostMemoryIsDeviceAddressable` is false and the lane cannot serve here at all +(#2515 confirmed). Porting ROCm i-quant `vec_dot` (#1940): a placed tower is +executed by the CPU, and an UNPLACED tower on this board cannot be executed at +all because `GlmResidentExpertSlice` refuses a non-host-addressable device before +any kernel is asked for — so those kernels would not have produced a token here. +Correcting the device budget (#2518): the credited footprint is 11.620 GiB paged +in against a 58.000 GiB managed ceiling and a 64.00 GiB reported pool, so the +6.00 GiB of optimism between them cannot decide this load in either direction. + +**One measurement about the instrument, not the code.** `vllm-cli`'s sha256 is +BYTE-IDENTICAL across the base and patched builds +(`ac37fb6d11ba3c1ae7d3d931777ac32ef6fe6bee00231ba18470330740192529`) while +`libvllm.so.0.0.3` moves — it is a thin ABI client and nothing in it changed. A +binary-identity guard that digests only the executable reports two different +builds as one, and this row measured that rather than inheriting it. From 938c951e4591ece5b4f53685d930215a01b0d9d4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 10:06:20 +0000 Subject: [PATCH 4/4] docs(BACKEND-ROCM-IQ-EXPERT-RESIDENCY): say what GLM-5.3 does on gfx1151, which is load and not generate The checkpoint row's build-requirements cell said `--device cuda` and named the CPU queue as the only alternative. That is now incomplete: with #2562 the same artifact LOADS on `strix:gpu0` under `VT_CPU_MOE=1 --device auto`, all 228 routed towers stay compressed, 11.620 GiB is paged in, and the engine sizes its KV cache. It then emits NOTHING, because the first forward throws in the MLA block (#2564), and the sentence says that in the same breath rather than leaving a reader to infer a token from a successful load. The three surrounding facts a reader on an AMD box needs are stated with it and each carries its issue: `auto` is the only route to ROCm (#2505), `cpu_moe` is required because `--fit` leaves towers on a device that cannot hold them (#2565), and the streamed-expert lane is not what serves them and cannot be on this board (#2515). No speed number is offered, because the run reaches no token at all. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/USAGE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index 2c76dfb86..b0c646552 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -655,7 +655,7 @@ repository in this project's history. | GLM-5.3-Flash FP8 source | `model-000{01..62}-of-00062.safetensors` | 328,326,771,576 bytes total (305.78 GiB) | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-26 | Owed: no byte of payload has been fetched, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | Declared source of `scripts/convert-glm5-next-gguf.py`. Only the safetensors HEADERS were read, by HTTP RANGE over all 62 shards: 76,108 tensors, `F8_E4M3` block-quantized at `weight_block_size: [128, 128]` with `weight_scale_inv` companions, plus BF16 and F32 scales | **Nothing has been converted.** The download needs explicit developer authority and a box with room for 305.78 GiB of source and ~100.35 GiB of output at once; owed as O7 on [#2011](https://github.com/mudler/vllm.cpp/issues/2011). The revision is a branch name and not a commit, which is NOT a pin: it is what was read, and W7b re-reads and records the commit when it stages the bytes | | GLM-5.3-Flash GGUF | `GLM-5.3-Flash-UD-Q2_K_XL-0000{1..4}-of-00004.gguf` | 108,720,071,427 bytes total (101.2535 GiB) across four shards; 1412 tensors | `unsloth/GLM-5.3-Flash-GGUF` @ `d425e572fb9686125831f476129e51cea34bc5b4`, path `UD-Q2_K_XL`, staged 2026-08-28 | Owed for this row: the shards are staged and were sha256-verified when they were fetched, but **W5c consumed only the four GGUF HEADERS** and states no hash of its own. W7b ([#2225](https://github.com/mudler/vllm.cpp/issues/2225)) records the per-shard sha256 alongside the load it measures | **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)). **A MATERIALIZED LOAD EXISTS** -- driven at this artifact on `dgx:gpu0` 2026-08-30, all four shards load and the engine sizes its caches in under 26 minutes wall ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)). At that change the first step threw at the `multi_kv` guard above the model's own hook; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) writes the consuming forward that guard was waiting for and it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT, and peak RSS is MEASURED** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). On `dgx:gpu0` 2026-08-30, in the SHIPPED configuration with no diagnostic env set, `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is` emits ` Paris.` at `rc=0`, and `VmHWM` peaks at 104,792,300 kB = 99.94 GiB. Two instrumented `thor:gpu0` runs the same day supply the bisect: four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427), ` one`, ` located`, ` known`, ` a` at margin 1.279, and none of 180 per-layer readings over four steps carries a NaN. The first attempt emitted token id 0 eight times, because the loader repacked this file's 346 q8_0 tensors into the i8mm interleave that the host bridge reads as plain blocks (spec `## Owed` O30). **No speed number is claimed, and the earlier ones are void** -- they were taken from an all-NaN forward. The GB10 arm is the one measured above. The GGUF arm of `load_weights` resolves all 1383 backbone tensors of this file (W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242)); `blk.45`, the multi-token-prediction block, is read, counted and DROPPED, as the transformers reference does. `ModelRegistry::Forward` dispatches to the model as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which bridges ONE decoder layer at a time out of the block-resident tower and decodes only the 8 of 288 experts a token selects — a float tower is 426.72 GiB against ~119.63 GiB usable. **A MATERIALIZED LOAD NOW EXISTS**: driven at this artifact on `dgx:gpu0` 2026-08-30, all four shards load and the engine sizes its caches in under 26 minutes wall. **NO TOKEN WAS GENERATED** — the first step throws at the `multi_kv` guard above the model's own hook ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)) — and **peak RSS and speed are still unmeasured**, because the staging run did not sample them. The vision tower (a separate `mmproj-BF16.gguf`) and the safetensors arm still refuse by name, as does a multi-request step; a non-CPU queue is admitted as of W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) for the routed-expert GEMM alone, and a device that is neither CPU nor CUDA is refused by name; **the KV-cache spec does not**, as of W5 ([#2223](https://github.com/mudler/vllm.cpp/issues/2223)), which publishes its three groups through the production factory hook | **The earlier row here said `none exists`, and that was true when it was written (2026-08-26) and is not now.** "UD-Q2_K_XL" names a TARGET AVERAGE and not a format: the census over all 1412 tensors is F32 638, Q8_0 346, Q5_K 181, Q6_K 117, IQ2_XS 82, IQ3_XXS 41, IQ4_XS 3, Q2_K 2, Q4_K 1, Q3_K 1 — **two** Q2_K tensors in a file named Q2_K. It fits `dgx:gpu0` only because IQ2_XS and IQ4_XS keep their blocks ([#2247](https://github.com/mudler/vllm.cpp/issues/2247)); both now have a CUDA keep-quant kernel too ([#2260](https://github.com/mudler/vllm.cpp/issues/2260)), so the expert GEMM no longer drains the stream to the host and the fused seam no longer throws. W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then built a device arm for this artifact's routed-expert GEMM and MEASURED it end to end, where it **SEGFAULTED**: both `--device cuda` legs on `dgx:gpu0` died with rc=139 emitting no token, reproducibly (spec O46). The split is therefore OPT-IN and defaults OFF, so `--device cuda` refuses exactly as it did before. **Use `--device cpu`** -- measured on that artifact it emits ` Paris.` at rc=0, 1176 s wall of which 169 s is generation. Every OTHER primitive of this model is still a host reference on an interposed CPU queue (spec O43), so what `--device cuda` reaches is one arm of eleven and not a device arm. **A materialized load NOW exists and a token still does not** — `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches, then the first step throws at the `multi_kv` guard above the model's own hook. **Peak RSS and speed remain unmeasured** | | GLM-5.3-Flash config | `config.json` | 69,416 bytes | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-27 | sha256 `bb8f01c42cb92a52ca72e65afb4d5bd8d11aef083cd210e8de25dfb904f23e9f` | The ONLY byte of this checkpoint any change on this row has consumed. Checked in verbatim as `tests/vllm/models/fixtures/glm5_next/config.json` and used as W1's gate fixture, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | **Arms refused by name:** the SAFETENSORS one, which is what this row is, because every published safetensors artifact of this model exceeds every device this project owns. `Glm5NextForConditionalGeneration` is REGISTERED, its config RESOLVES, and the GGUF arm both loads and forwards ([#2067](https://github.com/mudler/vllm.cpp/issues/2067), [#2242](https://github.com/mudler/vllm.cpp/issues/2242), [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). The revision is a branch name and not a commit, which is NOT a pin for the WEIGHTS; for this one file the sha256 above is the pin | -| GLM-5.3 GGUF (`glm-dsa`) all six shards | `GLM-5.3-UD-IQ1_S-0000{1..6}-of-00006.gguf` | 9,428,677 B (shard 1, metadata only, 0 tensors) and 49,968,868,928 B (shard 2); the six shards total 216,715,365,893 B = 201.83 GiB across 1809 tensors | `unsloth/GLM-5.3-GGUF` @ `346b3591c7f28d1a23716f97a065ecf12ec14771`, path `UD-IQ1_S`, staged 2026-08-30, completed and verified 2026-08-31 | shard 1 `ff3adab0853dfb00bdf3889ec3f5556196f56b65783115720d57767bbd760dd9`; shard 2 `659d04cf4fc0b6026944f34c0b590a635803bff06c1775361e28490db7b168f8`; shard 3 `433302bac0e2d54da64c7c2f28509fa1b235aeccdf5b215a8a446ebaad1b5b27`; shard 4 `d0a6f19452d5b5cd498e1eb8fbe856e00aed7da1f80c27c095301eabe81e9bc1`; shard 5 `2ea1537ffab40fa8b8584a8647ec10fbaa6199dfed45e4019b822da2b319db37`; shard 6 `42a76ef04ffc5e321e1240f4e572b6fa6fc3315da5bea22fb598d7460db210fe`. **All six are complete and each was hashed TWICE** — once by the fetch script as it landed and once independently off the same share afterwards — and the two readings agree. **The DERIVED metadata shard has a hash of its own:** `scripts/glm-dsa-write-indexer-types.py` run against the staged shard 1 with `zai-org/GLM-5.3`'s own `config.json` produces a 9,428,810-byte file, 64 keys becoming 65, 21 `full` of 78, sha256 `b3e9838651a5c279533c98390ab4bc03cf1d8c176d5be0754180f07d9ed85c01`, reproduced identically by three independent runs. **That is a DERIVED artifact and must never be quoted as `unsloth/GLM-5.3-GGUF`'s shard 1** | **THIS ARTIFACT GENERATES THROUGH THE EXPERT-STREAMING LANE: `The capital of France is` -> ` Paris`.** On `dgx:gpu0` (GB10, 20 cores, 119 GB, compute capability 12.1) under an `rc` lease, 2026-08-31, a build with `-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_FLASH_ATTN=ON` and CUTLASS 4.5.0: `VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=4096 vllm-cli --model --device cuda --prompt "The capital of France is" --max-tokens 1 --temperature 0` returns `rc=0`, `prompt_tokens=5 completion_tokens=1`, and seven bytes of stdout: a space, `Paris`, a newline. Wall 1154 s for the process, `generate` 852.330 s, `VmHWM` 60,512,268 kB = 57.71 GiB against 119.631 GiB of device and 201.83 GiB of artifact. **The lane's own counters are the streaming evidence, and they include one number that must travel with them:** `[expert-stream] ON slots=4096 slot_bytes=6684672 resident=25.50 GiB`, then `steps=1 hits=0 misses=6399 evictions=0 fills=4096 bytes=13939408896 exhausted=2303 advised=0`. 4096 slices were paged out of the file into slots and 12.98 GiB moved through them with zero evictions, and the 187.312 GiB of towers were never materialized — but the step needed 6399 distinct slices, so **2303 of them (36%) were read in place out of the mapping instead of streamed**. That is a PREFILL working set exceeding any slot budget by construction (spec R2, O34), it is counted rather than silent, and no figure here may be quoted as a fully-streamed step. **No speed number is claimed:** one token, a CIFS-backed artifact, and 2303 in-place fallbacks in the measurement. `--device cpu` on the same box and artifact also emits ` Paris` (`rc=0`, `generate` 950.249 s, `VmHWM` 44.46 GiB), and **that arm does NOT stream at all** — a CPU queue builds no slot lane, so every routed-expert slice is read in place. On `thor:gpu0` (sm_110a) the CUDA arm cannot reach a token: MLA prefill on this family IS FlashAttention, the vendored FA2 covers `8.0,8.6,8.7,8.9,12.0a,12.1a`, and sm_110a is outside it. Also gated on a complete synthetic model of the same shape: `test_glm_moe_dsa_gguf_load.cpp` 5 cases / 228 assertions, `test_glm_moe_dsa_forward.cpp` 7 / 5258, `test_glm_moe_dsa_schedule.cpp` 12 / 533, and the real file's census from its headers (`test_glm_moe_dsa_gguf_census.cpp` 3 / 3831): 1809 tensors, 228 expert towers at 187.312 GiB, 1581 resident at 14.511 GiB, largest per-expert slice 6,684,672 B. What the forward still refuses BY NAME is a step in which any request RESUMES while its selection PRUNES — that needs the indexer KV side cache `KV-DSV4-MULTICACHE` owns (spec O4, #1925/#2323), so a FIRST token on a fresh prompt is reachable and a SECOND is not — and sparse prefill (spec O6) is still W6's. No speed axis has a denominator (spec O10) | **THIS FILE CANNOT BE FED AS PUBLISHED**, and that is a property of the file rather than of the port: its 64 metadata keys carry neither `glm-dsa.attention.indexer.types` nor `index_topk_freq`/`index_skip_topk_offset`, so it states its per-layer indexer schedule nowhere, and it broadcasts `indexer.*` onto all 79 blocks while the checkpoint ships them on 22. The loader refuses it by name rather than substituting llama.cpp's hardcoded table (spec D3). **The repair is one command and it rewrites the 9.4 MB metadata shard only:** `scripts/glm-dsa-write-indexer-types.py --shard --from-config --out /GLM-5.3-UD-IQ1_S-00001-of-00006.gguf`, with the five payload shards hard-linked beside the output, then `--model` that directory's shard 1. It transcribes the schedule from the model author's own `config.json` and derives nothing; the result is a DERIVED artifact with its own sha256 and is not `unsloth/GLM-5.3-GGUF`. **Build requirements this model does not degrade past:** `--device cuda` (the expert-streaming lane is not built on a CPU queue, and the towers would then be read in place out of a 201.83 GiB mmap), and a build with the vendored FlashAttention-2, which needs CUTLASS headers and an arch in `8.0,8.6,8.7,8.9,12.0a,12.1a` — MLA prefill IS FlashAttention here and has no fallback below it. **Arms refused by name:** the SAFETENSORS one, permanently (spec D1 — 703.74 GiB across 141 shards, no streaming loader, no MoE block-fp8 rung), and `UD-IQ1_M`, which refuses at file open because `IQ1_M` (ggml id 29) has no reader traits (spec O3) | +| GLM-5.3 GGUF (`glm-dsa`) all six shards | `GLM-5.3-UD-IQ1_S-0000{1..6}-of-00006.gguf` | 9,428,677 B (shard 1, metadata only, 0 tensors) and 49,968,868,928 B (shard 2); the six shards total 216,715,365,893 B = 201.83 GiB across 1809 tensors | `unsloth/GLM-5.3-GGUF` @ `346b3591c7f28d1a23716f97a065ecf12ec14771`, path `UD-IQ1_S`, staged 2026-08-30, completed and verified 2026-08-31 | shard 1 `ff3adab0853dfb00bdf3889ec3f5556196f56b65783115720d57767bbd760dd9`; shard 2 `659d04cf4fc0b6026944f34c0b590a635803bff06c1775361e28490db7b168f8`; shard 3 `433302bac0e2d54da64c7c2f28509fa1b235aeccdf5b215a8a446ebaad1b5b27`; shard 4 `d0a6f19452d5b5cd498e1eb8fbe856e00aed7da1f80c27c095301eabe81e9bc1`; shard 5 `2ea1537ffab40fa8b8584a8647ec10fbaa6199dfed45e4019b822da2b319db37`; shard 6 `42a76ef04ffc5e321e1240f4e572b6fa6fc3315da5bea22fb598d7460db210fe`. **All six are complete and each was hashed TWICE** — once by the fetch script as it landed and once independently off the same share afterwards — and the two readings agree. **The DERIVED metadata shard has a hash of its own:** `scripts/glm-dsa-write-indexer-types.py` run against the staged shard 1 with `zai-org/GLM-5.3`'s own `config.json` produces a 9,428,810-byte file, 64 keys becoming 65, 21 `full` of 78, sha256 `b3e9838651a5c279533c98390ab4bc03cf1d8c176d5be0754180f07d9ed85c01`, reproduced identically by three independent runs. **That is a DERIVED artifact and must never be quoted as `unsloth/GLM-5.3-GGUF`'s shard 1** | **THIS ARTIFACT GENERATES THROUGH THE EXPERT-STREAMING LANE: `The capital of France is` -> ` Paris`.** On `dgx:gpu0` (GB10, 20 cores, 119 GB, compute capability 12.1) under an `rc` lease, 2026-08-31, a build with `-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_FLASH_ATTN=ON` and CUTLASS 4.5.0: `VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=4096 vllm-cli --model --device cuda --prompt "The capital of France is" --max-tokens 1 --temperature 0` returns `rc=0`, `prompt_tokens=5 completion_tokens=1`, and seven bytes of stdout: a space, `Paris`, a newline. Wall 1154 s for the process, `generate` 852.330 s, `VmHWM` 60,512,268 kB = 57.71 GiB against 119.631 GiB of device and 201.83 GiB of artifact. **The lane's own counters are the streaming evidence, and they include one number that must travel with them:** `[expert-stream] ON slots=4096 slot_bytes=6684672 resident=25.50 GiB`, then `steps=1 hits=0 misses=6399 evictions=0 fills=4096 bytes=13939408896 exhausted=2303 advised=0`. 4096 slices were paged out of the file into slots and 12.98 GiB moved through them with zero evictions, and the 187.312 GiB of towers were never materialized — but the step needed 6399 distinct slices, so **2303 of them (36%) were read in place out of the mapping instead of streamed**. That is a PREFILL working set exceeding any slot budget by construction (spec R2, O34), it is counted rather than silent, and no figure here may be quoted as a fully-streamed step. **No speed number is claimed:** one token, a CIFS-backed artifact, and 2303 in-place fallbacks in the measurement. `--device cpu` on the same box and artifact also emits ` Paris` (`rc=0`, `generate` 950.249 s, `VmHWM` 44.46 GiB), and **that arm does NOT stream at all** — a CPU queue builds no slot lane, so every routed-expert slice is read in place. On `thor:gpu0` (sm_110a) the CUDA arm cannot reach a token: MLA prefill on this family IS FlashAttention, the vendored FA2 covers `8.0,8.6,8.7,8.9,12.0a,12.1a`, and sm_110a is outside it. Also gated on a complete synthetic model of the same shape: `test_glm_moe_dsa_gguf_load.cpp` 5 cases / 228 assertions, `test_glm_moe_dsa_forward.cpp` 7 / 5258, `test_glm_moe_dsa_schedule.cpp` 12 / 533, and the real file's census from its headers (`test_glm_moe_dsa_gguf_census.cpp` 3 / 3831): 1809 tensors, 228 expert towers at 187.312 GiB, 1581 resident at 14.511 GiB, largest per-expert slice 6,684,672 B. What the forward still refuses BY NAME is a step in which any request RESUMES while its selection PRUNES — that needs the indexer KV side cache `KV-DSV4-MULTICACHE` owns (spec O4, #1925/#2323), so a FIRST token on a fresh prompt is reachable and a SECOND is not — and sparse prefill (spec O6) is still W6's. No speed axis has a denominator (spec O10) | **THIS FILE CANNOT BE FED AS PUBLISHED**, and that is a property of the file rather than of the port: its 64 metadata keys carry neither `glm-dsa.attention.indexer.types` nor `index_topk_freq`/`index_skip_topk_offset`, so it states its per-layer indexer schedule nowhere, and it broadcasts `indexer.*` onto all 79 blocks while the checkpoint ships them on 22. The loader refuses it by name rather than substituting llama.cpp's hardcoded table (spec D3). **The repair is one command and it rewrites the 9.4 MB metadata shard only:** `scripts/glm-dsa-write-indexer-types.py --shard --from-config --out /GLM-5.3-UD-IQ1_S-00001-of-00006.gguf`, with the five payload shards hard-linked beside the output, then `--model` that directory's shard 1. It transcribes the schedule from the model author's own `config.json` and derives nothing; the result is a DERIVED artifact with its own sha256 and is not `unsloth/GLM-5.3-GGUF`. **Build requirements this model does not degrade past:** `--device cuda` (the expert-streaming lane is not built on a CPU queue, and the towers would then be read in place out of a 201.83 GiB mmap), and a build with the vendored FlashAttention-2, which needs CUTLASS headers and an arch in `8.0,8.6,8.7,8.9,12.0a,12.1a` — MLA prefill IS FlashAttention here and has no fallback below it. **On ROCm `gfx1151` (`strix:gpu0`, Radeon 8060S) this artifact LOADS and DOES NOT GENERATE, as of [#2562](https://github.com/mudler/vllm.cpp/pull/2562).** The route is `VT_CPU_MOE=1 vllm-cli --model --device auto` -- `auto` because no `--device` value names ROCm ([#2505](https://github.com/mudler/vllm.cpp/issues/2505)), and `cpu_moe` because `--fit`'s default placement leaves 22 layers on a device whose keep-quant set cannot hold their IQ1_S towers ([#2565](https://github.com/mudler/vllm.cpp/issues/2565)). All 1809 tensors resolve, all 228 routed-expert towers stay compressed, 11.620 GiB is paged in (at 11.5 MiB/s off the CIFS share the artifact lives on, which is a property of the share), and the engine auto-fits `max_model_len` to 8192 against 256 blocks of 32 tokens. **NO TOKEN COMES OUT:** the first forward throws in the MLA block, because `vt::OpRegistered(kFusedNormRope, ...)` is false on ROCm and cannot see the reference tier, so the split A-projection path is taken and refuses this checkpoint's block-quantized `kv_a_proj_with_mqa` ([#2564](https://github.com/mudler/vllm.cpp/issues/2564)). The streamed-expert lane is NOT what serves the towers here and cannot be: `pageableMemoryAccess` is 0 on this board, so `host_memory_is_device_addressable()` is false ([#2515](https://github.com/mudler/vllm.cpp/issues/2515)). No speed number is admissible from this board for this model. **Arms refused by name:** the SAFETENSORS one, permanently (spec D1 — 703.74 GiB across 141 shards, no streaming loader, no MoE block-fp8 rung), and `UD-IQ1_M`, which refuses at file open because `IQ1_M` (ggml id 29) has no reader traits (spec O3) | | GLM-5.3 config | `config.json` | 29,464 bytes | `zai-org/GLM-5.3` @ `935644c05e76fc198714f4cca449fd8b970ff6d7` | Committed verbatim in-tree as `tests/vllm/models/glm_moe_dsa_config_glm53.inc`, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | It is the ONLY authoritative source of the 78-entry `indexer_types` list — 21 `full`, at layers {0,1,2} and every fourth from 6 to 74 — which three independent derivations agree on bit for bit (the list itself, vLLM's rule at `deepseek_v2.py:1097-1101`, and llama.cpp's `GLM_5_2_DEFAULT_INDEXER_TYPES`) | The GGUF above does not carry this list, which is why it cannot be fed as published | | Qwen3.5-0.8B (Tenstorrent P150 arm) | `model.safetensors-00001-of-00001.safetensors` | 1,746,942,600 bytes | `Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17`, authorized 2026-08-23 | `04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696` (non-quantized; hashed anyway from the local bytes the gates and the eager profile consumed) | bf16 on the Tenstorrent P150: the sacred greedy pair, both ambient legs, and the #1715/#2107 profile legs all ran from this snapshot | **Arms refused by name:** GGUF k-quant arms on TT — no TT kernels exist for them, refused at load; Qwen3.8-27B on TT — no arm fits the P150 (bf16 53.8 GB), refused at load | | dots3-note bf16 language tower | `model-000{01..131}-of-00131.safetensors` | 561,371,869,568 bytes total (522.82 GiB), of which the MoE is 545,823,175,680 | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed: **no tensor byte has been fetched**, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | The bf16 text tower this port loads: 46 backbone layers, both MLA geometries, and since W5 the 45 MoE layers — the ungrouped noaux_tc router at 256/8 plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536. Everything except `mlp.gate.e_score_correction_bias` is BF16; that one is F32, on both sides | **Nothing has ever loaded these bytes.** The tower alone is 522.82 GiB against a 122 GiB ceiling on the largest host this project reaches (spec §6.2), so the arm is representable and unfeedable, and the e2e gate is an OPEN GAP by construction. GGUF k-quants are refused by name (W9). The 19-tensor nextn tail is a NAMED W10 deferral rather than a refusal since #2176 |