diff --git a/.agents/specs/rocm-placed-expert-residency.md b/.agents/specs/rocm-placed-expert-residency.md new file mode 100644 index 000000000..7524b27e6 --- /dev/null +++ b/.agents/specs/rocm-placed-expert-residency.md @@ -0,0 +1,334 @@ +# BACKEND-ROCM-IQ-EXPERT-RESIDENCY — route a placed routed-expert tower's residency through the device that will RUN it + +Issue: [#2516](https://github.com/mudler/vllm.cpp/issues/2516) (the routing), +[#2517](https://github.com/mudler/vllm.cpp/issues/2517) (the fit check that +ignores the plan it just installed). +Related, and deliberately NOT repaired here: +[#2515](https://github.com/mudler/vllm.cpp/issues/2515) (the host-slot lane +cannot serve on `gfx1151`; this row's answer does not need it), +[#2518](https://github.com/mudler/vllm.cpp/issues/2518) (the managed ceiling; +measured below to be unable to decide this load), +[#1940](https://github.com/mudler/vllm.cpp/issues/1940) (ROCm i-quant +`vec_dot`; measured below to be unnecessary for this load), +[#2505](https://github.com/mudler/vllm.cpp/issues/2505) (no `--device rocm`). + +Base SHA: `35116605bb294e78f1be103cd040c378c6d00e20`. + +## Scope + +Make GLM-5.3 (`GlmMoeDsaForCausalLM`) `UD-IQ1_S` load and generate on +`strix:gpu0` (`gfx1151`, ROCm 7.2.4) with its routed experts placed on the +host, by fixing the two predicates that stop a HYBRID-PLACED load from +reaching a forward. It does not port a kernel, does not move a platform +predicate, and claims no performance number. + +IN SCOPE: + +- `GgufLoadPolicy::Route` / `PeekRoute`: for a `kStackedExpertWeight`, resolve + the residency against the device the installed `MoePlacementPlan` says will + COMPUTE that tower, instead of always against the engine device. +- `GgufStagedWeightFootprint` / `CheckDeviceWeightFit`: exclude the + routed-expert tensors of layers the installed plan places off the engine + device, and say so in the refusal. +- `tests/vllm/model_executor/test_gguf_device_fit.cpp`'s IQ case, which is red + on every ROCm build for a reason the case does not state (#2516). + +OUT OF SCOPE: + +- ROCm i-quant `vec_dot` kernels (#1940). Measured unnecessary: a tower placed + on the CPU is executed by `vt::cpu`'s `vec_dot`, and + `vt::cpu::HasQuantDotKernel` already answers true for all six encodings this + checkpoint's experts use (`cpu_quant_dot.cpp:995,996,1000,1001,1004,1005`). + The ROCm gap is what makes an UNPLACED tower expand, and this checkpoint has + no unplaced-tower arm on this board for an unrelated reason (see W2). +- `RocmPlatform::residency_policy().device_memory_total_bytes` (#2518). Left at + the `hipMemGetInfo` probe. See W4 for why it cannot decide this load. +- The host-slot expert-streaming lane (#2515). Not repaired and not needed: see + W2. +- `--fit`'s own resolver. It sizes a placement from bytes and cannot know that + an UNPLACED tower expands on this device, so its default answer on this board + is a placement that still refuses. Recorded under `## Owed`. + +## W1. The chain, measured rather than assumed + +Five facts, each read off the tree at the base SHA or off the board: + +1. **GLM-5.3's expert towers are never uploaded to any device.** + `LoadStackedExperts` (`glm_moe_dsa_loader.cpp:381-421`) takes no `Dev` and + no queue; it returns an mmap-borrowed, unprefaulted `OwnedTensor`. The + registry's `prepare` hook is a no-op (`glm_moe_dsa_registry.cpp:88-93`), and + the forward reaches a tower only through `GlmExpertSlice` + (`glm_moe_dsa_forward.cpp:293,294,297-300`), never through + `dense_attn::ResidentWeight` — which is the one place `d.b.Alloc` + + `d.b.Copy` live (`dense_attn_block.h:250-268`). The file says so itself at + `glm_moe_dsa_forward.cpp:94-113`. + +2. **On ROCm the unplaced MoE forward cannot run at all.** + `GlmResidentExpertSlice` throws unless the platform is CPU or host-addressable + (`glm_moe_dsa_forward.cpp:112-122`), and `expert_stream::ExpertSlice` admits + the streaming lane on the same predicate (`expert_stream_seam.cpp:431`). + `host_memory_is_device_addressable()` is FALSE on `gfx1151` + (#2515, re-measured in W4), so BOTH arms are closed. GLM-5.3's routed + experts on this board can only run on a CPU `Dev` handed to `MoeBlock` by + `RunMoePlaced` — which the model already composes, unconditionally placeable + (`glm_moe_dsa_forward.cpp:471-475`). + +3. **The loader asks the wrong device about a placed tower.** The one policy is + `GgufLoadPolicy::FromEnv(source.device)` (`glm_moe_dsa_registry.cpp:80`), and + `Route` hands `policy.device` straight to `RouteGgufTensor` + (`gguf_keep_quant.cpp:401-408`). `DeviceKeepQuantSupported` serves exactly + {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm (`gguf_keep_quant.cpp:135-147`), so every + IQ1_S / IQ2_XXS / IQ3_XXS / IQ4_XS / Q2_K / Q3_K tower routes to + `kExpandBf16` and `LoadStackedExperts` refuses by name + (`glm_moe_dsa_loader.cpp:392-400`) — for a tower the plan has already + decided will execute on the CPU and whose bytes will never touch the GPU. + +4. **The fit check ignores the plan installed 220 lines above it.** + `InstallMoePlacementPlan` runs at `model_loader.cpp:2672`, + `CheckDeviceWeightFit` at `:2895`. Neither `GgufStagedWeightFootprint` nor + `CheckDeviceWeightFit` takes any representation of a resolved placement, so + the 187.3125 GiB of towers is charged to the device pool whatever the plan + says. This is #2517, and on this board it is not cosmetic: it is the line + between the placed load and a forward. + +5. **The CPU can execute every encoding this checkpoint's experts use.** + `vt::cpu::HasQuantDotKernel` (`cpu_quant_traits.cpp:165-170`) accepts 17 + block dtypes including all six here. So a tower routed against `kCPU` keeps + its blocks and is read in place out of the mapping. + +The chain therefore closes with two edits and no new kernel. + +## W2. Design + +**One question, asked of the right device.** `GgufLoadPolicy::device` is +documented as "the device the ENGINE resolved for this load", and the field's +own comment states that EVERY device-dependent decision reads it. Hybrid +placement (#2023/#2314) introduced a second, narrower answer for exactly one +class of tensor: `ActiveMoePlacementPlan().DeviceForLayer(l)` is the device a +layer's routed experts are COMPUTED on. A residency decision for a +`kStackedExpertWeight` is a question about that device, not about the engine. +This is the same defect as #1136 (the bound and the policy naming different +devices) and #2406 (a host-ISA repack applied to a weight staged elsewhere), +one seam further along. + +`GgufLoadPolicy::Route` and `PeekRoute` resolve through ONE new private helper +so the two entry points cannot answer differently — the property +`GgufExpertTowersReachSlotLane` depends on, since it peeks the same tensors the +loader routes. + +**Inert by construction, not by a flag.** The override applies only when + +``` +role == kStackedExpertWeight + && ActiveMoePlacementPlan().PlacesAnything() + && the tensor names a block index + && that block's device != the plan's engine device +``` + +`PlacesAnything()` is false on every load that configured no placement, which is +every CUDA and CPU load in the tree's gates today, so those loads route +byte-for-byte as they did. The last term is what stops a default-constructed +global (whose `engine_device_` is `kCPU`) from silently answering `kCPU` on a +CUDA load: the answer is only ever ADOPTED when the plan actively moves that +tensor somewhere else. + +**The block index comes from the name**, because that is the only key the loader +has. `MoePlacementPlan::DeviceForRoutedExpertTensor(name)` parses the +`blk..` prefix llama.cpp's GGUF export writes — the same spelling +`RoutedExpertTensorNames` and `LlmFfnExpsBlockRegex` already compose — and +answers the engine device for a name that carries none. + +**The fit check credits the same plan.** `GgufStagedWeightFootprint` gains a +`const MoePlacementPlan*` (null = today's behaviour, byte for byte) and moves a +placed routed-expert tensor into two new fields, `placed_tensor_count` / +`placed_bytes`, exactly as `StreamedExpertLane` already moves a streamed one. +The justification is the placement seam's own: `ResidentWeight` aliases host +bytes on a CPU `Dev`, so a layer whose block only ever runs there is never +staged to the accelerator (`moe_placement_seam.h`, the "ACROSS" comment). The +refusal appends a NOTE naming the placed count and bytes, so an operator can +never again read two contradictory numbers in consecutive lines. + +**Why not make the lane serve instead.** #2515 is right and this row does not +contest it: `HostMemoryIsDeviceAddressable` is `integrated && +pageable_memory_access`, `pageableMemoryAccess` is 0 on `gfx1151`, and the slot +arena is a pageable `std::vector`. Forcing the lane on would delete a +correct refusal. The device-side slot store that would fix it is +`ENG-EXPERT-STREAM-DEVICE` W2 and a different row. + +**Why not widen ROCm keep-quant.** Writing IQ `vec_dot` in +`rocm_grouped_gemm.hip` (#1940) would keep an UNPLACED tower compressed — and +W1 fact 2 says an unplaced tower cannot be executed on this board at all, +because `GlmResidentExpertSlice` refuses a non-host-addressable device before +any kernel is asked for. So #1940 would not produce a token here. It stays owed +for the discrete-ROCm arm and for performance. + +## W3. Tests + +Red-first, and each named test fails for its intended reason before the change. + +**`tests/vllm/test_gguf_keep_quant.cpp`** (new cases): + +- An IQ1_S stacked-expert tower under a policy with `device = kROCM` and an + installed plan that places its layer on `kCPU` routes `kKeepQuant`. RED + before (`kExpandBf16`). +- The SAME tensor with NO plan installed still routes `kExpandBf16` on + `kROCM` — the inertness pin, green before and after. +- A tensor of a layer the plan does NOT place still routes against the engine + device while a placed sibling does not, in one assertion pair, so the + override is proved per-layer rather than per-load. +- `Route` and `PeekRoute` agree on every case above. +- `kMatmulWeight` and `kEmbeddingTable` are UNMOVED by an installed plan: the + override is scoped to the one role whose compute the plan actually moves. + +**`tests/vllm/model_executor/test_device_placement.cpp`**: +`DeviceForRoutedExpertTensor` on `blk.7.ffn_gate_exps.weight`, on a name with +no block index, and on an out-of-range block. + +**`tests/vllm/model_executor/test_gguf_device_fit.cpp`**: + +- A footprint with a plan that places the trailing N layers reports + `placed_bytes` equal to those towers and drops them from + `lower_bound_bytes`; with a null plan it is byte-identical to today. +- `CheckDeviceWeightFit` REFUSES without the plan and PASSES with it, on one + synthetic file whose numbers straddle the budget. RED before (the parameter + does not exist). +- The #2516 case is re-pointed from `CurrentPlatform().device_type()` to the + two devices it is actually about: `kCPU` routes IQ towers `kKeepQuant`, and + `kROCM` routes them `kExpandBf16`. Platform-independent, and it PINS the + ROCm gap #1940 owns instead of failing over it. + +**Reachability (`.agents/reachability.md`).** The production entry point is +`vllm_engine_load` → `ModelLoader::FromModelDir`'s GGUF branch. The mutation +deletes the placement term at BOTH production call sites (the `Route` helper +and the `CheckDeviceWeightFit` argument), one at a time, and requires the +focused suites to go red; a by-hand construction of the policy is not the +proof and is not offered as one. + +## W4. Gates + +- `scripts/agent-preflight.sh --fail-on-skip` on the branch. +- Hand-run, with case AND assertion counts read together: + `test_gguf_keep_quant`, `test_gguf_device_fit`, `test_gguf_device_fit_reach`, + `test_device_placement`, `test_glm_moe_dsa_gguf_load`, `test_expert_stream_wiring`. +- On `strix:gpu0` inside an `rc` lease: the same suites on a `gfx1151` HIP + build, plus the product legs below. +- CUDA/CPU inertness: every suite above on a non-HIP build, and the assertion + that a load with no placement installed routes and prices byte-identically. + +**The product bar.** `vllm-cli --model --device auto` with +`VT_CPU_MOE=1` on `strix:gpu0`, against +`/workspace/ckpt/GLM-5.3-UD-IQ1_S` (six shards, first shard, never the +directory; shard 1 derived by `scripts/glm-dsa-write-indexer-types.py`). +Reported with it: the emitted text verbatim, the resident footprint, whether +the towers stayed compressed, and `GetReferenceTierHits()` / the +`[vt reference-tier]` distinct count — because #2505 means no `--device` value +names ROCm and configuration alone cannot separate a GPU run from a host +fallback. + +**No speed number, either way.** `docs/ROCM.md:60-61` disqualifies a +performance result from a run with a non-zero reference-tier count, and this +model's MLA/DSA arm is eight ops short on ROCm +(`.agents/specs/rocm-glm53-dsa.md` W1.3), every one of which lands on that +tier. + +## W5. Risks + +- **A fourth predicate.** Three predicates on this row have each looked like + the last one (#2507, #2515, #2516). W1 traces the whole chain from load to + `MoeBlock` rather than the next hop, but the MLA/DSA arm is unexercised on + this board and may refuse for its own reasons. The stop condition below says + what happens then. +- **A global read inside a routing function.** `Route` becomes dependent on + process state. Argued for rather than against: the plan IS the resolved + answer to "which device runs this tensor", it is installed before any weight + I/O by construction (`model_loader.cpp:2668-2672`), and the alternative — + threading a plan pointer through twelve `FromEnv` call sites — widens the + change without making any of them able to answer differently. The inertness + term keeps a never-installed global from ever being consulted. +- **`quant_repack` is still resolved from the ENGINE device**, so a tower + placed on the CPU is not i8mm-repacked. That is a performance gap, not a + correctness one (the CPU quant GEMM reads unrepacked blocks), and it is + listed as owed rather than fixed, because moving it means moving a flag + `FromEnv` resolves once for the whole file. + +## W6. Stop conditions + +Stop and report `NEEDS_DECISION` rather than widening scope if the placed load +reaches a forward and then refuses inside the MLA/DSA arm: porting those eight +ops is `.agents/specs/rocm-glm53-dsa.md`'s recorded campaign, not this row. + +Stop rather than forcing the streaming lane on, or lowering +`host_memory_is_device_addressable()`'s bar, if the placed route turns out to +need it. #2515 records why that deletes a correct refusal. + +## Owed + +- **ROCm i-quant `vec_dot`** (#1940), for a discrete ROCm board and for the + performance arm. Unnecessary for this row's result, for the reason in W2. +- **The ROCm MLA/DSA arm**, eight ops (`.agents/specs/rocm-glm53-dsa.md`). + Every one of them runs on the reference tier in this row's result, which is + why no speed number is quoted. +- **`--fit`'s resolver is not residency-aware**, filed as + [#2565](https://github.com/mudler/vllm.cpp/issues/2565) and CONFIRMED on + hardware: with no placement configured it places the trailing 56 of 78, the + fit check then passes (this row's own repair), and the load refuses on + `blk.3`, one of the 22 it left on the device. On this board the operator has + to say `cpu_moe`. +- **The MLA `fused_nr` availability probe**, + [#2564](https://github.com/mudler/vllm.cpp/issues/2564) — the blocker that + now stands between this checkpoint and a token on `gfx1151`. Owner + `BACKEND-ROCM`. +- **`quant_repack` for a placed tower** (W5). +- **The device budget** (#2518) and **the host-slot lane** (#2515), both + untouched and both re-measured here. + +## Now + +`ACTIVE` — implemented, gated, and measured on `strix:gpu0`; open as #2562 for a +fresh review. The row's own scope is complete and the model's remaining blocker +is named and filed (#2564), not left to be discovered. + +## Outcome + +**What was measured, on the real artifact.** Both predicates this row names are +real, and they are SEQUENTIAL rather than alternative: on the base build with +`VT_CPU_MOE=1` the fit check refuses quoting the un-reduced 216433205760 B with +all 78 layers already placed (#2517), and with that refusal suppressed the load +gets one stage further and refuses on +`blk.3.ffn_gate_exps.weight routed to an EXPAND residency` (#2516). With both +fixed, the same command LOADS: 1809 tensors, `[vt load] weights 1372.153 s`, +`gguf prefault spans=589 paged_in=11.620 GiB in 1037.930 s (11.5 MiB/s)`, the +engine auto-fits `max_model_len` from 1048576 to 8192 against 256 blocks of 32 +tokens, and the scheduler starts. **All 228 routed-expert towers stayed +compressed**, which is not an inference: `LoadStackedExperts` throws by name on +any tower that expands, so a load that completes is the proof. + +**NO TOKEN CAME OUT, and none is claimed.** The first forward throws inside the +MLA block. `fused_nr` (`mla_attention.cpp:550-551`) has a +`vt::OpRegistered(kFusedNormRope, ...)` term, ROCm registers no +`kFusedNormRope`, and `OpRegistered` is deliberately a native-only probe that +cannot see the reference tier — so the split A-projection path is taken and +refuses this checkpoint's block-quantized `kv_a_proj_with_mqa`. The comment at +`:628-631` states that path is reachable "only with VT_MLA_FUSED_NORM_ROPE=0"; +this run had it unset and is the counterexample. Filed as +[#2564](https://github.com/mudler/vllm.cpp/issues/2564). That is exactly this +spec's W6 stop condition, and the row stops there rather than widening into the +MLA arm. + +**What was rejected, with the reason measured rather than assumed.** Repairing +the host-slot lane: `pageableMemoryAccess = 0` was read off this board, so +`HostMemoryIsDeviceAddressable` is false and the lane cannot serve here at all +(#2515 confirmed). Porting ROCm i-quant `vec_dot` (#1940): a placed tower is +executed by the CPU, and an UNPLACED tower on this board cannot be executed at +all because `GlmResidentExpertSlice` refuses a non-host-addressable device before +any kernel is asked for — so those kernels would not have produced a token here. +Correcting the device budget (#2518): the credited footprint is 11.620 GiB paged +in against a 58.000 GiB managed ceiling and a 64.00 GiB reported pool, so the +6.00 GiB of optimism between them cannot decide this load in either direction. + +**One measurement about the instrument, not the code.** `vllm-cli`'s sha256 is +BYTE-IDENTICAL across the base and patched builds +(`ac37fb6d11ba3c1ae7d3d931777ac32ef6fe6bee00231ba18470330740192529`) while +`libvllm.so.0.0.3` moves — it is a thin ABI client and nothing in it changed. A +binary-identity guard that digests only the executable reports two different +builds as one, and this row measured that rather than inheriting it. diff --git a/docs/USAGE.md b/docs/USAGE.md index 2c76dfb86..b0c646552 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -655,7 +655,7 @@ repository in this project's history. | GLM-5.3-Flash FP8 source | `model-000{01..62}-of-00062.safetensors` | 328,326,771,576 bytes total (305.78 GiB) | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-26 | Owed: no byte of payload has been fetched, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | Declared source of `scripts/convert-glm5-next-gguf.py`. Only the safetensors HEADERS were read, by HTTP RANGE over all 62 shards: 76,108 tensors, `F8_E4M3` block-quantized at `weight_block_size: [128, 128]` with `weight_scale_inv` companions, plus BF16 and F32 scales | **Nothing has been converted.** The download needs explicit developer authority and a box with room for 305.78 GiB of source and ~100.35 GiB of output at once; owed as O7 on [#2011](https://github.com/mudler/vllm.cpp/issues/2011). The revision is a branch name and not a commit, which is NOT a pin: it is what was read, and W7b re-reads and records the commit when it stages the bytes | | GLM-5.3-Flash GGUF | `GLM-5.3-Flash-UD-Q2_K_XL-0000{1..4}-of-00004.gguf` | 108,720,071,427 bytes total (101.2535 GiB) across four shards; 1412 tensors | `unsloth/GLM-5.3-Flash-GGUF` @ `d425e572fb9686125831f476129e51cea34bc5b4`, path `UD-Q2_K_XL`, staged 2026-08-28 | Owed for this row: the shards are staged and were sha256-verified when they were fetched, but **W5c consumed only the four GGUF HEADERS** and states no hash of its own. W7b ([#2225](https://github.com/mudler/vllm.cpp/issues/2225)) records the per-shard sha256 alongside the load it measures | **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)). **A MATERIALIZED LOAD EXISTS** -- driven at this artifact on `dgx:gpu0` 2026-08-30, all four shards load and the engine sizes its caches in under 26 minutes wall ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)). At that change the first step threw at the `multi_kv` guard above the model's own hook; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) writes the consuming forward that guard was waiting for and it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT, and peak RSS is MEASURED** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). On `dgx:gpu0` 2026-08-30, in the SHIPPED configuration with no diagnostic env set, `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is` emits ` Paris.` at `rc=0`, and `VmHWM` peaks at 104,792,300 kB = 99.94 GiB. Two instrumented `thor:gpu0` runs the same day supply the bisect: four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427), ` one`, ` located`, ` known`, ` a` at margin 1.279, and none of 180 per-layer readings over four steps carries a NaN. The first attempt emitted token id 0 eight times, because the loader repacked this file's 346 q8_0 tensors into the i8mm interleave that the host bridge reads as plain blocks (spec `## Owed` O30). **No speed number is claimed, and the earlier ones are void** -- they were taken from an all-NaN forward. The GB10 arm is the one measured above. The GGUF arm of `load_weights` resolves all 1383 backbone tensors of this file (W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242)); `blk.45`, the multi-token-prediction block, is read, counted and DROPPED, as the transformers reference does. `ModelRegistry::Forward` dispatches to the model as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which bridges ONE decoder layer at a time out of the block-resident tower and decodes only the 8 of 288 experts a token selects — a float tower is 426.72 GiB against ~119.63 GiB usable. **A MATERIALIZED LOAD NOW EXISTS**: driven at this artifact on `dgx:gpu0` 2026-08-30, all four shards load and the engine sizes its caches in under 26 minutes wall. **NO TOKEN WAS GENERATED** — the first step throws at the `multi_kv` guard above the model's own hook ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)) — and **peak RSS and speed are still unmeasured**, because the staging run did not sample them. The vision tower (a separate `mmproj-BF16.gguf`) and the safetensors arm still refuse by name, as does a multi-request step; a non-CPU queue is admitted as of W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) for the routed-expert GEMM alone, and a device that is neither CPU nor CUDA is refused by name; **the KV-cache spec does not**, as of W5 ([#2223](https://github.com/mudler/vllm.cpp/issues/2223)), which publishes its three groups through the production factory hook | **The earlier row here said `none exists`, and that was true when it was written (2026-08-26) and is not now.** "UD-Q2_K_XL" names a TARGET AVERAGE and not a format: the census over all 1412 tensors is F32 638, Q8_0 346, Q5_K 181, Q6_K 117, IQ2_XS 82, IQ3_XXS 41, IQ4_XS 3, Q2_K 2, Q4_K 1, Q3_K 1 — **two** Q2_K tensors in a file named Q2_K. It fits `dgx:gpu0` only because IQ2_XS and IQ4_XS keep their blocks ([#2247](https://github.com/mudler/vllm.cpp/issues/2247)); both now have a CUDA keep-quant kernel too ([#2260](https://github.com/mudler/vllm.cpp/issues/2260)), so the expert GEMM no longer drains the stream to the host and the fused seam no longer throws. W9c-3a ([#2464](https://github.com/mudler/vllm.cpp/issues/2464)) then built a device arm for this artifact's routed-expert GEMM and MEASURED it end to end, where it **SEGFAULTED**: both `--device cuda` legs on `dgx:gpu0` died with rc=139 emitting no token, reproducibly (spec O46). The split is therefore OPT-IN and defaults OFF, so `--device cuda` refuses exactly as it did before. **Use `--device cpu`** -- measured on that artifact it emits ` Paris.` at rc=0, 1176 s wall of which 169 s is generation. Every OTHER primitive of this model is still a host reference on an interposed CPU queue (spec O43), so what `--device cuda` reaches is one arm of eleven and not a device arm. **A materialized load NOW exists and a token still does not** — `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches, then the first step throws at the `multi_kv` guard above the model's own hook. **Peak RSS and speed remain unmeasured** | | GLM-5.3-Flash config | `config.json` | 69,416 bytes | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-27 | sha256 `bb8f01c42cb92a52ca72e65afb4d5bd8d11aef083cd210e8de25dfb904f23e9f` | The ONLY byte of this checkpoint any change on this row has consumed. Checked in verbatim as `tests/vllm/models/fixtures/glm5_next/config.json` and used as W1's gate fixture, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | **Arms refused by name:** the SAFETENSORS one, which is what this row is, because every published safetensors artifact of this model exceeds every device this project owns. `Glm5NextForConditionalGeneration` is REGISTERED, its config RESOLVES, and the GGUF arm both loads and forwards ([#2067](https://github.com/mudler/vllm.cpp/issues/2067), [#2242](https://github.com/mudler/vllm.cpp/issues/2242), [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). The revision is a branch name and not a commit, which is NOT a pin for the WEIGHTS; for this one file the sha256 above is the pin | -| GLM-5.3 GGUF (`glm-dsa`) all six shards | `GLM-5.3-UD-IQ1_S-0000{1..6}-of-00006.gguf` | 9,428,677 B (shard 1, metadata only, 0 tensors) and 49,968,868,928 B (shard 2); the six shards total 216,715,365,893 B = 201.83 GiB across 1809 tensors | `unsloth/GLM-5.3-GGUF` @ `346b3591c7f28d1a23716f97a065ecf12ec14771`, path `UD-IQ1_S`, staged 2026-08-30, completed and verified 2026-08-31 | shard 1 `ff3adab0853dfb00bdf3889ec3f5556196f56b65783115720d57767bbd760dd9`; shard 2 `659d04cf4fc0b6026944f34c0b590a635803bff06c1775361e28490db7b168f8`; shard 3 `433302bac0e2d54da64c7c2f28509fa1b235aeccdf5b215a8a446ebaad1b5b27`; shard 4 `d0a6f19452d5b5cd498e1eb8fbe856e00aed7da1f80c27c095301eabe81e9bc1`; shard 5 `2ea1537ffab40fa8b8584a8647ec10fbaa6199dfed45e4019b822da2b319db37`; shard 6 `42a76ef04ffc5e321e1240f4e572b6fa6fc3315da5bea22fb598d7460db210fe`. **All six are complete and each was hashed TWICE** — once by the fetch script as it landed and once independently off the same share afterwards — and the two readings agree. **The DERIVED metadata shard has a hash of its own:** `scripts/glm-dsa-write-indexer-types.py` run against the staged shard 1 with `zai-org/GLM-5.3`'s own `config.json` produces a 9,428,810-byte file, 64 keys becoming 65, 21 `full` of 78, sha256 `b3e9838651a5c279533c98390ab4bc03cf1d8c176d5be0754180f07d9ed85c01`, reproduced identically by three independent runs. **That is a DERIVED artifact and must never be quoted as `unsloth/GLM-5.3-GGUF`'s shard 1** | **THIS ARTIFACT GENERATES THROUGH THE EXPERT-STREAMING LANE: `The capital of France is` -> ` Paris`.** On `dgx:gpu0` (GB10, 20 cores, 119 GB, compute capability 12.1) under an `rc` lease, 2026-08-31, a build with `-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_FLASH_ATTN=ON` and CUTLASS 4.5.0: `VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=4096 vllm-cli --model --device cuda --prompt "The capital of France is" --max-tokens 1 --temperature 0` returns `rc=0`, `prompt_tokens=5 completion_tokens=1`, and seven bytes of stdout: a space, `Paris`, a newline. Wall 1154 s for the process, `generate` 852.330 s, `VmHWM` 60,512,268 kB = 57.71 GiB against 119.631 GiB of device and 201.83 GiB of artifact. **The lane's own counters are the streaming evidence, and they include one number that must travel with them:** `[expert-stream] ON slots=4096 slot_bytes=6684672 resident=25.50 GiB`, then `steps=1 hits=0 misses=6399 evictions=0 fills=4096 bytes=13939408896 exhausted=2303 advised=0`. 4096 slices were paged out of the file into slots and 12.98 GiB moved through them with zero evictions, and the 187.312 GiB of towers were never materialized — but the step needed 6399 distinct slices, so **2303 of them (36%) were read in place out of the mapping instead of streamed**. That is a PREFILL working set exceeding any slot budget by construction (spec R2, O34), it is counted rather than silent, and no figure here may be quoted as a fully-streamed step. **No speed number is claimed:** one token, a CIFS-backed artifact, and 2303 in-place fallbacks in the measurement. `--device cpu` on the same box and artifact also emits ` Paris` (`rc=0`, `generate` 950.249 s, `VmHWM` 44.46 GiB), and **that arm does NOT stream at all** — a CPU queue builds no slot lane, so every routed-expert slice is read in place. On `thor:gpu0` (sm_110a) the CUDA arm cannot reach a token: MLA prefill on this family IS FlashAttention, the vendored FA2 covers `8.0,8.6,8.7,8.9,12.0a,12.1a`, and sm_110a is outside it. Also gated on a complete synthetic model of the same shape: `test_glm_moe_dsa_gguf_load.cpp` 5 cases / 228 assertions, `test_glm_moe_dsa_forward.cpp` 7 / 5258, `test_glm_moe_dsa_schedule.cpp` 12 / 533, and the real file's census from its headers (`test_glm_moe_dsa_gguf_census.cpp` 3 / 3831): 1809 tensors, 228 expert towers at 187.312 GiB, 1581 resident at 14.511 GiB, largest per-expert slice 6,684,672 B. What the forward still refuses BY NAME is a step in which any request RESUMES while its selection PRUNES — that needs the indexer KV side cache `KV-DSV4-MULTICACHE` owns (spec O4, #1925/#2323), so a FIRST token on a fresh prompt is reachable and a SECOND is not — and sparse prefill (spec O6) is still W6's. No speed axis has a denominator (spec O10) | **THIS FILE CANNOT BE FED AS PUBLISHED**, and that is a property of the file rather than of the port: its 64 metadata keys carry neither `glm-dsa.attention.indexer.types` nor `index_topk_freq`/`index_skip_topk_offset`, so it states its per-layer indexer schedule nowhere, and it broadcasts `indexer.*` onto all 79 blocks while the checkpoint ships them on 22. The loader refuses it by name rather than substituting llama.cpp's hardcoded table (spec D3). **The repair is one command and it rewrites the 9.4 MB metadata shard only:** `scripts/glm-dsa-write-indexer-types.py --shard --from-config --out /GLM-5.3-UD-IQ1_S-00001-of-00006.gguf`, with the five payload shards hard-linked beside the output, then `--model` that directory's shard 1. It transcribes the schedule from the model author's own `config.json` and derives nothing; the result is a DERIVED artifact with its own sha256 and is not `unsloth/GLM-5.3-GGUF`. **Build requirements this model does not degrade past:** `--device cuda` (the expert-streaming lane is not built on a CPU queue, and the towers would then be read in place out of a 201.83 GiB mmap), and a build with the vendored FlashAttention-2, which needs CUTLASS headers and an arch in `8.0,8.6,8.7,8.9,12.0a,12.1a` — MLA prefill IS FlashAttention here and has no fallback below it. **Arms refused by name:** the SAFETENSORS one, permanently (spec D1 — 703.74 GiB across 141 shards, no streaming loader, no MoE block-fp8 rung), and `UD-IQ1_M`, which refuses at file open because `IQ1_M` (ggml id 29) has no reader traits (spec O3) | +| GLM-5.3 GGUF (`glm-dsa`) all six shards | `GLM-5.3-UD-IQ1_S-0000{1..6}-of-00006.gguf` | 9,428,677 B (shard 1, metadata only, 0 tensors) and 49,968,868,928 B (shard 2); the six shards total 216,715,365,893 B = 201.83 GiB across 1809 tensors | `unsloth/GLM-5.3-GGUF` @ `346b3591c7f28d1a23716f97a065ecf12ec14771`, path `UD-IQ1_S`, staged 2026-08-30, completed and verified 2026-08-31 | shard 1 `ff3adab0853dfb00bdf3889ec3f5556196f56b65783115720d57767bbd760dd9`; shard 2 `659d04cf4fc0b6026944f34c0b590a635803bff06c1775361e28490db7b168f8`; shard 3 `433302bac0e2d54da64c7c2f28509fa1b235aeccdf5b215a8a446ebaad1b5b27`; shard 4 `d0a6f19452d5b5cd498e1eb8fbe856e00aed7da1f80c27c095301eabe81e9bc1`; shard 5 `2ea1537ffab40fa8b8584a8647ec10fbaa6199dfed45e4019b822da2b319db37`; shard 6 `42a76ef04ffc5e321e1240f4e572b6fa6fc3315da5bea22fb598d7460db210fe`. **All six are complete and each was hashed TWICE** — once by the fetch script as it landed and once independently off the same share afterwards — and the two readings agree. **The DERIVED metadata shard has a hash of its own:** `scripts/glm-dsa-write-indexer-types.py` run against the staged shard 1 with `zai-org/GLM-5.3`'s own `config.json` produces a 9,428,810-byte file, 64 keys becoming 65, 21 `full` of 78, sha256 `b3e9838651a5c279533c98390ab4bc03cf1d8c176d5be0754180f07d9ed85c01`, reproduced identically by three independent runs. **That is a DERIVED artifact and must never be quoted as `unsloth/GLM-5.3-GGUF`'s shard 1** | **THIS ARTIFACT GENERATES THROUGH THE EXPERT-STREAMING LANE: `The capital of France is` -> ` Paris`.** On `dgx:gpu0` (GB10, 20 cores, 119 GB, compute capability 12.1) under an `rc` lease, 2026-08-31, a build with `-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_FLASH_ATTN=ON` and CUTLASS 4.5.0: `VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=4096 vllm-cli --model --device cuda --prompt "The capital of France is" --max-tokens 1 --temperature 0` returns `rc=0`, `prompt_tokens=5 completion_tokens=1`, and seven bytes of stdout: a space, `Paris`, a newline. Wall 1154 s for the process, `generate` 852.330 s, `VmHWM` 60,512,268 kB = 57.71 GiB against 119.631 GiB of device and 201.83 GiB of artifact. **The lane's own counters are the streaming evidence, and they include one number that must travel with them:** `[expert-stream] ON slots=4096 slot_bytes=6684672 resident=25.50 GiB`, then `steps=1 hits=0 misses=6399 evictions=0 fills=4096 bytes=13939408896 exhausted=2303 advised=0`. 4096 slices were paged out of the file into slots and 12.98 GiB moved through them with zero evictions, and the 187.312 GiB of towers were never materialized — but the step needed 6399 distinct slices, so **2303 of them (36%) were read in place out of the mapping instead of streamed**. That is a PREFILL working set exceeding any slot budget by construction (spec R2, O34), it is counted rather than silent, and no figure here may be quoted as a fully-streamed step. **No speed number is claimed:** one token, a CIFS-backed artifact, and 2303 in-place fallbacks in the measurement. `--device cpu` on the same box and artifact also emits ` Paris` (`rc=0`, `generate` 950.249 s, `VmHWM` 44.46 GiB), and **that arm does NOT stream at all** — a CPU queue builds no slot lane, so every routed-expert slice is read in place. On `thor:gpu0` (sm_110a) the CUDA arm cannot reach a token: MLA prefill on this family IS FlashAttention, the vendored FA2 covers `8.0,8.6,8.7,8.9,12.0a,12.1a`, and sm_110a is outside it. Also gated on a complete synthetic model of the same shape: `test_glm_moe_dsa_gguf_load.cpp` 5 cases / 228 assertions, `test_glm_moe_dsa_forward.cpp` 7 / 5258, `test_glm_moe_dsa_schedule.cpp` 12 / 533, and the real file's census from its headers (`test_glm_moe_dsa_gguf_census.cpp` 3 / 3831): 1809 tensors, 228 expert towers at 187.312 GiB, 1581 resident at 14.511 GiB, largest per-expert slice 6,684,672 B. What the forward still refuses BY NAME is a step in which any request RESUMES while its selection PRUNES — that needs the indexer KV side cache `KV-DSV4-MULTICACHE` owns (spec O4, #1925/#2323), so a FIRST token on a fresh prompt is reachable and a SECOND is not — and sparse prefill (spec O6) is still W6's. No speed axis has a denominator (spec O10) | **THIS FILE CANNOT BE FED AS PUBLISHED**, and that is a property of the file rather than of the port: its 64 metadata keys carry neither `glm-dsa.attention.indexer.types` nor `index_topk_freq`/`index_skip_topk_offset`, so it states its per-layer indexer schedule nowhere, and it broadcasts `indexer.*` onto all 79 blocks while the checkpoint ships them on 22. The loader refuses it by name rather than substituting llama.cpp's hardcoded table (spec D3). **The repair is one command and it rewrites the 9.4 MB metadata shard only:** `scripts/glm-dsa-write-indexer-types.py --shard --from-config --out /GLM-5.3-UD-IQ1_S-00001-of-00006.gguf`, with the five payload shards hard-linked beside the output, then `--model` that directory's shard 1. It transcribes the schedule from the model author's own `config.json` and derives nothing; the result is a DERIVED artifact with its own sha256 and is not `unsloth/GLM-5.3-GGUF`. **Build requirements this model does not degrade past:** `--device cuda` (the expert-streaming lane is not built on a CPU queue, and the towers would then be read in place out of a 201.83 GiB mmap), and a build with the vendored FlashAttention-2, which needs CUTLASS headers and an arch in `8.0,8.6,8.7,8.9,12.0a,12.1a` — MLA prefill IS FlashAttention here and has no fallback below it. **On ROCm `gfx1151` (`strix:gpu0`, Radeon 8060S) this artifact LOADS and DOES NOT GENERATE, as of [#2562](https://github.com/mudler/vllm.cpp/pull/2562).** The route is `VT_CPU_MOE=1 vllm-cli --model --device auto` -- `auto` because no `--device` value names ROCm ([#2505](https://github.com/mudler/vllm.cpp/issues/2505)), and `cpu_moe` because `--fit`'s default placement leaves 22 layers on a device whose keep-quant set cannot hold their IQ1_S towers ([#2565](https://github.com/mudler/vllm.cpp/issues/2565)). All 1809 tensors resolve, all 228 routed-expert towers stay compressed, 11.620 GiB is paged in (at 11.5 MiB/s off the CIFS share the artifact lives on, which is a property of the share), and the engine auto-fits `max_model_len` to 8192 against 256 blocks of 32 tokens. **NO TOKEN COMES OUT:** the first forward throws in the MLA block, because `vt::OpRegistered(kFusedNormRope, ...)` is false on ROCm and cannot see the reference tier, so the split A-projection path is taken and refuses this checkpoint's block-quantized `kv_a_proj_with_mqa` ([#2564](https://github.com/mudler/vllm.cpp/issues/2564)). The streamed-expert lane is NOT what serves the towers here and cannot be: `pageableMemoryAccess` is 0 on this board, so `host_memory_is_device_addressable()` is false ([#2515](https://github.com/mudler/vllm.cpp/issues/2515)). No speed number is admissible from this board for this model. **Arms refused by name:** the SAFETENSORS one, permanently (spec D1 — 703.74 GiB across 141 shards, no streaming loader, no MoE block-fp8 rung), and `UD-IQ1_M`, which refuses at file open because `IQ1_M` (ggml id 29) has no reader traits (spec O3) | | GLM-5.3 config | `config.json` | 29,464 bytes | `zai-org/GLM-5.3` @ `935644c05e76fc198714f4cca449fd8b970ff6d7` | Committed verbatim in-tree as `tests/vllm/models/glm_moe_dsa_config_glm53.inc`, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | It is the ONLY authoritative source of the 78-entry `indexer_types` list — 21 `full`, at layers {0,1,2} and every fourth from 6 to 74 — which three independent derivations agree on bit for bit (the list itself, vLLM's rule at `deepseek_v2.py:1097-1101`, and llama.cpp's `GLM_5_2_DEFAULT_INDEXER_TYPES`) | The GGUF above does not carry this list, which is why it cannot be fed as published | | Qwen3.5-0.8B (Tenstorrent P150 arm) | `model.safetensors-00001-of-00001.safetensors` | 1,746,942,600 bytes | `Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17`, authorized 2026-08-23 | `04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696` (non-quantized; hashed anyway from the local bytes the gates and the eager profile consumed) | bf16 on the Tenstorrent P150: the sacred greedy pair, both ambient legs, and the #1715/#2107 profile legs all ran from this snapshot | **Arms refused by name:** GGUF k-quant arms on TT — no TT kernels exist for them, refused at load; Qwen3.8-27B on TT — no arm fits the P150 (bf16 53.8 GB), refused at load | | dots3-note bf16 language tower | `model-000{01..131}-of-00131.safetensors` | 561,371,869,568 bytes total (522.82 GiB), of which the MoE is 545,823,175,680 | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed: **no tensor byte has been fetched**, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | The bf16 text tower this port loads: 46 backbone layers, both MLA geometries, and since W5 the 45 MoE layers — the ungrouped noaux_tc router at 256/8 plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536. Everything except `mlp.gate.e_score_correction_bias` is BF16; that one is F32, on both sides | **Nothing has ever loaded these bytes.** The tower alone is 522.82 GiB against a 122 GiB ceiling on the largest host this project reaches (spec §6.2), so the arm is representable and unfeedable, and the e2e gate is an OPEN GAP by construction. GGUF k-quants are refused by name (W9). The 19-tensor nextn tail is a NAMED W10 deferral rather than a refusal since #2176 | diff --git a/include/vllm/model_executor/device_placement.h b/include/vllm/model_executor/device_placement.h index 4f2e5d2d3..ea09ffe87 100644 --- a/include/vllm/model_executor/device_placement.h +++ b/include/vllm/model_executor/device_placement.h @@ -171,6 +171,19 @@ class MoePlacementPlan { // gets the inert answer rather than an exception on the decode path. vt::DeviceType DeviceForLayer(int64_t l) const; + // The device THIS ROUTED-EXPERT TENSOR's block runs on, keyed by the only + // thing a weight loader has: the name. + // + // WHY THE LOADER NEEDS THIS. `RunMoePlaced` hands the placed device to the + // forward, but the RESIDENCY of a routed-expert tower is decided far earlier, + // by `GgufLoadPolicy::Route`, and that decision is a question about the device + // that will EXECUTE the tower — whether it has a `vec_dot` for the encoding. + // Asking the engine device about a tower the plan has already sent elsewhere + // is how a checkpoint whose experts every placement device can execute is + // refused at load. A name with no `blk..` prefix, and a layer this plan was + // not resolved against, both answer the engine device. + vt::DeviceType DeviceForRoutedExpertTensor(const std::string& name) const; + // True when at least one layer runs its experts somewhere other than the // engine device. The forward reads this to decide whether any of the placement // machinery runs at all, so an unplaced model pays nothing. @@ -212,6 +225,12 @@ class MoePlacementPlan { // of them. std::vector RoutedExpertTensorNames(int64_t layer); +// The decoder-block index a llama.cpp GGUF tensor name carries (`blk..…`), +// or -1 when it carries none. The inverse of `RoutedExpertTensorNames`' own +// prefix, exposed so a caller resolves the layer with the same rule the plan was +// built with instead of a private copy of it. +int64_t GgufBlockIndexFromTensorName(const std::string& name); + // ── The placement device's queue, and the active plan ──────────────────────── // The queue a placed group runs on. One per device type, created on FIRST USE and diff --git a/include/vllm/model_executor/model_loader/gguf_device_fit.h b/include/vllm/model_executor/model_loader/gguf_device_fit.h index f791c49bf..706f581f0 100644 --- a/include/vllm/model_executor/model_loader/gguf_device_fit.h +++ b/include/vllm/model_executor/model_loader/gguf_device_fit.h @@ -35,6 +35,7 @@ #include #include "vllm/model_executor/model_loader/gguf_keep_quant.h" +#include "vllm/model_executor/device_placement.h" #include "vllm/model_executor/model_loader/gguf_reader.h" namespace vllm { @@ -131,6 +132,15 @@ struct GgufStagedFootprint { // The arena term actually added (0 when the lane is off, and 0 when it is on // but nothing matched). size_t arena_bytes = 0; + // #2517. How many routed-expert tensors the installed hybrid PLACEMENT took + // out of the sum, and how many bytes they would otherwise have contributed. + // Both 0 with no plan, and 0 with a plan that places nothing. + // + // Reported rather than merely subtracted, for the same reason the streamed + // pair is: a figure that shrank by 137 GiB without saying which tensors left + // it is a number an operator cannot check. + size_t placed_tensor_count = 0; + size_t placed_bytes = 0; }; // The per-expert SLICE size of a stacked expert tower, which is the slot size @@ -254,10 +264,28 @@ inline bool GgufPolicyForcesFullExpand(const GgufLoadPolicy& policy) { // available on that device) where the min-based bound under-counted by // roughly 4x and the load reached an uncaught `hipMalloc: out of memory` // instead of this refusal. +// +// `placement` (#2517) is the resolved hybrid-placement plan, or null for no +// plan. A routed-expert tensor whose layer the plan places away from the plan's +// own engine device is moved into `placed_*` and leaves the bound, because a +// layer whose MoE block only ever runs on the placement device is never staged +// to the accelerator -- `RunMoePlaced` hands that device to the block and +// `ResidentWeight` aliases host bytes on a CPU `Dev` rather than uploading them +// (`moe_placement_seam.h`). NULL, or a plan that places nothing, reproduces the +// pre-#2517 answer byte for byte, which is every load in this tree that +// configured no placement. +// +// WHY THIS IS NOT OPTIONAL BOOKKEEPING. `InstallMoePlacementPlan` runs 220 lines +// ABOVE the one `CheckDeviceWeightFit` call site (`model_loader.cpp:2672` and +// `:2895`), and before this parameter existed the refusal quoted the UN-reduced +// footprint on the line after the plan announced it had reduced it. On +// `strix:gpu0` that contradiction was the whole distance between a placed +// GLM-5.3 load and a forward. GgufStagedFootprint GgufStagedWeightFootprint( const GgufFile& gguf, size_t model_dtype_bytes = 2, const StreamedExpertLane& lane = {}, - bool policy_forces_full_expand = false); + bool policy_forces_full_expand = false, + const MoePlacementPlan* placement = nullptr); // The budget to compare a footprint against, in bytes, or 0 for UNKNOWN. // @@ -383,6 +411,7 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, size_t budget_bytes, size_t model_dtype_bytes = 2, const StreamedExpertLane& lane = {}, - bool policy_forces_full_expand = false); + bool policy_forces_full_expand = false, + const MoePlacementPlan* placement = nullptr); } // namespace vllm diff --git a/include/vllm/model_executor/model_loader/gguf_keep_quant.h b/include/vllm/model_executor/model_loader/gguf_keep_quant.h index 831cdb4ed..96ce9a08a 100644 --- a/include/vllm/model_executor/model_loader/gguf_keep_quant.h +++ b/include/vllm/model_executor/model_loader/gguf_keep_quant.h @@ -381,6 +381,24 @@ struct GgufLoadPolicy { // Route one tensor and notify `audit`. This is the ONLY entry point the // loader uses, so every routed tensor is observable. GgufResidency Route(const GgufTensorInfo& tensor, GgufTensorRole role) const; + + // The device that will EXECUTE this tensor, which is what a residency decision + // is about and is not always `device`. + // + // It differs for exactly one role. A routed-expert tower whose layer the + // installed `MoePlacementPlan` places away from the engine is computed on the + // PLACEMENT device (`RunMoePlaced` hands that device to the MoE block), so + // asking the engine whether its `vec_dot` covers the encoding refuses a + // checkpoint the placement device can execute perfectly well. Every other + // role, and every load with no placement installed, answers `device` + // unchanged -- see the implementation for the four inertness terms and why the + // last one is required rather than defensive. + // + // Public because `PeekRoute` must resolve the device THE SAME WAY `Route` + // does; a second spelling is how a bound and a forward come to disagree about + // one file. + vt::DeviceType ComputeDeviceFor(const std::string& name, + GgufTensorRole role) const; }; // A policy copy with the KEEP-QUANT residency disabled. diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 06fce92f0..ffbf2488e 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -2892,12 +2892,29 @@ std::unique_ptr LoadedEngine::FromModelDir( // defaults) should run. Using the narrower predicate here is what makes // this refusal reachable on ROCm without moving any of the other one's // consumers; the row's spec records why that flag stays untouched. + // #2517: THE PLAN INSTALLED 220 LINES ABOVE, credited here. + // + // `InstallMoePlacementPlan` runs at the top of this branch, before any + // weight I/O, and announces what it placed. Until this argument existed + // the refusal on the very next line quoted the UN-reduced footprint: one + // load printed "56 layers run their routed experts on cpu ... to bring a + // 216433205760 B footprint under a 68719476736 B budget" and then refused + // needing 216433205760 B. Two lines of one load contradicting each other, + // and on `strix:gpu0` the contradiction was the whole distance between a + // placed GLM-5.3 load and a forward. + // + // The GLOBAL rather than a local, because `SetActiveMoePlacementPlan` is + // where the plan lives and the forward reads the same object; a copy taken + // here could be the one the seam never sees. It is installed + // unconditionally, including when nothing is placed, so this is never a + // stale plan from a previous load in the same process. const DeviceWeightFit fit = CheckDeviceWeightFit( gguf, vt::DeviceTypeName(target.device_type()), target.allocates_bounded_device_memory(), DeviceWeightBudgetBytes( target.residency_policy().device_memory_total_bytes), - /*model_dtype_bytes=*/2, lane, policy_forces_full_expand); + /*model_dtype_bytes=*/2, lane, policy_forces_full_expand, + &vllm::ActiveMoePlacementPlan()); if (fit.refuse) throw std::runtime_error(fit.message); } // QUANT-QWEN38-27B-GGUF-ARM (#821): refuse a qwen3_5-family GGUF carrying diff --git a/src/vllm/model_executor/device_placement.cpp b/src/vllm/model_executor/device_placement.cpp index 9aba4cd87..5db8c0318 100644 --- a/src/vllm/model_executor/device_placement.cpp +++ b/src/vllm/model_executor/device_placement.cpp @@ -142,6 +142,46 @@ vt::DeviceType MoePlacementPlan::DeviceForLayer(int64_t l) const { return per_layer_[static_cast(l)]; } +int64_t GgufBlockIndexFromTensorName(const std::string& name) { + // `blk..` is llama.cpp's GGUF spelling for a decoder block, and it is the + // ONLY key a loader has: `GgufTensorInfo` carries a name, never a layer. + // `RoutedExpertTensorNames` composes the same prefix and + // `LlmFfnExpsBlockRegex` matches it, so this is the inverse of two functions + // that already exist rather than a new naming convention. + // + // ANCHORED AT THE START, deliberately. `regex_search`'s unanchored semantics + // are right for an operator's `-ot` pattern and wrong here: a name that merely + // CONTAINS `blk.3.` is not a block-3 tensor, and answering as though it were + // would move a weight no plan claimed. + static constexpr char kPrefix[] = "blk."; + const size_t plen = sizeof(kPrefix) - 1; + if (name.size() <= plen || name.compare(0, plen, kPrefix) != 0) return -1; + size_t i = plen; + int64_t idx = 0; + bool any = false; + while (i < name.size() && name[i] >= '0' && name[i] <= '9') { + // A count that cannot be a layer index is not a layer index. Refusing + // instead of wrapping keeps a malformed name out of `per_layer_`'s range + // rather than aliasing it onto a real layer. + if (idx > (INT64_MAX - 9) / 10) return -1; + idx = idx * 10 + (name[i] - '0'); + any = true; + ++i; + } + if (!any || i >= name.size() || name[i] != '.') return -1; + return idx; +} + +vt::DeviceType MoePlacementPlan::DeviceForRoutedExpertTensor( + const std::string& name) const { + const int64_t layer = GgufBlockIndexFromTensorName(name); + // A name carrying no block index belongs to no layer, so no placement claims + // it. The engine's own device is the inert answer, which is `DeviceForLayer`'s + // own out-of-range behaviour rather than a second convention. + if (layer < 0) return engine_device_; + return DeviceForLayer(layer); +} + std::string MoePlacementPlan::Describe() const { if (placed_ == 0) return ""; std::string out = std::to_string(placed_); diff --git a/src/vllm/model_executor/model_loader/gguf_device_fit.cpp b/src/vllm/model_executor/model_loader/gguf_device_fit.cpp index a15736897..bb6c3ba5d 100644 --- a/src/vllm/model_executor/model_loader/gguf_device_fit.cpp +++ b/src/vllm/model_executor/model_loader/gguf_device_fit.cpp @@ -131,12 +131,47 @@ bool GgufExpertTowersReachSlotLane(const GgufFile& gguf, return matched; } +namespace { + +// #2517. Is this tensor a ROUTED-EXPERT tower whose layer the installed plan +// runs somewhere other than the engine device? +// +// The name test is `_exps.weight`, which is the class the plan's own +// `RoutedExpertTensorNames` composes and the class `RunMoePlaced` moves. The +// layer term goes through `DeviceForRoutedExpertTensor`, so the bound and the +// forward resolve the placement with ONE rule rather than two. +// +// The engine-device comparison is against the PLAN's engine device, not the +// caller's: a plan resolved for a different device than the one being priced is +// a caller bug, and answering "placed" for a tensor that will in fact stage here +// would DELETE a correct refusal -- the unsafe direction #1378 names. +bool PlacedAwayFromEngine(const MoePlacementPlan* plan, + const std::string& name) { + if (plan == nullptr || !plan->PlacesAnything()) return false; + if (!NameHasSuffix(name, "_exps.weight")) return false; + return plan->DeviceForRoutedExpertTensor(name) != plan->engine_device(); +} + +} // namespace + GgufStagedFootprint GgufStagedWeightFootprint(const GgufFile& gguf, size_t model_dtype_bytes, const StreamedExpertLane& lane, - bool policy_forces_full_expand) { + bool policy_forces_full_expand, + const MoePlacementPlan* placement) { GgufStagedFootprint out; for (const GgufTensorInfo& t : gguf.Tensors()) { + // BEFORE the lane test, because a placed tower does not reach the lane + // either: `RunMoePlaced` hands the block a CPU `Dev`, and the expert-stream + // seam's own admission test (`expert_stream_seam.cpp:431`) is about the + // ENGINE platform. Counting one tensor in both classes would report the same + // bytes twice in a message whose whole job is to account for them once. + if (PlacedAwayFromEngine(placement, t.name)) { + ++out.placed_tensor_count; + out.placed_bytes += + StagedBytes(t, model_dtype_bytes, policy_forces_full_expand); + continue; + } // W0d: a tensor the slot lane serves is never staged, so it contributes // nothing to the bound and cannot be the largest single allocation either. // It is still COUNTED, in its own two fields, so the caller can say what was @@ -216,7 +251,8 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, size_t budget_bytes, size_t model_dtype_bytes, const StreamedExpertLane& lane, - bool policy_forces_full_expand) { + bool policy_forces_full_expand, + const MoePlacementPlan* placement) { DeviceWeightFit fit; fit.budget_bytes = budget_bytes; // A platform that does not stage weights reads them where they already are, so @@ -229,7 +265,7 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, if (budget_bytes == 0) return fit; const GgufStagedFootprint fp = GgufStagedWeightFootprint( - gguf, model_dtype_bytes, lane, policy_forces_full_expand); + gguf, model_dtype_bytes, lane, policy_forces_full_expand, placement); fit.needed_bytes = fp.lower_bound_bytes; if (fp.lower_bound_bytes <= budget_bytes) return fit; @@ -282,6 +318,21 @@ DeviceWeightFit CheckDeviceWeightFit(const GgufFile& gguf, // correction is appended rather than left to be read as a stale claim. The // lane-OFF message is untouched, byte for byte, because every CPU and discrete // user reads that one and it is still true for them. + // #2517. The plan announced a reduction on the line above this refusal and + // the refusal used to quote the un-reduced figure, which is two contradictory + // numbers in consecutive lines of one load. Whatever the verdict, say what the + // placement took out of it. + if (fp.placed_tensor_count > 0) { + fit.message += + " NOTE: an installed hybrid MoE placement runs " + + std::to_string(fp.placed_tensor_count) + + " routed-expert tensors (" + std::to_string(fp.placed_bytes) + + " bytes, " + Gib(fp.placed_bytes) + ") on another device, so they are " + "ALREADY EXCLUDED from the figure above rather than being a reduction " + "still available. What does not fit is the remainder. Placing more " + "layers (\"cpu_moe\", \"n_cpu_moe\", VT_CPU_MOE=1) is the knob that " + "shrinks it further."; + } if (fp.streamed_tensor_count > 0) { fit.message += " NOTE: the expert-stream lane IS active for this load, so the " diff --git a/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp b/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp index d2c5e7146..65cc5db45 100644 --- a/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp +++ b/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp @@ -7,6 +7,7 @@ #include #include "vllm/config/weight_residency.h" +#include "vllm/model_executor/device_placement.h" #include "vt/ops.h" #include "vt/quant.h" @@ -398,11 +399,52 @@ GgufLoadPolicy GgufLoadPolicy::FromEnv(vt::DeviceType dev) { return p; } +// THE DEVICE A RESIDENCY DECISION IS ACTUALLY ABOUT (#2516). +// +// `GgufLoadPolicy::device` is the ENGINE's resolved device, and its own field +// comment says every device-dependent decision in the struct reads it. Hybrid +// placement (#2023, #2314) introduced a SECOND, narrower answer for exactly one +// class of tensor: `ActiveMoePlacementPlan().DeviceForLayer(l)` is the device a +// layer's routed experts are COMPUTED on, and `RunMoePlaced` hands that device +// to `MoeBlock`. A keep-quant decision asks "does the device that will execute +// this GEMM have a `vec_dot` for this encoding", so for a PLACED routed-expert +// tower it is a question about the placement device, not about the engine. +// +// Measured, on GLM-5.3 `UD-IQ1_S` on `strix:gpu0`: `DeviceKeepQuantSupported` +// serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, so all 228 IQ1_S/IQ3_XXS/IQ2_XXS/ +// IQ4_XS/Q2_K/Q3_K towers routed `kExpandBf16` and `LoadStackedExperts` refused +// the load by name -- for towers whose bytes never reach the GPU at all (that +// model reads a tower only through `GlmExpertSlice`, never through +// `ResidentWeight`) and which the installed plan had already sent to the CPU, +// whose `vec_dot` table covers every one of those six encodings. +// +// THIS IS #1136 AND #2406 ONE SEAM FURTHER ALONG. Both were the same shape: a +// residency decision resolved against a device other than the one that would +// read the bytes. The repair is the same shape too -- ask the device that will +// run it. +// +// INERT BY CONSTRUCTION, in four terms, and the last one is the load-bearing +// one. The plan is a process-global that a load with no placement never writes, +// and a DEFAULT-CONSTRUCTED plan's `engine_device()` is `kCPU`. Returning its +// answer unconditionally would therefore route a CUDA load's experts as though +// they were on the host. So the placed answer is ADOPTED only when the plan +// actively moves THIS tensor somewhere other than the device the plan itself was +// resolved for; every other path returns the policy's own device unchanged. +vt::DeviceType GgufLoadPolicy::ComputeDeviceFor(const std::string& name, + GgufTensorRole role) const { + if (role != GgufTensorRole::kStackedExpertWeight) return device; + const MoePlacementPlan& plan = ActiveMoePlacementPlan(); + if (!plan.PlacesAnything()) return device; + const vt::DeviceType placed = plan.DeviceForRoutedExpertTensor(name); + if (placed == plan.engine_device()) return device; + return placed; +} + GgufResidency GgufLoadPolicy::Route(const GgufTensorInfo& tensor, GgufTensorRole role) const { - const GgufResidency r = - RouteGgufTensor(keep_quant, keep_f16, nvfp4_fp4, cpu_ref, role, - tensor.ggml_type, tensor.shape, device); + const GgufResidency r = RouteGgufTensor( + keep_quant, keep_f16, nvfp4_fp4, cpu_ref, role, tensor.ggml_type, + tensor.shape, ComputeDeviceFor(tensor.name, role)); if (audit) audit(tensor.name, role, r); return r; } @@ -415,9 +457,15 @@ GgufLoadPolicy NoKeepQuant(const GgufLoadPolicy& policy) { GgufResidency PeekRoute(const GgufLoadPolicy& policy, const GgufTensorInfo& tensor, GgufTensorRole role) { + // THROUGH THE SAME RESOLVER `Route` USES, not a copy of the expression. This + // function exists so a caller can ask what the loader WILL do without + // notifying the audit, and `GgufExpertTowersReachSlotLane` is one such caller + // -- it peeks the very tensors the loader then routes. Two spellings of the + // device term is exactly how the bound and the forward come to disagree about + // one file, which is the defect #1378 records. return RouteGgufTensor(policy.keep_quant, policy.keep_f16, policy.nvfp4_fp4, policy.cpu_ref, role, tensor.ggml_type, tensor.shape, - policy.device); + policy.ComputeDeviceFor(tensor.name, role)); } } // namespace vllm diff --git a/tests/vllm/model_executor/test_device_placement.cpp b/tests/vllm/model_executor/test_device_placement.cpp index adacef1b5..1adc23de6 100644 --- a/tests/vllm/model_executor/test_device_placement.cpp +++ b/tests/vllm/model_executor/test_device_placement.cpp @@ -439,3 +439,55 @@ TEST_CASE("placement seam: an UNPLACEABLE arm REFUSES a real placement, and stay vllm::ResetActiveMoePlacementPlanForTesting(); vt::DestroyQueue(q); } + +// ── BACKEND-ROCM-IQ-EXPERT-RESIDENCY (#2516) ──────────────────────────────── +// The name -> layer inverse. It exists because a weight LOADER has a tensor +// name and no layer index, and the residency of a routed-expert tower is +// decided there. +TEST_CASE("GgufBlockIndexFromTensorName reads llama.cpp's blk.. prefix") { + CHECK(vllm::GgufBlockIndexFromTensorName("blk.0.ffn_gate_exps.weight") == 0); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.7.ffn_up_exps.weight") == 7); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.77.ffn_down_exps.weight") == 77); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.3.attn_q.weight") == 3); + + // ANCHORED. A name that merely CONTAINS the prefix is not that block's + // tensor, and answering as though it were would move a weight no plan + // claimed. This is the one place this rule differs from an operator's `-ot` + // pattern, which is deliberately `regex_search`. + CHECK(vllm::GgufBlockIndexFromTensorName("x.blk.3.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("token_embd.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("output_norm.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.3") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("blk.x.weight") == -1); + CHECK(vllm::GgufBlockIndexFromTensorName("") == -1); + // An index no `per_layer_` can hold is refused rather than wrapped onto a + // real layer. + CHECK(vllm::GgufBlockIndexFromTensorName( + "blk.99999999999999999999999.ffn_gate_exps.weight") == -1); +} + +TEST_CASE("MoePlacementPlan::DeviceForRoutedExpertTensor answers by name") { + std::vector ov; + for (int64_t l = 2; l < 4; ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + const vllm::MoePlacementPlan plan = vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, vt::DeviceType::kCUDA), + /*num_hidden_layers=*/4); + REQUIRE(plan.PlacesAnything()); + REQUIRE(plan.placed_layer_count() == 2); + + CHECK(plan.DeviceForRoutedExpertTensor("blk.2.ffn_gate_exps.weight") == + vt::DeviceType::kCPU); + CHECK(plan.DeviceForRoutedExpertTensor("blk.3.ffn_down_exps.weight") == + vt::DeviceType::kCPU); + CHECK(plan.DeviceForRoutedExpertTensor("blk.1.ffn_gate_exps.weight") == + vt::DeviceType::kCUDA); + // A name with no block index, and a block beyond what the plan was resolved + // against, both take the inert answer rather than throwing on a load path. + CHECK(plan.DeviceForRoutedExpertTensor("token_embd.weight") == + vt::DeviceType::kCUDA); + CHECK(plan.DeviceForRoutedExpertTensor("blk.9.ffn_gate_exps.weight") == + vt::DeviceType::kCUDA); +} diff --git a/tests/vllm/model_executor/test_gguf_device_fit.cpp b/tests/vllm/model_executor/test_gguf_device_fit.cpp index 2757ee8cd..2d3d89a9c 100644 --- a/tests/vllm/model_executor/test_gguf_device_fit.cpp +++ b/tests/vllm/model_executor/test_gguf_device_fit.cpp @@ -19,6 +19,7 @@ #include "support/test_env.h" #include "vllm/gguf_builder.h" +#include "vllm/model_executor/device_placement.h" #include "vllm/model_executor/model_loader/gguf_device_fit.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" @@ -700,12 +701,24 @@ TEST_CASE( // And the residency itself, stated directly in the same role, so a reader // does not have to infer it from the predicate above. + // + // #2516: NAMED DEVICES, not `CurrentPlatform()`. This case is about the CPU + // and CUDA `vec_dot` kernels #2247 added, and asking the running platform made + // it RED on every ROCm build for a reason the case did not state -- + // `DeviceKeepQuantSupported` serves exactly {Q8_0, Q4_K, Q5_K, Q6_K} there + // (#1940 owns the gap). Both answers are pinned instead, because the ROCm one + // is a real property of this tree and a case that merely skipped it would stop + // noticing when #1940 lands. for (const vllm::GgufTensorInfo& t : gguf.Tensors()) { CAPTURE(t.name); CHECK(vllm::RouteGgufTensor(true, false, false, false, vllm::GgufTensorRole::kStackedExpertWeight, - t.ggml_type, - t.shape, vllm::platforms::CurrentPlatform().device_type()) == vllm::GgufResidency::kKeepQuant); + t.ggml_type, t.shape, vt::DeviceType::kCPU) == + vllm::GgufResidency::kKeepQuant); + CHECK(vllm::RouteGgufTensor(true, false, false, false, + vllm::GgufTensorRole::kStackedExpertWeight, + t.ggml_type, t.shape, vt::DeviceType::kROCM) == + vllm::GgufResidency::kExpandBf16); } } @@ -775,3 +788,175 @@ TEST_CASE( CHECK(on.streamed_bytes == kF16TowerStaged); CHECK(on.lower_bound_bytes == 256); } + +// ── #2517: the fit check credits the plan installed 220 lines above it ─────── +// +// Measured on `strix:gpu0`, three consecutive lines of ONE load: +// "device placement INSTALLED: 56 layers run their routed experts on cpu" +// "--fit placed 56 layer(s) (147798884352 B) to bring a 216433205760 B +// footprint under a 68719476736 B budget" +// "cannot serve this GGUF: staging its weights needs at least 216433205760 +// bytes" +// The refusal quoted the UN-reduced figure. Nothing in `CheckDeviceWeightFit` +// carried any representation of a resolved placement, so there was no path by +// which the placed bytes could be subtracted. +namespace { + +// Three layers, one routed-expert tower each, plus one non-expert tensor. +// tower: Q8_0 [E=4, N=2, K=32] = 256 elems = 8 blocks * 34 = 272 B on disk, +// 512 B expanded to bf16, so `min` takes 272. +// embd: F32 8 elems = 32 B on disk, 16 B expanded, so `min` takes 16. +// Whole-file bound with no plan: 3*272 + 16 = 832. +constexpr size_t kPlaceTowerStaged = 272; +constexpr size_t kPlaceEmbdStaged = 16; +constexpr size_t kPlaceLayers = 3; +constexpr size_t kPlaceWholeBound = + kPlaceLayers * kPlaceTowerStaged + kPlaceEmbdStaged; + +std::string BuildGgufWithPerLayerExpertTowers() { + GgufModelBuilder b; + b.AddKv(StrKv("general.architecture", "qwen35moe")); + for (size_t l = 0; l < kPlaceLayers; ++l) { + b.AddTensor("blk." + std::to_string(l) + ".ffn_gate_exps.weight", + {32, 2, 4}, /*ggml_type=*/8, [] { + std::string s; + for (int i = 0; i < 8; ++i) s += Q8Block(); + return s; + }()); + } + b.AddTensor("token_embd.weight", {4, 2}, /*ggml_type=*/0, + std::string(32, '\1')); + return b.Build(); +} + +// Built through the production resolver, not by hand, so a pass cannot come +// from a per-layer vector this test invented. +vllm::MoePlacementPlan PlanPlacingTrailing(vt::DeviceType engine, + int64_t first_placed) { + std::vector ov; + for (int64_t l = first_placed; l < static_cast(kPlaceLayers); ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + return vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, engine), + static_cast(kPlaceLayers)); +} + +} // namespace + +TEST_CASE("#2517: a placed routed-expert tower leaves the staged footprint") { + TempFile f(BuildGgufWithPerLayerExpertTowers()); + const vllm::GgufFile gguf = vllm::GgufFile::Open(f.path()); + + // NULL PLAN: byte-for-byte the pre-#2517 answer, which is what every load in + // this tree that configured no placement gets. + const vllm::GgufStagedFootprint bare = vllm::GgufStagedWeightFootprint(gguf); + CHECK(bare.lower_bound_bytes == kPlaceWholeBound); + CHECK(bare.placed_tensor_count == 0); + CHECK(bare.placed_bytes == 0); + CHECK(bare.tensor_count == kPlaceLayers + 1); + + // A plan that PLACES NOTHING is the same answer again, through the other + // door: `cpu_moe` on a CPU engine resolves to exactly this. + const vllm::MoePlacementPlan trivial = + PlanPlacingTrailing(vt::DeviceType::kCPU, /*first_placed=*/0); + REQUIRE_FALSE(trivial.PlacesAnything()); + const vllm::GgufStagedFootprint inert = vllm::GgufStagedWeightFootprint( + gguf, 2, {}, false, &trivial); + CHECK(inert.lower_bound_bytes == kPlaceWholeBound); + CHECK(inert.placed_tensor_count == 0); + + // TWO OF THREE LAYERS PLACED. The two towers move into `placed_*` and leave + // the bound; the third tower and the vocabulary table stay. + const vllm::MoePlacementPlan plan = + PlanPlacingTrailing(vt::DeviceType::kCUDA, /*first_placed=*/1); + REQUIRE(plan.PlacesAnything()); + const vllm::GgufStagedFootprint placed = vllm::GgufStagedWeightFootprint( + gguf, 2, {}, false, &plan); + CHECK(placed.placed_tensor_count == 2); + CHECK(placed.placed_bytes == 2 * kPlaceTowerStaged); + CHECK(placed.lower_bound_bytes == kPlaceTowerStaged + kPlaceEmbdStaged); + CHECK(placed.tensor_count == 2); + // The bytes are ACCOUNTED, not lost: what left plus what stayed is what was + // there. A subtraction that merely shrank the figure would pass the line + // above and fail this one. + CHECK(placed.lower_bound_bytes + placed.placed_bytes == kPlaceWholeBound); +} + +TEST_CASE("#2517: CheckDeviceWeightFit refuses without the plan and passes " + "with it") { + TempFile f(BuildGgufWithPerLayerExpertTowers()); + const vllm::GgufFile gguf = vllm::GgufFile::Open(f.path()); + // A budget that straddles the two figures: 832 does not fit, 288 does. One + // file, one budget, one difference. + constexpr size_t kBudget = 500; + REQUIRE(kPlaceWholeBound > kBudget); + + const vllm::DeviceWeightFit no_plan = vllm::CheckDeviceWeightFit( + gguf, "cuda", /*needs_weight_staging=*/true, kBudget, 2, {}, false, + nullptr); + CHECK(no_plan.refuse); + CHECK(no_plan.needed_bytes == kPlaceWholeBound); + + const vllm::MoePlacementPlan plan = + PlanPlacingTrailing(vt::DeviceType::kCUDA, /*first_placed=*/1); + const vllm::DeviceWeightFit with_plan = vllm::CheckDeviceWeightFit( + gguf, "cuda", /*needs_weight_staging=*/true, kBudget, 2, {}, false, + &plan); + CHECK_FALSE(with_plan.refuse); + CHECK(with_plan.needed_bytes == kPlaceTowerStaged + kPlaceEmbdStaged); + + // A plan that does not place enough still refuses -- and then SAYS what the + // placement already took out, so the operator can never again read two + // contradictory numbers in consecutive lines of one load. + const vllm::MoePlacementPlan partial = + PlanPlacingTrailing(vt::DeviceType::kCUDA, /*first_placed=*/2); + const vllm::DeviceWeightFit still = vllm::CheckDeviceWeightFit( + gguf, "cuda", /*needs_weight_staging=*/true, /*budget_bytes=*/200, 2, {}, + false, &partial); + CHECK(still.refuse); + CHECK(still.message.find("ALREADY EXCLUDED") != std::string::npos); + CHECK(still.message.find(std::to_string(kPlaceTowerStaged)) != + std::string::npos); +} + +TEST_CASE( + "#2516: PeekRoute follows the placement, so the lane predicate and the " + "loader agree about a placed tower") { + // `GgufExpertTowersReachSlotLane` is the PRODUCTION consumer of `PeekRoute`, + // and it asks about the same tensors, in the same role, that + // `LoadStackedExperts` then routes. This case enters through that predicate + // rather than through `Route` directly, so it measures the pair rather than + // the function. + TempFile f(BuildGgufWithIQuantExpertTowers()); + const vllm::GgufFile gguf = vllm::GgufFile::Open(f.path()); + vllm::GgufLoadPolicy rocm = PolicyWith(true, false, false, false); + rocm.device = vt::DeviceType::kROCM; + + // NO PLAN: ROCm has no IQ `vec_dot` (#1940), so the towers expand and the + // predicate is false. This is the state #2516 reports and the inertness pin + // for every ROCm load that configures no placement. + vllm::ResetActiveMoePlacementPlanForTesting(); + CHECK_FALSE( + vllm::GgufExpertTowersReachSlotLane(gguf, "_exps.weight", rocm)); + + // A PLAN over both layers, placing them on the CPU whose `vec_dot` table + // covers IQ2_XXS and IQ4_XS: the same file, the same policy, the same + // predicate, and now true. + { + std::vector ov; + for (int64_t l = 0; l < 2; ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + const vllm::MoePlacementPlan plan = vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, vt::DeviceType::kROCM), + /*num_hidden_layers=*/2); + REQUIRE(plan.PlacesAnything()); + vllm::SetActiveMoePlacementPlan(plan); + CHECK(vllm::GgufExpertTowersReachSlotLane(gguf, "_exps.weight", rocm)); + } + vllm::ResetActiveMoePlacementPlanForTesting(); + + // ...and it is false again once the plan is gone, so the override belongs to + // the plan and is not a latch some earlier case set. + CHECK_FALSE( + vllm::GgufExpertTowersReachSlotLane(gguf, "_exps.weight", rocm)); +} diff --git a/tests/vllm/test_gguf_keep_quant.cpp b/tests/vllm/test_gguf_keep_quant.cpp index a8ee0fd58..c7f1e7448 100644 --- a/tests/vllm/test_gguf_keep_quant.cpp +++ b/tests/vllm/test_gguf_keep_quant.cpp @@ -35,6 +35,7 @@ #include "gguf_builder.h" #include "vllm/config/weight_residency.h" +#include "vllm/model_executor/device_placement.h" #include "vllm/model_executor/model_loader/gguf_dequant.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" #include "vllm/model_executor/model_loader/gguf_reader.h" @@ -2520,3 +2521,164 @@ TEST_CASE("gguf residency: the policy carries its device, and Route uses it") { CHECK(keep_metal.Route(t, vllm::GgufTensorRole::kEmbeddingTable) == vllm::GgufResidency::kExpandBf16); } + +// ── BACKEND-ROCM-IQ-EXPERT-RESIDENCY (#2516) ──────────────────────────────── +// +// A residency decision is a question about the device that will EXECUTE the +// tensor. Hybrid placement (#2023/#2314) made that device differ from the engine +// for exactly one role, and the loader kept asking the engine. +// +// The file this exists for: GLM-5.3 `UD-IQ1_S` on `strix:gpu0`. Its routed +// experts are IQ1_S/IQ3_XXS/IQ2_XXS/IQ4_XS/Q2_K/Q3_K, +// `DeviceKeepQuantSupported` serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, and the +// towers are never uploaded to any device — so every one of them expanded and +// `LoadStackedExperts` refused a load whose experts the CPU can execute. +namespace { + +// A routed-expert tower as the file stores it: [E, N, K] with a K that is a +// whole number of 256-element super-blocks, which is what `KeepQuantKDim` reads. +vllm::GgufTensorInfo IqTower(const std::string& name) { + vllm::GgufTensorInfo t; + t.name = name; + t.ggml_type = 19u; // IQ1_S + t.shape = {4, 2, 256}; + return t; +} + +// The plan the loader installs: `cpu_moe` over `layers` layers on an engine that +// is NOT the CPU, built through the SAME resolver production uses so a pass here +// cannot come from a hand-made per-layer vector. +vllm::MoePlacementPlan CpuMoePlan(vt::DeviceType engine, int64_t layers, + int64_t first_placed) { + std::vector ov; + for (int64_t l = first_placed; l < layers; ++l) + ov.push_back({vllm::LlmFfnExpsBlockRegex(l), "cpu"}); + return vllm::MoePlacementPlan::Resolve( + vllm::DevicePlacement::FromOverrides(ov, engine), layers); +} + +// RAII, because the plan is a process global and a case that leaked one would +// change the answer of every case that ran after it — including the inertness +// pins, which would then pass for the wrong reason. +struct ScopedPlan { + explicit ScopedPlan(const vllm::MoePlacementPlan& p) { + vllm::SetActiveMoePlacementPlan(p); + } + ~ScopedPlan() { vllm::ResetActiveMoePlacementPlanForTesting(); } +}; + +} // namespace + +TEST_CASE( + "#2516: a routed-expert tower PLACED on the CPU keeps its blocks on a ROCm " + "engine") { + vllm::ResetActiveMoePlacementPlanForTesting(); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + const vllm::GgufTensorInfo t = IqTower("blk.3.ffn_gate_exps.weight"); + + // NO PLAN INSTALLED — the inertness pin, and the state every load in this + // tree that configured no placement is in. IQ1_S has no ROCm `vec_dot` + // (#1940), so the tower expands, exactly as before this row. + CHECK(rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); + + // WITH the plan installed, the same tensor under the same policy keeps its + // blocks, because the device that will run it is the CPU and + // `vt::cpu::HasQuantDotKernel(kIQ1_S)` is true. + { + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/0)); + CHECK(rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kKeepQuant); + } + + // ...and the plan going away restores the old answer, so the override is the + // plan's and not a latch. + CHECK(rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); +} + +TEST_CASE("#2516: the override is PER LAYER, not per load") { + // Layers 4..7 placed, 0..3 not. One assertion pair on one file under one + // policy: a change that keyed the override on "a plan exists" rather than on + // THIS tensor's layer passes the case above and fails this one. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/4)); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + + CHECK(rocm.Route(IqTower("blk.5.ffn_up_exps.weight"), + vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kKeepQuant); + CHECK(rocm.Route(IqTower("blk.1.ffn_up_exps.weight"), + vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); +} + +TEST_CASE("#2516: only the STACKED-EXPERT role moves; the plan places nothing " + "else") { + // The plan moves a routed-expert BLOCK. An attention weight and the vocabulary + // table of a placed layer's block stay on the engine and must keep answering + // the engine's own capability — widening the override to every role would + // keep a Q8_0 attention weight quantized for a device that will execute it. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/0)); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + + vllm::GgufTensorInfo w; + w.name = "blk.3.attn_q.weight"; + w.ggml_type = 19u; // IQ1_S: kept on the CPU, expanded on ROCm + w.shape = {2, 256}; + CHECK(rocm.ComputeDeviceFor(w.name, vllm::GgufTensorRole::kMatmulWeight) == + vt::DeviceType::kROCM); + CHECK(rocm.Route(w, vllm::GgufTensorRole::kMatmulWeight) == + vllm::GgufResidency::kExpandBf16); + CHECK(rocm.ComputeDeviceFor("blk.3.ffn_gate_exps.weight", + vllm::GgufTensorRole::kStackedExpertWeight) == + vt::DeviceType::kCPU); +} + +TEST_CASE("#2516: PeekRoute and Route resolve the device the same way") { + // `GgufExpertTowersReachSlotLane` peeks the very tensors the loader routes. + // Two spellings of the device term is how a bound and a forward come to + // disagree about one file (#1378), so this pins that they cannot. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kROCM, /*layers=*/8, + /*first_placed=*/4)); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + for (const char* name : {"blk.5.ffn_gate_exps.weight", + "blk.1.ffn_gate_exps.weight", + "blk.5.attn_q.weight"}) { + CAPTURE(name); + const vllm::GgufTensorInfo t = IqTower(name); + CHECK(vllm::PeekRoute(rocm, t, + vllm::GgufTensorRole::kStackedExpertWeight) == + rocm.Route(t, vllm::GgufTensorRole::kStackedExpertWeight)); + } +} + +TEST_CASE("#2516: a plan that places NOTHING never overrides the engine") { + // `cpu_moe` on a CPU engine resolves to a plan over every layer that places + // nothing (`DevicePlacement::IsTrivial`). A default-constructed plan's engine + // device is also `kCPU`. Either one, adopted unconditionally, would route a + // ROCm load's experts as though they were on the host — which is the same + // silent wrong answer in the opposite direction. + ScopedPlan guard(CpuMoePlan(vt::DeviceType::kCPU, /*layers=*/8, + /*first_placed=*/0)); + CHECK_FALSE(vllm::ActiveMoePlacementPlan().PlacesAnything()); + vllm::GgufLoadPolicy rocm; + rocm.keep_quant = true; + rocm.device = vt::DeviceType::kROCM; + CHECK(rocm.ComputeDeviceFor("blk.3.ffn_gate_exps.weight", + vllm::GgufTensorRole::kStackedExpertWeight) == + vt::DeviceType::kROCM); + CHECK(rocm.Route(IqTower("blk.3.ffn_gate_exps.weight"), + vllm::GgufTensorRole::kStackedExpertWeight) == + vllm::GgufResidency::kExpandBf16); +}