perf(GFX1100-TG200): the 200 tok/s gfx1100 decode campaign and fp8 KV cache decode attention

.agents/benchmark-record.md

@@ -28723,13 +28723,12 @@ killed every measured leg mid-load on the previous attempt.
    0.774 GiB on disk in bf16 and 1.547 GiB resident, because `qwen3_vl.cpp`
    widens it to host f32 —
    [#1359](https://github.com/mudler/vllm.cpp/issues/1359), which also affects
-   the Qwen3.6-27B path. #1359's Qwen3-VL half has since LANDED, and the
-   2026-08-28 rerun recorded later in this file MEASURED the consequence:
-   826916864 B = 0.770 GiB, **0.499x this figure**. The HALVING IS CORRECT
-   rather than a regression — the flag now frees the tower the checkpoint ships
-   instead of the tower plus our widening. The figure recorded here stands
-   unaltered as what the run at `41ab550b9` measured, and is superseded for
-   current behaviour; `muse-glimmer-30b` still widens, blocked on
+   the Qwen3.6-27B path. #1359's Qwen3-VL half has since LANDED, so this leg
+   rerun should read about 0.774 GiB rather than 1.542, and that HALVING IS
+   CORRECT rather than a regression — the flag now frees the tower the
+   checkpoint ships instead of the tower plus our widening. The figure recorded
+   here stands as what the run at `41ab550b9` measured; `muse-glimmer-30b` still
+   widens, blocked on
    [#2166](https://github.com/mudler/vllm.cpp/issues/2166).
 2. **Load-time residency, not a served request.**
    `ForwardQwen3VLForConditionalGeneration` `VT_CHECK`s `input.mm.has_value()`,

.agents/roadmap_v1.md

@@ -546,7 +546,7 @@ degraded run — it is no run at all.
 | `--tool-call-parser` | 42 names, **84/90 recipe uses (93%)** — the healthy axis; `inkling` (2 uses) landed 2026-08-13 under #608 W1 | [#608](https://github.com/mudler/vllm.cpp/issues/608) for the last 6 |
 | `--reasoning-parser` | 10 of 28 names, **15/76 uses (20%)**; `qwen3` (18) rejected on our own gate models | [#605](https://github.com/mudler/vllm.cpp/issues/605) |
 | `--enable-auto-tool-choice`, `--trust-remote-code` | no-ops for us, yet **abort startup** on 89 and 82 recipes | [#606](https://github.com/mudler/vllm.cpp/issues/606) |
-| `--language-model-only`, `--limit-mm-per-prompt` | **ACCEPTED + ENFORCED 2026-08-14** (#607 L2): the 43 recipes that pass the flag reach model load, and it zeroes every modality limit so mm requests are REFUSED with upstream's message and HTTP 400. L3 landed the tower skip, and on 2026-08-24 its HOST RSS was measured on ONE model: Qwen3-VL-4B-Instruct freed 1655791616 B = 1.542 GiB at load, `--device cpu`, MET on both pairs against a threshold declared before the run (`multimodal-track.md` §1.5 L3). #1359's Qwen3-VL half has since landed, and the **2026-08-28 rerun at `525d2b991` on `dgx:gpu0` MEASURED the consequence: 826916864 B = 0.770 GiB, MET on both pairs against the live 747625881 B threshold** — a 0.499x fall that is correct rather than a regression, and no longer a prediction. That rerun also verifies #1359's Qwen3-VL half: the default arm recovered 828219392 B = 99.7% of prediction while the tower-free `--language-model-only` control arm moved +0.0077% against a 2% bound. #1359 stays OPEN for #2166 (muse-glimmer) and #2173 (Gemma-4 vision, unreached). **Freed encoder VRAM is still entirely unmeasured** — that figure is host RAM on a `VLLM_CPP_CUDA=OFF` build — as is `muse-glimmer-30b` on either axis, so the flag may be described as freeing memory only with the model, the host and the load-time window named beside it | [#607](https://github.com/mudler/vllm.cpp/issues/607) |
+| `--language-model-only`, `--limit-mm-per-prompt` | **ACCEPTED + ENFORCED 2026-08-14** (#607 L2): the 43 recipes that pass the flag reach model load, and it zeroes every modality limit so mm requests are REFUSED with upstream's message and HTTP 400. L3 landed the tower skip, and on 2026-08-24 its HOST RSS was measured on ONE model: Qwen3-VL-4B-Instruct freed 1655791616 B = 1.542 GiB at load, `--device cpu`, MET on both pairs against a threshold declared before the run (`multimodal-track.md` §1.5 L3). #1359's Qwen3-VL half has since landed, so a rerun should read about 0.774 GiB and that fall is correct rather than a regression. **Freed encoder VRAM is still entirely unmeasured** — that figure is host RAM on a `VLLM_CPP_CUDA=OFF` build — as is `muse-glimmer-30b` on either axis, so the flag may be described as freeing memory only with the model, the host and the load-time window named beside it | [#607](https://github.com/mudler/vllm.cpp/issues/607) |
 | `--kv-cache-dtype` | not a serve flag; residual on the `KV-FP8` row | — |
 | `--speculative-config` | MTP + DFlash land; `eagle`/`eagle3` (7 uses) do not | — |
 | TP / EP / multi-node (`--tensor-parallel-size`, `--enable-expert-parallel`, `--mm-encoder-tp-mode`) | absent by scope, not by defect — single-box engine | see the TP W-plan above |

.agents/specs/dsv4-dsa-compose.md

@@ -7,9 +7,8 @@ Oracle: vLLM at the parity pin `5559679229`, `vllm/models/deepseek_v4/`.
 ## Now
 
 `READY` — this spec is the deliverable of the scoping wave. No implementation
-has started. W1 cannot begin until `KV-DSV4-MULTICACHE` **W5** lands, and W5 has
-no owner today (#2302; W3, which an earlier revision of this spec named, landed
-as `ca3dcda21` on 2026-08-27). See `## Dependencies`.
+has started. W1 cannot begin until `KV-DSV4-MULTICACHE` W3 lands (see
+`## Dependencies`).
 
 ## Scope
 
@@ -27,23 +26,6 @@ the cache topology (`KV-DSV4-MULTICACHE`, #1925); residency (#2283); the
 attention sink, which is a loaded per-head weight and not cache state
 (`attention.py:218-222`).
 
-**The boundary with `KV-DSV4-MULTICACHE` W5, which this row overlapped when it
-was created** (#2302). W5's scope lived in a single wave-table cell reading "the
-DSA-sparse attention path reading the published caches" and removing the
-`!is_indexer && !is_comp` refusal -- the ALGORITHM, not the plumbing -- and it has
-never had a design section, so this row was specced over it. The split, recorded
-in both specs:
-
-| row | owns |
-|---|---|
-| `KV-DSV4-MULTICACHE` W5 | the caches REACHING the model and each layer routing to its own: `attn_kv` consumed rather than `(void)`-ed, and `ModelRegistry::Forward`'s `input.multi_kv` guard (`model_registry.cpp`) stopping its refusal |
-| **this row** | what RUNS on those caches: the three layer shapes, the compressor's two stages, the `coff` role selection and boundary emission, the indexer's `qr`-sourced query, and the `!is_indexer && !is_comp` refusal at `deepseek_v4.cpp:786-787` |
-
-W5 lands first and deliberately does NOT remove the DSA refusal; it makes a cache
-reachable for this row. Neither row is gateable end-to-end alone: W5's
-synthetic-config gate proves routing, and the token-exact oracle gate above 512
-tokens belongs here.
-
 ## Upstream chain
 
 Read at `5559679229`. **The path is `vllm/models/deepseek_v4/`, NOT
@@ -74,21 +56,10 @@ nothing here and reads as "upstream does not implement it", which is wrong.
 | SWA-only | 0 | 5 | neither |
 
 Counts are `config.json`'s `compress_ratios` histogram `{0: 5, 4: 21, 128: 20}`
-= 46 entries.
-
-**RESOLVED, and it was already answered elsewhere in this row family.**
-`dsv4-dsa-geometry.md` read the artifact's own `config.json` and records that the
-46 entries are **43 layers + 3 MTP blocks**: layers 0 and 1 are `0`, layers 2..42
-alternate `4` and `128`, and the MTP tail is `0`. So `num_hidden_layers == 43`,
-2 layers are dense, 21 carry an indexer and 20 a compressor-only -- which is
-exactly the "41 of 43 carry a compressor, 21 carry an indexer" the row already
-recorded. There is no contradiction: 43 counts LAYERS, 46 counts the config list
-INCLUDING the MTP tail, and the trellis shard count matches the layers.
-
-W1 therefore inherits 43 and does not need to reconcile anything. The entry is
-kept rather than deleted because #2186 raised it as open and a reader who saw
-that deserves to find the answer here, with its source, rather than a silent
-deletion.
+= 46 entries. **46 is not 43**, and the row's older records say "43 layers" /
+"41 of 43"; 43 is the trellis shard count (`exl3-layer-000..042`). W1 must
+reconcile which number each claim means rather than inherit either (#2186
+raised this and it is still open).
 
 ### D2. The 3-way stream overlap is performance, not correctness
 
@@ -137,100 +108,6 @@ forward refuses because `AttentionBlock` indexes the COLLAPSED geometry. So W1
 is a forward change, not a loader change — and the refusal's own text is the
 specification of what to build.
 
-### W1 design — the compressor-only shape is COMPOSITION, not new kernels
-
-Traced before estimating, because every earlier estimate on this row family moved
-once the tree was read.
-
-**Every primitive W1 needs already exists, on CPU and CUDA.**
-
-| the shape needs | what exists |
-|---|---|
-| the window pass | `vt::MlaDecodeAttention` with `window_size` (`left == sliding_window - 1`) -- landed by `KV-DSV4-MULTICACHE` W5 |
-| the compressed-history pass | the SAME op's SELECTED-SLOT arm, `topk_indices` + `valid_counts` |
-| combining the two | `vt::MergeAttnStates` -- an LSE merge with both `+inf` and both-`-inf` edge cases ported |
-| the pool | `CompressorPoolNorm` -- per-column softmax over the window, then RMSNorm |
-| the APE save | `CompressorSaveScoreApe` |
-| the per-head sink | `MlaDecodeAttentionArgs::attn_sink`, landed by W5 |
-
-So W1 composes: save state each step, pool at a boundary into the compressed
-cache, then TWO attention passes merged by their LSEs -- rather than the single
-fused two-cache kernel upstream calls
-(`flash_mla_with_kvcache(k_cache=swa, extra_k_cache=compressed, ...)`). The
-composition is mathematically the same; only the kernel fusion differs, and that
-is a performance question for a later wave, not a correctness one.
-
-**`compress_ratio == 128` FIRST because `coff == 1` there.** `overlap` is
-`compress_ratio == 4`, so the 128 shape has no overlapping windows and no
-`head_offset` role selection -- the mechanism W5-4 of `dsv4-dsa-compose.md`
-describes. It exercises the state cache, the boundary gate and the two-pass merge
-without the hardest part.
-
-#### The `c128a` selection is ARITHMETIC, not a learned top-k
-
-The last unknown in W1's shape, and it resolves in W1's favour. The compressed
-pass needs an index list, and the name upstream gives it --
-`c128a_global_decode_topk_indices` -- reads like the Lightning Indexer's output.
-It is not.
-
-`sparse_mla.py:126-129` calls the field "Pre-computed C128A metadata
-(compress_ratio == 128 only). Decode: global slot ids + valid-entry counts
-**(fused from positions)**", and `_build_c128a_metadata` asserts
-`cm.positions is not None` because positions are its only input. The selection is
-therefore arithmetic over the current position -- which compressed windows have
-CLOSED -- and carries no learned component at all.
-
-That is what makes `compress_ratio == 128` the right first shape. It needs:
-
-- the compressor cycle (landed: `CompressorStepCycle`),
-- a window pass and a compressed pass merged by LSE (proven equivalent above),
-- and an index list computable from `positions` alone.
-
-The Lightning Indexer, which DOES learn its selection, belongs only to the
-`compress_ratio == 4` layers and therefore to W3. A reader who assumed "topk
-implies indexer" would have pulled W3's hardest dependency into W1 for no reason.
-
-#### THE SINK MUST ENTER EXACTLY ONE PASS
-
-The trap, written down before anyone hits it. A sink is one extra logit in the
-DENOMINATOR. `MergeAttnStates` combines two states by their LSEs, and each pass's
-LSE is `log sum exp(its scores)`. **If both passes seed the denominator with the
-sink, the merged denominator counts it TWICE**, and the result is a plausible,
-slightly-too-small attention output that no token gate would catch.
-
-This is the same defect family as the split double-count `MlaDecodeAttentionArgs`
-already documents: a sink added per split rather than in the final reduction. It
-has now appeared twice in this design, which is why it is stated as a rule --
-**the sink belongs to exactly one contributor to any merged denominator** --
-rather than as a note about one kernel.
-
-The gate must therefore compare a two-pass merged result against a SINGLE-pass
-reference over the union of both key sets, with a non-zero sink, at a length
-where both passes are non-empty. A gate where either pass is empty cannot see a
-double-count.
-
-#### The composition primitive now exists, and the rule is executable
-
-`MergeWindowAndCompressed` in `src/vllm/model_executor/models/deepseek_v4_dsa.cpp`
-is that composition: it attends the compressed rows with NO sink, and merges
-against the window pass's output and LSE. The window pass carries the sink, so
-exactly one contributor seeds the denominator. `PagedCausalMlaAttention` grew an
-optional `out_lse` for this, because a merge needs both sides' LSEs.
-
-Two constraints are asserted rather than assumed. Every query sees EVERY
-compressed row -- a closed window is history, so no causal bound applies among
-them -- and `VT_CHECK(num_tokens == 1 || num_heads == 1)` holds the point where
-the two LSE layouts coincide, since `MergeAttnStates` wants `[H, T]` and the
-decode op emits `[T, H]`. A general prefill step needs a transpose there and
-does not get one yet; it is listed under `## Owed`.
-
-The gate this section demanded is `tests/vllm/models/test_deepseek_v4_paged_equiv.cpp`,
-"W1: two LSE-merged passes equal one pass over the union". Three mutations prove
-it discriminates: seeding the compressed pass with the same sink (the
-double-count itself), returning the window output unmerged, and bounding the
-compressed rows causally. Each was built before it was read -- a mutation that
-fails to compile leaves a stale binary reporting a pass.
-
 ## Our baseline
 
 What this tree has TODAY, so a later reader does not re-derive it:
@@ -287,45 +164,20 @@ gate is token-exactness against the pinned oracle ABOVE 512 tokens
 |---|---|
 | W1 (#1960, `c1e6f3fb9`) | LANDED — `SlidingWindowMLASpec`, the four `MLAAttentionSpec` fields |
 | W2 (#1973, `6b18829bc`) | LANDED — all seven groups / 167 entries published; runner refuses an unallocated published group |
-| W3 (#2068, `ca3dcda21`) | LANDED 2026-08-27 — the runner allocates a buffer for EVERY published cache instead of one per hidden layer, and `ModelForwardInput` gained the third channel |
-| W4 | proposal, **no owner** — non-uniform `block_size` |
-| W5 | proposal, **no owner** — consumption |
-| W6-W7 | proposals, **no owner** |
+| W3 | OWED — the third `ModelForwardInput` channel |
+| W4 | OWED — non-uniform `block_size` |
+| W5 | OWED — consumption |
 
-**W1 of this row cannot start before that row's W5.** The composition writes a
+**W1 of this row cannot start before that row's W3.** The composition writes a
 separate compressed cache beside a sliding-window raw cache, and it cannot reach
 a cache the forward is not handed. This is a hard ordering, not a preference.
 
-**W5, not W3** (#2302). An earlier revision of this spec named W3, which had
-already landed when it was written. The wall today is the one the code names
-itself, in `ModelRegistry::Forward`
-(`src/vllm/model_executor/models/model_registry.cpp`, the `input.multi_kv` guard at the top of `ModelRegistry::Forward`):
-
-> `... and no registered forward consumes a cache set keyed by layer name.
-> Refusing rather than discarding an allocated KV topology in silence (row
-> KV-DSV4-MULTICACHE W5 owns the consuming forward; #1925, #2068)`
-
-A DeepSeek-V4 engine therefore constructs, publishes and ALLOCATES all 167
-buffers today, and refuses at the first forward. **This makes the ordering
-harder than the earlier revision claimed, not softer:** W3 had an owner and
-landed, while W4 through W7 are proposals with no owner at all. Nothing in this
-row can begin until W5 acquires one.
-
-The error is recorded rather than quietly corrected because its cause is
-reusable: two stale records agreed with each other and neither was the tree.
-#1925's index row predates W3, and `kv-dsv4-multicache.md` `## Now` opened with
-"W3 (#2068) is claimed" while its own closing paragraph already said the engine
-allocates all 167 buffers and refuses naming W5. AGENTS.md `## History is git`
-is explicit -- "Before you conclude anything about past work, read the spec and
-run `git log -S`" -- and `git log --oneline --grep '2068'` shows `ca3dcda21`
-immediately. It was not run.
-
 ## Work breakdown
 
 | wave | scope | depends on |
 |---|---|---|
 | W0 | this spec | — |
-| W1 | reconcile 43 vs 46; layer-shape dispatch in `AttentionBlock`, SEQUENTIAL, replacing the refusal for the `compress_ratio == 128` (compressor-only) shape first | multicache W5 |
+| W1 | reconcile 43 vs 46; layer-shape dispatch in `AttentionBlock`, SEQUENTIAL, replacing the refusal for the `compress_ratio == 128` (compressor-only) shape first | multicache W3 |
 | W2 | the compressor's two stages: `save_partial_states`, then boundary-gated compress/norm/RoPE/quant/store | W1 |
 | W3 | the overlapped window and role selection; the `compress_ratio == 4` shape; the indexer's `qr`-sourced query | W2, and both kernel `SPIKE` rows promoted |
 | W4 | the stream overlap, as a measured performance wave | W3 |
@@ -365,18 +217,10 @@ failure mode this row is most exposed to.
 - The `43` vs `46` layer-count reconciliation, raised by #2186 and still open.
 - Promotion of `KERNEL-ATTN-DSA-SPARSE-INDEX` and `KERNEL-ATTN-DSA-COMPRESSOR`
   out of `SPIKE`.
-- The `[T, H]` to `[H, T]` LSE transpose `MergeWindowAndCompressed` refuses, so a
-  PREFILL step with more than one token and more than one head can compose. A
-  decode step is unaffected: it carries one token.
-- The `AttentionBlock` compressor arm itself. The primitive above is reached only
-  by its gate until that arm calls it, and the `compress_ratio == 128` refusal
-  stays in place until then.
 
 ## Stop conditions
 
-- Stop if `KV-DSV4-MULTICACHE` W5 does not land: W1 has no cache to READ. W3
-  (the allocation and the forward channel) landed as `ca3dcda21`; W5 is the
-  consuming forward, and it has no owner (#2302).
+- Stop if `KV-DSV4-MULTICACHE` W3 does not land: W1 has no cache to write to.
 - Stop before claiming any speed number. This row makes the model RUN; a
   throughput comparison against SparkInfer's 44-47 tok/s additionally needs
   `nvfp4_ds_mla` and K5 speculative decoding, neither of which exists here.