feat(KV-FP8): W6 — the ROCm fp8-e4m3 KV cache store and read

.agents/feature-matrix.md

@@ -82,7 +82,7 @@ Confirmed NON-gap: vLLM has removed prompt adapters.
 | SGLang RadixAttention behavior parity (fuse-or-flag) | SGLang v0.5.15 `f63458b` `mem_cache/radix_cache.py`, `managers/schedule_policy.py`, `constrained/outlines_jump_forward.py` | `ACTIVE` T2 | **Scoped 2026-07-27 (`CLAIM-SGLANG-RADIX-SCOPE`); W1+W2 IMPLEMENTED 2026-07-27 (`CLAIM-SGLANG-IMPL`, rows now `ACTIVE`).** VERDICT: SGLang's radix TREE == our block-hash APC ⇒ RadixAttention is **already FUSED**, `--enable-radix-attention` is an ALIAS for the APC toggle (LANDED: server alias + C-ABI `enable_prefix_caching` tri-state, ABI v7). Genuinely-distinct behavior = cache-aware **LPM scheduling `--schedule-policy=lpm`** (LANDED: `SchedulerPolicy::kLPM` reorders the FCFS waiting deque by APC longest-match, ported FROM `schedule_policy.py:205,229`, output-neutral; gate `test_scheduler_lpm` 6/6). **SW2 in-batch prefix-collision de-prioritization LANDED 2026-07-27 (`CLAIM-SGLANG-SW2`)** inside the `kLPM` reorder (block-hash APC keys, no second trie; ported FROM `schedule_policy.py:253-301,311`), output-neutral; its throughput lever is NOT-APPLICABLE — our APC caches at allocation time so the 2nd same-step collider already hits (within-step dedup subsumes it). overlap scheduler == `ENG-ASYNC-SCHED` (fused). **SW3 jump-forward decoding — safe TOKEN-UNIQUE subset LANDED 2026-07-28 (`CLAIM-SGLANG-SW3`)**: forced-token detection hook `StructuredOutputGrammar::forced_token()` + opt-in driver `DrainForcedTokens` (env `VT_ENABLE_JUMP_FORWARD`, default OFF), provably byte-identical to per-token constrained decode (jumps only where the grammar leaves exactly one valid token — no re-tokenization); gate `test_jump_forward` 5/5 (RED-first). Residual: SW4 + the general re-tokenization span + production scheduler splice (named). **ABI/API/flag EXPOSURE 2026-07-28 (`CLAIM-SGLANG-ABI-DOCS`, reconciled to ABI v10):** LPM + jump-forward made first-class DOCUMENTED knobs on ALL THREE surfaces (were server-only / env-only) — LPM via the concurrent session's C-ABI **string** field `vllm_model_params.scheduling_policy="lpm"` (ABI v9; NO duplicate int knob) + C++ `EngineParams::policy=kLPM` + server `--scheduling-policy lpm`; jump-forward via new C-ABI `vllm_model_params.enable_jump_forward` (tri-state int, ABI **v10** appended after the v9 fields) + C++ `EngineParams::enable_jump_forward` + server `--[enable\|disable]-jump-forward`. `VT_ENABLE_JUMP_FORWARD` retained as env override. User docs [docs/SGLANG-COMPAT.md](../docs/SGLANG-COMPAT.md) + spec [sglang-enablement.md](specs/sglang-enablement.md); ABI e2e `tests/capi/test_capi.cpp` (2 v10 jump-forward cases; `vllm_abi_version()`==10). Default-inert (all-zero ⇒ byte-identical). Rows `KV-SGLANG-RADIX-CACHE` + `ENG-SGLANG-BEHAVIOR-FLAG`. Sibling benchmark track = `BACKEND-GATE-CUDA-SGLANG*` (unchanged) | [sglang-radixattention.md](specs/sglang-radixattention.md) |
 | **SGLang parity PROGRAM** (whole-surface inventory + oracle) | SGLang v0.5.15 `f63458b` — full runtime surface | `SPIKE` T2 | **Elevated 2026-07-27 (`CLAIM-SGLANG-PARITY-PROGRAM`).** The vLLM-parity approach replicated for SGLang: a tabular whole-surface inventory (44 rows) classifying every SGLang capability **FUSED (23) / SGLANG-DISTINCT (8) / INVENTORIED (5) / OUT-OF-SCOPE (8)**, plus SGLang stood up as a correctness + performance ORACLE (dgx GB10 via the arm64 cu130 image — no from-source build needed). SGLang is a competitor perf FLOOR + correctness cross-check, NOT the mirror source (vLLM stays behavior truth). Headline SGLANG-DISTINCT opt-ins: LPM scheduling, in-batch prefix de-prioritization, radix eviction strategies, jump-forward, custom logit processors, batch-invariant determinism, PD disaggregation, two-batch EP overlap. Full map + ranked plan in the matrix. Sibling benchmark rows `BACKEND-GATE-CUDA-SGLANG*` unchanged | [sglang-matrix.md](sglang-matrix.md); [sglang-parity-oracle.md](specs/sglang-parity-oracle.md) |
 | SlidingWindowSpec + ChunkedLocalAttentionSpec | `v1/kv_cache_interface.py` | `PARTIAL` T1 | Both execution leaves are implemented: W1 sliding-window and W3 chunked-local sizing, registry/grouping, manager prefix/recycling policy, admission and hybrid-disabled conversion pass their ported CPU/property/sanitizer gates (G1/G2). The compute-locality consumers are now GPU-gated (2026-07-27 `CLAIM-ROADMAP-C5`, dgx GB10: Gemma-2/Gemma-3 sliding-window model gates 48/48; `test_chunked_local_attention` 5/5). The KV memory-OPTIMIZATION path (optimized-manager held-block cap vs the full-allocation fallback the current model gates use) still needs a model-level hybrid-manager memory gate (G8) — kept `PARTIAL` honestly | [sliding-local-yarn-long-context.md](specs/sliding-local-yarn-long-context.md) |
-| fp8 KV cache (`cache_dtype=fp8*`) | `layers/quantization/kv_cache.py` | ◐ T1 | `KV-FP8` ACTIVE / `QUANT-KV-FP8` PARTIAL — W1 CPU fp8-e4m3 store+read+config-parse landed; CUDA + memory-halving e2e later | [fp8-kv-cache](specs/fp8-kv-cache.md) |
+| fp8 KV cache (`cache_dtype=fp8*`) | `layers/quantization/kv_cache.py` | ◐ T1 | `KV-FP8` ACTIVE / `QUANT-KV-FP8` PARTIAL — W1 CPU fp8-e4m3 store+read+config-parse landed; W2 CUDA store+read landed; W6 ROCm store+read landed; memory-halving e2e later | [fp8-kv-cache](specs/fp8-kv-cache.md) |
 | nvfp4 / per-token-head / turboquant KV | `config/cache.py` | ☐ T2 | | `planned: specs/nvfp4-kv-cache.md` |
 | KV offload (CPU tiering, LRU/ARC) | `v1/kv_offload/` | ☐ T2 | | `planned: specs/kv-offload.md` |
 | External KV-cache provider ABI + LMCache (MP service and in-process connectors) | `config/kv_transfer.py`, `distributed/kv_transfer/kv_connector/v1/{base,lmcache_connector,lmcache_mp_connector}.py` | ☐ T2 | explicit roadmap outcome `KV-EXTERNAL-CACHE`: mirror `kv_producer`/`kv_consumer`/`kv_both`, scheduler/worker metadata, async layer load/store, dynamic external connector modules, failure policy, metrics and cache-lifecycle ownership; gate the official LMCache shared-prefix quickstart plus Qwen3.6 hybrid behavior | `planned: specs/external-kv-cache-lmcache.md` |

.agents/issue-index.md

@@ -692,6 +692,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1843](https://github.com/mudler/vllm.cpp/issues/1843) | `FIX-FP8-PLAN-CAPTURE-1843` | The fp8 cuBLASLt lane still queries `cublasLtMatmulAlgoGetHeuristic` inside CUDA-graph capture because `VT_FP8_PLAN_CACHE` ships OFF (`fp8_plan_cache.h:49-59` @ `364f2a898`), so a captured decode on an fp8-tower model dies on CUDA 13.3 even with PR #1741 -- measured on `dgx:gpu0` (GB10, staged CUDA 13.3.73): #1741 alone fails on the fp8 lane, `VT_FP8_PLAN_CACHE=1` alone fails on the bf16-TN lane, both together pass the graphed 35B gate token-exact on all three arms. The fix is the default flip #1741's spec owed, with the same not-a-performance-knob polarity argument its `gemm_plan_cache.h` records. Claimed by row `FIX-FP8-PLAN-CAPTURE-1843` ([spec](specs/fix-fp8-plan-capture.md)) | bug |
 | [#1849](https://github.com/mudler/vllm.cpp/issues/1849) | `SPEC-DFLASH2` | **The DFlash2 draft step costs a flat ~23 ms at EVERY K, and the two levers #1849 names resolve differently once read from the records.** Lever A (quantize the shared head) is ALREADY LANDED for the measured subject: `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` stores `lm_head` as W4A16_NVFP4 g16 (header-verified 2026-08-21, quantization-matrix `QUANT-QWEN38-27B-NVFP4-ARM`), upstream computes with it packed through `lm_head.quant_method.apply`, and both our reads have been packed since #1628 — so the head traffic is ~2×0.72 GB not 2×2.54, the draft-phase floor re-derives to ~9 ms, and the unattributed residual GROWS to ~13-14 ms. Lever B (launch/sync trim) is counted in code at one replay + ~10 launches + ~76 B up / 64 B down + one sync — well under 0.5 ms, so the residual sits INSIDE kernels and needs on-box attribution. W9 lands `VT_SPEC_TRACE=2` (the `[spec-phase-dev]` pre/fwd/select/walk split) as the instrument, and borrow-first loading for the draft's shared bf16 embed+head (~5.1 GB host on the bf16 arm, ~2.5 GB on the r0b0tlab arm; memory only, no step-time claim). The bf16-target arm's 2×2.54 GB head reads are upstream's own serving dtype and stand as a recorded ceiling. Wave spec [dflash2-draft-fixed-cost.md](specs/dflash2-draft-fixed-cost.md); the K-ladder rerun, the `ncu`/`nsys` attribution and any step delta are owed there, operator-run | perf |
 | [#1844](https://github.com/mudler/vllm.cpp/issues/1844) | `ENG-MM-INPUT-PIPELINE` | **`scripts/mm/tower_skip_rss.sh` killed every measured leg mid-load, because `run_arm`'s `/health` poll was answered by the PREVIOUS leg's server.** First real run (`thor:gpu0`, worker `rc-worker-kk96r`, `d60692c8`): checkpoint staged and verified (29 files, 8887294190 B), both binaries built sha256-identical (`78d582e4...`), live target query green on both build dirs -- and then **five 0-byte `.time` files** and `VOID` on both pairs. `warmup` reached `listening on http://0.0.0.0:18607` (1286 B log); all four measured legs stopped at `loading model from ...` (363/345 B) inside one minute. TWO defects, one shape. (a) `$PORT` is fixed and the poll starts immediately, so a stale listener answers it and the leg is ready before it has read a tensor. (b) `kill "$pid"` signals `/usr/bin/time`, which installs no handler: the timer dies before writing its `-o` file and the server is reparented to init and KEEPS THE PORT -- which is what was answering. Measured: `/usr/bin/time -v -o f sleep 100 & kill $!` leaves `f` at 0 B and `sleep` alive with ppid 1; signalling the CHILD leaves `f` at 752 B with a `Maximum resident set size` line. **Nothing could catch it**: `test_tower_skip_rss_report.py` was 60/60 green over finished files, and `run_arm`, the poll and the teardown ran only under a lease -- the residual [#1819](https://github.com/mudler/vllm.cpp/issues/1819) recorded verbatim, now observed. FIXED IN FLOW: a leg refuses to start into an occupied port; readiness requires the banner in the leg's OWN log (its stdout, which no other server can write) before `/health`; SIGTERM goes to the SERVER so the timer survives to write; the port must stop accepting before the next leg; each of the three waits is bounded and each bound REFUSES; and a leg whose `.time` carries no `Maximum resident set size` line fails AT that leg rather than as VOID four legs later. GATED: `TOWER_SKIP_RSS_SOURCE_ONLY=1` sources the harness for its functions alone and `tests/scripts/test_tower_skip_rss_arm.py` drives `run_arm` against a fake server on a scratch port -- stale listener, absent banner, death during load, the happy path's non-empty `.time`, five legs in the declared order, and both halves restored as mutations. RED-first: 11 of 14 cases fail against the pre-fix `run_arm`, the stale-listener case reporting `LEG default OK` with a 0-byte `.time` | bug |
+| [#2065](https://github.com/mudler/vllm.cpp/issues/2065) | `KV-FP8` | The ROCm fp8-e4m3 KV cache arm: the store kernel (`vt::ReshapeAndCacheFp8` for `DeviceType::kROCM`), the fp8 dequant on the ROCm paged-attention read, the `OpId::kReshapeAndCacheFp8` registration for `kROCM`, and the widening of the `src/vt/ops.cpp` fp8 read refusal to admit `kROCM`. Mirrors the CUDA W2 arm element-for-element; the CPU kernels remain the oracle. Spec [`fp8-kv-cache.md`](specs/fp8-kv-cache.md) `## W6` | feature |
 | [#1846](https://github.com/mudler/vllm.cpp/issues/1846) | `MODEL-MM-dots3-note-dots3-note-for-causal-lm` | **The released `dots3-note-prev` shard index declares `indexer_rope_layout: "leading"` and `indexer_rope_converted_from: "tail"` in its `metadata` block, and NOTHING reads either key** — `git grep indexer_rope_layout` over vLLM `origin/main` returns nothing. Measured at W2 while reading the whole index. It is the publisher stating how the DSA indexer's `wq_b`/`wk` are laid out along the 128-wide index head, and it agrees with what upstream's code does anyway: `DeepseekV2Indexer` rotates `[..., :rope_dim]` and leaves `[..., rope_dim:]` (`deepseek_v2.py:805,:814`, `rope_dim` 64 of `index_head_dim` 128), which is a LEADING slice. NOT spec §4 trap 2: that one is about which PAIRS rope rotates (GPT-J vs NeoX), this one about which HALF of the head it rotates, and both are numerically silent on a row spec §6.4 says has no oracle. W2 pins both values in an assertion so a re-published checkpoint cannot flip the layout silently; W2 consumes neither, because W2 writes no maths. W3 owes the slice | feature |
 | [#1857](https://github.com/mudler/vllm.cpp/issues/1857) | `SPEC-DFLASH2` | **The q>1 DFlash2 verify rides the PREFILL flash lane (`is_prefill = num_tokens > num_reqs`, `fa2_decode` hard-requires `num_tokens == num_reqs`), costing +9 ms/step from q=2 to q=9 on the #1574 K-ladder — the last attributed gap against SGLang (27.60 vs 25.07 tok/s at equal acceptance, ~109 vs ~122 ms/step, both paying the same ~20 ms draft).** W10 mirrors upstream's spec-as-decode: the reorder-threshold policy `1 + (parallel_drafting ? 2 : 1) * K` (`backend.py:718-736` @ `b389ac2946`, identical at the pin) classifies the runner's already-verified uniform verify length onto the decode class, the classification travels `CommonAttentionMetadata -> PagedAttentionArgs`, and a new ADDITIVE d256 launcher serves it with the exact presentation upstream `mha_fwd_kvcache` uses at seqlen_q>1 — batched split-KV, bottom-right causal against `seqused_k` (the draft mask with no new mask code), `set_params_splitkv` heuristic. The shipped q==1 arms and every unclassified batch stay dispatch-identical; `VT_FA2_SPEC_DECODE=0` restores the prefill route for a same-binary A/B. Wave spec [dflash2-spec-as-decode.md](specs/dflash2-spec-as-decode.md); the GPU step-time delta (the −8-9 ms claim), the GPU token gates and the first CUDA compile are owed there, operator-run | perf |
 | [#1853](https://github.com/mudler/vllm.cpp/issues/1853) | `LTX25-DIT-ATTN-FLASH` | **`PENDING` on a `dgx:gpu0` lease: the arithmetic-perturbation reference render that would make [#1743](https://github.com/mudler/vllm.cpp/issues/1743)'s criterion RELATIVE.** §11 of [`ltx25-dit-attn-flash.md`](specs/ltx25-dit-attn-flash.md) relocates the pixel verdict onto **correspondence** and **incoherence**, which discriminate a degraded render from a separated trajectory. It deliberately does NOT answer the other half of #1743: is the swap's divergence no worse than this pipeline's own divergence under an arithmetic perturbation of comparable size. That needs one further arm - the **naive** path at `768x448/49f`, seed `20260820`, on §10.7's pinned binary and checkpoints, with a bounded `+/-1` bf16 ULP dither injected at the DiT attention output at the `8.6e-05` to `3.7e-04` per-element flip rate §10.2 derives - after which `D(flash, naive) <= D(dither, naive)` is a bound with NO chosen constant. **No lease was authorised for #1743, so this is PENDING and not skipped.** The cross-build `baseline-20260820` vs `naive` figure (mean \|delta\| **9.452407**, LARGER than the swap's **6.414156**) is NOT that control and is not used as one: the binary lineage differs, so every other commit between `a50c57d69` and `3e2961ef0` sits inside it, which §10.8 already records. NOT FIXED IN FLOW: it needs a GPU lease this work does not have. Owned by row `LTX25-DIT-ATTN-FLASH` and listed under `## Owed` | bug |