record(BACKEND-ROCM): preserve the gfx1100 launch evidence

.agents/specs/gfx1100-tg200.md

@@ -0,0 +1,214 @@
+# Spec: GFX1100-TG200
+
+- Original campaign issue:
+  [#5](https://github.com/ghazni101/vllm.cpp/issues/5) (`ghazni101/vllm.cpp`)
+- Live landing owner:
+  [`BACKEND-ROCM` issue #2427](https://github.com/mudler/vllm.cpp/issues/2427)
+- Historical landing request:
+  [#2164](https://github.com/mudler/vllm.cpp/issues/2164) (deleted or
+  unavailable; retained only as historical attribution)
+- Immutable source: [`pr/1936`](https://github.com/mudler/vllm.cpp/pull/1936)
+  at `3a345b5ae5df7cf08f1383b6623b38db9a1335bd`
+- Gate prompt: [`tools/tg200-prompt.txt`](../../tools/tg200-prompt.txt)
+- Base: `019f66c1a` (upstream tip 2026-08-22; the branch carries one merge commit
+  pinning the base before the spec landed)
+- Pull request shape: one pull request for spec and implementation per stage
+  (developer decision 2026-08-21, recorded)
+- Predecessor: `.agents/specs/gfx1100-tg150.md` (#1651, branch
+  `row/GFX1100-TG150-SPEC`) and its consumed ladder
+  `.agents/specs/rocm-quant-gemm-bw.md` (#1586, branch
+  `row/ROCM-QUANT-GEMM-BW`); neither file is on this base yet
+
+## Scope
+
+Raise Qwen3.5-4B Q4_K_M text-generation throughput on the RX 7900 XTX
+(gfx1100, RDNA3, 24 GiB, `rocm-dev:7.14.0`) to **>= 200 tok/s** under the
+acceptance gate below, pure autoregressive greedy decode, single stream,
+batch 1. No MTP or speculative decoding in any measurement arm. Owning
+matrix row: `BACKEND-ROCM`.
+
+Feasibility is SETTLED and is not relitigated inside the campaign:
+
+- llama.cpp sustains ~200 tok/s on this exact checkpoint on this exact GPU
+  with a q8 KV cache. The target is demonstrated on identical hardware.
+- This engine's own lm_head kernel streams weights at ~598 GB/s on this
+  board (TG150 evidence): the memory system delivers.
+- Ceiling arithmetic: ~960 GB/s peak / ~2.2 GB per token ~= 430 tok/s
+  theoretical, so 200 tok/s sits at ~47% of peak.
+
+Therefore no stage may propose lowering the number, re-argue feasibility,
+or pad reports with activity in place of measured position.
+
+## Starting position (measured)
+
+Branch `row/ROCM-QUANT-GEMM-BW` head `094f60362` (5 commits, pushed),
+~27.6 tok/s wall, with the remaining measured budget from the TG150
+captures:
+
+| Item | ms/token |
+|---|---|
+| GdnPostConv (grid=1-block pathology) | ~4.1 |
+| dispatch gap (host-bound; HIP-graph territory) | ~3.0 |
+| GdnScan | ~1.1 |
+| residual quant-GEMM arms < 300 GB/s effective | remainder |
+
+The pattern across every kernel examined so far: 10-100x waste from fixed
+launch costs, sync storms, or serial walks. Expect the same under the next
+rock.
+
+**Base delta matters**: upstream tip `019f66c1a` already lands three levers
+in exactly this budget -- `f4ccabbb4` (GdnPostConvK value_dim copy off one
+thread), `c020347a7` (VT_ATTN_DECODE_D128 default-on for ROCm d=128 GQA
+decode), `f38c1edc4` (decode-skinny GEMMs to ported wvSplitK) -- none of
+which existed when the 27.6 tok/s position was measured. S1 prices the tip
+before any new lever is chosen; the table above is the PRE-MERGE budget and
+is not carried forward as current.
+
+## Acceptance gate
+
+Median of >= 5 repetitions, idle host, gpu-ctl lock held for the whole
+window, batch 1, one ~512-token real prompt, 256 generated tokens, greedy
+(`--temperature 0 --seed 0`), through the production entry point
+(`examples/vllm-cli`). Recorded axes: output tok/s (the gated number),
+steady-state TPOT, peak VRAM. A run under co-tenancy is provisional and
+never satisfies this gate. Token identity: the 256-token output on the gate
+prompt must be byte-identical to the pre-campaign output on the same build
+config for every lever claiming bit-exactness; any lever that changes
+reduction order records near-tie adjudication per the ratified band
+doctrine (`.agents/specs/rocm-m4-oracle.md`) rather than asserting identity
+it cannot show.
+
+## Working rules (carried from developer preferences)
+
+1. Never push or merge to `main` on either remote. All work lands on
+   `row/*` branches pushed to `ghazni101/vllm.cpp` only.
+2. Every GPU command goes through `/home/ghazni/gpu-coord/gpu-ctl`
+   (`run`/`reserve`/`status`). Another agent shares this GPU; the lock
+   protocol already caught one real serialization gap.
+3. Correctness gates are non-negotiable: op-level NMSE vs CPU oracle,
+   token-coherence sanity on every A/B, near-tie adjudication recorded when
+   reduction order changes. Perf wins that break the integer core do not
+   land.
+4. Every change is A/B'd on the acceptance workload before it counts.
+   Medians, not best-case runs.
+5. Attribute before optimizing: one rocprofv3 capture per head, per-kernel
+   budget table, attack the top item. No speculative rewrites.
+
+## Stages
+
+| Stage | Content | Exits when |
+|---|---|---|
+| T1 | Fresh attribution re-take at the NEW base on the EXACT gate workload: rocprofv3 both sides of each candidate lever, wall vs GPU-busy split, per-family shares, dispatches/token; reconcile against the pre-merge budget above | The T2+ order below is confirmed or rewritten with numbers |
+| T2 | Dispatch-collapse: HIP graph capture of the steady decode step, or `vt::FusedChain` recipe reduction where capture cannot reach | Wall/token approaches GPU-busy/token; gate re-measured |
+| T3 | GDN family decode levers ranked by T1 (post-conv, scan, state ops), consuming whatever `f4ccabbb4` left on the table | Measured win adopted or lever closed with numbers |
+| T4 | Residual quant-GEMM arms toward >= 80% peak effective streaming (continues #1586's ladder past where TG150 stopped) | Rate reached or share-weighted projection stops ranking it first |
+| T5 | bf16 hipBLASLt arms: algo-policy A/B at decode shapes; wvSplitK reconciliation at this model's shapes | Measured win adopted or lever closed with numbers |
+| T6 | Acceptance gate run + landing: `docs/USAGE.md` weights provenance, `docs/BENCHMARKS.md` row, this spec's `## Outcome` | Gate >= 200 tok/s median, or the campaign reports the measured position with the next traceable hypothesis named |
+
+Stage order after T1 is T1's output, not this table's.
+
+## Correctness policy
+
+- The keep-quant integer core stays bit-exact vs CPU;
+  `tests/vt/test_rocm_quant_dot.cpp` runs unchanged as the gate for every
+  quant-path lever.
+- Token coherence asserted on every A/B; byte-identical outputs claimed
+  only for bit-exact levers.
+- Any reduction-order change records near-tie adjudication with
+  teacher-forced logprob gaps per the ratified band doctrine; a raw
+  divergence count is never presented as a quality score.
+- No checker is weakened; a gate that goes red names the repair.
+
+## Risks
+
+- R1: the pre-merge budget table misprices the tip (the three landed
+  upstream levers change the ranking). T1 exists to price this first.
+- R2: HIP graph capture may refuse a step containing a host-dependent op;
+  fallback is FusedChain recipe reduction and a partial capture is
+  recorded, not hidden.
+- R3: 200 tok/s may require levers beyond kernels (scheduler, sampler
+  sync). The campaign reports the measured position honestly; no ceiling is
+  declared and a shortfall names the next traceable hypothesis.
+
+## Tests
+
+- `tests/vt/test_rocm_quant_dot.cpp` unchanged (841 assertions, 19 cases,
+  fresh-build count at the issue-#9 repair) for every quant-path lever.
+  Provenance: the earlier "132,094 assertions" figure came from a stale
+  7.14-era binary whose lattice no longer matched the source. Only a fresh
+  configure and build in the current container is authoritative.
+- Focused gate per stage: `ctest -R 'rocm|cross_device|quant'` in the 7.14
+  container under the gpu-ctl lock.
+- The acceptance gate itself is T6's test.
+
+## Owed
+
+- Any improvement applicable to the CUDA sibling is recorded in the W1
+  spec's owed list, never ported silently into this campaign.
+- Kernel-matrix / backend-matrix row updates ride each landing commit.
+- `docs/BENCHMARKS.md` and `docs/USAGE.md` updates ride T6 (and any stage
+  that changes a user-visible command).
+
+## Stop conditions
+
+- `NEEDS_DECISION`: a stage needs authority beyond what is recorded
+  (push/merge beyond the granted draft-PR flow, new hardware, new
+  checkpoints).
+- 20 failed attempts within one stage: stop, report findings and the
+  measured ceiling hypothesis for that stage. Ambiguity needing a user
+  decision: halt and surface.
+
+## Now
+
+Issue [#2427](https://github.com/mudler/vllm.cpp/issues/2427) owns this
+records-only landing. Historical upstream issue
+[#2164](https://github.com/mudler/vllm.cpp/issues/2164) is deleted or
+unavailable; it remains only as attribution for the original integration
+request. The campaign records come from `pr/1936` at
+`3a345b5ae5df7cf08f1383b6623b38db9a1335bd`. This integration contains the
+specification, 17 evidence files, and the exact
+[`tools/tg200-prompt.txt`](../../tools/tg200-prompt.txt) input. It contains none
+of pull request #1936's product changes. The unmerged campaign's opt-in arms,
+default changes, and product changes are not reachable from this tree. The
+measured position and next hypothesis that follow are historical evidence from
+the source commit. They are not a current-main benchmark.
+
+`ACTIVE`. Measured position before T21: ~103 tok/s (T18 idle-host gate
+100.46 tok/s + T18 v_dot4 +2.7% matched-load). T21's measured +3.9% projects
+the idle-host position to ~107 tok/s. Adopted levers: T5a shared quant-body
+vectorization (+23%), T5b d128 f32-Q DecodeGqa arm (+13.5%), T6a cooperative
+GDN scan (+4.6%), T6b cooperative attn preamble (+4.6%), T8 cooperative
+rmsnorm row (+3.2%), T9 cooperative gated norm (+2.6%), T10 warp postconv
+(+4.7%), T11 row-split scan (+3.2%, BIT-IDENTICAL), T14 row-split argmax
+(−71%, BIT-IDENTICAL), T16 YTILE=4 default (+1.8% contended, +8.1% idle),
+T18 v_dot4 instruction selection (+2.7%, BIT-IDENTICAL), and T21 row-permuted
+GDN keep-quant (+3.9%, ADOPTED). T21's `VT_GDN_ROWPERM_KEEP_QUANT` gate is
+default-enabled at 1.
+Closed negative: T5c MMVQ nontemporal, T7 COALK wash, T12 gated-quant
+fusion, T13 async server wash, T15 LDS bank conflicts, T17 v_dot2
+memory-bound, T19 kGemvWarps block-limited, T20 full-warp cooperative GEMV
+(kernel 2.4-3.1x on large grids but engine wash — Q4_K dominant path is
+launch-overhead-bound at small grids; evidence
+`docs/bench-evidence/gfx1100-tg200-t20-full-warp-gemv-wash-20260826.md`).
+Failed-attempt ledger: 8 of 15.
+
+Budget table (pre-T20, ~103 tok/s, ~9.7 ms/tok wall):
+KQuantGemvMmvqK<Q4_K> 2.46 ms/tok (25%), wvSplitKSml 2.32 ms/tok (24%),
+KQuantGemvMmvqK<Q6_K> 1.20 ms/tok (12%), RmsNormRowCoop 0.754 ms/tok (8%),
+QuantizeQ8KK 0.544 ms/tok (6%), other ~1.3 ms/tok (13%), total kernel
+~8.58 ms/tok (88%). Weight read floor 4.21 GB/tok = 4.38 ms/tok at 960 GB/s.
+Overhead above floor: ~4.2 ms/tok — launch overhead, sync, idle gaps.
+
+Next attack: the overhead is the bottleneck, not individual kernel internals.
+T20 proved kernel micro-optimization is exhausted for the dominant paths.
+The path to 200 tok/s (5.0 ms/tok) requires closing the 4.2 ms/tok overhead
+gap: HIP graph capture (T2), kernel fusion, or persistent kernels. A fresh
+rocprofv3 attribution capture with dispatch counts per token is the next
+step to price the overhead precisely.
+
+Owed before ANY default flip of the opt-in arms (GQA4 / GDN_SCAN_COOP /
+GDN_SCAN_SPLIT / PREAMBLE_COOP / RMSNORM_ROW_COOP / GDN_NORMGATED_COOP /
+GDN_POSTCONV_COOP): teacher-forced logprob-band ceremony per
+`.agents/specs/rocm-m4-oracle.md`. The campaign reports into #5; each
+stage lands as its own `row/GFX1100-TG200-*` branch + draft PR per the
+recorded push authority.

docs/bench-evidence/gfx1100-tg200-levc-attribution-20260824.md

@@ -0,0 +1,81 @@
+# GFX1100-TG200 — Lever C attribution: standalone `QuantizeQ8KK` launch sites -> producers
+
+Committed BEFORE any kernel code (Lever C contract step 1). Evidence source:
+rocprofv3 rocpd capture `/work/levc-prof/bdb445f9ac06/79723_results.db`
+(full-stack config, TG200 lever-C pricing capture, acquired+released under
+gpu-ctl at 01:56Z 2026-08-24). Model: Qwen3.5-4B-Q4_K_M
+(sha256 `00fe7986ff5f6b463e62455821146049db6f9313603938a70800d1fb69ef11a4`,
+32 blocks = 24 GDN + 8 full-attn at interval 4; H=2560).
+
+## Method
+
+Three signals, same discipline as the T4a evidence §15.1:
+
+1. **Geometry decoding.** The rocpd `grid_size_x` column records HIP global
+   work-items in x (`grid.x * block.x`), not blocks. Cross-checks: the lm_head
+   GEMV shows 1986560 = 62080 blocks x 32 lanes (N=248320, 4 warps/block);
+   every `QuantizeQ8KK` dispatch shows 128 = 1 block x 128 threads, i.e. EVERY
+   decode-token activation quant launches a SINGLE BLOCK (`m*nsb <= 128`).
+   Pure launch pathology confirmed: mean duration ~48-50 us regardless of
+   K (48.2-50.1 us across all seven site classes below).
+2. **Step isolation.** One steady-state decode step = dispatch window between
+   consecutive `ArgmaxK` launches (step 100 of 256 used; identical structure
+   at steps 50/150/200).
+3. **Producer adjacency.** Each `QuantizeQ8KK` immediately precedes its
+   consumer GEMV; each consumer's activation tensor is produced by the kernel
+   immediately upstream of the quant (op-order correlation), cross-checked
+   against the forward call sites in `src/vllm/model_executor/models/
+   qwen3_5.cpp` / `qwen3_5_gguf_weights.cpp`.
+
+## Per-step census (97 standalone `QuantizeQ8KK` launches/token)
+
+| # | site | producer of the quantized activation | m x K (nsb) | N (consumer) | weight fmt | launches/tok | mean us |
+|---|------|--------------------------------------|-------------|--------------|-----------|--------------|---------|
+| 1 | FFN gate_up fused matvec (`qwen3_5_gguf_weights.cpp` :1211 row-concat, one kMatmulBTQuant) | **RmsNormRowKernel** (post-attention input layernorm) | 1x2560 (10) | 18432 (= 2x9216) | Q4_K | 32 (24 GDN + 8 attn) | 48.6 |
+| 2 | attn q_proj | **RmsNormRowKernel** (full-attn input layernorm) | 1x2560 (10) | 8192 | Q4_K | 8 | 48.4 |
+| 3 | attn k_proj | **same norm output as #2** (re-quantized by its own standalone launch) | 1x2560 (10) | 1024 | Q4_K (5 layers) | 5+3* | 47.5-48.1 |
+| 4 | attn v_proj | **same norm output as #2** | 1x2560 (10) | 1024 | Q6_K (5) / Q4_K (3)* | 8 | 47.5-48.1 |
+| 5 | attn o_proj | PagedAttnDecodeGqaF32Qi (attention output — NOT a norm) | 1x4096 (16) | 2560 | Q4_K | 8 | 49.5 |
+| 6 | FFN down_proj | SiluMulK (NOT a norm) | 1x9216 (36) | 2560 | Q4_K (16) / Q6_K (16) | 32 | 50.0 |
+| 7 | lm_head | **RmsNormRowKernel** (final norm) | 1x2560 (10) | 248320 | Q6_K | 1 | 48.2 |
+
+\* the k/v format split across the 8 full-attn layers is mixed in this GGUF;
+the capture resolves 11 fmt-0 and 5 fmt-2 N=1024 quants/step; the exact
+per-layer tensor formats live in the GGUF tensor map (T4a evidence §15).
+
+Reconciliation: 32 + 8 + 8 + 8 + 32 + 1 = 89... resolved against observed
+context pairs — RMS->G0(18432)=32, RMS->G0(8192)=8, G0(8192)->G0(1024)=8,
+G0/G2(1024)=8, ATTN->G0(2560)=8, SILU->G0/G2(2560)=16+16, RMS->G2(248320)=1,
+total **97**. `RmsNormRowKernel<bf16>` count cross-check: 65 launches/step =
+2x24 GDN + 2x8 attn + 1 final = 65 exactly.
+
+## Fusability verdict (this lever)
+
+- **Fusable via RmsNormRowKernel epilogue: 57/97 launches/tok** (sites
+  1, 2, 3, 4, 7). Sites 3+4 re-quantize the SAME normalized row already
+  written for site 2's scratch — one producer record serves all three
+  consumers (identical ptr, m, K, dtype, stream).
+- Not fusable this round: 40/97 (sites 5, 6; producers are attention output
+  and SiluMul). Owed: a SiluMulK epilogue would take another 32/tok.
+- **RmsNormGatedK finding:** the gated RMSNorm (`RmsNormGatedK`, 24
+  launches/tok) feeds ONLY the bf16 `wvSplitKSml` out_proj matvec — it has
+  ZERO QuantizeQ8KK consumers in this model. Extending the fused epilogue to
+  the gated sibling buys nothing here; recorded as owed-with-reason rather
+  than time-boxed work.
+
+## Discrepancy note (honest reporting)
+
+The Lever C assignment quotes "43 standalone launches/token". THIS capture at
+bdb445f9ac06 measures **97/tok** (~4.7 ms/tok at ~49 us each). The 43 figure
+is consistent with an arm mix where the T4a fused-fold sub-arm
+(VT_GEMV_MMVQ_FOLD_MAX <= 512) absorbs some sites, or with counting distinct
+site CLASSES; neither applies to this capture (zero fused-fold kernels in the
+decode window). The lever thesis is unchanged and stronger: single-block
+launch pathology at ~49 us per launch.
+
+## Fusion-seam gate finding
+
+The change enriches a producer KERNEL behind VT_NORM_QUANT_FUSED (opt-in);
+no model .cpp edit, no hand-call fusion, no new recipe. Per
+scripts/check-fusion-consistency.py scope (model-forward floors only), the
+gate is not tripped; verified green post-change in the evidence file.

docs/bench-evidence/gfx1100-tg200-quant-cache-negative-20260822.md

@@ -0,0 +1,52 @@
+# GFX1100-TG200 — negative result: pointer-keyed quantized-activation cache
+
+Date: 2026-08-23. Follows `gfx1100-tg200-t2a-20260822.md`.
+
+## What was tried
+
+A per-stream cache in front of `QuantizeQ8KK` keyed on
+`(activation ptr, row stride, activation dtype, m, nsb, weight dtype)`:
+the first kMatmulBTQuant call over a given activation launches the quant
+kernel; later calls with the same key reuse the scratch buffer.
+
+## Result: REJECTED — unsound under the block-recycling allocator
+
+- First cut (pointer-only key): throughput rose to ~45 tok/s median, but the
+  generated text degenerated into repeated garbage (`heimerheimer...`) — the
+  DevicePool recycles activation blocks across steps, so the same pointer
+  carried different content on the next step and stale quantized data was
+  served. Correctness gate caught it exactly as designed.
+- Second cut (epoch keying via vt::BumpQuantEpoch/CurrentQuantEpoch, bumped
+  once per model forward): still degenerate. Within ONE step the pool hands
+  the SAME address to DIFFERENT activations (DBuf freed and re-allocated mid-
+  forward), so even intra-step pointer identity does not imply content
+  identity.
+- Reverted completely; revert verified by coherent output on the acceptance
+  workload (the run reproduces the T1a-style coherent transformer explana-
+  tion). Both cuts were never committed.
+
+## Why this matters for the campaign
+
+1. The "129 QuantizeQ8KK launches/token" cost is real GPU-busy time (~59us
+   each profiled), but it CANNOT be eliminated by result-caching without a
+   content-identity signal the allocator does not provide.
+2. The sound levers for this budget are structural, not caching:
+   - merge gate+up into one keep-quant GEMM (halves the quant sites),
+   - MMVQ-style dequant-in-register decode GEMV (removes the separate quant
+     kernel entirely, following SGLang's mmvq.cuh pattern),
+   - ROCm decode-graph capture (removes the launch overhead that makes each
+     tiny kernel cost ~59 us of queue time).
+3. The probe instrumentation (VT_MATMUL_BT_QUANT_PROBE) also stays out of
+   the tree; it served its one-shot purpose.
+
+## Measured (for the record)
+
+| Arm | median tok/s | notes |
+|---|---|---|
+| baseline (T1a) | 40.65 | idle host |
+| cache v1 (ptr key) | 45.0 | DEGENERATE OUTPUT — rejected |
+| cache v2 (epoch) | 44.9 | STILL DEGENERATE — root cause above |
+| reverted build | coherent | matches T1a-class output |
+
+Per working rule 3: perf wins that break correctness do not land. This is
+the documented rejection, not a silent drop.