perf(ds4): accelerate ROCm decode and long-context selection

[davide221]

Status

Draft only. This is a stacked, hardware-targeted optimization series and is not ready to merge.

Base: #664 (codex/ds4-pflash-long-context). All full-model measurements retain DeepSeek V4 Flash's six routed experts. The historical q=4 result below means speculative verification width four, not four experts.

What changed

• Route the exact gfx1151 HC q=4 projection through MMVF instead of hipBLAS.
• Tune ROCmFP2/ROCmFP3 MoE loads and launch shapes for RDNA Wave32.
• Build long verify masks as negative spans and upload only those spans after a device clear.
• Add an exact hierarchical top-512 selector for long direct-indexer rows.
• Add an exact direct-top-k decode attention specialization that skips a redundant visibility scan, mask loads, and shared bound atomics when all selected rows are already known visible.
• Add selector parity/microbench coverage and expand ROCmFP2/3 tests.
• Add a shared, per-request adaptive speculative-width controller for every linear-chain decoder: generic/layer-split DFlash, DS4/DSpark, Gemma4, Laguna, Qwen3.5, and Qwen3.5-MoE.

Current-head autoregressive result

This is plain autoregressive decode: pflash/drafting off, no speculative verification, expert top-k 6, 32,767 prompt tokens, 256 output tokens, gfx1151, ROCm 7.2.

32K AR run	Server decode	Output SHA-256
Current-head control	15.82 tok/s	a91b1261...506ec43
Direct-visible specialization	16.03 tok/s	a91b1261...506ec43

That is a measured +1.3% structural AR gain with an identical 256-token response. The specialization remains opt-in with GGML_DS4_FA_DIRECT_TOPK_VISIBLE=1 while the branch is draft.

A follow-up that retained the exact-zero value envelope measured 16.00 tok/s, so the original profiled specialization was kept.

Matched LuceGraph evidence

The normal and ROCprof runs use the same input and exact response hash. Profiler throughput is not used as the performance claim.

• Profiling worktree: ecae4618198a7b0ac42bda8a99b2cbe4dd803b85 plus the candidate fattn.cu, byte-identical to commit 2f13618.
• Normal AR: 15.82 -> 16.03 tok/s.
• ROCprof twin: 15.15 -> 15.36 tok/s.
• Marked decode span: 16.896 -> 16.665 s.
• Marked kernel-busy time: 14.482 -> 14.296 s.
• The 201.102 ms visibility-bounds kernel disappears. The specialized attention kernel adds 32.326 ms of value work, leaving a net kernel saving of about 169 ms plus about 45 ms less non-kernel time.
• Complete JSONL footers, zero dropped records, matched inputs, and exact output hash.

Retained captures:

• /data/lucegraph-ds4-ar-ecae461-runs/baseline-ar-32k-normal
• /data/lucegraph-ds4-ar-ecae461-runs/baseline-ar-32k-rocprof
• /data/lucegraph-ds4-ar-ecae461-runs/candidate-direct-visible-ar-32k-normal
• /data/lucegraph-ds4-ar-ecae461-runs/candidate-direct-visible-ar-32k-rocprof

The remaining 256-token profiled round is dominated by generic dense ROCmFP4 matvec (4.525 s), routed ROCmFP2/3 mix kernels (2.909 s), indexed attention (1.760 s), and 2.414 s of non-kernel gaps. The published 38.41 tok/s Vulkan point used DSpark speculation, so it is not an AR baseline; a matched Vulkan AR-only run is still required.

Rejected current-head experiments

• HIP graph replay: warm 4K AR regressed 18.53 -> 18.32 tok/s.
• ROCmFP4 MMVQ two/four waves per row: flat versus one wave.
• Full Wave32 per-row attention dot: -5.2%.
• Attention row-split, fused MoE combine, and q4 VDR=4: <=0.7%, inside the rejection/noise threshold.
• Forced hipMalloc instead of managed UMA: model load stalled near 98 GiB GTT on this 128 GiB system.

None of these rejected variants is included.

Historical speculative result

The earlier q=4 verification run is separate from the AR result above:

Workload	Before this stack	This branch	Change
122,879-token prompt, 256 output tokens	22.700 tok/s	26.714 tok/s	+17.7%
Historical Lucebox ROCm chart at ~123K	16.20 tok/s	26.714 tok/s	+64.9%

The matched 32K speculative normal capture measured 32.026 tok/s. Its ROCprof twin produced the identical response hash at 29.978 tok/s. These results predate the general adaptive-width commit; no adaptive-width speedup is claimed from them.

Adaptive speculative width

The controller starts conservatively, backs off from the exact accepted prefix after a rejection, and probes one row wider after a fully accepted block. Full acceptance is censored feedback rather than an exact sample, avoiding a narrow-width trap.

• Enabled by default; DFLASH_ADAPTIVE_SPEC_WIDTH=0 restores prior fixed-width behavior.
• Explicit backend width overrides still win.
• DS4 q4 retains its calibrated confidence policy and uses acceptance feedback as fallback. DS4 q5 adapts from acceptance feedback rather than extrapolating the q4 confidence head.
• DDTree is unchanged because it controls a branching topology/budget rather than a linear verification width.

Policy reference: LaurentZuijdwijk/llama.cpp@ca26169911e8

Verification

• Byte-for-byte match between the committed kernel and the profiled candidate.
• Exact full-model hashes at 4K, 8K, and 32K during the candidate sweep.
• HIP_VISIBLE_DEVICES=1: test_rocmfpx_mmq and deepseek4_unit pass on Strix Halo.
• Shared adaptive-width controller: 8/8 focused unit cases.
• Strix ROCm suite: test_server_unit 431/431 before the final DS4 policy-selection tweak.
• dflash_server builds successfully on gfx1151.

Running the two focused GPU tests without isolating a device currently segfaults during multi-GPU initialization on lucebox8 (R9700 + Strix); the matching Strix-only invocation passes.

Controls

• GGML_DS4_FA_DIRECT_TOPK_VISIBLE=1 enables the new AR attention specialization.
• DFLASH_ADAPTIVE_SPEC_WIDTH=0 disables acceptance-feedback width adaptation globally.
• DFLASH_GFX1151_HC_MMVF_Q4=0 opts out of the gfx1151 HC MMVF route.
• DFLASH_DS4_INCREMENTAL_VERIFY_MASK=0 opts out of incremental mask upload.
• GGML_DS4_TOPK_BLOCK_RADIX=1 enables the hierarchical selector.

Before merge

• Qualify the direct-visible shape guards across wrap boundaries and non-default cache settings before deciding its default policy.
• Qualify adaptive q5 and each non-DS4 backend with correctness hashes and matched fixed/adaptive runs.
• Repeat long-context and power/clock-controlled runs.
• Continue structural work on the generic ROCmFP4 matvec, routed FP2/3 mix, and launch gaps identified by the current trace.
• Diagnose the multi-GPU CTest initialization crash separately.

[davide221]

lucebox8 gfx1151 A/B (ecae461)

Matched cold-process runs; only DFLASH_MOE_FUSED_COMBINE=0|1 changed. Both
used the 0731 ROCmFPX MIX target, DSpark draft, q4 K/V, sparse prefill, full
6/6 routing, q=4 cap, and 256 output tokens.

Prompt	Metric	OFF	ON	Delta
32,768	server prefill (median, n=2)	184.56	191.23 tok/s	+3.62%
32,768	client decode (median, n=2)	31.771	31.938 tok/s	+0.53% (noise)
122,879	server prefill	129.46	133.43 tok/s	+3.07%
122,879	client decode	26.875	26.540 tok/s	-1.25% (no gain)

Correctness/integrity:

• All normal runs returned the identical expected response hash
  a91b12613458d22a3fbf6d66ff2fba42124b46ea0e032f68c3564cc0c506ec43.
• Captures completed with zero dropped steps, requests, or token bursts.
• 431/431 server tests and 5/5 focused profiling/GPU tests passed, including
  GPU numerical equivalence for ggml_moe_combine.

Matched ROCprof at 32K confirms the intended replacement: roughly 3,675 old
reduce/multiply launches disappear and 3,643 moe_combine_kernel launches are
added, saving about 1.36 s of sampled GPU work. Normal runs—not ROCprof—are the
throughput source.

Conclusion: reusable prefill win, decode-neutral. Keep this draft and
opt-in while the next gfx1151 work targets dominant quantized matmul and
grouped compact attention kernels.

[pepuscz]

Thank you, @davide221, for your work on Lucebox, for pushing this optimization series, and for acting on the earlier Strix Halo feedback. We qualified PR #667 head 2f136181b0b387592e413a81dedc0cd79991e128.

Test configuration

• AMD Ryzen AI Max+ 395, 128 GiB unified memory
• Lucebox ROCm/ROCmFPX MIX target
• Q4RMFP4 DSpark drafter, six routed experts, speculative verification width q=4
• Q4_0 K/V cache, sparse prefill, 131,072-token context, one prefix-cache slot
• 120/120/120 W package limits
• Baseline: our previous Lucebox ROCm ROCmFPX setup, built from Lucebox f686c447f067a04ea100a996e4c826e8cc4decc1 with four pinned patches for 128K bounds, gfx1151 MMQ tuning, ROCmFPX dense matvec, and current DeepSeek V4 reasoning behavior. The exact Ansible configuration pins every patch and artifact hash, build input, runtime setting, and hardware policy.
• Candidate source: PR #667 head 2f136181b0b387592e413a81dedc0cd79991e128
• Identical model, prompts, generation limits, runtime options, and hardware profile

Matched results

Workload	Previous Lucebox ROCm ROCmFPX setup	PR #667	Change
2,048-token prompt, up to 510 output tokens — median decode	30.50 tok/s	38.60 tok/s	+26.6%
122,879-token prompt — cold prefill	133.223 tok/s	145.682 tok/s	+9.4%
122,879-token prompt, 256 output tokens — decode	18.10 tok/s	26.50 tok/s	+46.4%
Deterministic quality suite	30/30	29/30	one regression

The PR #667 build also fit the full 131,072-token configuration in 128 GiB. Minimum effective non-CMA headroom was 4.984 GiB. There was no OOM, GPU reset, server restart, or kernel memory-pressure event. For comparison, the stacked PR #664 base could not complete the same exact-128K fit because of a global-memory OOM.

Quality finding

Quality was evaluated with client_test_runner.py and all 30 fixtures in the Lucebox harness at commit 90f85fa. The only failed fixture was math_10: it asks for f(84) in the specified recursive integer function, with expected boxed answer 997.

The previous setup stopped naturally after 1,974 generated tokens and returned \boxed{997}. PR #667 reached the 2,048-token output limit and ended mid-formula without returning the answer. The other 29 fixtures passed the corrected quality gate.

For independent reproduction, our v1.5.0 release packages the exact Ansible configurations for both tested Strix Halo systems together with the benchmark definitions, pinned revisions, result data, and comparison chart.

[davide221]

Split-KV MLA follow-up (0f1264c), still draft and opt-in.

• Adds a native HIP two-way split-KV schedule for indexed D512 MLA verification (q <= 8), enabled with GGML_CUDA_MLA_SPLIT_KV=1.
• Keeps all six routed experts and the same adaptive DSpark configuration.
• 8K normal, 3-run medians: 36.239 -> 36.996 tok/s (+2.1%); target verify 108.1 -> 105.8 ms (-2.1%). The final decode-only build separately confirmed 37.628 tok/s and 104.0 ms.
• 32K normal, 3-run medians: 33.676 -> 34.671 tok/s (+3.0%); target verify 116.7 -> 113.4 ms (-2.8%). The final decode-only build separately confirmed 34.415 tok/s and 115.7 ms against the 33.645 tok/s / 118.3 ms matched control.
• Every run produced the identical response SHA-256 0f785a7f...d104d194.
• Decode-only guard leaves prefill flat: 8K 207.11 vs 207.39 tok/s; 32K 183.51 vs 183.86 tok/s, and removes the prototype extra ~0.9 GB allocation.
• Matched ROCprof: indexed-attention cost per layer fell 0.293 -> 0.127 ms (-56.5%), saving ~3.5 ms per verification round. Median marked verify range fell 106.30 -> 102.60 ms. Split launches were exactly 672 = 32 rounds x 21 attention layers; none occurred in prefill.

Retained captures:

• /data/lucegraph-ds4-splitkv-runs/rocprof-control-8k
• /data/lucegraph-ds4-splitkv-runs/rocprof-split2-decodeonly-8k

Rejected: four-way split (repeat client deltas were only +0.17%/+0.32%) and the earlier 2-wave ROCmFP MMQ tile (+0.07%).