perf(ds4): elide intermediate prefill logits

[cheese-cakee]

Status

Draft: fresh current-main qualification does not support merging this as a performance PR.

The old +4.1% dual-GPU and +5.7% Strix-only measurements were made against an older main. Since then, #640 and #647 merged and changed the relevant DeepSeek4 paths. The old gfx1151 MMQ tile edits also overlap newer code in current main and are no longer part of the reconciled candidate.

Remaining candidate scope

The current-main reconciliation keeps only the DeepSeek4 logits-elision work:

• avoid materializing vocabulary logits on intermediate prefill chunks that do not consume them;
• keep terminal and snapshot-boundary logits;
• distinguish logits and no-logits layer-major graph variants;
• allow repeated terminal-only prompts to populate the graph cache while protecting an established bulk no-logits graph.

A focused unit test was added for the graph-cache replacement policy. That local candidate has not been pushed because its fresh performance result is not acceptable.

Fresh current-main results

Base: current main at c994209a2c03c1bf3426924aba52b5edc05501ff
Local reconciled candidate: d38bb5dc5d883b37f3ba7822bf864d51c501070d

Matched Lucebox6 runs used the same ROCmFP2 model, placement, prompt, runtime settings, cache policy, two warmups, and five measured requests per arm. Main was bracketed around the candidate.

Workload	Pooled current-main median	Candidate median	Delta
Dual GPU, 401 tokens, chunk 512	178.43 tok/s	178.51 tok/s	+0.04%, neutral
Dual GPU, 2048 tokens, chunk 512	213.91 tok/s	198.49 tok/s	-7.21%
Strix only, 401 tokens, chunk 512	178.365 tok/s	178.27 tok/s	-0.05%, neutral

Every request produced the same exact response SHA-256:

cd5cb9fb5ac3c4f4007e8b41d117da21622439cd05c1728f3e82f90e4f869dad

The 2048-token candidate arm had high variance, but it did not show a positive signal and therefore fails closed. No performance claim is made from it.

Validation completed on the local reconciliation

• clean Release HIP build for gfx1151;gfx1201, HIP graphs enabled, ROCm 7.2.4;
• test_deepseek4_unit exited OK on physical gfx1201 and gfx1151;
• the two-device scratch subtest self-skipped in the deliberately single-device invocations and is not represented as passed;
• exact full-model response parity across all measured cells;
• git diff --check clean;
• KFD clean after testing.

Evidence root: /home/cheese/pr590-pr633-ready-20260825T1815Z
Manifest SHA-256: 26e936d5ad47d26cc9b45ce9c01d87d9e6e83543c0435faeaa2780678a4b497f

[cubic-dev-ai]

All reported issues were addressed across 4 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[Graffioh]

does this mean single-chunk prompts never populate the cache? eg --chunk 512, a 5–512 token prompt goes straight to the terminal/logits path, with logits_needed always true, so can_cache_dominant is false and every request rebuilds the layer-major graph

repeated single-chunk prompts with the same shape rebuild the layer-major graph on every request...was that case benchmarked, or is the build cost small enough that it does not matter?

extra: A snapshot boundary also sets need_logits to true because restoring that snapshot later requires the logits from that exact position.