perf(ds4): enable sparse gate-up MMQ prefill

[dusterbloom]

Summary

• default the existing qtype-106 ROCmFP2 MIX MMQ path only for the routed gate/up pair on gfx1151
• leave qtype-106 down projections and every qtype-105 projection on the existing generic policy
• preserve the existing RDNA4 opt-in and DFLASH_DS4_MIX_MMQ_PREFILL=0 opt-out
• add no flag, kernel, graph operation, or model-specific scheduler

The role boundary is structural: ggml_cuda_try_fuse_mul_mat_glu() sees the matched routed gate/up pair. Down is a later independent MUL_MAT_ID and never enters this default.

Result

Ryzen AI Max+ 395 / gfx1151, ROCm 7.2.4, monolithic sparse prefill, chunk 512, all six routed experts, caches disabled:

Prompt	Baseline	This branch	Gain	Lower time
8K	109.548 tok/s	146.474 tok/s	+33.71%	25.21%
16K	108.691 tok/s	143.697 tok/s	+32.21%	24.36%
64K	100.128 tok/s	128.580 tok/s	+28.42%	22.13%
128K	85.236 tok/s	106.166 tok/s	+24.56%	19.71%

The 8K row is a same-binary default-versus-explicit-opt-out A/B. The longer baseline rows are retained from the frozen clean-environment campaign on the same hardware, model, chunk, and request family; every final role-scoped candidate was rerun after this fix.

Why this path

An 8K request-scoped profile attributed 45.28% of GPU time to MIX dequantization and 17.54% to its GEMMs. Flash attention was 12.87%; the Lightning Indexer was 0.67%.

The faster type-wide experiment was rejected. It admitted qtype-106 down projections too, exceeding the qualified gate/up boundary. Enabling both MIX types was also rejected after failing the retained long code-audit prompt.

Dispatch telemetry over the final 8K run recorded all 688 down projections on the native fallback: 448 qtype-105 and 240 qtype-106. Gate/up was consumed by the paired fusion. Explicit DFLASH_DS4_MIX_MMQ_PREFILL=0 restored the 109.548 tok/s baseline.

Verification

• dflash_server builds on gfx1151 with -j4
• test_rocmfpx_mmq: all cases pass on gfx1151
• test_rocmfp3_mix_registry: pass on gfx1151
• long memory prompt: exact expected answer
• long code-audit prompt: correct consolidation diagnosis
• no graph-capture eligibility change: native wide down remains truthfully graph-ineligible

One file, 8 insertions, 1 deletion.

[cubic-dev-ai]

1 issue found across 1 file

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/deps/llama.cpp/ggml/src/ggml-cuda/mmq.cu">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml-cuda/mmq.cu:668">
P1: When a model contains qtype-106 down experts, this default also sends the down projection through MMQ, contrary to the gate/up-only production choice. Restrict the default qualification to the gate/up dispatch or pass the projection role into the predicate so down projections retain their prior path.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/deps/llama.cpp/ggml/src/ggml-cuda/mmq.cu">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml-cuda/mmq.cu:668">
P2: This case drops RDNA 4 and removes the opt-out. On main, gfx12xx (RDNA 4) could use qtype-106 MMQ by setting DFLASH_DS4_MIX_MMQ_PREFILL; now `GGML_CUDA_CC_IS_RDNA3_5(cc)` makes it never selectable on RDNA4 even with the env set, contradicting the PR's stated 'RDNA 3.5/4' scope. It also makes qtype-106 MMQ unconditional on gfx1151 with no way to disable it, where the env var previously controlled it. If RDNA4 was excluded because it is unvalidated (the comment says only RDNA 3.5 is qualified), gate the RDNA4 removal on that explicitly, and keep the upcoming MIX mmq_info GGML_ASSERT in mmq.cu guarded by a real opt-out so an unregistered model or a regression does not hard-abort prefill.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>