feat(ds4): add concurrent serving on Strix Halo and R9700 + Strix

[Graffioh]

Summary

Adds DeepSeek4 paged continuous serving for up to six concurrent lanes on the two qualified resident HIP deployments:

• monolithic Strix Halo (gfx1151), with every expert on the target; and
• in-process R9700 (gfx1201) target plus Strix Halo (gfx1151) route-level expert parallelism.

It also lets the existing DSpark q=4 path remain resident beside the paged engine. A request admitted alone after the 20 ms idle coalescing window uses the classic speculative singleton path; a coalesced cohort uses the paged autoregressive engine.

Qwen concurrency remains unchanged at up to 64 slots.

Note

The absolute performance table below was measured at retained candidate 92b53237, four commits ahead of the current GitHub PR head dcf895cc. Those commits must be pushed to codex/ds4-concurrency-strix before this is merge qualification rather than candidate qualification.

System boundary

Request shape	Execution path	Concurrency
One request after idle admission	Contiguous target cache + DSpark q=4	1
Coalesced or already-active requests	Persistent paged cache + gathered autoregressive graph	2-6

The DSpark and paged target caches are independent. DSpark is restricted to the idle singleton path; it is not mixed into a paged cohort.

Implementation

• Keeps each sequence's raw MLA rows, compressed MLA rows, indexer state, length, and block table in a persistent 128-token paged cache.
• Lowers each scheduler plan into one exact gathered graph with up to six independent lanes. Decode rows share the weight pass while each selected sequence advances by one token.
• Reuses the existing in-process MoE split only for the qualified R9700-primary plus Strix-secondary topology. This is route-level expert parallelism, not a layer split or remote target shard.
• Charges the heterogeneous paged cache and full-context prefill staging allocation against the R9700 before resident-expert placement.
• Uses one shared DeepSeek4 slot limit across CLI validation, feature gates, scheduler admission, backend state, and gathered graph construction.
• Keeps cancellation, slow-client isolation, and fair continuous admission in the shared HTTP scheduler.

The retained optimization stack:

• stops padding the model-default top-6 tail;
• skips non-owner expert routes by default;
• batches route materialization and exact split-copy readiness checks while preserving peer-copy event dependencies;
• adds the classic DSpark singleton fast path to paged serving;
• removes redundant hybrid output copies;
• groups four expert pairs per MMVQ block; and
• tunes the gfx1151 ROCmFP2/ROCmFP3 routed decode kernels.

The four candidate-head additions are:

• 3c4dbbe8 — DSpark singleton fast path;
• e30e23e8 — redundant hybrid output-copy removal;
• 1bb7e054 — four expert pairs per MMVQ block; and
• 92b53237 — gfx1151 routed-kernel tuning.

Current measured performance

Profile:

• target: R9700 (hip:0), dense work and selected experts;
• secondary: Strix Halo, remaining materialized experts;
• model: DeepSeek-V4-Flash-0731-ROCMFPX-MIX-STRIX.gguf;
• paged attention, six slots, 24,576-token KV pool, 4K max context;
• exact prefill, prefix and prefill caches disabled;
• --ds4-expert-top-k 4;
• DSpark q=4 enabled for idle singleton requests;
• temperature zero, three HumanEval waves, maximum 512 generated tokens.

HumanEval here is a throughput workload; retained gold tests were not executed.

Fixed concurrency	Requests	Runtime path	Aggregate tok/s	Output-window tok/s	Median TTFT	p95 TTFT
C1	3	DSpark singleton	14.148	41.083	7.619 s	8.005 s
C2	6	Paged autoregressive	27.337	36.973	6.651 s	6.932 s
C3	9	Paged autoregressive	33.752	45.469	7.985 s	8.353 s
C4	12	Paged autoregressive	36.876	49.630	9.709 s	10.163 s

Aggregate throughput scales to 1.93x, 2.39x, and 2.61x the C1 result at C2, C3, and C4. aggregate_tok_s includes prefill and TTFT; output_window_tok_s measures generation after first-token arrival.

The retained C1 output contract is 455 completion tokens across the three cases, with matching content, reasoning, finish reason, and per-case hashes across retained comparisons.

Long-context exact prefill

A representative 1,195-token C1 request on the retained tokenwise exact path measured:

• TTFT: 72.70 s;
• output-window throughput: 26.07 tok/s; and
• end-to-end aggregate throughput: 0.433 tok/s.

Exact prefill is therefore the primary remaining latency bottleneck.

Rejected batching experiments

No experimental prefill batching code is included in the retained candidate:

• Direct q=4 exact batching improved aggregate throughput from 12.64 to 25.99 tok/s and median TTFT from 8.09 to 2.28 s, but changed completion length from 455 to 597 tokens and changed output hashes.
• Batched HC readbacks preserved output bytes but regressed the same-binary control from 13.90 to 13.46 aggregate tok/s and from 7.746 to 7.837 s median TTFT.
• Batching only the prefix while keeping the final 128-token DSpark capture window singleton reduced the 1,195-token TTFT from 72.70 to 63.94 s, but the generated output diverged late in the response.

These candidates were discarded rather than weakening the fixed-profile output contract.

Prior paged-cohort qualification

Before the DSpark singleton extension, the retained paged optimization sequence improved mean C1-C4 output-window throughput by 32.93% over the original 4fa7112d PR head in chained fixed-concurrency A/B/A comparisons. Every retained transition matched its corresponding fixed-C baseline byte-for-byte. The final scheduler step reduced steady-round hipStreamSynchronize calls from 695 to 443 while leaving peer copies, event records, stream waits, and kernel count unchanged.

This historical index used model-default top-6 routing, no speculative drafter, and at most 128 generated tokens. It is provenance for the paged scheduler changes, not a direct comparison to the current top-4 DSpark profile.

Guardrails

Paged concurrency fails closed for:

• unqualified primary/secondary GPU topology pairs;
• CUDA or out-of-process expert ownership;
• explicit layer splits or remote target shards;
• DDTree, PFlash, or KVFlash;
• fused target decode or approximate prefill;
• windowed attention;
• mutable expert caching; and
• prefix-cache parking.

Four-expert routing is an explicit approximation from the model-default six experts. Exact prefill describes the prefill implementation, not equivalence to model-default top-6 routing.

Validation

• dual-architecture HIP build of dflash_server: passed;
• test_deepseek4_unit: passed on R9700 and Strix;
• test_rocmfp_mix_gateup_glu: passed on R9700 and Strix;
• test_rocmfpx_mmq: passed on R9700 and Strix;
• test_deepseek4_mmid_grouped_cuda: passed on R9700 and Strix;
• fixed-C A/B/A qualification for the retained paged scheduler sequence: passed C1-C4;
• latest paged scheduler step: 30/30 candidate responses matched corresponding fixed-C baselines; and
• current C1 retained output signatures: matched.

Draft limitations

• The current GitHub head does not yet contain the four candidate-head commits named above.
• Exact-prefill TTFT remains high, especially beyond the 128-token DSpark feature window.
• C2-C4 intentionally use paged autoregressive execution, not DSpark, so fixed-concurrency cohorts are qualified independently rather than compared across widths.
• Four-expert routing trades model-default exactness for throughput.
• Output dependence on gathered graph width remains a release blocker.
• The PR remains draft.

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

3 issues found across 25 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_graph.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_graph.cpp:7249">
P2: For long-running ratio-4 lanes, crossing each compression boundary rebuilds the whole-model graph instead of replaying it, causing recurring latency spikes and reducing serving throughput. Use a capacity-padded compressed-history topology with runtime visibility masking, or otherwise keep the graph shape stable across row-count changes.</violation>
</file>

<file name="server/src/common/concurrency/seq_slot_manager.cpp">

<violation number="1" location="server/src/common/concurrency/seq_slot_manager.cpp:175">
P2: When a stochastic request uses `seed=0`, this branch replaces the requested deterministic stream with `std::random_device`, so repeated requests cannot reproduce. Use `needs_logit_processing()` alone as the seed gate and seed the RNG with zero.</violation>
</file>

<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:527">
P2: When hybrid placement uses ratio-4 layers, `estimate_ds4_cache_bytes` underestimates the indexer compressor state by one factor of two. Charge the actual `2 * index_dim` state width so the expert budget does not leave the subsequent cache allocation without enough VRAM.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: For long-running ratio-4 lanes, crossing each compression boundary rebuilds the whole-model graph instead of replaying it, causing recurring latency spikes and reducing serving throughput. Use a capacity-padded compressed-history topology with runtime visibility masking, or otherwise keep the graph shape stable across row-count changes.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_graph.cpp, line 7249:

<comment>For long-running ratio-4 lanes, crossing each compression boundary rebuilds the whole-model graph instead of replaying it, causing recurring latency spikes and reducing serving throughput. Use a capacity-padded compressed-history topology with runtime visibility masking, or otherwise keep the graph shape stable across row-count changes.</comment>

<file context>
@@ -6854,6 +7156,259 @@ static bool initialize_layer_range_cache(
+                cache.plan.physical_blocks, ratio, prepared[(size_t) il])) return false;
+        for (const auto & row : prepared[(size_t) il]) {
+            key.push_back((int64_t) row.raw_history.size());
+            key.push_back((int64_t) row.compressed_history.size());
+            key.push_back(row.slot < 0 ? -1 :
+                (ratio ? row.position % ratio : row.position % DS4_PAGE_TOKENS));
</file context>

[cubic-dev-ai]

P2: When a stochastic request uses seed=0, this branch replaces the requested deterministic stream with std::random_device, so repeated requests cannot reproduce. Use needs_logit_processing() alone as the seed gate and seed the RNG with zero.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/concurrency/seq_slot_manager.cpp, line 175:

<comment>When a stochastic request uses `seed=0`, this branch replaces the requested deterministic stream with `std::random_device`, so repeated requests cannot reproduce. Use `needs_logit_processing()` alone as the seed gate and seed the RNG with zero.</comment>

<file context>
@@ -0,0 +1,278 @@
+    s.sample_history = prompt;
+    // Same predicate the engine uses to pick CPU sampling over GPU argmax:
+    // a seed only means anything when the sampler actually draws.
+    if (sampler.needs_logit_processing() && sampler.seed != 0) {
+        s.rng.seed(sampler.seed);
+    } else {
</file context>

[cubic-dev-ai]

P2: When hybrid placement uses ratio-4 layers, estimate_ds4_cache_bytes underestimates the indexer compressor state by one factor of two. Charge the actual 2 * index_dim state width so the expert budget does not leave the subsequent cache allocation without enough VRAM.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_backend.cpp, line 527:

<comment>When hybrid placement uses ratio-4 layers, `estimate_ds4_cache_bytes` underestimates the indexer compressor state by one factor of two. Charge the actual `2 * index_dim` state width so the expert budget does not leave the subsequent cache allocation without enough VRAM.</comment>

<file context>
@@ -503,20 +515,19 @@ static uint64_t estimate_ds4_cache_bytes(const DeepSeek4Weights & w, int max_ctx
+            // only in fixed-size state scratch and does not scale with context.
+            const size_t index_dim = (size_t) w.n_indexer_head_dim;
+            total_bytes += comp_cap * index_dim * sizeof(uint16_t);
+            total_bytes += state_rows * index_dim * sizeof(float) * 2;
+            total_bytes += (size_t) 2 * 2 * ratio *
+                           (size_t) w.n_indexer_head * index_dim *
</file context>

Suggested change

	total_bytes += state_rows * index_dim * sizeof(float) * 2;
	total_bytes += state_rows * (2 * index_dim) * sizeof(float) * 2;