DSpark net-negative on M3 Ultra + MXFP4 despite 82.8% accept rate — independent confirmation of #695 replay/verify overhead

[lifodetails]

Summary

Adding a cross-configuration data point for #695: DSpark is net-negative on M3 Ultra 256GB + MXFP4 (the recommended quantization), even though the draft acceptance rate is 82.8%. This confirms the core finding of #695 from a different hardware generation and quantization format: the bottleneck is replay + verify overhead, not draft quality.

Environment

Item	Value
Chip	Apple M3 Ultra (256 GB unified memory, 819 GB/s)
Engine	ds4 commit b030961
Model GGUF	DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf (145 GB)
DSpark support	DeepSeek-V4-Flash-DSpark-support-0731.gguf (5.6 GB)
Quantization	MXFP4 experts (native, bit-for-bit) + F16/Q8 non-expert
Context	8192, --nothink --temp 0
Prompt	"Write a Python quicksort function with comments." (100 tokens)

Controlled Experiment (4 modes, same prompt)

Mode	Generation (t/s)	Decode vs baseline
Baseline (no DSpark)	39.78	—
--dspark-strict (loaded, target-only)	39.17	-1.5% (loading overhead ≈ 0)
--dspark (confidence 0.7, default)	25.47	-36%
--dspark --dspark-confidence 0 (forced 5-token blocks)	21.58	-46%

The strict experiment isolates the overhead: loading the DSpark support model costs ~1.5% (within noise). The regression is entirely in the runtime decode path.

DSpark Telemetry (DS4_DSPARK_STATS=1)

Default confidence (0.7)

cycles=46 proposed=58 accepted_draft=48 accept_rate=82.76% avg_accept=1.043
full=12 partial=4 miss_first=2 no_draft=28 scheduler_skips=23
time_ms propose=379.8 verify=1236.4 replay=1206.8 spec_total=2460.6
        target=1571.3 saved=1631.7 net_saved=-1208.7
draft_len_hist=1:2,2:4,3:5,4:2,5:5
accepted_len_hist=0:30,1:3,2:3,3:5,4:1,5:4

Forced 5-token blocks (confidence 0)

cycles=24 proposed=120 accepted_draft=70 accept_rate=58.33% avg_accept=2.917
full=7 partial=16 miss_first=1 no_draft=0 scheduler_skips=0
time_ms propose=221.2 verify=1928.3 replay=1746.3 spec_total=3698.1
        target=760.6 saved=1794.5 net_saved=-2124.8
draft_len_hist=5:24
accepted_len_hist=0:1,1:5,2:6,3:2,4:3,5:7

Analysis: Where the Loss Comes From

All three time components in this run are in the same order of magnitude:

Component	Time (ms)	Per-cycle (ms)	Notes
propose	379.8	8.3	Draft generation (cheap)
verify	1236.4	26.9	Target model verification
replay	1206.8	26.2	Partial-accept replay (7fb2830)

replay (1207 ms) ≈ verify (1236 ms). The replay cost — introduced by commit 7fb2830 ("Replay partial DSpark accepts through ordinary decode") for correctness — is as expensive as the verify pass itself. Combined, replay + verify = 2443 ms, while accepted tokens only save 1632 ms.

This is consistent with the finding in #695: the scheduler's break-even model may not account for replay cost, preventing convergence to baseline even when draft quality is good.

What This Confirms (and What It Doesn't)

Confirms: Draft quality is not the bottleneck on MXFP4. An 82.8% acceptance rate — higher than expected — still yields a net loss. The regression is driven by replay + verify overhead.

Does not claim: This report does not compare absolute timing against #695, as the test scales differ (#695 covers a multi-thousand-token corpus; this is a 100-token single prompt). Per-cycle costs are not directly comparable across different draft block sizes and corpus lengths. The shared conclusion is qualitative: both reports independently observe net-negative results despite healthy acceptance rates.

Scheduler Behavior

In default mode (confidence 0.7), the scheduler skips 23 of 46 cycles (no_draft=28). This is the correct behavior — when forced to propose every cycle (confidence 0), net loss worsens from -1209 ms to -2125 ms. The scheduler is already doing damage control, but as #695 notes, the break-even model may not fully account for replay cost, preventing convergence to baseline.

Repro

# Baseline
./ds4 -m <MXFP4-0731.gguf> --ctx 8192 --nothink --temp 0 \
  -p "Write a Python quicksort function with comments." --tokens 100

# DSpark with telemetry
DS4_DSPARK_STATS=1 ./ds4 \
  -m <MXFP4-0731.gguf> \
  --mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
  --dspark --ctx 8192 --nothink --temp 0 \
  -p "Write a Python quicksort function with comments." --tokens 100

Happy to run additional experiments (longer prompts, different domains, longer contexts) if useful.

[trueimage]

I also found DSpark to be detrimental on my M3 Ultra 512GB

[GiorgioOppo]

Would you like to try my implementation of dspark with a requantized GGUF? I’ve noticed an acceptable acceptance rate of around 80%.

#621

[lifodetails]

Would you like to try my implementation of dspark with a requantized GGUF? I’ve noticed an acceptable acceptance rate of around 80%.

#621

Thanks for the suggestion, @GiorgioOppo! The ~80% acceptance rate you observed is actually consistent with what I measured (82.76% via DS4_DSPARK_STATS), so we seem to agree that draft quality is not the bottleneck here.

The loss in my setup is driven entirely by the replay + verify cost structure, not by acceptance rate. From the telemetry in this report:

Component	Time (ms)
replay (7fb2830)	1207
verify	1236
combined	2443
tokens saved by accepts	1632

replay alone (~26 ms/cycle) is nearly as expensive as the verify pass itself. Even at 82.8% acceptance, the combined replay + verify cost exceeds the time saved by accepted tokens. This matches what @datanerdie found in #695 (replay was also the largest single cost there).

My understanding is that AProjQ4 changes the attention projection quantization (Q8→Q4K), which should reduce model size and may slightly affect per-layer read bandwidth. But it's unclear whether it would materially change the replay or verify absolute cost, since those are properties of the correctness replay path (7fb2830) and the Metal verify kernel, not of the projection quantization format.

That said, if you have DS4_DSPARK_STATS=1 telemetry from your AProjQ4 + DSpark runs (especially the replay and verify fields), that would be very valuable to compare. If the per-cycle replay/verify costs differ from ~26 ms, that would point to a quantization-dependent effect worth investigating.

[GiorgioOppo]

I’ve currently implemented it using SSD streaming due to my machine's limited memory, so I can't run a test like that myself. However, if you do run a test and notice any worsening or improvement in the DsPark performance, please let me know. Thanks in advance.

[lifodetails]

Thanks @GiorgioOppo — I went ahead and ran the test. Here are the results.

Setup

Since the prebuilt AProjQ4 GGUF on HF uses IQ2XXS experts (84 GB), which would confound the comparison against my MXFP4 baseline (different expert quantization = different replay/verify byte counts), I used the --source-gguf requantize tool from #621 to produce a clean version: MXFP4 experts preserved bit-for-bit, only the 215 attention projection tensors requantized Q8_0 → Q4_K. Same model, same experts, only attention projection precision differs.

Token throughput

Same prompt, --ctx 8192 --nothink --temp 0 --tokens 100:

Mode	MXFP4 Q8Attn baseline	AProjQ4	Delta
baseline (no DSpark)	39.78 t/s	41.77 t/s	+5.0%
--dspark-strict	39.17 t/s	40.74 t/s	+4.0%
--dspark (conf 0.7)	25.47 t/s	26.77 t/s	+5.1%
--dspark --dspark-confidence 0	21.58 t/s	22.09 t/s	+2.4%

AProjQ4 is consistently a few percent faster (attention projection reads half the bytes), but the regression pattern is unchanged: conf 0.7 is still -36% vs baseline, identical to MXFP4.

Telemetry comparison

The key question is whether AProjQ4 changes the per-cycle replay/verify cost. I ran control groups to isolate the variable: same binary (#621 build) running both MXFP4 Q8Attn and AProjQ4, same confidence (0, full 5-token drafts, no pruning).

conf 0 (full drafts, cleanest comparison):

Per-cycle	MXFP4 Q8Attn	AProjQ4	Delta
replay	73.4 ms	65.8 ms	-10.4%
verify	81.7 ms	81.5 ms	-0.2%

replay improved ~10% (attention projection is part of the replay forward pass), verify essentially unchanged. Both within ±15%.

But replay + verify combined (3534 ms) still vastly exceeds the 1591 ms saved by accepted tokens. net_saved = -2186 ms.

Conclusion

AProjQ4 delivers the expected small improvement in attention-projection-bound paths (~10% on replay, consistent with attention projections being ~4% of the model), but it does not change the DSpark cost structure. The dominant cost is reading routed experts during verify and replay, which AProjQ4 doesn't touch. The net-negative result holds across all configurations.

The bottleneck, as #695 also identified, is the replay path itself (commit 7fb2830) and the verify kernel cost — both are structural to the current DSpark implementation, not properties of the attention quantization format.

Appreciate the suggestion and the requantize tool though — it was clean to use and the Q4_K attention path works correctly. Good luck with #621.

[GiorgioOppo]

Tanks for shearing it, I’ll work on it

[lifodetails]

Quick update for anyone following this thread: PR #755 (ivanfioravanti's pre-M5 decode optimizations, currently OPEN) also partially benefits the DSpark replay path.

Tested on my M3 Ultra + MXFP4 setup (same as the original report):

DSpark metric	main b030961	PR #755 5b73804
replay	1234.6 ms	1070.6 ms (-13.3%)
verify	1046.2 ms	1060.2 ms (+1.3%)
generation	26.09 t/s	28.58 t/s (+9.5%)

replay improves because it shares the decode forward path that #755 optimizes. However, net_saved remains negative (replay + verify still far exceeds token savings from accepts), so this doesn't resolve the core DSpark issue — it just makes the loss slightly smaller.

The real fix still needs to happen at the replay mechanism (7fb2830) and/or the verify kernel cost structure, as #695 identified.

[GiorgioOppo]

@lifodetails can I ask to retest #621 with some implementation takes from #755?

[lifodetails]

After updating to 84cc882 (the pipelined verify from commit 42033ee), replay dropped from 1207ms to 0ms as expected — that's a real structural fix, thanks.

But with a longer 500-token generation to reach steady state (100-token runs showed misleading +1.4% from small-sample variance), DSpark is still net-negative at -7.5%:

Tokens	DSpark gen	baseline gen	net_saved
100	45.62 t/s	44.99 t/s	+10ms (noise)
500	43.21 t/s	46.71 t/s	-782ms

The 500-token telemetry makes the bottleneck unambiguous — verify is now 99% of spec_total:

cycles=205 accept_rate=83.77% avg_accept=1.410
no_draft=98 scheduler_skips=76
time_ms verify=5925.2 replay=0.0 spec_total=5985.2

replay=0 confirms the pipeline fix worked. But verify alone (5925ms over 205 cycles ≈ 29ms/cycle) still exceeds what accepts can save. The scheduler correctly skips 76 of 205 cycles where draft confidence is too low to break even — it's doing damage control against verify cost, not against replay.

The next target for making DSpark net-positive on Metal appears to be verify_layer (5922ms, essentially all of verify). Whatever the M5 TensorOps path does for prefill speedup, M3 Ultra's legacy verify path reads the same routed-expert bytes at the legacy rate, and that's the wall DSpark hits here.