Following up on the aside in #150 with actual numbers, since they may be useful for the default.
On Apple Silicon the shipped default of block_size: 16 was consistently well below the throughput optimum in our runs. Smaller blocks won despite lower acceptance, because verify cost grows faster with row count than acceptance does at this scale.
Setup
M5 Max 128 GB, MLX 0.32, dflash/model_mlx.py at 94e4abc (plus #150 for the 35B drafter). Greedy draft, 24 coding prompts, 160–192 token caps, seed 0, single stream. Throughput is mean_tok_s as reported by stream_generate.
Qwen3.6-35B-A3B + z-lab/Qwen3.6-35B-A3B-DFlash (4-bit target)
| block |
tok/s |
| 4 |
191.1 |
| 5 |
193.8, 194.2 |
| 6 |
193.7 |
| 8 |
188.2, 188.3 |
| 16 (default) |
147.6 |
Block 16 is 24% below the block-5/6 peak. Acceptance rises monotonically with block size across that range (τ 2.95 → 4.30), so this is purely the verify-cost side.
Qwen3.6-27B + z-lab/Qwen3.6-27B-DFlash (4-bit target)
| block |
tok/s |
| 4 |
50.8 |
| 5 |
52.0 |
| 6 |
43.5 |
| 8 |
40.1 |
| 16 (default) |
36.8 |
Block 16 is 29% below peak here.
Caveat on protocol
Both tables above are with thinking enabled. We have a partial thinking-disabled set on the MoE — block 5 → 207.6, block 8 → 217.1 — where the ordering flips and 8 beats 5. We did not measure 16 under that setting, so I can't claim the 24% gap holds there; I'd rather report that than imply a complete sweep. The consistent part across both settings is that 16 was never the winner in anything we measured.
Why this might matter beyond our machine
The cost curve is hardware-shaped: a verify pass on Apple Silicon is memory-bandwidth-bound and grows steeply past a handful of rows, whereas on a B200 the same rows are close to free. So 16 may well be right on datacenter GPUs and wrong on unified-memory hardware — in which case the fix is a note in the MLX docs rather than a change to the default.
Worth mentioning: deepseek-ai/DeepSeek-V4-Flash-0731 ships dspark_block_size: 5, and its card recommends num_speculative_tokens: 7, which brackets what we measured as optimal.
Happy to run any additional configuration you'd like data on.
Following up on the aside in #150 with actual numbers, since they may be useful for the default.
On Apple Silicon the shipped default of
block_size: 16was consistently well below the throughput optimum in our runs. Smaller blocks won despite lower acceptance, because verify cost grows faster with row count than acceptance does at this scale.Setup
M5 Max 128 GB, MLX 0.32,
dflash/model_mlx.pyat 94e4abc (plus #150 for the 35B drafter). Greedy draft, 24 coding prompts, 160–192 token caps, seed 0, single stream. Throughput ismean_tok_sas reported bystream_generate.Qwen3.6-35B-A3B +
z-lab/Qwen3.6-35B-A3B-DFlash(4-bit target)Block 16 is 24% below the block-5/6 peak. Acceptance rises monotonically with block size across that range (τ 2.95 → 4.30), so this is purely the verify-cost side.
Qwen3.6-27B +
z-lab/Qwen3.6-27B-DFlash(4-bit target)Block 16 is 29% below peak here.
Caveat on protocol
Both tables above are with thinking enabled. We have a partial thinking-disabled set on the MoE — block 5 → 207.6, block 8 → 217.1 — where the ordering flips and 8 beats 5. We did not measure 16 under that setting, so I can't claim the 24% gap holds there; I'd rather report that than imply a complete sweep. The consistent part across both settings is that 16 was never the winner in anything we measured.
Why this might matter beyond our machine
The cost curve is hardware-shaped: a verify pass on Apple Silicon is memory-bandwidth-bound and grows steeply past a handful of rows, whereas on a B200 the same rows are close to free. So 16 may well be right on datacenter GPUs and wrong on unified-memory hardware — in which case the fix is a note in the MLX docs rather than a change to the default.
Worth mentioning:
deepseek-ai/DeepSeek-V4-Flash-0731shipsdspark_block_size: 5, and its card recommendsnum_speculative_tokens: 7, which brackets what we measured as optimal.Happy to run any additional configuration you'd like data on.