Skip to content

fix: make DFlash2 drafts work on M1/M2 (fp32 quant metadata + GEMV block cap) - #10

Open
boojongmin wants to merge 1 commit into
jundot:mainfrom
boojongmin:main
Open

fix: make DFlash2 drafts work on M1/M2 (fp32 quant metadata + GEMV block cap)#10
boojongmin wants to merge 1 commit into
jundot:mainfrom
boojongmin:main

Conversation

@boojongmin

@boojongmin boojongmin commented Aug 22, 2026

Copy link
Copy Markdown

Problem

On bf16-emulated chips (M1/M2), running the published DFlash2 recipe for Qwen3.8-27B (incoai/Qwen3.8-27B-DFlash2 + w4a16 draft quant, block 5) is ~2.5x slower than plain decoding (3.3 vs 8.25 tok/s on M1 Max) instead of faster. Two independent causes, both verified on M1 Max + Qwen3.8-27B-4bit:

1. All-NaN draft logits from the fp16 cast

resolve_draft_load_dtype returns float16 on M1/M2 to dodge the bf16-emulation slowdown, and load_draft_bundle applied it with a blanket cast that also downcasted the per-group quantization scales/biases produced by nn.quantize. That metadata loss makes DFlash2 block forwards numerically unstable: with real target-hidden magnitudes the dequantized matmul overflows and the draft emits all-NaN logits — acceptance ~3%, every drafted token rejected, decode 3.3 tok/s while still paying draft+verify+replay.

Root-caused by casting order: nn.quantize → naive fp16 cast reproduces NaN on 50/50 synthetic seeds; casting weights to fp16 while pinning scales/biases to fp32 is finite on 30/30 seeds with identical acceptance.

2. Verify block lands on the small-M GEMM cliff

MLX dispatches quantized_matmul (transpose=True, the serving layout) through a batched-GEMV kernel only while the row count stays below a per-chip limit (get_qmv_batch_limit in mlx/backend/metal/quantized.cpp: 6 on M1/M2 non-Ultra, 13 on M3/M4, 33 on M5). At the limit it falls to a bm=32 tiled GEMM whose bandwidth collapses on decode shapes — measured on a 4-bit 5120×248320 matmul: 245 GB/s at M=1 vs 48 GB/s at M=6; an end-to-end verify pass costs ~55 ms/row flat for M=3..5 and 79 ms/row at M=6. The blog's block-5 recipe (verify M=6) lands exactly on the M1 cliff.

Fix

  • is_dflash2_config() detects DFlash2DraftModel checkpoints; _cast_floating_model_preserving_quant_metadata() casts DFlash2 drafts to fp16 while pinning quant scales/biases to fp32 (norm gains stay fp16 so draft_backend's compute-dtype probe picks the fast path). DFlash (1) drafts keep the plain fp16 cast, which was never unstable; M3+ is untouched.
  • resolve_speculative_cycle_config caps effective_block_tokens to the chip's GEMV-safe row count (limit − 1) when given the real chip profile (threaded through SpeculativeSession). An explicitly smaller block is never raised, M3/M4/M5 keep their full requested block, and DFLASH_DISABLE_GEMV_BLOCK_CAP=1 opts out. Unit tests stay chip-independent (the three block-size policy tests opt out explicitly).

Measurements (M1 Max, Qwen3.8-27B-4bit + incoai/Qwen3.8-27B-DFlash2, w4:gs64, verify=dflash)

config gen tok/s acceptance
main (fp16 cast, block 5) 3.33 3.1%
this PR (block 5, capped verify) 15.3 74.2%
this PR (block 4, GEMV-safe) 20.9–30.4 71–74%
plain baseline 8.25

Test suite: 1322 passed (unchanged pre-existing failure in test_verify_kernel_contract deselected — M1 numeric deviation in the m16_gate_up shape, unrelated).

Fixes the M1/M2 half of the underperformance discussed in z-lab/omlx-fork#2 (that PR works around it from the omlx side with w4a32; this fixes the runtime so the default w4a16 path is both correct and fast, with ~10% faster drafts than fp32).

…ock cap)

Two bf16-emulated-chip fixes, measured on M1 Max with
Qwen3.8-27B-4bit + incoai/Qwen3.8-27B-DFlash2 (w4:gs64):

1. NaN draft logits: the old-Apple fp16 draft cast also downcasted the
   nn.quantize scales/biases, and DFlash2 block forwards overflow in
   that regime (all-NaN logits on 50/50 synthetic seeds, acceptance ~3%,
   3.3 tok/s). Cast DFlash2DraftModel checkpoints with weights in fp16
   but scales/biases pinned to fp32: 0/30 NaN, acceptance 74-77%.

2. Verify-block GEMM cliff: MLX dispatches quantized_matmul through a
   batched-GEMV kernel only while M stays below a per-chip limit
   (get_qmv_batch_limit: 6 on M1/M2 non-Ultra, 13 on M3/M4, 33 on M5);
   at the limit it falls to a bm=32 tiled GEMM at 48 GB/s vs 245 GB/s
   at M=1 (5120x248320 4-bit). Verify costs ~55 ms/row flat for M=3..5
   and 79 ms/row at M=6, so the published block-5 recipe lands exactly
   on the M1 cliff. resolve_speculative_cycle_config now caps the
   effective block to the chip's GEMV-safe row count when given the
   real chip profile; smaller explicit blocks are never raised, M3+
   keeps its full block, DFLASH_DISABLE_GEMV_BLOCK_CAP=1 opts out, and
   unit tests stay chip-independent.

End to end on M1 Max: 3.3 -> 15.3 tok/s at the blog's block-5 settings
(plain baseline 8.25), 20.9-30.4 tok/s at the GEMV-safe block 4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant