fix: make DFlash2 drafts work on M1/M2 (fp32 quant metadata + GEMV block cap) - #10
Open
boojongmin wants to merge 1 commit into
Open
fix: make DFlash2 drafts work on M1/M2 (fp32 quant metadata + GEMV block cap)#10boojongmin wants to merge 1 commit into
boojongmin wants to merge 1 commit into
Conversation
…ock cap) Two bf16-emulated-chip fixes, measured on M1 Max with Qwen3.8-27B-4bit + incoai/Qwen3.8-27B-DFlash2 (w4:gs64): 1. NaN draft logits: the old-Apple fp16 draft cast also downcasted the nn.quantize scales/biases, and DFlash2 block forwards overflow in that regime (all-NaN logits on 50/50 synthetic seeds, acceptance ~3%, 3.3 tok/s). Cast DFlash2DraftModel checkpoints with weights in fp16 but scales/biases pinned to fp32: 0/30 NaN, acceptance 74-77%. 2. Verify-block GEMM cliff: MLX dispatches quantized_matmul through a batched-GEMV kernel only while M stays below a per-chip limit (get_qmv_batch_limit: 6 on M1/M2 non-Ultra, 13 on M3/M4, 33 on M5); at the limit it falls to a bm=32 tiled GEMM at 48 GB/s vs 245 GB/s at M=1 (5120x248320 4-bit). Verify costs ~55 ms/row flat for M=3..5 and 79 ms/row at M=6, so the published block-5 recipe lands exactly on the M1 cliff. resolve_speculative_cycle_config now caps the effective block to the chip's GEMV-safe row count when given the real chip profile; smaller explicit blocks are never raised, M3+ keeps its full block, DFLASH_DISABLE_GEMV_BLOCK_CAP=1 opts out, and unit tests stay chip-independent. End to end on M1 Max: 3.3 -> 15.3 tok/s at the blog's block-5 settings (plain baseline 8.25), 20.9-30.4 tok/s at the GEMV-safe block 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On bf16-emulated chips (M1/M2), running the published DFlash2 recipe for Qwen3.8-27B (incoai/Qwen3.8-27B-DFlash2 +
w4a16draft quant, block 5) is ~2.5x slower than plain decoding (3.3 vs 8.25 tok/s on M1 Max) instead of faster. Two independent causes, both verified on M1 Max + Qwen3.8-27B-4bit:1. All-NaN draft logits from the fp16 cast
resolve_draft_load_dtypereturnsfloat16on M1/M2 to dodge the bf16-emulation slowdown, andload_draft_bundleapplied it with a blanket cast that also downcasted the per-group quantization scales/biases produced bynn.quantize. That metadata loss makes DFlash2 block forwards numerically unstable: with real target-hidden magnitudes the dequantized matmul overflows and the draft emits all-NaN logits — acceptance ~3%, every drafted token rejected, decode 3.3 tok/s while still paying draft+verify+replay.Root-caused by casting order:
nn.quantize→ naive fp16 cast reproduces NaN on 50/50 synthetic seeds; casting weights to fp16 while pinning scales/biases to fp32 is finite on 30/30 seeds with identical acceptance.2. Verify block lands on the small-M GEMM cliff
MLX dispatches
quantized_matmul(transpose=True, the serving layout) through a batched-GEMV kernel only while the row count stays below a per-chip limit (get_qmv_batch_limitinmlx/backend/metal/quantized.cpp: 6 on M1/M2 non-Ultra, 13 on M3/M4, 33 on M5). At the limit it falls to a bm=32 tiled GEMM whose bandwidth collapses on decode shapes — measured on a 4-bit 5120×248320 matmul: 245 GB/s at M=1 vs 48 GB/s at M=6; an end-to-end verify pass costs ~55 ms/row flat for M=3..5 and 79 ms/row at M=6. The blog's block-5 recipe (verify M=6) lands exactly on the M1 cliff.Fix
is_dflash2_config()detectsDFlash2DraftModelcheckpoints;_cast_floating_model_preserving_quant_metadata()casts DFlash2 drafts to fp16 while pinning quant scales/biases to fp32 (norm gains stay fp16 sodraft_backend's compute-dtype probe picks the fast path). DFlash (1) drafts keep the plain fp16 cast, which was never unstable; M3+ is untouched.resolve_speculative_cycle_configcapseffective_block_tokensto the chip's GEMV-safe row count (limit − 1) when given the real chip profile (threaded throughSpeculativeSession). An explicitly smaller block is never raised, M3/M4/M5 keep their full requested block, andDFLASH_DISABLE_GEMV_BLOCK_CAP=1opts out. Unit tests stay chip-independent (the three block-size policy tests opt out explicitly).Measurements (M1 Max, Qwen3.8-27B-4bit + incoai/Qwen3.8-27B-DFlash2, w4:gs64, verify=dflash)
Test suite: 1322 passed (unchanged pre-existing failure in
test_verify_kernel_contractdeselected — M1 numeric deviation in the m16_gate_up shape, unrelated).Fixes the M1/M2 half of the underperformance discussed in z-lab/omlx-fork#2 (that PR works around it from the omlx side with
w4a32; this fixes the runtime so the defaultw4a16path is both correct and fast, with ~10% faster drafts than fp32).