metal: multi-column Q1_0 mat-vec for small verify batches (+ opt-in word-parallel popcount path) - #142
metal: multi-column Q1_0 mat-vec for small verify batches (+ opt-in word-parallel popcount path)#142bri-prism wants to merge 5 commits into
Conversation
Speculative-decode verify batches put N = k+1 rows (2..8) through the
target. For Q1_0 that hit the generic small-batch mul_mv_ext path, which
dequantizes every 1-bit weight through the float path and runs at a flat
~2.5 TFLOPS for every type: a 4-row forward on the 27B Q1_0 target cost
3.5 decode steps on M5 Pro (pp4 44.7 tok/s vs tg128 39.1).
Add kernel_mul_mv_q1_0_f32_nr1_{2,3,4}: each q1_0 block is read once and
dotted against nr1 src1 columns (tpb threads per block, SW = QK1_0/tpb).
Route Q1_0 off the ext path, pick nr1 = 3 for exactly 3 columns and 2
otherwise, and keep Q1_0 on mul_mv up to 16 rows before mul_mm. The
single-column kernel is untouched, so decode numerics are unchanged.
M5 Pro, 27B Q1_0, -fa 1 (tok/s, 3-run medians, interleaved A/B):
pp2 29.6 -> 54.2 pp4 44.7 -> 69.2 pp8 48.6 -> 77.4
pp12 49.1 -> 79.6 pp16 65.3 -> 80.4 tg128 unchanged (38.8 vs 37.7, thermal spread)
test-backend-ops MUL_MAT q1_0: 45/45 vs CPU.
Env knobs for A/B: GGML_METAL_Q1_0_NR1 (1 disables, 2..4 forces),
GGML_METAL_Q1_0_EXT_ENABLE, GGML_METAL_Q1_0_MV_MAX (default 16).
…llel popcount) Both opt-in and default-off, kept for the measurements they produced. The multi-column kernel in the parent commit is still ALU-bound: a timing probe that removes only the per-weight bit test runs pp4 69.2 -> 118.9 tok/s, so the bit extraction, not the weight read, is what is left to attack. 1. GGML_METAL_Q1_0_H2: two src1 columns per half2 lane, fp32 block sums, fp32 promotion every q1_0 block. Correct (45/45 MUL_MAT q1_0 vs CPU) and worth pp2 55.7 -> 57.7, pp4 69.2 -> 72.1, pp8 78.0 -> 81.1 = +3.5 to +4.5%. Too small to justify the fp16 accumulator, so it stays off. 2. GGML_METAL_Q1_0_POPCNT: quantize each activation column to int8 once per matmul and store it as 8 bit-planes, then consume 32 weights per AND+popcount (~0.8 integer ops per weight per column against ~3 for select-per-weight). Adds kernel_q1_0_build_planes, a scratch record in the dst buffer tail sized by ggml_metal_op_mul_mat_extra_q1_0_planes, and a barrier between the two dispatches. Correct (45/45) and end-to-end accept counters are unchanged (80.645%, identical to the fp32 path). Timing is pending an uncontended GPU. Note for whoever touches Metal type gating next: this backend's supports_op accepts every type for MUL_MAT and only rejects NVFP4, so a type with no good kernel silently takes a generic path. That is the same shape as the TQ1_0 and iq4_nl gaps found the same day; those abort, Q1_0 small batch merely ran slow.
…y batch height) pp4 84.4 vs 62.8 tok/s and pp8 102.7 vs 82.3 against nr1=4, which spills registers (pl[32] + w[nr0][4] + sumf). GGML_METAL_Q1_0_PC_NR1 forces either.
The scratch record lives in the dst-buffer tail, so sizing it unconditionally would grow every small-batch Q1_0 mat-mul destination even with the path off. Gate the size helper on the same knob the dispatch reads.
There was a problem hiding this comment.
Pull request overview
Optimizes Metal Q1_0 matrix-vector multiplication for speculative-decode verification batches.
Changes:
- Adds multi-column Q1_0 kernels and routing for batches up to 16 rows.
- Adds optional half2 and popcount implementations.
- Adds popcount scratch allocation and pipeline setup.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
ggml/src/ggml-metal/kernels/mul_mv.metal |
Implements the new Q1_0 kernels. |
ggml/src/ggml-metal/ggml-metal.cpp |
Reserves popcount scratch memory. |
ggml/src/ggml-metal/ggml-metal-ops.h |
Declares the scratch-size helper. |
ggml/src/ggml-metal/ggml-metal-ops.cpp |
Routes operations and dispatches kernels. |
ggml/src/ggml-metal/ggml-metal-impl.h |
Defines kernel arguments and constants. |
ggml/src/ggml-metal/ggml-metal-device.h |
Declares pipeline accessors. |
ggml/src/ggml-metal/ggml-metal-device.cpp |
Selects and compiles pipelines. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The plane-scratch size helper and the encoder disagreed. The encoder only takes the word-parallel path for F32 activations with no src0/src1 broadcast, while the size helper reserved a record for any Q1_0 mat-mul in the 2..16 column range, so unsupported shapes were padding their destination buffer for a path that would never run. Both now call one predicate, ggml_metal_op_mul_mat_q1_0_pc_supported, which keeps the two conditions from drifting apart again. Remove the half2 kernel family: kernel_mul_mv_q1_0_f32_h2_impl, both _h2_nr1_2 and _h2_nr1_4 entry points, and the GGML_METAL_Q1_0_H2 routing branch. It was worth 3.5 to 4.5% at the batch heights it targeted, which does not pay for an fp16 accumulator sitting in the tree with nothing enabling it by default. The multi-column and word-parallel paths are unchanged. Also give the plane-scratch declaration its own comment in ggml-metal-ops.h. It had landed under the "tokens per expert" label that belongs to the declaration below it. test-backend-ops MUL_MAT q1_0 on Metal: 45 OK / 0 FAIL, both with GGML_METAL_Q1_0_POPCNT unset and with it set. Interleaved before/after llama-bench at 2, 4 and 8 columns plus tg128 is unchanged on both arms, all deltas inside run-to-run spread.
|
Thanks, all three are fair. Addressed in the follow-up commit. 1. Scratch reserved for shapes the encoder will not take. Correct, the two conditions had drifted. The encoder only takes the word-parallel Rather than copy the missing terms into the allocator and leave two lists to drift 2. Header comment mislabelled the new declaration. Fixed. The plane helper carries its own label now, and the tokens per expert comment 3. The half2 kernel family. Agreed, and it is removed. That covers The multi-column path and the opt-in word-parallel path are untouched. One thing the review did not catch that I found while making the change: folding Verification after the change: |
There was a problem hiding this comment.
🔵 Needs a closer look
The description advertises a half2 variant that is absent from the implementation.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
ggml/src/ggml-metal/ggml-metal-device.cpp:920
- The PR description advertises an opt-in
GGML_METAL_Q1_0_H2half2 variant, but this routing exposes only the NR1 variants and there is no H2 switch or kernel anywhere in the Metal backend. Please either include that promised variant or remove it from the description so the documented scope matches the code.
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Balanced
|
Good catch, and it was my inconsistency rather than a missing feature. I dropped the half2 kernels in 443f016 in response to your earlier point that a second kernel family with no supported production use is permanent maintenance surface, but I left the description advertising The description now matches the code: the half2 item is gone from the How section, and the 3.5 to 4.5% measurement survives as a past-tense note saying it was built, measured, and removed, so nobody re-explores it from scratch. I also added No code changed for this, description only. Confirmed there is no |
What
Four Metal kernels and one routing change for
Q1_0mat-vec at the small batch heights aspeculative-decode verify step uses (N = drafted tokens + 1, so 2 to 8 rows).
Only the first commit changes default behaviour. The other three are opt-in and default-off.
Why
A verify step runs k drafted tokens plus the anchor through the target in one forward. On a
bandwidth-bound model that should cost about one plain decode step until compute takes over,
but on this backend a 4-row forward cost 3.5 decode steps, so speculation could not win at
block 4 no matter how good the drafter was.
The cause is routing, not the weights. For
ne11in [2, 8]Q1_0was going to the genericmul_mv_extpath, which dequantizes each 1-bit weight through the float path and runs at aflat ~2.5 TFLOPS for every quant type. Past 8 rows it fell to
mul_mm, whose per-tile dequantof the whole model costs a flat ~245 ms. Measured with
test-backend-ops perf, that FLOP ratealone predicts the pp2 / pp4 / pp8 timings within 5%.
Worth noting for anyone touching type gating here:
supports_opaccepts every type forMUL_MATand rejects only NVFP4, so a type with no good kernel silently takes a generic path.The same shape caused a TQ1_0 pipeline abort and an iq4_nl flash-attention abort found the same
day. Those abort loudly; this one just ran slow.
How
1. Multi-column
mul_mv(default-on).kernel_mul_mv_q1_0_f32_nr1_{2,3,4}read eachq1_0block once and dot it againstnr1src1 columns.Q1_0is routed off the ext path andstays on
mul_mvup to 16 rows.2. Word-parallel popcount (
GGML_METAL_Q1_0_POPCNT, off). A timing probe that removed onlythe per-weight bit test ran pp4 at 118.9 tok/s, so bit extraction, not the weight read, was what
remained. This path quantizes each activation column to int8 once per matmul and stores it as 8
bit-planes, so 32 weights are consumed per AND plus popcount:
That is about 11 integer ops per streamed weight byte against roughly 16 fp32 lane-ops per byte
of budget on this hardware, which is why it clears a bar that an exact-arithmetic form does not.
Plane construction is its own dispatch into a scratch record in the dst-buffer tail, sized by
ggml_metal_op_mul_mat_extra_q1_0_planes; building planes per threadgroup instead would costmore than it saves.
nr1 = 2is required,nr1 = 4spills registers.A third variant, two columns per half2 lane with fp32 block sums, was built and measured at 3.5
to 4.5%. That does not justify an fp16 accumulator on this path, so it was removed in 443f016
and is not part of this diff. Recording the number here so the option does not get re-explored
from scratch.
Measured
M5 Pro, 27B
Q1_0,-fa 1, 3-run medians, arms interleaved.A 4-row forward goes from 3.50 to 2.26 to 1.77 plain-decode steps.
End to end on the speculative path (
--spec-type draft-dflash, block-4 drafter,--spec-draft-p-min 0.75, 128 tokens greedy), against plain decode measured in the same runs:30.1 tok/s before, 38.6 to 39.4 with the multi-column kernel, and 42.0 to 43.1 with popcount,
against 36.8 to 37.8 plain. That is the first net-positive block-4 speculative result on this
machine.
Decode and prefill are unchanged: pp512 421.5 / 420.5 and tg128 40.1 / 40.0 for old and new
routing in an interleaved A/B. The single-column kernel is untouched.
Testing
test-backend-ops test -o MUL_MAT: 45/45q1_0cases pass against CPU, both with thepopcount path on and off.
(82.857%), so the int8 activation step does not move the greedy trajectory on the prompt tested.
GGML_METAL_Q1_0_NR1=1,GGML_METAL_Q1_0_EXT_ENABLE=1,GGML_METAL_Q1_0_MV_MAX=8. The popcount path's own rowcount is
GGML_METAL_Q1_0_PC_NR1.Open item before the popcount path could ever be default
It needs a measured logit-KLD gate. The int8 activation step is the same numeric class as the
CUDA mat-vec path but it is not free, and the KLD noise floor here is around 4.3e-4, so that gate
needs a same-path control run to be readable. Until then it stays off by default, which is why
this PR does not turn it on.