perf(cpu/matmul): Q8_0 outer-product GEMM via AVX2 VNNI (VPDPBUSD-256) — unblock #61 register pressure - #323
Draft
jamesburton wants to merge 2 commits into
Draft
Conversation
…#312) Add a new F32 outer-product GEMM kernel as a companion to the existing Q8_0/Q5_0/K-quant outer-product kernels landed in PR #61. The Q8_0 outer-product (4×3 AVX2 tile) is currently blocked on RyuJIT register pressure (23 YMM needed, 16 available) due to the Q8_0-specific overhead — `ones` mask vector, scale extraction, `Half→float` conversion. The F32 path has none of these artifacts: a 4×3 tile reaches exactly 12 + 3 + 1 = 16 YMM (12 accumulators + 3 token vectors + 1 reloaded weight vector), which fits the AVX2 register file naturally. Kernel design - 4 weight rows × 3 input tokens register tile. - Vectorises along K (8 floats/lane via Vector256<float>). - At each K-step, 3 token vectors are held in registers and reused across 4 FMAs per row (weight-vector reused 3× across the tokens). - Horizontal-reduces 12 accumulators into the C tile. - Scalar K-tail, row tail, and token tail handle non-tile-aligned shapes. - AVX2/FMA detection at the public entry point; falls back to a scalar reference implementation otherwise. Tests (tests/DotLLM.Tests.Unit/Cpu/Kernels/OuterProductGemmTests.cs) - 25 new test cases extending the existing Q8_0 outer-product test class. - Scalar path: **bit-exact** equality with MatMul.GemmF32Scalar (same accumulation order, no auto-FMA contraction). - Vector path: tolerance ≤ 4e-6·√K vs MatMul.GemmF32 (FMA + horizontal- reduction reorders rounding). - Coverage: tile-aligned shapes, all-tail-combination shapes (row tail, token tail, K tail, all three combined), and edge cases (M=1, N=1, K=1, pure inner product). Benchmark (benchmarks/DotLLM.Benchmarks/OuterProductGemmF32Benchmark.cs) - Compares OuterProductGemmF32 vs production MatMul.GemmF32 at three prefill profiles (M ∈ {128, 512, 2048}, K=4096, N=32). - Intel Core Ultra 7 155H (AVX2, no AVX-512), single-threaded, 15 iterations: | M | Baseline | Outer-product | Speedup | |------|-----------|---------------|----------| | 128 | 2.507 ms | 1.238 ms | 2.02× | | 512 | 10.216 ms | 5.588 ms | 1.83× | | 2048 | 47.070 ms | 32.714 ms | 1.44× | Scope - New, independent kernel — no callers re-wired. Production prefill continues to use MatMul.GemmF32 / GemmF32(..., pool). Caller switching (and threaded outer-product dispatch) is a separate PR after broader benchmark validation. - F32 only — Q8_0/Q5_0/K-quant outer-product re-enablement remains blocked on AVX2 register pressure as documented in the roadmap. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…) — unblocks #61 register pressure - Add OuterProductQ8_0Vnni_4x3: VPDPBUSD-256 (AvxVnni.MultiplyWideningAndAdd) fuses the AVX2 maddubs + madd(ones) integer reduction into one instruction, dropping the `ones` register and the `prod` temporary. This lifts the safe accumulator residency from the AVX2 kernel's conservative 3 to 6 (2 rows × 3 tokens), the exact register relief that blocked PR #61 (~23 YMM at 4×3). Peak ≈ 13 YMM ≤ 16; full 12-accumulator 4×3 stays infeasible (~18+ YMM). - Per-block dw·dx float fold preserved (convert → FMA), so results match the AVX2/scalar paths to FP rounding. abs/sign idiom reused unchanged. - Wire a VNNI branch ahead of Avx2 in both OuterProductGemmQ8_0 dispatch paths (single-threaded + parallel worker); tail tokens/rows unchanged. - Tests: VNNI microkernel vs scalar reference and vs AVX2; GEMM-dispatch parity over discriminating shapes (multi-block K, full tile, all tail combos, large M≥128/N≥32); plus a self-checking discrimination test. Verified the suite FAILS under a deliberate tile bug (22 fail), then passes after revert (75/75). - Benchmark: add OuterProduct4x3_Vnni to OuterProductDisasmBenchmarks. Public GemmQ8_0 (n>1) wiring DEFERRED with reason: OuterProductGemmQ8_0 needs R4-repacked weights and has no production callers; GemmQ8_0 routes row-major weights through ComputeGemmTiled. Wiring would require a per-call full-matrix R4 repack — architecturally significant and risks the path — out of scope for this kernel PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
Benchmark — V256 VNNI path (
|
| Run (warmup/iter) | OuterProduct4x3 (AVX2 maddubs+ones) |
OuterProduct4x3_Vnni (VPDPBUSD-256) |
Speedup |
|---|---|---|---|
| 10 / 30 | 2.343 µs ± 0.58 | 1.726 µs ± 0.41 | 1.36× |
| 2 / 5 | 2.665 µs ± 0.41 | 1.653 µs ± 0.44 | 1.61× |
VNNI is consistently faster — ~1.3–1.6× — from replacing the maddubs + madd(ones) pair with a single VPDPBUSD, which drops the ones mask and prod temporaries and lifts safe accumulator residency from 3 → 6 rows. Code size grows 904 → 1,490 B from the wider unroll (expected).
Methodology caveat: absolute times are sub-3 µs and StdDev is ~25 % of mean — this microbenchmark is noisy on a hybrid-core mobile CPU (P/E-core migration + frequency scaling). The direction and rough magnitude are solid; a definitive figure wants affinity-pinning on a quieter / server-class part. The AVX-512 VNNI (Avx512Vnni) path is benchmarked separately on AVX-512 hardware.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #321. Stacks on #315 (#312 — F32 outer-product kernel).
What
Adds the Q8_0 outer-product GEMM microkernel using AVX2 VNNI (
AvxVnni.MultiplyWideningAndAdd→VPDPBUSD-256), completing the Q8_0 half of ROADMAP Phase 3 Step 26.PR #61 left the Q8_0 outer-product family register-pressure-bound: the AVX2 microkernel holds only 3 live accumulators (one weight row at a time) because the s8×s8 reduction needs
maddubs+madd(ones)plus aonesmask and aprodtemporary. VNNI fuses multiply + widen + accumulate into a singleVPDPBUSD, dropping theonesandprodregisters and lifting safe accumulator residency 3 → 6 rows (~13–14 YMM ≤ 16).Kernel
OuterProductQ8_0Vnni_4x3insrc/DotLLM.Cpu/Kernels/MatMul.cs:acc = AvxVnni.MultiplyWideningAndAdd(Vector256<int>.Zero, absX.AsByte(), adjW). Same 4-row × 3-token output tile; abs/sign idiom and per-blockdw·dxfloat fold preserved → bit-identical per cell to the AVX2 path (same FMA order).OuterProductGemmQ8_0(single-threaded) andOuterProductGemmQ8_0Worker(parallel); not taken on AVX512BW CPUs (they keep the 4×6 AVX512 path) — targets AVX2+VNNI-without-AVX512.Tests — 75/75 pass
Deferred (tracked)
GemmQ8_0: the Q8_0 outer-product family needs R4-repacked weights and has no production callers yet (public n>1 path routes row-major weights throughComputeGemmTiled); wiring needs a per-call R4 repack — same shape as kernels(cpu)(matmul): F32 outer-product tiled GEMM kernel for prefill (ROADMAP Phase 3 Step 26) #312/kernels(cpu)(matmul): F32 outer-product tiled GEMM kernel for prefill #315's deferred F32 caller-wiring.AvxVnni.V512shipped in Add AvxVnni.V512 hardware intrinsics dotnet/runtime#128365, not yet in a net10.0-consumable SDK).