Support AProjQ4 GGUFs: Q4_K dense attention projections; Metal, ROCm and CUDA performance improvements; SSD streaming support for speed-bench; quality-of-life improvements; bugfixes - #621
Conversation
|
DS4_METAL_STREAMING_EXPERT_NOCACHE=1 DS4_METAL_STREAMING_EXPERT_PREAD_SPLIT=4 ./ds4 --metal --ssd-streaming --ssd-streaming-cache-experts 448 -m gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix.gguf |
|
@antirez , |
|
Volunteering as a tester for the CUDA path — this is the configuration your table lists as implemented but untested. Machine: stock NVIDIA DGX Spark, GB10 (sm_121), 128 GB unified memory, aarch64, Ubuntu 24.04.4, driver 580.173.02, CUDA 13.0.88. Built with What I can run, using a harness already in place from measuring
Why the memory result is the interesting one here. Resident What blocks me: I could not find an AProjQ4 GGUF published (not in So: could you publish Context for the numbers above: #705. |
|
Hi, the download is blocked by a pr on hf, https://huggingface.co/antirez/deepseek-v4-gguf/discussions/22 You should be able to download the template from https://huggingface.co/antirez/deepseek-v4-gguf/blob/refs%2Fpr%2F22/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf |
|
Tested on a stock NVIDIA DGX Spark (GB10, sm_121, 128 GB unified, CUDA 13.0.88, driver 580.173.02), as offered above. Short version: the CUDA path is correct, the memory win is bigger than expected, but prefill regresses ~16x, which is a blocker for resident use on this machine. Setup: branch One caveat up front: the AProjQ4 file is from an older checkpoint (no 1. Correctness: the CUDA path looks good
Top-1 matches, top-10 sets match. The divergence is smaller than on the existing Q8_0 path, so the new kernel is not introducing numerical error. 2. Memory: a bigger win than the file-size delta suggests
But free memory after startup improves by ~11 GB, well beyond the 2.14 GiB the file shrank:
My guess is that the Q4_K tensors no longer exist simultaneously as an aligned artifact and a mapping, which accounts for roughly 6.7 GiB of it — I have not verified the rest, so treat the exact figure with caution. On GB10 (CPU and GPU share one 128 GB pool) this is a real change: 3. Prefill: ~16x slower, and this is the blocker
Decode is essentially free (−1.2%) — impressive for halving the precision of the attention projections. All the cost is in prefill, exactly where your comment above Practical effect for a long-context server: a full 194681-token prefill went at a steady 54.6 t/s, i.e. ~59 minutes instead of ~4.5 (I aborted the run at 17%). So on this machine the patch trades a context ceiling we can now afford for one we can no longer fill — a 15k-token page would go from ~17s to ~4.5 minutes. That makes the dequant + GEMM prefill path you mention in the comment the deciding factor here, rather than a later optimization. Is that something you're planning? If you push a prefill path, I'm happy to re-run this whole sweep on the same machine — the harness is set up and a rebuild plus full sweep is under an hour. 4. Perplexity — reported for completeness onlySame 1882-token sample: AProjQ4 12.730 (avg_nll 2.5439) vs AProjQ8-0731 11.108 (avg_nll 2.4077). Please do not read this as a +14.6% quantization cost — it is dominated by the checkpoint difference, and it is inconsistent with your own +0.99% measured within one checkpoint. I mention it only so the number isn't misread if it surfaces elsewhere. If you publish an AProjQ8 GGUF from the same older checkpoint, I'll rerun and give you a clean quality delta. Also worth noting for anyone reading this from the DGX Spark side: this is a different issue from #705, which is about routed Q4_K experts having no aligned repack builder. This PR touches dense tensors only and does not OOM. |
|
@adamlawi, |
|
Retested
Decode is unchanged (17.32 / 14.90 / 14.82 / 14.67 vs 17.55 / 15.08 / 14.97 / 14.83 for Q8_0, i.e. −1.1%), so the MMQ path did not cost anything on the decode side. Two more checks, since an optimization like this could plausibly trade accuracy for speed — it did not:
No Memory is unchanged from my previous report, as expected: startup spans 6.06 GiB (vs 8.20 GiB for AProjQ8), 259 aligned artifacts / 74.16 GiB, and ~11 GB more free memory at every context size. So on a 128 GB GB10 this now looks like a straight win: ~2 GiB smaller resident model plus ~11 GB more free memory, for ~4.5% prefill and ~1% decode — and Nice work — that was a fast turnaround. One question about the other half of the commit, |
|
Standing offer, so you don't have to ask: when you publish the GGUF for the newer DeepSeek checkpoint, I'll run the full suite on the DGX Spark and post the numbers here — That run would also finally give a clean same-checkpoint quality delta, which neither of my perplexity numbers so far can provide (the AProjQ4 file I tested is from the older checkpoint, and I don't have a matching AProjQ8 baseline for it). The same applies to No rush from my side — the machine is a stock DGX Spark that stays available, and a rebuild plus the whole measurement suite takes well under an hour. Just ping me here. |
|
The new gguf is online https://huggingface.co/antirez/deepseek-v4-gguf/tree/8268ad9384d11cbb63f472d097e499a9c59709ec. It's made by requantizing the q8. I know it's not the best choice, but on my hardware it's the only thing I can do. |
|
try use DS4_CUDA_NO_Q4_ATTN_GROUPED_TOK8=1 DS4_CUDA_NO_Q4_QKV_PAIR=1 in the next run |
|
Ran
So They also hurt numerics, which I did not expect:
With the new paths active the divergence is worse than the Q8_0 baseline; with them off it is back to the The good news — decode improved, and it is not from these two paths:
Decode is ~+5% over One more thing worth a look: perplexity on the same 1882-token sample is 12.7836 (A) and 12.7904 (B), versus 12.7302 on Summary from this machine: keep whatever gave the decode win, and either fix or gate the grouped-tok8 / QKV-pair paths — on GB10 they cost 38% of prefill and worsen decode-consistency. Happy to re-run anything on request. (And the |
|
@adamlawi Feel free to try the requantization test
|
|
1. Requantization./deepseek4-quantize \
--source-gguf DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
--template <same file> \
--attention-proj q4_k \
--out ...-AProjQ4-SExpQ8-OutQ8-chat-v2-0731-requant.ggufConverted 215 tensors Two notes for anyone repeating this:
2.
|
| avg_nll | top1 | first_match | avg_lcp | |
|---|---|---|---|---|
| yours: Q8 (macOS, older ckpt) | 0.632957 | 82.79% | 44 | 3.830 |
| yours: Q4 (macOS, older ckpt) | 0.653162 | 82.88% | 41 | 3.420 |
mine: Q8 -0731 (CUDA) |
0.404811 | 85.86% | 55 | 5.150 |
mine: Q4 requant -0731 (CUDA) |
0.407383 | 85.34% | 56 | 5.540 |
Q8 → Q4 delta: +3.19% for your pair, +0.64% for mine.
Please don't compare the absolute values across our two pairs — the data/flash fixtures were collected from the official DeepSeek V4 Flash 0731 API, so they match my checkpoint and not the older one. That alone explains most of the level difference (0.405 vs 0.633). Only the within-pair deltas are meaningful, and even those differ in backend and checkpoint.
Worth noting: on my pair first_match and avg_lcp actually go up slightly with Q4 (56 vs 55, 5.540 vs 5.150), while top1 drops 0.5 pp — so greedy behaviour is essentially preserved.
Classic perplexity on my own 1882-token sample agrees: 11.108 → 11.316, i.e. +1.87%. That also settles the confusion from my earlier comments — the +14.6% I reported before was almost entirely the checkpoint difference, not the quant.
3. Speed and memory on the requantized -0731
| ctx | prefill Q8 | prefill Q4 | decode Q8 | decode Q4 |
|---|---|---|---|---|
| 2048 | 817.32 | 789.63 | 17.55 | 18.41 |
| 4096 | 892.89 | 850.92 | 15.08 | 15.71 |
| 6144 | 883.59 | 842.99 | 14.97 | 15.61 |
| 8192 | 877.75 | 839.80 | 14.83 | 15.44 |
Decode is ~5% faster than Q8_0 (that gain is from the rest of 8b84e878, as reported earlier), prefill is ~4% slower. Startup spans 8.20 → 6.06 GiB, aligned artifacts 474/78.71 GiB → 259/74.16 GiB, and free memory after startup:
| ctx | Q8 -0731 |
Q4 requant -0731 |
|---|---|---|
| 200000 | 9.8 GB | 23.1 GB |
| 300000 | 8.1 GB | 20.3 GB |
So on a 128 GB GB10 this variant now gives us --ctx 300000 with more headroom than --ctx 200000 had before, at +0.64% avg_nll and with faster decode. That is a straightforward win for long-context serving, and it is only possible because --source-gguf exists — thanks for adding it.
Happy to re-run any of this, or to try an imatrix-guided requantization if you think it is worth measuring the difference.
|
Can you share th GGUF? |
|
Final data point on the requantized Full-window load test, one session, prompt sized to fill the window, sampling
So it is 1.5x more usable context with ~5.8x more headroom at the worst moment. The memory profile over the run: start 21.9 GB → min 14.6 GB → median 19.0 GB → end 17.4 GB. The minimum is a brief spike, not a plateau — we were within 1 GiB of it for only about 6 seconds. That matters here: on the Q8_0 model, The model also answered correctly from the full 293k-token context (a sensible summary of the input), so this is not just an allocation test. One caveat on the prefill number, since it looks lower than my earlier sweep: 293,390 tokens in 449 s is ~653 t/s effective, versus 840 t/s on the short bench frontiers. That is the attention cost growing with window length, not a regression — the Q8_0 baseline shows the same pattern (735 t/s when filling 200k, versus 878 t/s on the bench). Don't compare those two full-window numbers across the models, they cover windows of different sizes; the like-for-like prefill comparison is still the bench table (−4%). Summary of where this lands for a 128 GB DGX Spark, all on the same
For long-context serving on this hardware that is a clear win, and none of it would have been reachable without Still happy to test the newer-checkpoint GGUF when you publish it, and to re-run the grouped-tok8 / QKV-pair comparison if you change those paths. |
|
Sorry for the delay — I posted the load-test results before spotting your question. I'd rather give you the recipe than the file, because the GGUF is fully reproducible from a source you already have, and it saves you an 78 GiB download. It is a deterministic transform of cd gguf-tools && make # the checked-in binary predates --source-gguf
SRC=DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf
./deepseek4-quantize \
--source-gguf "$SRC" \
--template "$SRC" \
--attention-proj q4_k \
--out DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-0731-requant.ggufExpected result, so you can confirm you got the same thing:
If the sha256 matches, you have a bit-identical copy of what produced all the numbers I posted. Note there is no imatrix involved (the KV block only stores the imatrix filename, not the vectors), so this is the unweighted double-quantization path — which is what makes the +0.64% avg_nll result interesting rather than disappointing. If you specifically want our file rather than your own rebuild — e.g. to rule out a toolchain difference on aarch64 — say the word and I'll arrange an upload; I just didn't want to push 78 GiB at you for something one command reproduces. |
|
I see your point; I just wanted to make a direct comparison. Could you upload the result in TSV format? I’d like to compare it with the requant using imatrix. |
|
Here you go: https://gist.github.com/adamlawi/bf83e3b7efb90fd28fa9db0281c8589e Both files are per-case
26 columns as emitted by One caveat for the comparison: only the delta inside a pair is meaningful here. The fixtures come from the 0731 API, so they match this checkpoint — absolute values won't line up against a run on a different one. And a question, since you mention a requant using imatrix: where do you get imatrix vectors for the dense tensors? On my side |
|
It seems to improve even further with the imatrix; I'm still finishing up the tests, but the results look positive. Unfortunately, it takes hours on my Mac. If you want to try the imatrix I created, it's available here: https://huggingface.co/antirez/deepseek-v4-gguf/discussions/22/files |
|
Pushed follow-up commit This commit:
The DGX A/B in the previous comment disabled both CUDA paths together, so this does not claim that either path was independently isolated. It makes the known-good fallback permanent while preserving the unrelated decode gain. Official-continuation quality scoreLower
The clean same-backend CUDA comparison is Q8 vs Q4 without imatrix: Q4 changes Q4+imatrix TSV SHA-256: Imatrix: Score command: gguf-tools/quality-testing/score_official "$Q4" \
gguf-tools/quality-testing/data/flash/manifest.tsv \
/tmp/score-q4-imatrix-0731.tsv 4096 \
--ssd-streaming --ssd-streaming-cache-experts 16Machine/backend: Apple M1 Pro, 16 GB, Metal, SSD streaming with 16 cached experts. The first scorer process was SIGKILLed after 72 cases from cumulative memory pressure; cases 72-99 were resumed in a fresh process and merged. The final TSV was validated as exactly 100 unique case IDs with no duplicates or manifest mismatches. Local checks run on this commit: make clean
make
make ds4_test
./ds4_test --metal-kernels
make -C gguf-tools
git diff --check
CUDA compilation/regression and the post-change DGX benchmark still need to be rerun on the DGX Spark. |
|
Hi @adamlawi, would you like to run a test with dspark added on your machine? |
|
Ran the full set on the DGX Spark. Gist updated with the third TSV: https://gist.github.com/adamlawi/bf83e3b7efb90fd28fa9db0281c8589e 1. Q4 + imatrix, same backend, same buildYour imatrix comparison was cross-backend, so here is the single-backend version — all three on CUDA, one build (
Q8 → Q4 no imatrix: +0.635%. Q8 → Q4+imatrix: −2.070%. Q4 no imatrix → Q4+imatrix: −2.688%, and that number is now clean of any backend difference. So the weighted Q4 requant is better than the Q8 it came from, on a model 2.14 GiB smaller. Worth noting the Q8 baseline is itself an imatrix build, so this isn't "weighted vs unweighted" — I don't have an explanation for why it lands below Q8, only the measurement. Sample is small (100 cases, 2,313 target tokens). Our CUDA figure (0.396432) is also slightly better than your Metal run (0.398263), so the cross-backend comparison was understating the imatrix, not overstating it. 2.
|
| ctx | prefill no-imx | prefill +imx | decode no-imx | decode +imx |
|---|---|---|---|---|
| 2048 | 835.51 | 840.27 | 18.50 | 18.48 |
| 4096 | 906.55 | 905.78 | 15.78 | 15.83 |
| 8192 | 887.83 | 890.24 | 15.54 | 15.56 |
Differences are under 0.6% and go both ways. Memory identical: 259 aligned artifacts / 74.16 GiB. We've switched our production endpoint to the imatrix model on that basis.
4. DSpark — currently slower here
823186e2, Q4+imatrix, --temp 0, 128 tokens, 1,574-token prompt:
| prefill | decode | |
|---|---|---|
| without DSpark | 755.63 t/s | 18.71 t/s |
| with DSpark | 851.31 t/s | 14.08 t/s |
The support model loads cleanly (stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 metadata_errors=0, capture on layers 40,41,42), but decode drops ~25%.
Treat this as a signal rather than a verdict: it's a single run, the prompt is synthetic and highly repetitive (which should favour speculation), and I don't see accept-rate statistics in the output — so I can't tell whether drafts are being rejected or the overhead is in the module itself. The prefill difference (756 vs 851) is also unexplained, since DSpark shouldn't touch prefill; that suggests run-to-run variance of over 10%, which argues for caution on the decode figure too. Happy to rerun with a realistic agent prompt, or with accept-rate logging if you can point me at a flag.
|
@adammawi can you retake the test now? |
|
@OPS-NeoRetro Happy to help, but let me first say exactly what the tool in this branch can and cannot give you, so you can tell me whether it is what you need.
if (first_logits_path && case_n == 0 && i == 0 && !dump_logits(...))
Two practical consequences on our side:
There is also something I can give you immediately at zero cost: the full per-case TSV that So, to avoid burning windows on the wrong artifact:
Our platform, for context: GB10 / sm_121 (DGX Spark), CUDA, |
|
@GiorgioOppo Retested at PrefillSame binary throughout, interleaved DEFAULT /
These are round-2 numbers, not the two-round mean, and I want to be explicit about why. Round 1 carries a first-load artifact in two cells: Two results in one measurement:
Decode is unchanged (−0.7…+0.2%), as expected for a prefill-side change. Q4 still leads Q8 on decode by +13.9…+18.0%. Quality: no movementThe grouped prefill path is numerically neutral here. The small numerical shift I reported earlier today against the 11.08 baseline is still present and now confirmed by a third independent run, so it came in with the I verified the path is actually activeAfter being wrong about this once today, I am not asserting activation without proof. Two arms:
The control arm fails hard, which proves the code reaches that decision for our shapes, so For the merge decision, the two CUDA levers point opposite ways
So the configuration that ships by default is the good one, and the 16-warp kernel is a cost on this chip if anyone enables it. Both are numerically exact, so this is purely a scheduling question, not a correctness one. Window: 12 min 19 s of downtime, production restored automatically, zero |
Now that we are winning, extend the test to see if we can maintain the lead, thanks |
|
So at this point q4 is winning on both prefill and decode in both metal and cuda, we only have RoCm left on which we are winning only in decode but losing 2% on prefill |
|
@kyuz0 Do you have time to take a test on RoCm? |
It'll need to be a few days sorry, I have all my devices busy on long 24 hours benchmarks. |
|
@GiorgioOppo Extended range as you asked, at Before the numbers, one methodological finding that affects everyone benchmarking this PR, not just us. The arm that runs first wins by ~0.4–0.5 ppMy earlier extended-range runs put the arms in a fixed order and always measured Q8 last. That was wrong. Throughput declines monotonically over a measurement window on this box — in two earlier windows, all 24 arm-to-arm comparisons drifted downward, so whichever arm runs later is penalised. Measured directly, with everything else held constant:
The bias is larger than three of the four effects we are trying to measure. At 32768 the sign of the result depends entirely on which model you load first. This is not thermal: SM clock sat at 2405 MHz for the whole two hours with no throttling flags, temperature 31–61 °C. So the numbers below come from a design that balances it: round 1 runs Result: 10 windows over 2 hours, 20 paired samples per contextPure prefill (
All four intervals exclude zero, so the tilts are real, but they are fractions of a percent. Q4 is ahead at 8k and marginally behind above it. Nothing here approaches the ~2% you quote for ROCm. The series had converged well before it ended: adding the tenth window moved every mean by ≤0.02 pp. More repetitions would not sharpen this further. Two corrections to my own earlier postsThe −1.0% I reported at 16–64k was mostly positional bias, not a real deficit. With the order balanced it is −0.34%. I would drop that earlier figure entirely. My What is unchanged and solid
Suggestion for the other testersIf @iammac2 or @trueimage are comparing Q4 against Q8 at differences below ~1%, the fixed-order effect will dominate their numbers too. Alternating the order between rounds costs nothing and removes it. Happy to share the harness if useful. Raw CSVs, per-run SM clock and temperature, and the full log are kept on our side — say the word if you want any of it attached. |
|
I'm planning to close this PR and open a cleaner one without the various intermediate tests. What do you think? |
Do you think it's noise or a solid result? |
|
@evandhoffman Could you do another test on the final commit 6a20b13? |
M5 Max 128 GB re-test at
|
6a20b13 cold start |
6a20b13 hot start |
2669a8e (2026-08-30) |
|
|---|---|---|---|
decode q4/q8, 3-rep paired median |
1.155 | 1.147 | 1.155 |
| frontiers where q4 > q8 | 32 / 32, every rep | 32 / 32 | 32 / 32 |
prefill q4/q8 |
1.003 (0.980–1.027) | 1.021 | 0.998 |
The ratio reproduces to three decimals across five weeks and 226 commits. Prefill is indistinguishable for the third independent time — AProjQ4's gain remains decode, and only decode.
| ctx | q4 gen | q8 gen | ratio |
|---|---|---|---|
| 2,048 | 48.80 | 41.62 | 1.173 |
| 18,432 | — | — | 1.154 |
| 34,816 | — | — | 1.145 |
| 51,200 | — | — | 1.135 |
| 65,536 | — | — | 1.129 |
Same shape as before: widest at short context, narrowing with depth, never inverting.
Something worth knowing before quoting any single number
The advantage shrinks within a session, and it is not heat.
Per-rep paired median, both runs:
| rep 1 | rep 2 | rep 3 | |
|---|---|---|---|
| run 1 | 1.187 | 1.147 | 1.139 |
| run 2 | 1.194 | 1.156 | 1.132 |
Absolute sweep-median decode falls q4 −10.0%, q8 −4.1% across the three reps (run 1: −9.5% / −5.1%). q4 loses more, so the ratio narrows.
I assumed thermal throttling and measured it. Die temperature, sampled every 60 s, after a 300 s idle to a stable baseline:
14:21:38 34.85 C baseline, idle
14:25:39 78.11 C peak, 4 minutes in
14:38:40 69.55 C
14:46:41 70.43 C still under full load, sweep 6 of 6
The machine saturates in about two minutes and then runs cooler for the rest of the session while throughput keeps falling. After minute four, temperature and decode move in opposite directions. So this is not die temperature — sustained-power limits on a longer time constant, or accumulated memory-system state across repeated 80 GiB model loads, are both still candidates and this run does not separate them.
Practical consequence: a cold single-rep benchmark of this pair reports about 1.19; three reps report 1.155. If a contributor posts a number near 1.19, that is consistent with these results and not a disagreement — it is one cold rep.
Method
- Worktree detached at
6a20b13,make clean && make -j18, Metal. - Weights from
refs/pr/22ofantirez/deepseek-v4-gguf, unmodified. AProjQ4 is 84,420,584,288 bytes, matching the PR. Resident 78.62 GiB. ds4-bench --prompt-file speed-bench/promessi_sposi.txt --ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128, fully resident, no SSD streaming.- Arms interleaved at sweep level (q4, q8, q4, q8, q4, q8), 3 reps. Not a block design — with 10% within-session drift, blocks would confound the ratio with the drift.
sysctl iogpu.wired_limit_mb=114688→ 112.00 GiB Metal working set.- Run 2 additionally: 300 s idle to a stable baseline first, temperature logged throughout.
New in the runtime since 2669a8e, recorded because it changes what is being measured, not because anything is attributed to it:
ds4: Metal 4 tensor API enabled for Tensor kernels
ds4: drift-patch flags hc_stable=on norm_unify=on kv_raw_f32=off
rope_exp2_log2=off math_safe=off tensor_matmul=on
What this does not cover
- Decode only. No agent loop, no end-to-end wall time.
- No quality scoring this pass. I did not re-run the
--temp 0coherence check; the August run's no-drift result is not re-certified here. - This GGUF pair only —
IQ2XXS-w2Q2K-AProj{Q4,Q8}-SExpQ8-OutQ8,OutQ8on both arms, so nothing is said about quantizing the output head. - I could not answer the open checklist item. From rep 1 alone it looked like AProjQ8 had regressed against my August numbers; three reps showed both arms had simply drifted together, so this run gives no evidence either way on Q8-on-PR versus Q8-on-
main. That still wants a dedicated interleaved run, which I am happy to do if it is useful — say the word.
Full write-up, per-rep tables, the complete temperature log and raw CSVs: evanwtf/local-llm#91 — evanwtf/local-llm#91
I'm late, but I need the full logits before the sampler. Like, every token generated. I need to find if the difference between AProjQ4 and AProjQ8 are just quantization noise and/or FP accumulation order in kernels. I need to find MSE, RMS, SAD, Hamadard SATD, square sum, sum, dot product, negatively-accumulated dot product, average, mean absolute differences, median absolute differences, mean differences, sum of differences, median differences, minimum, maximum, negative sum, exponential sum and square root sum. I'm kind of scientific here, but those are among the most useful metrics to know what a very long vector actually is. |
|
@OPS-NeoRetro more than anything, perhaps now we should make a benchmark like @kyuz0's mini bench If I'm not mistaken, @trueimage did it with only 1 incomplete pass and showed excellent results |
|
@evandhoffman would you have 24 hours to let the benchmark run? |
I probably can kick it off right now and let it go 24h. Is there an existing script for 24h benchmark run? |
@kyuz0 Can you help him, you who have more experience with the benchmark? |
|
I would continue to use this pr for general info to keep the real pr clean |
try to take a look here https://www.tbench.ai/run, but not having the right hardware I've never even looked at how it works |
|
Cross-referencing, since this PR is now the benchmark archive: the M5 Max re-test at Note that |
10 participants
PR #621 — consolidated AProjQ4 results
This comment supersedes the exploratory and intermediate benchmark reports in the thread. The tables below retain only the latest meaningful matched AProjQ4/AProjQ8 comparisons and exclude known first-load artefacts from steady-state conclusions.
Note
Current PR head:
6a20b131e18f6a1650706d8f76e15f60ac2bc91f.Some of the latest complete backend A/B runs predate this exact head. Before merge, the compact validation matrix should therefore be rerun once on the final SHA.
What this PR adds
This PR implements explicit support for the DeepSeek V4 Flash AProjQ4 layout.
The following five dense attention-projection tensor families are converted from
Q8_0toQ4_Kin each of the 43 layers:attn_q_aattn_q_battn_kvattn_output_aattn_output_bThis represents 215 converted tensors in total. Routed experts, shared experts, output quantization and checkpoint identity remain unchanged.
The implementation includes:
This is support for one explicit DS4 model layout, not generic support for arbitrary Q4 GGUFs.
Executive summary
Across the three GPU backends, AProjQ4 now has a consistent and substantial decode advantage. CUDA has also crossed Q8 in steady-state prefill. ROCm has reduced its original large prefill deficit to a small residual gap.
Model footprint
The saved memory can be used as:
Metal results
M5 Max, resident execution
Three interleaved repetitions per model at the 2K frontier:
AProjQ4 exceeds 50 generated tokens/s and is faster than AProjQ8 at every measured decode frontier.
Full Metal decode sweep
Context range: 2,048 to 65,536 tokens, 32 frontiers, three interleaved repetitions.
Summary:
CUDA / GB10 results
The default grouped attention-A prefill path introduced at
35b87ddais the first CUDA configuration in this PR where Q4 clearly exceeds Q8 in steady-state prefill.The 2K row is omitted here because Q8 showed a first-load artefact. The 4K–8K rows are the reliable steady-state result.
Decode remains substantially faster with Q4:
The default path is the correct one for GB10:
The 16-warp path should therefore remain experimental or architecture-gated; the grouped path is the production win on GB10.
ROCm / gfx1151 results
The original ROCm implementation had a large Q4 prefill penalty. After TILE8, direct-WMMA, staging and scheduling work, the residual deficit has been reduced to approximately 2–3%, while decode remains consistently faster.
Latest complete published steady-state A/B table:
The later K128 staging work is intended to close the remaining gap further; the current reported state is approximately 2% behind Q8 in prefill. The exact final-head table should replace the table above once that rerun is published.
ROCm conclusions:
k0GPU-hang bug is fixed;Startup, cold-start and time-to-stable-throughput
Q4 reaches usable and stable performance faster than Q8. These measurements should be kept separate from steady-state prefill throughput.
* The CUDA and ROCm 2K rows document first-load behaviour, not steady-state prefill. Q8 was still in a cold-start condition, while Q4 had already reached its normal throughput range. These rows are deliberately excluded from the main prefill comparison.
The practical conclusion is that Q4 not only uses less memory, but also tends to reach stable execution sooner and reduces time-to-first-token.
Quantization quality
Same-checkpoint CUDA comparison on the tracked official-continuation fixture:
avg_nll0.4048112510.8586250.4073833040.8534370.3964324030.860355On this fixture, the imatrix-built AProjQ4 model shows no measured quality regression and obtains the best measured NLL, first-token match, LCP and API top-1 values of the three tested models.
This is a limited fixture of 100 cases and 2,313 target tokens. It demonstrates preservation on the measured workload; it is not proof that Q4 is universally better than Q8.
Advantages
Lower memory footprint: 2.14 GiB saved without changing the routed experts, shared experts, output quantization or checkpoint identity.
Faster decode on every tested GPU backend:
CUDA prefill is now faster than Q8: approximately +2.0% to +2.4% at the reliable 4K–8K frontiers.
Faster startup and first response: lower model-preparation time, lower residency-request time and lower first-token latency.
Quality is preserved on the tracked fixture: the imatrix Q4 artifact produces the best measured values in the matched comparison.
Correctness gaps are fixed: Q4 tensors no longer enter Q8-only execution paths that could produce BOS loops, corrupted output, non-finite logits or failed prefills.
Broader validation: CPU oracles, backend parity tests, canaries, fail-closed
REQUIREcontrols and production-shape benchmark harnesses have been added.Useful tooling: direct same-checkpoint GGUF-to-GGUF requantization avoids rebuilding from hundreds of gigabytes of original safetensors.
Disadvantages and remaining work
ROCm prefill has not completely reached Q8. The remaining difference is approximately 2–3% at the longer reliable frontiers.
Published Metal data do not yet prove a prefill win. They show statistical parity on M5 Max and a clear decode win. A current-head A/B is needed before saying that Metal wins both phases.
The exact final SHA still needs one compact cross-backend rerun. The branch moved after some of the latest complete Metal, CUDA and ROCm comparisons.
The PR is very large: 189 commits, 117 changed files and roughly 78k additions. This raises review, maintenance and regression risk beyond the core AProjQ4 feature.
There is a large runtime-tuning surface. Many optimized paths have default, rollback, opt-in and strict diagnostic controls. The default/experimental distinction must remain clearly documented.
The experimental CUDA 16-warp path is slower on GB10. It is numerically correct but costs approximately 6.3% prefill there and should not become the GB10 default.
Mixed ROCm Q4-A/Q8-B parity cases still need an explicit final status. They do not represent the published all-Q4 AProjQ4 GGUF, but the remaining test result should not be hidden.
Coverage is not universal. CUDA SSD streaming, non-GB10 CUDA, multi-GPU/tensor-parallel execution and a full CPU end-to-end performance run have not all received equivalent model-backed validation.
A compiled benchmark binary is currently tracked in the PR.
speed-bench/metal_q4_attn_out_a_direct_benchshould be removed from the commit and generated locally from its.msource.This is not generic Q4 support. It intentionally recognizes and executes the specific AProjQ4 layout.
Benchmark provenance
2669a8e9…35b87dda…5c5b1d5…; final K128/current-head rerun still to be added6a20b131e18f6a1650706d8f76e15f60ac2bc91fConclusion
The central objective of this PR has been achieved:
CUDA now wins both prefill and decode. Metal has a strong and broad decode win, with published prefill currently at parity. ROCm also wins decode and has reduced its prefill deficit to a small remaining 2–3%.
The remaining work before merge is validation and cleanup rather than a redesign:
Quantization quality
Same-checkpoint CUDA comparison on the tracked official-continuation fixture:
avg_nll0.4048112510.8586250.4073833040.8534370.3964324030.860355On this fixture, AProjQ4 with the importance matrix has approximately 2.07% lower NLL than the matched Q8 control and does not show a measured quality regression.
This is a limited fixture of 100 cases and 2,313 target tokens. It is evidence that quality is preserved for this matched comparison, not proof that Q4 is universally better than Q8. The difference between the two Q4 rows also shows that the importance matrix is an important part of the final artifact.
Backend results
Metal — Apple M5 Max, resident
Isolated 2K result
AProjQ4 exceeds 50 generated tokens/s and produces the first token approximately 3.54 ms earlier.
Decode sweep, 2K–64K
Across the complete sweep:
The decode advantage becomes smaller as context increases, but it never reverses.
The published end-to-end Metal prefill measurement is effectively tied. Therefore the defensible Metal conclusion is decode win plus prefill parity, rather than a claimed prefill win.
Source: M5 Max Q4/Q8 benchmark
CUDA — NVIDIA GB10 /
sm_121The default grouped Q4 attention-A prefill path is the first published CUDA configuration in this PR where AProjQ4 exceeds the matched AProjQ8 model in prefill.
The 2K result is intentionally excluded from the headline comparison because the Q8 arm showed a first-load cold-start artifact.
Stable prefill frontiers
Average advantage across 4K–8K: approximately +2.24%.
Decode remains ahead of Q8 by approximately +13.9% to +18.0% across the measured frontiers.
The grouped prefill path itself was measured as approximately +4.4% to +4.6% over the previous default CUDA path, and the corresponding quality outputs were byte-identical.
An important configuration detail:
The default shipping configuration is therefore the faster one for GB10. The 16-warp path should remain experimental or architecture-gated rather than becoming the general default.
Source: final CUDA grouped-prefill benchmark
ROCm — AMD Strix Halo /
gfx1151The latest complete clean public short-context table shows:
The 2K frontier is omitted because Q8 repeatedly exhibited a first-frontier warm-up artifact, making that row unsuitable for the steady-state comparison.
Subsequent ROCm work has targeted the remaining gap with wider K64/K128 WMMA staging. The current status can reasonably be described as approximately 2–3% behind Q8 in prefill, pending publication of a final raw table for the latest head.
The decode result is already unambiguous: AProjQ4 is roughly 11–12% faster.
The previous deterministic GPU-hang issue in the
k0loop has been fixed. The default real-model path completed cleanly without GPU hangs, non-finite output, BOS loops or corrupted text.Source: ROCm clean Q4/Q8 rerun
Cold-start and time-to-first-token behavior
Q4 has a noticeably better startup profile than Q8. This should be described as a cold-start / model-preparation / first-token advantage, rather than claiming that every internal warm-up kernel is faster.
Q8 also repeatedly showed a large first-load or first-frontier prefill dip in several benchmark windows, while Q4 reached its normal throughput more directly. Those contaminated Q8 rows have been excluded from the steady-state performance conclusions, but they are relevant to interactive cold-start behavior.
In practical terms, Q4:
Advantages
Lower memory footprint. AProjQ4 saves approximately 2.14 GiB without changing the checkpoint or the routed/shared/output quantization mix.
Faster decode on every tested GPU backend. The gain is approximately 15.5% median on the M5 Max Metal sweep, 13.9–18.0% on CUDA/GB10, and 11.5–12.6% on ROCm/gfx1151.
CUDA prefill now exceeds Q8. The stable 4K–8K results show a 2.0–2.4% advantage.
Better cold-start behavior. Model preparation, residency and first-token latency all benefit from the smaller dense-attention representation.
No measured quality regression on the tracked fixture. The imatrix Q4 model produced the best measured NLL in the matched comparison.
Fixes a real correctness gap. DS4 could previously load Q4 dense-attention tensors while dispatching Q8-only assumptions, leading to BOS loops, corrupted output, non-finite logits, incompatible kernels or prefill failures.
Cross-backend implementation. CPU reference, Metal, CUDA and ROCm all understand the explicit AProjQ4 layout.
Direct requantization workflow. The final model can be produced directly from the matched Q8 GGUF using the dense importance matrix, without returning to the original full-precision checkpoint.
SSD-streaming support. The smaller footprint can be converted into additional expert-cache capacity on memory-constrained machines.
Substantial validation infrastructure. The PR adds backend-specific oracles, parity tests, canaries, boundary tests, benchmark harnesses and fail-closed
REQUIREcontrols.Disadvantages and remaining risks
The PR is extremely large. At the current snapshot it contains 189 commits, touches 117 files and reports approximately 78,663 additions and 8,167 deletions. This significantly increases review and integration risk.
Benchmarks are not all on the exact current head. The branch continued to change after the strongest Metal, CUDA and ROCm benchmark snapshots. A compact final rerun on
6a20b131or the actual merge SHA remains necessary.ROCm prefill is still slightly behind Q8. The remaining difference is small compared with earlier versions, but it has not yet crossed parity in the last complete published table.
Some mixed ROCm Q4-A/Q8-B parity cases remain unresolved or diagnostic. They do not affect the published all-Q4 AProjQ4 GGUF, but the final status should be explicitly documented.
The best CUDA results are GB10-specific. Non-GB10 CUDA, CUDA SSD streaming and broader architecture coverage are not established by the current benchmark set.
Multi-GPU and tensor-parallel coverage is incomplete. The PR touches shared engine and placement code, so this remains an integration-risk area.
CPU is primarily a correctness backend. End-to-end CPU AProjQ4 performance is not a demonstrated advantage.
Some optimized paths depend on architecture-specific policy and environment controls. This adds maintenance cost and makes it important to document which paths are default, opt-in, diagnostic or rollback-only.
Warning-clean ROCm builds still need attention. Several published ROCm builds completed successfully but emitted warnings, while the release QA document treats new warnings as failures.
Generated executables/blobs should not be committed. Tracked benchmark or test binaries should be removed from the PR and produced by the build instead.
The quality fixture is limited. Its positive result should not be generalized beyond the tested 100 cases and 2,313 target tokens.
Merge assessment
The core feature is valuable and has been demonstrated to work:
Before merge, the remaining work should be limited to release validation rather than additional feature expansion:
With those items recorded, the measured memory reduction, startup improvement and cross-backend decode gains provide a strong justification for the AProjQ4 layout.
Questa è la formulazione che userei adesso, senza pubblicarla. Il punto che non scriverei ancora è “Metal wins prefill”: con i numeri presenti nella PR, la frase tecnicamente sostenibile è “Metal prefill parity, strong decode win”. Analogamente, per scrivere ROCm −2% in modo netto sostituirei la tabella da −3% con il run più recente che produce quel risultato.
Non ho pubblicato nulla. La bozza sotto è quella che userei come commento principale, in inglese per coerenza con la PR.
Ho mantenuto due cautele importanti:
Consolidated AProjQ4 status — final results, benefits and trade-offs
1. What the PR changes
This PR adds explicit support for the DeepSeek V4 Flash AProjQ4 layout.
The 215 dense attention-projection tensors are converted from
Q8_0toQ4_K:attn_q_aattn_q_battn_kvattn_output_aattn_output_bThe routed experts, shared experts, output head, checkpoint identity and remaining quantization mix are unchanged.
This is support for one explicit DS4 model layout. It is not intended to be generic support for arbitrary Q4 GGUF files.
2. Executive performance summary
3. Model size and memory
The 2.14 GiB reduction is available as:
A CUDA long-context capacity test also demonstrated the practical effect:
These rows use different context sizes and are therefore a capacity demonstration, not a like-for-like speed comparison.
4. Quantization quality
Final same-checkpoint comparison using the tracked official-continuation fixture:
avg_nll0.4048112510.8586250.3964324030.860355On this fixture, AProjQ4 with the dense imatrix shows:
avg_nllapproximately 2.07% lower;0.94;0.00173.This demonstrates no measured quality regression on this fixture. The sample contains 100 cases and 2,313 target tokens, so it must not be interpreted as proof that Q4 is universally better than Q8.
5. Metal
M5 Max, resident, isolated 2K frontier
Three interleaved repetitions per model:
The published numerical evidence therefore establishes:
Full decode sweep
The resident M5 Max sweep covered every 2K frontier from 2,048 to 65,536 tokens:
The decode advantage narrows as context grows, but it never inverts across the tested frontiers.
The final current-head Metal prefill table should be added here before claiming a numerical prefill win. The already-published M5 data supports parity, not a statistically established lead.
6. CUDA / DGX Spark GB10
The final default grouped attention-A prefill path is the first CUDA configuration in this PR that puts AProjQ4 ahead of the matched AProjQ8 model.
The 2K row is intentionally excluded below because the Q8 first load showed a cold-start artefact. The reliable 4K–8K rows are:
Therefore:
CUDA scheduling note
Two CUDA optimization controls have opposite results on GB10:
The configuration that ships by default is the faster one. The 16-warp path should remain experimental or be gated by architecture-specific policy.
7. ROCm / Strix Halo gfx1151
The latest fully posted clean A/B table before the final K128 staging work was:
The subsequent K128 staging work is reported to have narrowed the remaining prefill difference to approximately 2%. That result should be accompanied by its exact current-head table in the final comment.
The stable conclusion is:
8. Startup, residency and warm-up behaviour
Q4 reaches useful steady-state operation sooner because less model data must be prepared and made resident.
Several benchmark windows also showed a substantial first-frontier depression when Q8 was loaded cold, while Q4 reached its normal throughput immediately or much sooner. Those cold Q8 points were deliberately excluded from steady-state performance comparisons rather than being counted as a Q4 throughput win.
The low-level Metal kernel warm-up itself was approximately 6–7 ms for both models. The practical Q4 advantage comes primarily from:
9. Main advantages
Lower memory use: 2.14 GiB less model memory, directly increasing context or cache headroom.
Faster decode on every tested GPU backend: Metal, CUDA and ROCm all show a clear and repeatable Q4 decode advantage.
CUDA wins both phases: the default grouped path now puts Q4 ahead in both prefill and decode on GB10.
Faster startup behaviour: lower preparation, residency and first-token times, with Q4 reaching steady throughput more quickly.
No measured quality regression: the imatrix-based Q4 model equals or improves the tracked Q8 metrics on the current 100-case fixture.
Correctness gap fixed: Q4 models no longer load successfully only to be executed through incompatible Q8-only assumptions, avoiding the previously observed BOS loops, corrupted output, non-finite logits and prefill failures.
Broader backend support: CPU reference, Metal, CUDA and ROCm implementations, including resident and SSD-streaming paths where supported.
Much stronger validation surface: CPU oracles, canaries, parity tests, decode-consistency tests, fail-closed
REQUIREcontrols, benchmark harnesses and expanded environment-variable/release documentation.10. Disadvantages and remaining risks
ROCm prefill is not yet fully at parity: the remaining difference is approximately 2%, despite the clear decode win.
The PR is very large: approximately 189 commits, 117 changed files, 78,663 additions and 8,167 deletions. This increases review, maintenance and regression risk.
Some optimizations are architecture-specific: a path that wins on one GPU can regress on another. The CUDA 16-warp experiment is a concrete example, costing approximately 6.3% on GB10.
The final head has moved since some backend benchmarks: the exact merge candidate SHA must receive one final compact validation run.
Mixed ROCm Q4-A/Q8-B parity remains a known diagnostic issue: it does not affect the shipped all-Q4 AProjQ4 layout, but it should remain documented rather than hidden.
Coverage is not universal: CUDA SSD streaming, non-GB10 CUDA hardware and complete multi-GPU/tensor-parallel execution have not received the same end-to-end coverage.
Generated binaries or other build artefacts must not remain in the diff: only source, tests, reproducible data and documentation should be committed.
Quality conclusions remain fixture-bound: 100 cases and 2,313 target tokens are useful evidence, but not a universal evaluation.
11. Final assessment
The core objective of this PR has been achieved:
Before merge, the remaining work should be limited to:
Subject to those final validation and cleanup steps, the performance, memory and correctness case for AProjQ4 is positive.
Verifica dei numeri usati nella bozza
La PR è attualmente aperta, mergeable, con head
6a20b131, 189 commit, 117 file modificati, 78.663 aggiunte e 8.167 rimozioni. La descrizione documenta i 215 tensori, il risparmio di 2,14 GiB e il confronto qualitativo.Il test Metal M5 Max riporta 53,35 contro 45,69 tok/s nel decode, 824 contro 825 nel prefill, 23,32 contro 26,86 ms sul primo token e un vantaggio decode in 32 frontiere su 32.
Il test CUDA sul percorso grouped riporta +1,98%, +2,35% e +2,39% nel prefill a 4K, 6K e 8K, con un vantaggio decode compreso tra +13,9% e +18,0%.
L’ultimo A/B ROCm completo pubblicato prima dell’ottimizzazione K128 riporta circa −3% nel prefill e +11,5–11,6% nel decode.
I dati di preparazione ROCm, 19,700 contro 17,583 secondi, e il risparmio di memoria sono riportati nella validazione Strix Halo; i tempi Metal di residency e warm-up provengono dal test interleaved M3 Ultra.
Prima di pubblicarla, completerei soltanto le due tabelle mancanti: il nuovo prefill Metal realmente vincente e il ROCm post-K128 che porta il divario dal 3% a circa il 2%.
Non è stato pubblicato nulla.
Questa è la versione che userei come commento iniziale. Farei però una correzione rispetto alla formulazione originale: i dati pubblicati non dimostrano ancora un vantaggio di Q4 nel prefill Metal. Sul test M5 Max il risultato è
824contro825 tok/s, quindi sostanziale parità; su CUDA, invece, il sorpasso è documentato tra+1,98%e+2,39%. Su ROCm l’ultimo sweep completamente tabellato mostra circa−3%; descriverei quindi il divario come circa 2–3%, non come un 2% esatto.Consolidated AProjQ4 results — final measurements only
This comment consolidates the relevant final AProjQ4 results and intentionally omits intermediate experiments, superseded regressions, unsuccessful tuning attempts, and cold-start-contaminated performance claims.
Important
The current PR head is
6a20b131e18f6a1650706d8f76e15f60ac2bc91f.The latest complete backend-specific A/B measurements were produced on different recent snapshots. A compact Metal, CUDA, ROCm and CPU validation should therefore still be repeated on the exact merge SHA.
What this PR changes
AProjQ4 keeps the same DeepSeek V4 Flash checkpoint and the same routed-expert, shared-expert and output quantization mix, but converts the five dense attention-projection families in all 43 layers from
Q8_0toQ4_K:attn_q_aattn_q_battn_kvattn_output_aattn_output_bThis corresponds to 215 converted tensors.
The PR adds tensor-type-aware execution for these projections on CPU, Metal, CUDA and ROCm, including prefill, decode and the relevant SSD-streaming paths. It also adds direct GGUF-to-GGUF requantization with dense-imatrix support, backend-specific tests, CPU oracles, canaries and fail-closed diagnostic controls.
Executive summary
The consistent result across all three GPU backends is that AProjQ4 materially improves decode. CUDA has now also crossed Q8 in prefill. ROCm is close, but still slightly behind in prefill.
Model size and memory
On fully resident systems, this becomes additional context and KV-cache headroom. Under SSD streaming, it can instead be used for a larger routed-expert cache and fewer repeated expert reads.
Quantization quality
Same-checkpoint CUDA comparison on the tracked official-continuation fixture:
avg_nll0.4048112510.8586250.4073833040.8534370.3964324030.860355On this fixture, AProjQ4 with the dense imatrix shows no measured quality regression and records the best measured NLL of the three arms.
The fixture contains 100 cases and 2,313 target tokens. These results demonstrate preservation on this test set; they should not be interpreted as proof that Q4 is universally better than Q8.
Metal
M5 Max, resident, isolated 2K frontier
AProjQ4 exceeds 50 generated tokens/s and is faster than AProjQ8 in decode.
Decode sweep, 2K–64K context
Summary:
1.155, or +15.5%.CUDA / DGX Spark GB10
The default grouped attention-A prefill path introduced at
35b87ddais the first published CUDA result where AProjQ4 overtakes AProjQ8 in prefill.The 2K row is omitted from the steady-state conclusion because the Q8 arm showed a first-load cold-start artifact.
Decode is unaffected by this prefill-only routing change, and AProjQ4 retains a +13.9% to +18.0% decode advantage over AProjQ8 across the measured frontiers.
Quality checks for the grouped path were byte-identical:
CUDA scheduling note
DS4_CUDA_ENABLE_Q4_GROUPED_ATTN_A_PREFILLDS4_CUDA_Q4_MMQ_16WARPThe shipping default is therefore the faster path on GB10. The experimental 16-warp implementation remains useful for investigation or other hardware, but should not be enabled by default on this device.
ROCm / Strix Halo gfx1151
ROCm now builds and runs the AProjQ4 model successfully. The current short-context picture is a clear decode win with a small remaining prefill gap.
The 2K prefill result is excluded from the steady-state conclusion because the Q8 arm shows a cold-first-frontier artifact.
The branch has reduced a much larger earlier ROCm prefill gap to only a few percent. The remaining difference should be summarized as approximately 2–3%, pending an exact-current-head rerun.
Correctness results on the default path are clean:
Startup and warm-up behavior
AProjQ4 also reaches useful execution more quickly than AProjQ8.
CUDA and ROCm measurements also repeatedly show a pronounced Q8 first-load or first-frontier throughput dip that is absent or much smaller with Q4. Those cold-start-contaminated rows are deliberately excluded from the steady-state prefill claims, but they support the operational observation that Q4 reaches stable throughput sooner.
Advantages
Disadvantages and remaining risks
Conclusion
The main objective of this PR has been achieved:
The PR provides a meaningful user-facing improvement, particularly for memory-constrained resident execution, SSD-streaming cache capacity and decode-heavy workloads.
Before merge, the remaining work should be limited to cleanup and exact-final-SHA validation rather than additional feature expansion.