Batch decode tuning - #7
Conversation
Raise rows-per-workgroup from 1 to 4 for the q8_1 integer-dot mat-vec pipelines on RDNA3, gated on column count: Q8_0 from 7 columns, the other ten types from 4. Radeon 8060S (gfx1151), pure quants, decode, mean over 11 types: n=1 0% (gate inactive below 4 columns) n=8 +25.2%
Raise rows-per-workgroup from 1 to 4 for the q8_1 integer-dot mul_mat_vec_id pipelines on RDNA3. These carry no column index, so a single row count applies at every batch size. Radeon 8060S (gfx1151), pure quants, decode, mean over 11 types: n=1 +0.3% (inside the 1.3% noise floor) n=8 +6.3%
IQ4_XS had no q8_1 mat-vec pipeline and fell back to the float dequant path. Add one, with K_PER_ITER=16. Radeon 8060S (gfx1151), pure IQ4_XS, decode vs the float dequant path: n=1 +7.8% n=8 +37.2%
Q6_K was excluded from the integer-dot mat-vec path on every non-Intel vendor. Allow it on RDNA3 for n > 1; n = 1 keeps the existing path. Radeon 8060S (gfx1151), pure Q6_K, decode: n=1 0% (code path unchanged) n=8 +29.3%
Q2_K's min correction recomputed the activation sum once per row, but it varies only with the column. Output is bit-identical. Radeon 8060S (gfx1151), pure Q2_K, decode: n=1 +0.3% (at the noise floor) n=8 +5.0%
|
Great find, and thanks for the PR! Theres a few of us working on merging improvements / toolboxes, youre welcome to open up a PR there too: https://github.com/halo-box/strix-llama.cpp |
|
I found a slight regression in MoE models. Numbers looked good overall;. If you want to target just Strix devices, you can also add device->uma, Validated on gfx1151 (Radeon 8060S, RADV, uma: 1, int dot: 1) against ad914eb. Interleaved arms, warm pass discarded, exclusive GPU, within-arm spreads 0.0 to 1.3%. Claude summary of suggested changes: The regression:
|
|
Good idea to have a fork to collaborate on. I don't see any of your commits there though, is that planned? Or has upstream llama.cpp caught up with your prefill optimizations? BTW, this PR looks interesting and could make that redundant anyway, needs testing: ggml-org#27952 I'll check the MoE regression. |
|
They're on their way, the repo was created a few days ago, and working on getting them in there as we speak, The longterm plan is for most of this to be upstreamed anyways, so these toolboxes will end up being either redundant or tweaks vs commit fixes, theres still a lot to squeze out of Vulkan, so not redundant quite yet, |
… columns Raise the Q6_K MMVQ threshold from n > 1 to n >= 4. rm_int_n only gives Q6_K four rows per workgroup at i >= 3, so at two and three columns a single output row amortizes sixteen half-word loads and the gain is flat; it steps at n=4. MoE expert mat-vec runs at those batch sizes and loses a little there. Restrict both this gate and the mat-vec row counts to UMA devices. Both fired on any AMD_RDNA3, which includes discrete N31/N32; every measurement behind them is gfx1151 with shared LPDDR5X.
|
I was able to reproduce it, and your Claude was right by the looks of it. I've pushed a fix. |
|
Ill close this off and cherry pick it into the repo from upstream, thanks! |
* Get started with Onyx * Add architecture * Skip keys handled in super() * Loading tensors * Shorten * Graph * Apply suggestion from @pcuenca * Remove norm now embedding in transformers weights * Add eot * Explicit output_multiplier * Handle post_norm_eps * No super call; unhardcode eot. The pattern `self._set_vocab_gpt2()` seems preferred throughout the codebase, and it allows `set_vocab()` to be called from a different part of the Python class hierarchy: the drafter model converter that we may need eventually. * Register for drafting * DFlash: inherit rope type from the linked target. Another option would be to store it in the gguf file itself. * mmproj conversion Note: some fields to be renamed after the implementation works. We are keeping compatibility with the reference Meta gguf for testing purposes. * "clip" header declarations * Load mmproj * Pre-processing * Graph * Go back to using delimiters. Otherwise our generations are worse. Transformers does not use them. We need to trace inputs to verify whether they are equivalent. * downsample_factor -> merge_size * Add vision graph lol, forgot from a previous commit * Additional renames, align with llama.cpp / transformers * Prefer _size instead of independent _h and _w * Fix token layout Co-authored-by: Young Han <younghan@fb.com> * onyx: bring the chat parser onto the onyx branch common/chat.cpp on this branch has no Onyx handling, so a converted model serves malformed chat: the assistant preamble leaks into content ("to=self<|message|>...") and tool calls fail with HTTP 500 "The model produced output that does not match the expected peg-native format" common_chat_params_init_onyx exists on onyx-fair-patch, added there by 8bb73dd3d. It was never on this branch, so this is not a regression -- the two lines developed independently. The code here is taken verbatim from that commit. It is the clean side of `git merge origin/onyx-fair-patch`: chat.cpp is one of the files that merges without conflict. The full merge is not viable -- it produces 13 conflicts, including add/add on conversion/onyx.py and src/models/onyx.cpp where the q_norm-folding and metadata-scale approaches contradict each other, and Nathanw1014#4/Nathanw1014#7 are stacked on this branch's side of that. Verified on this branch: builds with 0 errors, converts an Onyx checkpoint, and serving it gives "4" for "What is 2+2?" plus a correct get_weather {"city":"Paris"} tool call, where the unported branch gives the two failures above. No converter or runtime changes are included, so this should not interact with the q_norm work. Co-authored-by: Beto de Paola <betodepaola@meta.com> * Less params, bilinear pos-emb interpolation as a graph op instead of CPU * Map to symbolic V_MMPROJ instead of strings * Make a couple params explicit * Patchify via build_inp() * No param for rope_theta * Small cleanup * Restore blank line * Unpermute, to adapt to the latest transformers checkpoint * Apply norm after token embeddings This follows the latest transformers approach. * Remove duplicated function * build_vit * onyx: use the model rope theta on sliding-window layers * DFlash: conversion from transformers drafter * Revert rope_type derivation from target NOTE: this breaks compatibility with Meta's distributed DFlash GGUFs, as the Q/K are stored in "NEOX" (rotated half) format, like in transformers. * Apply suggestion from @pcuenca * Set model type * Remove comment that will become obsolete * Hardcode post_norm_rms_eps instead of new param * Derive SWA+RoPE pattern from gguf array or scalar * Fix model type <-> number of layers * Reorder * Rename * Fix typo * DFlash: seed the draft KV cache from multimodal embedding batches `common_speculative_impl_draft_dflash::process()` returned early on any batch carrying embeddings, so an image prefill never had its target-layer features fused through the DFlash encoder and injected into the draft's KV cache. That left a hole spanning the image's positions, and the next injection at a post-image position failed to initialize its batch: ``` decoding image batch 1/1, n_tokens_batch = 256 decode: failed to initialize batch llama_decode: failed to decode, ret = -1 process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=17, offset=0) srv decode: failed to process speculative batch ``` Every image request with `--spec-type draft-dflash` failed with HTTP 500. Text-only was unaffected, since those batches carry token ids and were let through. Restore the earlier condition, which admits a batch that is either tokens or embeddings and skips only the degenerate neither/both cases. The rest of `process()` is already layout-agnostic -- it gathers features via `llama_get_embeddings_layer_inp()` and indexes `batch_in.pos[]` / `batch_in.seq_id[]`, none of which assume token ids -- so this is the whole fix. Validated against `muse-glimmer-30B-bf16.gguf` + `mmproj-muse-glimmer-30B-bf16.gguf` + a DFlash draft head, on an image describe-the-shapes request: - before: HTTP 500, `failed to process speculative batch` - after: HTTP 200, draft acceptance 0.34012 (167 accepted / 491 generated), mean len 3.04 Output equivalence holds, which is the property that matters: at temperature 0 the drafted response is byte-identical to the same request served with no draft attached (1213/1213 chars), so the draft is drafting correctly through the image context rather than merely not crashing. * Conversion: prefer rewrite to mapping * Revert "Conversion: prefer rewrite to mapping" This reverts commit a92d0ac. * fix lint * sliding_window metadata is not optional * disable state save/load * Apply suggestion from @pcuenca --------- Co-authored-by: Young Han <younghan@fb.com> Co-authored-by: Beto de Paola <betodepaola@meta.com> Co-authored-by: Daniel Han <michaelhan2050@gmail.com> Co-authored-by: ruanrms <ruanslv@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2 participants
These commits tune batch decode on Strix Halo.
The first two commits are basically the same PR I opened upstream (ggml-org#27909), except we can tune more specifically on this branch which is strix halo specific. Here I tune them by type instead of setting a fixed threshold/width for all types like in the upstream PR.
The integer-dot mat-vec pipeline for IQ4_XS is entirely Claude's work, I wouldn't have been able to do it myself. I noticed that IQ4_XS was slower than other types and asked Claude to figure out why, and he came up with this.
MMVQ for Q6_K was disabled on all platforms except Intel - I tested, and found that on Strix Halo it appears to be faster for batch sizes larger than 1, so it is now enabled under those conditions.
Q2_K is similar to the IQ4_XS work - entirely Claude, I noticed that Q2_K was unusually slow compared to other types and Claude found a fix.
Correctness
I've compared KLD and PPL between your branch and this PR branch and 10/13 types show no differences at all.
We change how each is computed (just the row count for Q5_0, and I've confirmed that changing just the row count on llama.cpp master gives the same result, so it's not something introduced in this PR), so small differences are probably expected. I notice that the prefill and decode flows for Q5_0 already differs by more than what was measured here, so I consider it a non-issue.
Performance improvements, per change
Measured using
llama-batched-benchon pure quants of Qwen3-4B (so basically, for each type I touch, create a pure quant of this model and benchmark that) and Ling-3.0-tiny for themul_mat_vec_idbench (same process, pure quant per type):mul_mat_vec_idrow countK_PER_ITER=16Each figure is measured on top of the previous commit, so they are not independent results we can sum up. See the next section for total end to end improvement.
Performance improvements, end to end
The end goal of this series of optimizations was to get better performance in single stream speculative decoding workloads (which is effectively batched decoding). Tested here with Qwen3.8-27B-Q5_K_S-bartowski.gguf with Qwen3.8-27B-DFlash2-Q4_K_M.gguf and spec-draft-n-max = 7 (so batch size 8). These figures are from my laptop, on a Strix Halo machine with more thermal headroom I'd expect higher absolute figures. Two prompts, one realistic/normal prompt, one designed to give high draft acceptance: