vulkan: tune mat-vec rows for batched inference on Strix Halo - #27909
Conversation
github-actions
Bot
added
Vulkan
| const bool rm_abs = device->vendor_id == VK_VENDOR_ID_AMD && device->architecture == AMD_RDNA3; | ||
| auto const &rm_int_n = [&](uint32_t rows, uint32_t i) { return (rm_abs && i >= 4) ? 4u : rows; }; | ||
| // RDNA3: Static 4 rows for all types bench faster than the default | ||
| auto const &rm_id = [&](uint32_t rows) { return rm_abs ? 4u : rows; }; |
There was a problem hiding this comment.
I did the simplest thing that worked for tuning this case here. If in the future this needs to be scaled differently for different hardware we need some better mechanism than my rm_abs flag with hard coded 4 as the only option. Not sure if I should bother to explore that now though?
0cc4m
left a comment
0cc4m
left a comment
There was a problem hiding this comment.
looks good otherwise, I can reproduce the perf increase.
On RDNA3 above four columns a static 4 rows for all types benches faster than the default.
mul_mat_vec_id has no column dimension to switch on. On my Strix Halo machine, a static 4 is faster here than the defaults across types and batch sizes.
9970e84 to
6709f1e
Compare
|
@ggerganov The Apple Vulkan CI is consistently broken with |
0cc4m
added
the
merge ready
Hm, weird. Running it manually on that machine does not trigger the problem. |
54 upstream commits since 1844325. Conflict resolutions keep the fork's validated mmid stack authoritative and graft upstream additions alongside: - mul_mat_id pc: union of our padded_N/use_row_lists/fusion_flags/tile_list_base and upstream's n_experts/hoist_row_ids (ggml-org#26686); mmq declares tile_list_base as layout filler so both shader blocks match the one host struct. - row-id loading: our row-lists prepass first, upstream's hoisted path second (their gate is n_as <= 256, so qwen4exp n_as=512 never hoists), ballot scan as the fallback; hoisting yields the prepass buffer to our row lists. - K-vs-N padding (ggml-org#27925): took upstream's y_staged_row_stride/K-pad scheme; padded_N stays as a pc-layout filler (no merged shader reads it). - Strix mat-vec tuning (ggml-org#27909): took upstream's rm_id/rm_int_n RDNA3 lambdas alongside our GGML_VK_MMV_RM_* sweep knobs. - kept our epilog-fusion ctx fields next to upstream's fused_topk_qsa (ggml-org#28032). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…VK_RDNA3_MMV_TUNE ggml-org#27909 benches faster on RDNA3 dGPUs but costs tg on gfx1151 (Strix Halo APU): post-merge decode -5% shallow / -13% @32k at equal draft acceptance. Default restores the pre-merge dispatch (generic 1/2 rows per WG); =1 re-enables the upstream tuning. Launcher forwards the knob. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@0cc4m Strangely, the failures seem to start happening after this PR: #28032 For example, here I just ran the CI on the commit right before that PR and the CI passes: https://github.com/ggml-org/llama.cpp/actions/runs/33434794357/job/99628492302 Any ideas? |
…rg#27909) * vulkan: RDNA3 static mat-vec rows above four columns On RDNA3 above four columns a static 4 rows for all types benches faster than the default. * vulkan: RDNA3 static mat-vec-id rows mul_mat_vec_id has no column dimension to switch on. On my Strix Halo machine, a static 4 is faster here than the defaults across types and batch sizes.
|
I think that might be coincidental, e.g. my PR #27952 also has the same failure, but it predates the radix PR and does not contain its commit yet: https://github.com/ggml-org/llama.cpp/actions/runs/33251797688/job/99098628443?pr=27952 |
|
I see. I'll try to restart the mac runners today and clear their caches. See if it makes a difference. |
3 participants
Overview
I found that by tuning
rm_kq_intandrm_stdq_intI'm able to get better performance for batch sizes 5-8 (mostly in that range, some operations get improvements on smaller batch sizes as well) on Strix Halo.Additional information
This is my first attempt at this kind of tuning, so it's certainly possible that I've done something wrong. I've tried to be as rigorous as possible in my testing and attach my findings here.
I investigated if
rm_int_ncould be a function of the batch size, but it appears to be a shoulder between n=4 and n=5, at least on this hardware. It seems to me like even more performance could be squeezed out by tuning on a per type basis, but that would require some kind of pattern/infrastructure for that.I have gated these changes on
device->vendor_id == VK_VENDOR_ID_AMD && device->architecture == AMD_RDNA3so it should not impact GPUs from other vendors/other generations of AMD GPUs. I only have a Strix Halo machine to test on though, so I can't say if these improvements hold up on other RDNA3 cards. Someone more experienced might be able to tell if this kind of tuning generally works across the whole architecture or not?KLD and PPL figures
To ensure that these changes didn't break anything, I calculated KLD and PPL figures against master.
Method:
Results:
On these models, nothing appears broken.
Micro benchmarks
Method:
test-backend-ops perf -o MUL_MAT -b Vulkan0on both branches, comparing results against each other.Results:
In this one, we expect speedup at n=5 and above.
Method:
test-backend-ops perf -o MUL_MAT_ID -b Vulkan0on both branches, comparing results against each other.Results:
But a big caveat! First, the results here swing wildly so the noise floor is high. Notably, q6_k has huge variance between runs, but doesn't reach this kernel on this hardware in practise anyway (ggml_vk_should_use_mmvq(...) returns false for q6_K on AMD). Second, the results are not really comparable to real inference on these types. The direction seems to agree between the micro bench and real inference though. I'm not able to test real inference speed on real quants in all those formats, so I ran the micro benchmarks and real inference for pure quants in a few of these formats.
Batch inference performance
Method:
llama-batched-bench -npp 256 -ntg 256 -npl 1,2,4,8 -c 8192 -b 2048 -ub 512 -ngl 99compared between master and branch.Result:
Good speedup at batch size 8.
Spec. decoding performance
This was my goal all along, with better batched performance we should be able to speed up single batch inference with spec decoding. I test this with two prompts, one that gives very high draft acceptance (copy) and one that gives a modest/normal (normal) draft acceptance:
muse-glimmer-30B-kquant-dynamic with Dflash drafter:
Qwen3.6 35B with MTP:
So at 7 draft tokens we see a nice boost on both models, and on the dense model we see it already from 4 draft tokens.
Requirements