Skip to content

CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 - #24546

Open
ravel7524 wants to merge 7 commits into
ggml-org:masterfrom
ravel7524:mmqx-rdna3-routed-moe-tiling
Open

CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3#24546
ravel7524 wants to merge 7 commits into
ggml-org:masterfrom
ravel7524:mmqx-rdna3-routed-moe-tiling

Conversation

@ravel7524

@ravel7524 ravel7524 commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

This PR improves routed MoE prefill performance on RDNA3 by changing how llama.cpp chooses MMQ N-tile sizes. Instead of using the worst case expert width, it estimates the typical routed expert width and uses that when it is smaller than the RDNA3 maximum tile width. The launch grid still uses the original worst case width, so output coverage and correctness are preserved. The change is limited to host side tile selection in mul_mat_q_case and does not modify structs, call sites, launch parameters, or device kernels. Benchmarks on a Radeon Pro W7800/gfx1100 with ROCm 7.2.3 show large prefill gains in many routed MoE cases, while decode and non RDNA3 architectures are mostly unchanged.

If maintainers or other contributors have access to NVIDIA, CDNA, RDNA2, or RDNA4 hardware, the natural follow up would be to test the same routed width picker under the corresponding MMQ/launch policy and enable it per architecture where it is beneficial.

Benchmark Summary

Hardware/software:

  • AMD Radeon Pro W7800, gfx1100 / RDNA3
  • ROCm 7.2.3
  • HIP backend, -ngl 99
  • GGML_CUDA_FA_ALL_QUANTS=ON
  • -fa 1 -ctk q8_0 -ctv f16 -sm none -t 4 --poll 0 -r 5

The improvement depends on the physical ubatch/chunk width that reaches routed MoE MMQ, not total context length:

win when:          ncols_typical < mmq_x_max
crossover_ubatch: 128 * n_experts / n_expert_used

So long prompts can still benefit when they are processed in smaller ubatch chunks.

Benchmark sweep
model / quant experts ubatch pp512 pp1024 pp2048 pp4096 pp8192 tg128
Qwen3.6-35B-A3B Q6_K 256 2048 +48.46% +22.15% +7.57% n/a n/a -0.67%
Qwen3-30B-A3B Q6_K 128 512 +35.50% +33.79% +32.15% n/a n/a -1.11%
Qwen3-30B-A3B Q6_K 128 2048 +36.62% +13.90% -0.75% n/a n/a -0.89%
Qwen3-30B-A3B Q4_K_M 128 512 +33.68% +31.54% +29.81% +27.70% +23.86% +0.35%
Qwen3-30B-A3B Q4_K_M 128 2048 +33.77% +14.00% -0.08% -0.06% -0.01% -0.20%
Qwen3.6-35B-A3B Q3_K_M 256 512 +23.48% +22.37% +22.73% +22.91% +21.18% -0.08%
Qwen3.6-35B-A3B Q3_K_M 256 2048 +22.93% +6.99% +4.02% +3.92% +3.69% +0.15%
Gemma-4-26B-A4B Q4_K_M 128 512 +29.53% +28.52% +26.65% +25.46% +23.19% +0.04%
Gemma-4-26B-A4B Q4_K_M 128 2048 +29.87% +13.16% +0.03% +0.05% +0.04% -0.09%
Gemma-4-26B-A4B Q6_K 128 512 +25.05% +24.05% +23.21% +22.51% +20.59% +0.08%
Gemma-4-26B-A4B Q6_K 128 2048 +25.53% +11.75% +0.11% -0.04% -0.00% +0.50%

The 128 expert models show the expected crossover behavior at ubatch 2048: pp2048 and larger are neutral because the typical width has reached mmq_x_max = 128. The 256-expert Qwen3.6 shape remains below crossover and still improves through pp8192.

Requirements

  • Yes, I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, AI was used to help review the change and summarize benchmark results. I reviewed the final diff and can explain all code changes.

@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Jun 12, 2026
@ravel7524
ravel7524 marked this pull request as ready for review June 12, 2026 21:04
@ravel7524
ravel7524 requested a review from a team as a code owner June 12, 2026 21:04
@ravel7524

ravel7524 commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

Here is an additional sweep extending the prior pp8192 table upward. Values are MMQX vs base throughput deltas. tg128 is included as the decode control.

Hardware/software: AMD Radeon Pro W7800 (gfx1100), ROCm 7.2.3, HIP backend, -ngl 99, GGML_CUDA_FA_ALL_QUANTS=ON, -fa 1 -ctk q8_0 -ctv f16 -sm none -t 4 --poll 0.

model / quant experts ubatch pp8192 pp16384 pp32768 pp65536 tg128 largest clean
Qwen3-30B-A3B Q4_K_M 128 512 1661.44 -> 2049.97 (+23.39%) 1349.44 -> 1599.57 (+18.54%) 990.08 -> 1118.48 (+12.97%) 633.79 -> 682.82 (+7.74%) 133.61 -> 134.08 (+0.35%) pp65536
Qwen3-30B-A3B Q4_K_M 128 1024 2043.18 -> 2243.77 (+9.82%) 1602.36 -> 1726.49 (+7.75%) 1121.08 -> 1179.00 (+5.17%) 691.16 -> 711.71 (+2.97%) 130.59 -> 131.20 (+0.46%) pp65536
Qwen3-30B-A3B Q4_K_M 128 2048 2265.11 -> 2259.45 (-0.25%) 1730.70 -> 1731.02 (+0.02%) 1178.53 -> 1177.80 (-0.06%) 718.44 -> 718.40 (-0.00%) 130.73 -> 130.73 (+0.00%) pp65536
Qwen3-30B-A3B Q4_K_M 128 4096 2312.65 -> 2308.85 (-0.16%) 1758.76 -> 1757.65 (-0.06%) 1192.18 -> 1192.16 (-0.00%) 720.98 -> 720.85 (-0.02%) 129.29 -> 129.89 (+0.46%) pp65536
Qwen3-30B-A3B Q4_K_M 128 8192 2087.19 -> 2150.76 (+3.16%) 1675.31 -> 1674.92 (-0.02%) 1154.52 -> 1154.02 (-0.04%) 705.68 -> 705.59 (-0.01%) 128.90 -> 129.74 (+0.65%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 512 1849.80 -> 2279.65 (+23.24%) 1662.29 -> 2004.19 (+20.57%) 1401.96 -> 1623.81 (+15.83%) 1054.25 -> 1171.73 (+11.14%) 85.08 -> 85.02 (-0.06%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 1024 2278.15 -> 2502.82 (+9.86%) 1990.69 -> 2164.03 (+8.71%) 1617.03 -> 1722.56 (+6.53%) 1169.98 -> 1224.99 (+4.70%) 78.59 -> 84.29 (+7.84%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 2048 2474.42 -> 2467.24 (-0.29%) 2124.07 -> 2124.74 (+0.03%) 1702.03 -> 1702.09 (+0.00%) 1213.12 -> 1211.99 (-0.09%) 78.47 -> 84.25 (+7.98%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 4096 2448.56 -> 2442.26 (-0.26%) 2070.54 -> 2069.72 (-0.04%) 1668.67 -> 1668.29 (-0.02%) 1193.49 -> 1195.54 (+0.17%) 78.54 -> 84.19 (+7.79%) pp65536
Gemma-4-26B-A4B Q4_K_M 128 8192 2122.56 -> 2118.94 (-0.17%) 1854.58 -> 1854.11 (-0.03%) 1510.94 -> 1511.74 (+0.05%) 1110.49 -> 1108.28 (-0.20%) 77.39 -> 84.41 (+9.99%) pp65536
Gemma-4-26B-A4B Q6_K 128 512 1605.25 -> 1935.92 (+20.60%) 1460.33 -> 1733.39 (+18.70%) 1267.12 -> 1444.12 (+13.97%) 983.37 -> 1086.60 (+10.50%) 88.47 -> 88.33 (-0.15%) pp65536
Gemma-4-26B-A4B Q6_K 128 1024 2046.42 -> 2238.64 (+9.39%) 1811.34 -> 1966.32 (+8.56%) 1503.42 -> 1603.36 (+6.65%) 1117.89 -> 1171.41 (+4.79%) 86.96 -> 87.59 (+0.73%) pp65536
Gemma-4-26B-A4B Q6_K 128 2048 2283.96 -> 2275.69 (-0.36%) 1981.77 -> 1982.07 (+0.01%) 1617.79 -> 1616.75 (-0.06%) 1180.25 -> 1180.33 (+0.01%) 87.59 -> 87.60 (+0.01%) pp65536
Gemma-4-26B-A4B Q6_K 128 4096 2301.54 -> 2293.49 (-0.35%) 1963.89 -> 1961.98 (-0.10%) 1608.37 -> 1608.08 (-0.02%) 1175.85 -> 1175.53 (-0.03%) 87.41 -> 87.65 (+0.27%) pp65536
Gemma-4-26B-A4B Q6_K 128 8192 2028.64 -> 2020.28 (-0.41%) 1780.78 -> 1784.42 (+0.21%) 1470.22 -> 1468.89 (-0.09%) 1089.65 -> 1088.48 (-0.11%) 87.63 -> 87.72 (+0.10%) pp65536

Across the tested 128 expert models, the gain persists at long context for smaller ubatches, and the larger ubatches are effectively neutral rather than regressing. I can do additional long context benchmarking for 256 expert models

@github-actions github-actions Bot added the CUDA Related to the CUDA backend label Jun 19, 2026
@ravel7524

ravel7524 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

@jiachengjason would you be able to benchmark this PR on your RDNA4 GPU when you have time? Since you tested #23685 on RDNA4, having the same kind of validation here would be very helpful.

If you know anyone who could also test on RDNA2 or CDNA, that would be great as well. No pressure.

@ravel7524

ravel7524 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@IMbackK when you have time, could you sanity check whether this RDNA3 routed MoE MMQ selector direction looks reasonable? The PR is scoped to gfx1100/RDNA3 and uses measured W7800 results; I mainly want to know if the shape is acceptable or if you would prefer a different architecture gating/config style.

@GZGavinZhao

GZGavinZhao commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Do you expect this to work well on RDNA3.5 / Strix Halo as well? Happy to test!

@ravel7524

ravel7524 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Yes, the PR’s RDNA3 predicate also includes RDNA3.5 / Strix Halo, so the routed-width selector should be active there. I expect the same general direction, although the performance magnitude may differ from gfx1100. Testing would be very helpful, ideally the same base vs PR prefill and decode comparison used in the PR.

@IMbackK IMbackK self-assigned this Jul 7, 2026
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Aug 3, 2026
@justinappler @claude
strix-halo: test routed-MoE ncols picker vs static J=48 (PR ggml-org#…
d12f0e2 
…24546)

Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the
typical expert width instead of the worst case. That is the same problem the
static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end:
the cap clamps the search ceiling, the picker fixes the search objective. This
commit swaps the cap for the picker so the two can be A/B'd on gfx1151.

The PR does not apply as written - it patches mmq_x/mmq_x_max/
mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same
refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto
mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the
only arch we can measure, and narrowing the predicate keeps the A/B
single-variable. Upstream's predicate already covers RDNA3.5 via
GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo
should benefit, but nobody has ever tested it there.

On the routed-MoE path ncols_max is ne12 - the worst case of one expert
receiving every token - so the search never reaches ntiles==1 and simply runs
to whatever ceiling it is given. The picker instead tiles from
ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q
still builds the grid from ncols_max, so worst-case coverage is unchanged.

  - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width
    picker; J_max is now a uniform 128 and the search terminates on its own.
  - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in
    lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE
    arg shape from mmq.cu and a crossover case.

For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the
typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating
point - narrow enough that it may land in noise. Verified host-side without
ROCm: MoE J=64, dense J=128, no aborts across all 21 types.

Above ub=4096 the typical width reaches 128, the picker disengages and the
search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured
the typical=64 regime, so that is untested on this chip; production ub=2048
cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and
the keep/revert criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Aug 3, 2026
@justinappler @claude
strix-halo: revert routed-MoE ncols picker, flat vs static J=48
c9da8a0 
Benched b47bb31e1 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6
35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap
at the production operating point:

  pp512  @ d=0      1428.13 -> 1388.93  (-2.7%)
  pp512  @ d=2048   1299.39 -> 1316.26  (+1.3%)
  pp512  @ d=8192   1135.42 -> 1142.23  (+0.6%)
  pp512  @ d=16384   971.25 ->  977.36  (+0.6%)

Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the
grid does still cover ncols_max while the tile is sized from the typical width.

The tg128 control fell ~1.6% at every depth. This change cannot move tg -
decode goes through MMVQ where ncols_dst==1, so the picker never runs - which
makes that a session drift between two builds measured on different days, the
same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw,
pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth
is inside the host's ~2% noise floor, so the effect is smaller than this rig can
resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%.

So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before
the numbers existed ("narrow enough that it may well land in noise"), which is
the only reason that reading is credible rather than post-hoc. Reverting to the
static cap because it is the validated status quo, not because it won.

Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5,
on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is
gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/
nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width,
so there is much less worst-case over-sizing left to recover. That is the data
ravel7524 asked for on 2026-07-03.

Doc kept with the numbers and the untested ub>=4096 crossover regime recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@IMbackK

IMbackK commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

I think this is a reasonable idea, but it would have to be updated for the changes in #24127

@ravel7524
ravel7524 force-pushed the mmqx-rdna3-routed-moe-tiling branch from 9ab2b07 to 3087cea Compare August 5, 2026 18:48
@ravel7524

ravel7524 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

I adapted the implementation to the MMQ configuration changes from #24127 and I provisionally enabled the selector for NVIDIA Volta and newer so those architectures can be benchmarked.

IMbackK
IMbackK requested changes Aug 5, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh Outdated
@IMbackK

IMbackK commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

I tested it on CDNA 1 and it seams that its not entirely a win in all cases.

Model Test Profile b10288 t/s PR t/s Delta (%)
qwen35moe 122B.A10B Q4_K - Medium pp32 244.15 ± 2.58 291.14 ± 1.43 +19.25%
pp512 649.31 ± 3.81 733.93 ± 3.05 +13.03%
pp2048 1128.57 ± 7.95 1245.77 ± 2.55 +10.39%
tg32 52.26 ± 0.78 52.69 ± 0.87 +0.82%
deepseek2 30B.A3B Q8_0 pp32 402.87 ± 3.37 441.00 ± 8.24 +9.46%
pp512 1508.40 ± 5.74 1494.33 ± 4.37 -0.93%
pp2048 1327.65 ± 2.52 1308.98 ± 2.13 -1.41%
tg32 84.63 ± 1.21 86.51 ± 1.28 +2.22%
deepseek2 30B.A3B Q8_0 pp32 @ d16384 212.65 ± 1.43 223.26 ± 1.42 +4.99%
pp512 @ d16384 334.95 ± 0.46 323.71 ± 28.41 -3.36%
pp2048 @ d16384 315.96 ± 9.35 315.71 ± 9.81 -0.08%
tg32 @ d16384 56.34 ± 0.64 57.04 ± 0.63 +1.24%
gpt-oss 120B MXFP4 MoE pp32 @ d16384 327.72 ± 5.80 369.25 ± 3.87 +12.67%
pp512 @ d16384 912.97 ± 5.61 831.88 ± 0.86 -8.88%
pp2048 @ d16384 1257.41 ± 178.26 1183.48 ± 174.00 -5.88%
tg32 @ d16384 99.70 ± 0.73 99.47 ± 0.86 -0.23%

IMbackK
IMbackK previously approved these changes Aug 6, 2026
View reviewed changes

@IMbackK IMbackK left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont bother cleaning this up if its not uniformly performance positive at least on rdna3

Comment thread ggml/src/ggml-cuda/mmq.cuh Outdated
@IMbackK
IMbackK self-requested a review August 6, 2026 19:15
@IMbackK
IMbackK dismissed their stale review August 6, 2026 19:16

missclick

@github-actions github-actions Bot added OpenCL Issues specific to the OpenCL backend IBM zDNN issues specific to IBM zDNN Accelerator Hexagon mtmd Related to multimodal functionality (video/image/audio) jinja parser Issues related to the jinja parser AMD ZenDNN Issues related to the AMD ZenDNN backend OpenVINO WebGPU server/ui conversion vendor labels Aug 30, 2026
@ravel7524

ravel7524 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

missclick

@ravel7524
ravel7524 force-pushed the mmqx-rdna3-routed-moe-tiling branch from 9c9bcc8 to 7d8bc65 Compare August 30, 2026 12:30
@ravel7524

ravel7524 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Merged master and the conflict resolved.
Apologies for the churn in between, a bad push briefly put ~350 unrelated commits on this branch. That's been corrected; it's back to the original commits plus the merge, 10 files, +36/−12.
One thing on the results: 0b5be7e4a (#26284) retuned the RDNA3 CASE ladder earlier today, which is the table this PR builds on. My benchmarks predate it, so I'm re-running the A/B on gfx1100 against the new baseline and will post updated numbers before this is worth a review pass.

@ggerganov ggerganov removed the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 30, 2026
@ggerganov

ggerganov commented Aug 30, 2026

Copy link
Copy Markdown
Member

Ok, I will remove the "merge ready" label for now. Let us know when your A/B tests are complete.

@CISC CISC removed documentation Improvements or additions to documentation model Model specific build Compilation issues testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs Vulkan Issues specific to the Vulkan backend devops improvements to build systems and github actions server labels Aug 30, 2026
@ravel7524

ravel7524 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author
model / quant experts ubatch pp32 pp512 pp2048 pp8192 tg128
Qwen3-30B-A3B Q4_K_M 128 512 733.20 → 863.53 (+17.78%) 2289.62 → 3189.41 (+39.30%) 2197.17 → 2965.48 (+34.97%) 1819.55 → 2303.36 (+26.59%) 122.44 → 122.48 (+0.03%)
Qwen3-30B-A3B Q4_K_M 128 1024 733.06 → 860.32 (+17.36%) 2283.84 → 3182.35 (+39.34%) 2911.50 → 3308.42 (+13.63%) 2265.44 → 2506.82 (+10.65%) 122.89 → 122.29 (-0.49%)
Qwen3-30B-A3B Q4_K_M 128 2048 736.29 → 861.27 (+16.97%) 2280.25 → 3183.38 (+39.61%) 3321.97 → 3320.29 (-0.05%) 2528.55 → 2530.69 (+0.08%) 122.36 → 122.29 (-0.06%)
Qwen3-30B-A3B Q4_K_M 128 4096 733.61 → 860.68 (+17.32%) 2280.76 → 3186.56 (+39.71%) 3322.12 → 3320.10 (-0.06%) 2581.54 → 2572.63 (-0.35%) 122.36 → 123.23 (+0.71%)
Qwen3-30B-A3B Q4_K_M 128 8192 737.28 → 861.10 (+16.79%) 2283.13 → 3182.73 (+39.40%) 3325.78 → 3316.32 (-0.28%) 2391.97 → 2408.30 (+0.68%) 122.42 → 122.55 (+0.11%)
Gemma-4-26B-A4B Q4_K_M 128 512 797.88 → 890.61 (+11.62%) 2459.39 → 3233.34 (+31.47%) 2279.49 → 2932.43 (+28.64%) 2045.13 → 2560.17 (+25.18%) 81.47 → 81.64 (+0.21%)
Gemma-4-26B-A4B Q4_K_M 128 1024 797.64 → 890.94 (+11.70%) 2458.11 → 3226.82 (+31.27%) 2943.36 → 3260.88 (+10.79%) 2525.81 → 2762.04 (+9.35%) 81.54 → 81.62 (+0.10%)
Gemma-4-26B-A4B Q4_K_M 128 2048 797.25 → 889.69 (+11.59%) 2456.71 → 3228.90 (+31.43%) 3330.54 → 3324.15 (-0.19%) 2723.18 → 2701.49 (-0.80%) 81.39 → 81.52 (+0.16%)
Gemma-4-26B-A4B Q4_K_M 128 4096 795.16 → 890.82 (+12.03%) 2457.95 → 3229.32 (+31.38%) 3328.93 → 3325.16 (-0.11%) 2662.87 → 2664.16 (+0.05%) 81.54 → 81.56 (+0.02%)
Gemma-4-26B-A4B Q4_K_M 128 8192 795.25 → 887.73 (+11.63%) 2455.83 → 3230.34 (+31.54%) 3331.74 → 3327.79 (-0.12%) 2300.03 → 2298.01 (-0.09%) 81.47 → 81.48 (+0.01%)
Gemma-4-26B-A4B Q6_K 128 512 623.94 → 780.25 (+25.05%) 2005.69 → 2642.57 (+31.75%) 1880.54 → 2425.13 (+28.96%) 1741.74 → 2168.59 (+24.51%) 79.73 → 79.96 (+0.29%)
Gemma-4-26B-A4B Q6_K 128 1024 623.48 → 776.50 (+24.54%) 2000.23 → 2630.37 (+31.50%) 2499.55 → 2832.09 (+13.30%) 2189.62 → 2446.66 (+11.74%) 79.80 → 79.90 (+0.13%)
Gemma-4-26B-A4B Q6_K 128 2048 623.42 → 774.26 (+24.20%) 1997.23 → 2630.83 (+31.72%) 2921.14 → 2915.42 (-0.20%) 2449.62 → 2433.62 (-0.65%) 79.73 → 79.85 (+0.15%)
Gemma-4-26B-A4B Q6_K 128 4096 625.23 → 774.59 (+23.89%) 1996.45 → 2626.36 (+31.55%) 2921.40 → 2913.61 (-0.27%) 2459.81 → 2439.36 (-0.83%) 79.80 → 79.87 (+0.09%)
Gemma-4-26B-A4B Q6_K 128 8192 621.83 → 773.96 (+24.46%) 1999.66 → 2629.99 (+31.52%) 2917.04 → 2912.77 (-0.15%) 2149.42 → 2161.37 (+0.56%) 79.68 → 79.95 (+0.34%)
Qwen3.6-35B-A3B Q3_K_M 256 512 642.02 → 724.77 (+12.89%) 2137.51 → 2631.25 (+23.10%) 2075.64 → 2548.21 (+22.77%) 1857.17 → 2282.59 (+22.91%) 97.24 → 97.30 (+0.06%)
Qwen3.6-35B-A3B Q3_K_M 256 1024 641.49 → 721.41 (+12.46%) 2133.10 → 2626.16 (+23.11%) 2736.59 → 3064.64 (+11.99%) 2422.21 → 2692.86 (+11.17%) 97.24 → 97.36 (+0.12%)
Qwen3.6-35B-A3B Q3_K_M 256 2048 643.06 → 721.57 (+12.21%) 2132.36 → 2632.68 (+23.46%) 3195.90 → 3406.64 (+6.59%) 2820.19 → 2974.40 (+5.47%) 97.39 → 97.24 (-0.15%)
Qwen3.6-35B-A3B Q3_K_M 256 4096 641.20 → 722.80 (+12.73%) 2137.60 → 2630.12 (+23.04%) 3201.18 → 3411.91 (+6.58%) 3091.48 → 3075.36 (-0.52%) 97.28 → 97.47 (+0.20%)
Qwen3.6-35B-A3B Q3_K_M 256 8192 643.08 → 723.63 (+12.53%) 2134.86 → 2633.19 (+23.34%) 3206.19 → 3417.45 (+6.59%) 3046.17 → 3027.84 (-0.60%) 97.52 → 97.32 (-0.21%)
GPT-OSS-20B MXFP4 32 512 670.76 → 1062.08 (+58.34%) 3027.83 → 3670.29 (+21.22%) 2990.12 → 3567.64 (+19.31%) 2750.83 → 3213.06 (+16.80%) 151.38 → 151.72 (+0.22%)
GPT-OSS-20B MXFP4 32 1024 673.09 → 1066.60 (+58.46%) 3030.37 → 3667.32 (+21.02%) 3711.92 → 3708.66 (-0.09%) 3326.50 → 3308.92 (-0.53%) 151.30 → 151.54 (+0.16%)
GPT-OSS-20B MXFP4 32 2048 672.65 → 1064.39 (+58.24%) 3028.62 → 3667.55 (+21.10%) 4081.59 → 4085.46 (+0.09%) — 151.12 → 151.51 (+0.26%)
GPT-OSS-20B MXFP4 32 4096 672.28 → 1066.36 (+58.62%) 3030.76 → 3668.58 (+21.04%) 4080.02 → 4078.49 (-0.04%) — 151.37 → 151.75 (+0.25%)
GPT-OSS-20B MXFP4 32 8192 672.23 → 1064.85 (+58.41%) 3026.77 → 3673.09 (+21.35%) 4084.70 → 4081.69 (-0.07%) — 150.97 → 151.28 (+0.21%)

No regressions larger than 0.83% were observed. Results at and beyond the expected ubatch crossover were effectively neutral, while smaller ubatches showed consistent gains; tg128 also remained neutral.

@Dev-next-gen

Dev-next-gen commented Aug 31, 2026

Copy link
Copy Markdown

Independent validation on multi-GPU RDNA3 (gfx1101) with a 512-expert MoE. Short version: I reproduce large prefill gains, I see no correctness change, and four of your five failing checks cannot be caused by this PR — I could not reproduce the fifth either.

Setup

  • 6× Radeon RX 7800 XT (gfx1101, RDNA3), ROCm 7.2.2, HIP backend
  • Dual Xeon E5-2698 v4; the GPUs straddle two NUMA nodes, PCIe only, no peer-to-peer (rocm-smi --showtopoaccess reports False for every cross-GPU pair)
  • Model: Qwen3-Coder-Next-abliterated 80B-A3B, Q5_K_M, 52.94 GiB / 79.67 B params
    • qwen3next.expert_count = 512, qwen3next.expert_used_count = 10
    • so crossover_ubatch = 128 × 512 / 10 ≈ 6554, and -ub 512 sits ~13× inside the win region your formula predicts
  • Base: master c589f0e with this PR's changes applied (note on that at the end)
llama-bench -m <model> -ngl 999 -sm layer -ts 1/1/1/1/1/1 \
  -fa 1 -ctk q4_0 -ctv q4_0 -b 2048 -ub 512 \
  -p 512,2048,8192 -n 128 -r 3

Results

test master c589f0e + this PR delta
pp512 920.17 ± 76.37 1285.94 ± 142.80 +39.8 %
pp2048 1031.34 ± 3.30 1463.23 ± 1.58 +41.9 %
pp8192 989.09 ± 0.97 1374.96 ± 0.33 +39.0 %
tg128 55.07 ± 0.71 54.98 ± 0.64 −0.2 % (noise)

Decode is untouched, as expected. On pp2048 and pp8192 the standard deviation is under 0.4 %, so the gain is well outside the noise. This is on a different die from your W7800 (gfx1101 vs gfx1100) and across 6 GPUs with -sm layer, which I believe had not been covered yet.

Correctness

test-backend-ops test -o MUL_MAT_ID -b ROCm0, run on both builds:

master + this PR
MUL_MAT_ID cases 875 875
failures 0 0
unsupported 3 3
verdict 7/7 backends passed 7/7 backends passed

The set of case descriptors is identical between the two runs. I am not claiming a byte-identical log — the async CUDA graph warmup lines interleave into the verdict lines non-deterministically, so a raw diff of the output is not meaningful. Zero failures on both is the claim.

No regression on the routed-MoE path here.

I also checked end-to-end generation, since test-backend-ops compares ops in isolation with a tolerance and never looks at what the model actually emits. Same server config on both builds, greedy decoding (temperature 0, top_k 1, seed 42):

prompt prefill tokens outputs
4 short prompts (code, arithmetic, recall, exposition) a few dozen each character-identical
1 long prompt (a source file plus three questions on it) 5048 character-identical, 178 completion tokens on both

The long one matters because the short prompts barely exercise the changed path — at a few dozen tokens the prefill is one small ubatch, whereas the 5048-token prompt runs just under ten full -ub 512 batches through routed-MoE MMQ with the new tile selection. Byte-identical greedy output there is a stronger statement than op-level tolerance checks: tile size changes the reduction order, so float non-associativity could legitimately have flipped a near-tied logit and diverged. It did not.

CI triage

This is probably the more useful part, and it argues that most of the red is not yours.

gpu-rocm is failing on every open PR I sampled, with an identical signature — Process completed with exit code 8 after roughly two minutes:

PR gpu-rocm gpu-cuda
#27754 — model: add GLM-5-Next failure (2m13s, exit 8) success (37m)
#26438 — opencl: tune quant paths for Intel failure success
#27310 — spec: fuse the encoder into KV inject failure success
#24546 (this PR) failure (2m29s, exit 8) failure (29m, exit 8)

Both pass on master (9723942), so gpu-rocm looks like pre-existing infrastructure breakage rather than anything in this branch.

More importantly, three of your five failing checks are on backends this PR cannot reach. The diff is confined to ggml/src/ggml-cuda/ (nine mmq-config-*.cuh plus mmq.cuh), yet gpu-vulkan-intel-linux, gpu-webgpu-nvidia and gpu-openvino-low-perf are all red. Those backends are compiled independently of ggml-cuda, so a tile-selection change there cannot alter their results.

That left two hypotheses:

  1. CI-wide infrastructure trouble on 2026-08-30 — in which case a re-run may simply turn this green.
  2. A compile error under nvcc — if these jobs build all backends, a failure inside ggml-cuda would take down every job regardless of which backend it exercises. nvcc is stricter than clang on some constexpr constructs, and my build here is HIP/clang, so I would not have caught that.

I then tested hypothesis 2 directly, using the trick from ikawrakow/ik_llama.cpp#2339 — you do not need an NVIDIA GPU to compile for one. I built the ggml-cuda target under nvcc 12.6 (-DCMAKE_CUDA_ARCHITECTURES=86) inside a nvidia/cuda:12.6.0-devel-ubuntu24.04 container, from master and from this branch:

tree configure build objects result
master c589f0e OK rc=0 153 libggml-cuda.so linked
+ this PR OK rc=0 153 libggml-cuda.so linked

No compile error under nvcc either way, so hypothesis 2 is out and the CI failures look environmental.

Going further, I compared the generated device code. Raw .o files differ (nvcc builds are not byte-reproducible), so I disassembled with cuobjdump -sass. Across every MMQ template instance the delta is a single immediate:

-  /*0050*/  MOV R10, 0x3c5 ;   // 965
+  /*0050*/  MOV R10, 0x3c6 ;   // 966

Same instruction count, same addresses, same scheduling; arange.cu.o is bit-identical as a control. The cause is that this PR adds one line to the struct at mmq.cuh:173, which shifts NO_DEVICE_CODE in mul_mat_q from line 965 to 966 — and NO_DEVICE_CODE expands to no_device_code(__FILE__, __LINE__, ...).

So the entire device-code footprint of this PR on NVIDIA is one line number inside an error path that only runs when a template specialisation is skipped. For sm_86 under nvcc 12.6, it is a no-op on CUDA at the SASS level.

Worth requesting a CI re-run — I do not think there is anything here to fix.

Note for anyone reproducing from the .diff

The branch itself is current and GitHub reports it mergeable. However the flat .diff no longer applies onto recent master: mmq-config-rdna3.cuh has diverged around your change (the GGML_TYPE_Q1_0 CASE lines were retuned upstream, e.g. 128, 2, 64, 64 → 256, 2, 128, 64). The two lines this PR actually modifies are unchanged, so it is a context-only conflict — I applied the other nine files with git apply and ported those two by hand.

For what it is worth, I have moved my own inference server onto this patch after the checks above.

Happy to re-run anything on this hardware: other ubatch sizes, other quants, more experts, or the full test-backend-ops suite.

AI usage disclosure: AI assistance was used under my supervision for the benchmark harness and the CI triage. The hardware, the measurements and the conclusions are mine.

@ravel7524

ravel7524 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Sorry @ggerganov, forgot to ping you, benchmark results are positive and the additional comment confirms the results

@Casten-Wang

Casten-Wang commented Sep 1, 2026

Copy link
Copy Markdown

I ran an independent Windows/gfx1151 backport experiment based on the typical-expert-width MMQ configuration idea in this PR.

Environment: Windows 11, AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151, 20 CUs), ROCm 7.1, driver 32.0.31021.5001, Qwen3.6-35B-A3B-UD-Q4_K_M, full GPU offload, and mmap=off.

Important limitation: this was implemented in an older MMQ scanner, so it is not a test of the current PR HEAD and was not rebased after PR #26284.

Operator correctness after making the loader, kernel and write-back wave counts consistent:

  • Q4_K: 153/153 passed
  • Q5_K: 23/23 passed
  • Q6_K: 23/23 passed

Operator median improvements for typical expert width 16:

  • Q4_K uniform routing: +21.66%
  • Q4_K skewed routing: +15.41%
  • Q5_K uniform routing: +7.45%
  • Q5_K skewed routing: +13.79%
  • width-64 negative controls remained within approximately ±0.33%

Q4_K_M model-level Prefill at p2048 / ubatch=512 / batch=4096 / threads=4, 12 samples per version:

  • baseline median: 1630.17 token/s
  • candidate median: 1667.29 token/s
  • change: +2.28%

The broad configuration initially regressed ubatch=1024 by -1.74% because it selected an unvalidated x32/y64 configuration. Restricting the typical-width override to the validated x16 case removed that regression:

  • ubatch=1024: +0.12%
  • ubatch=2048: -0.07%
  • Decode, 128 tokens: -0.20%

The final scoped candidate passed Q4_K 107/107, Q5_K 23/23, and Q6_K 23/23 checks.

These results support testing a narrowly scoped gfx1151/rdna3-5 configuration, especially for typical width 16. A fresh A/B against the current PR HEAD and current master is still required before treating this as direct validation.

@Casten-Wang

Casten-Wang commented Sep 2, 2026

Copy link
Copy Markdown

Tested on Windows 11, ROCm 7.1, Radeon 8060S (gfx1151).

  • Q4_K MUL_MAT_ID correctness: 73/73 passed.
  • Model: Qwen3.6-35B-A3B Q4_K_M
    • Prefill (input processing): +0.16% / +0.64% / -0.07% at ubatch = 512 / 1024 / 2048.
    • Decode (output generation): +0.02%, essentially flat.

One thing worth flagging: in the current source, gfx1151 sets use_typical_moe_ncols = false, so this PR's core typical-expert-width strategy isn't actually taken on gfx1151. That likely explains why I see no measurable change here — the target optimization path isn't enabled for this arch. No correctness regressions either way.

IMbackK
IMbackK approved these changes Sep 2, 2026
View reviewed changes

@IMbackK IMbackK left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is still beneficial against and when rebated on latest master.

Details
GPU Model Microbatch size Test t/s master t/s mmqx-rdna3-routed-moe-tiling Speedup
RX 7900 XTX gemma4 26B.A4B Q6_K 8 pp2048 430.12 430.35 1.00
RX 7900 XTX gemma4 26B.A4B Q6_K 64 pp2048 1058.57 1553.66 1.47
RX 7900 XTX gemma4 26B.A4B Q6_K 512 pp2048 2502.63 3197.90 1.28
RX 7900 XTX gemma4 26B.A4B Q6_K 1024 pp2048 3316.65 3773.25 1.14
RX 7900 XTX gpt-oss 20B MXFP4 MoE 8 pp2048 636.17 635.05 1.00
RX 7900 XTX gpt-oss 20B MXFP4 MoE 64 pp2048 1594.12 2116.75 1.33
RX 7900 XTX gpt-oss 20B MXFP4 MoE 512 pp2048 3846.41 4696.47 1.22
RX 7900 XTX gpt-oss 20B MXFP4 MoE 1024 pp2048 4809.61 4831.44 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 8 pp2048 402.41 402.97 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 64 pp2048 2451.46 2449.18 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 512 pp2048 6072.19 6058.35 1.00
RX 7900 XTX lfm2moe 8B.A1B F16 1024 pp2048 8385.86 8395.05 1.00
RX 7900 XTX lfm2moe 8B.A1B Q4_0 8 pp2048 1413.81 1417.80 1.00
RX 7900 XTX lfm2moe 8B.A1B Q4_0 64 pp2048 3433.56 4559.60 1.33
RX 7900 XTX lfm2moe 8B.A1B Q4_0 512 pp2048 9144.60 10884.34 1.19
RX 7900 XTX lfm2moe 8B.A1B Q4_0 1024 pp2048 11583.59 11661.76 1.01
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 8 pp2048 1003.92 1012.50 1.01
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 64 pp2048 3185.46 4461.54 1.40
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 512 pp2048 8429.84 10221.60 1.21
RX 7900 XTX lfm2moe 8B.A1B Q4_K_M 1024 pp2048 10737.57 10826.09 1.01
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 8 pp2048 985.99 989.12 1.00
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 64 pp2048 3128.37 4400.83 1.41
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 512 pp2048 8249.88 9973.57 1.21
RX 7900 XTX lfm2moe 8B.A1B Q5_K_M 1024 pp2048 10564.36 10640.39 1.01
RX 7900 XTX lfm2moe 8B.A1B Q6_K 8 pp2048 852.31 860.02 1.01
RX 7900 XTX lfm2moe 8B.A1B Q6_K 64 pp2048 2257.27 3378.81 1.50
RX 7900 XTX lfm2moe 8B.A1B Q6_K 512 pp2048 5863.58 7124.40 1.22
RX 7900 XTX lfm2moe 8B.A1B Q6_K 1024 pp2048 7587.75 7671.78 1.01
RX 7900 XTX lfm2moe 8B.A1B Q8_0 8 pp2048 1020.85 1024.86 1.00
RX 7900 XTX lfm2moe 8B.A1B Q8_0 64 pp2048 3461.44 4352.49 1.26
RX 7900 XTX lfm2moe 8B.A1B Q8_0 512 pp2048 9130.16 11053.51 1.21
RX 7900 XTX lfm2moe 8B.A1B Q8_0 1024 pp2048 11759.62 11835.76 1.01
RX 7900 XTX qwen35 27B Q5_K_M 8 pp2048 98.13 98.63 1.01
RX 7900 XTX qwen35 27B Q5_K_M 64 pp2048 557.11 558.72 1.00
RX 7900 XTX qwen35 27B Q5_K_M 512 pp2048 758.67 762.83 1.01
RX 7900 XTX qwen35 27B Q5_K_M 1024 pp2048 776.35 778.11 1.00
Backend GGML op Op parameters TFLOPS master TFLOPS mmqx-rdna3-routed-moe-tiling Speedup
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.85 0.89 1.04
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 5.07 5.13 1.01
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.49 6.39 0.98
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.64 1.76 1.07
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 0.91 0.83 0.91
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 7.07 7.13 1.01
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 2.94 2.83 0.96
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 0.80 0.83 1.04
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 0.87 0.89 1.02
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.07 6.66 1.10
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 8.04 8.04 1.00
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 2.78 2.71 0.98
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 0.88 0.94 1.07
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 14.21 14.20 1.00
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 4.77 4.73 0.99
ROCm0 MUL_MAT_ID type_a=f16,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 0.85 0.85 1.01
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.87 0.89 1.03
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 0.76 0.75 0.98
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 1.01 0.95 0.94
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 0.33 0.35 1.04
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 0.46 0.48 1.04
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 1.78 1.81 1.02
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 0.55 0.58 1.06
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 0.49 0.47 0.97
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 0.89 0.88 0.99
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 1.51 1.49 0.99
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 2.68 2.67 1.00
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 1.05 1.02 0.97
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 0.51 0.61 1.20
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 4.32 4.14 0.96
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 1.27 1.27 1.00
ROCm0 MUL_MAT_ID type_a=f32,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 0.51 0.51 1.01
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.87 0.89 1.03
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 2.58 7.93 3.07
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 4.87 11.11 2.28
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.79 2.40 1.34
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 1.65 1.74 1.06
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 9.06 15.63 1.73
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 2.20 3.87 1.76
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 1.08 1.07 0.99
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.32 1.32 1.00
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 4.99 11.13 2.23
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 9.48 16.40 1.73
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.12 3.75 1.20
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 1.65 1.70 1.03
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 18.59 21.45 1.15
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 4.20 7.23 1.72
ROCm0 MUL_MAT_ID type_a=iq2_xs,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 1.23 1.16 0.95
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=1,k=2880 3.20 3.20 1.00
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=4,k=2880 4.86 4.86 1.00
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=512,k=2880 24.47 31.41 1.28
ROCm0 MUL_MAT_ID type_a=mxfp4,type_b=f32,n_mats=32,n_used=4,b=0,m=2880,n=8,k=2880 4.43 4.61 1.04
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 1.35 1.41 1.04
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 3.39 11.61 3.43
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.25 16.16 2.59
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 2.15 3.79 1.76
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 4.87 5.07 1.04
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 12.03 17.68 1.47
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 3.09 6.46 2.09
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 6.12 6.23 1.02
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.44 1.46 1.01
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.71 17.51 2.61
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 12.84 21.70 1.69
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.90 6.66 1.71
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 5.34 5.18 0.97
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 24.57 28.93 1.18
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 6.06 12.65 2.09
ROCm0 MUL_MAT_ID type_a=q4_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 6.47 6.59 1.02
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 0.91 0.96 1.05
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 3.30 12.38 3.76
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.11 16.18 2.65
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 2.82 4.10 1.45
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 2.64 2.72 1.03
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 11.77 23.02 1.96
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 3.09 6.80 2.20
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 2.02 1.85 0.92
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.84 1.86 1.01
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.58 19.32 2.94
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 12.44 26.08 2.10
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 5.09 7.06 1.39
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 2.75 2.83 1.03
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 23.96 29.67 1.24
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 6.11 13.81 2.26
ROCm0 MUL_MAT_ID type_a=q4_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 2.41 2.63 1.09
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 1.31 1.34 1.03
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 2.02 8.37 4.14
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 3.79 11.79 3.11
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.78 2.48 1.39
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 1.94 2.03 1.04
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 7.47 15.10 2.02
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 1.89 4.47 2.37
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 1.18 1.34 1.13
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.26 1.38 1.10
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 3.98 11.99 3.01
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 7.42 16.34 2.20
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.03 4.21 1.39
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 2.15 2.13 0.99
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 14.78 18.03 1.22
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 3.60 8.56 2.38
ROCm0 MUL_MAT_ID type_a=q6_K,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 1.59 1.53 0.96
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=1,k=2048 1.28 1.33 1.04
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=128,k=2048 3.43 9.96 2.90
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=256,k=2048 6.40 16.13 2.52
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=32,k=2048 1.96 3.14 1.60
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=4,k=2048 2.82 2.87 1.02
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=512,k=2048 12.16 17.79 1.46
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=64,k=2048 3.23 5.47 1.69
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=128,n_used=8,b=0,m=768,n=8,k=2048 2.72 2.58 0.95
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=1,k=2048 1.35 1.36 1.01
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=128,k=2048 6.66 16.99 2.55
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=256,k=2048 12.76 19.17 1.50
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=32,k=2048 3.42 5.27 1.54
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=4,k=2048 3.89 3.90 1.00
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=512,k=2048 24.59 31.39 1.28
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=64,k=2048 6.17 10.11 1.64
ROCm0 MUL_MAT_ID type_a=q8_0,type_b=f32,n_mats=32,n_used=4,b=0,m=1792,n=8,k=2048 4.23 4.85 1.15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@IMbackK IMbackK IMbackK approved these changes
@ggerganov ggerganov Awaiting requested review from ggerganov

At least 2 approving reviews are required to merge this pull request.

Assignees

@IMbackK IMbackK

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@ravel7524 @GZGavinZhao @IMbackK @frizikk @ggerganov @Dev-next-gen @Casten-Wang @CISC