vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 by 0cc4m · Pull Request #27952 · ggml-org/llama.cpp

0cc4m · 2026-08-29T11:05:36Z

Overview

Based on @pwilkin's ideas in #27493, I finally managed to get coopmat1 int8 to work with good performance on RDNA3 and 4. This new MMQ cm1 shader supports q4_0, q4_1, q5_0, q5_1, q8_0, q3_k, q4_k, q5_k, q6_k, mxfp4, nvfp4 and iq4_nl on RDNA3 and RDNA4. Strix Halo prompt processing performance is significantly improved, RDNA4 is more neutral, but MoE prompt processing is also up significantly. The shader is limited to those two architectures since it hardcodes their specific coopmat access patterns. More architectures can be added if needed.

q4_1, q5_1, q4_k, q5_k and nvfp4 on RDNA4 are disabled for MUL_MAT since they run slower than existing fp16 matmul. Additionally nvfp4 is also disabled for MUL_MAT_ID.

Benchmarks

AMD Radeon 8060S (Strix Halo, RDNA3.5)

MUL_MAT test-backend-ops perf geomean us (lower = better)

quant	vk-master	vk-feat	rocm-master	feat vs master	feat vs rocm
q4_0	5183	4016	5064	1.29x	1.26x
q4_1	5123	4157	5189	1.23x	1.25x
q5_0	5881	4340	5273	1.35x	1.21x
q5_1	5574	4271	5362	1.31x	1.26x
q8_0	5645	4443	5421	1.27x	1.22x
iq4_nl	5471	4535	5061	1.21x	1.12x
mxfp4	5205	4329	4903	1.20x	1.13x
q3_K	6352	5718	5974	1.11x	1.04x
q4_K	5535	4749	5304	1.17x	1.12x
q5_K	5637	4940	5437	1.14x	1.10x
q6_K	6545	5747	5323	1.14x	0.93x
nvfp4	5262	5430	6290	0.97x	1.16x

MUL_MAT_ID test-backend-ops perf geomean us (lower = better)

quant	vk-master	vk-feat	rocm-master	feat vs master	feat vs rocm
q4_0	867	658	817	1.32x	1.24x
q4_1	869	663	875	1.31x	1.32x
q5_0	990	707	799	1.40x	1.13x
q5_1	962	691	785	1.39x	1.14x
q8_0	843	692	853	1.22x	1.23x
iq4_nl	951	723	747	1.32x	1.03x
mxfp4	925	703	727	1.32x	1.03x
q3_K	1138	856	853	1.33x	1.00x
q4_K	990	779	762	1.27x	0.98x
q5_K	994	824	844	1.21x	1.02x
q6_K	1094	899	1089	1.22x	1.21x
nvfp4	969	822	1121	1.18x	1.36x

llama-bench pp512 t/s (higher = better)

model	quant	kind	vk-master	vk-feat	rocm-master	feat vs master	feat vs rocm
Meta-Llama-3-8B	Q4_0	dense	862.7	1075.7	924.2	1.25x	1.16x
Meta-Llama-3-8B	Q4_1	dense	937.3	1073.2	897.9	1.15x	1.20x
Meta-Llama-3.1-8B	IQ4_NL	dense	901.7	1035.1	934.3	1.15x	1.11x
Meta-Llama-3.1-8B	Q3_K_S	dense	791.3	867.3	830.0	1.10x	1.04x
Meta-Llama-3.1-8B	Q4_K_S	dense	874.3	975.1	907.1	1.12x	1.08x
Meta-Llama-3.1-8B	Q6_K	dense	772.2	862.9	729.4	1.12x	1.18x
Meta-Llama-3.1-8B	Q8_0	dense	823.4	1030.5	874.4	1.25x	1.18x
llama2-13b-tiefighter	Q5_K_M	dense	515.0	547.1	519.6	1.06x	1.05x
Qwen3.8-27B	Q4_K_M	dense	186.5	254.1	237.6	1.36x	1.07x
gpt-oss-20b	MXFP4	MoE	1071.6	1410.5	1005.8	1.32x	1.40x
Qwen3.6-35B-A3B	Q4_0	MoE	883.7	1175.9	743.4	1.33x	1.58x
Qwen3.6-35B-A3B	Q4_K_M	MoE	777.8	1023.6	693.9	1.32x	1.48x
gemma-4-26B-A4B	Q6_K	MoE	737.2	925.5	723.1	1.26x	1.28x

AMD Radeon AI PRO R9700 (RDNA4)

MUL_MAT test-backend-ops perf geomean us (lower = better)

quant	vk-master	vk-feat	rocm-master	feat vs master	feat vs rocm	RDNA4 MMQ
q4_0	1146	1120	1036	1.02x	0.93x	on
q4_1	1123	1123	1189	1.00x	1.06x	off (fallback)
q5_0	1378	1234	1116	1.12x	0.90x	on
q5_1	1272	1273	1203	1.00x	0.94x	off (fallback)
q8_0	1324	1179	1024	1.12x	0.87x	on
iq4_nl	1304	1198	958	1.09x	0.80x	on
mxfp4	1326	1202	960	1.10x	0.80x	on
q3_K	1974	1616	1466	1.22x	0.91x	on
q4_K	1372	1373	1153	1.00x	0.84x	off (fallback)
q5_K	1433	1435	1180	1.00x	0.82x	off (fallback)
q6_K	1746	1583	2395	1.10x	1.51x	on
nvfp4	1330	1332	1554	1.00x	1.17x	off (fallback)

MUL_MAT_ID test-backend-ops perf geomean us (lower = better)

quant	vk-master	vk-feat	rocm-master	feat vs master	feat vs rocm	RDNA4 MMQ
q4_0	210	243	202	0.86x	0.83x	on
q4_1	205	253	219	0.81x	0.87x	on
q5_0	249	265	210	0.94x	0.79x	on
q5_1	227	260	219	0.87x	0.84x	on
q8_0	228	252	194	0.90x	0.77x	on
iq4_nl	237	259	195	0.92x	0.75x	on
mxfp4	241	261	195	0.93x	0.75x	on
q3_K	364	329	247	1.11x	0.75x	on
q4_K	251	307	216	0.82x	0.71x	on
q5_K	261	316	218	0.83x	0.69x	on
q6_K	312	327	343	0.95x	1.05x	on
nvfp4	242	243	282	1.00x	1.16x	off (fallback)

llama-bench pp512 t/s (higher = better)

model	quant	kind	vk-master	vk-feat	rocm-master	feat vs master	feat vs rocm
Llama-3-8B	Q4_0	dense	4656	4584	4370	0.98x	1.05x
Llama-3-8B	Q4_1	dense	4751	4771	3911	1.00x	1.22x
Llama-3-8B	Q4_K_S	dense	3881	3968	3887	1.02x	1.02x
Llama-3.1-8B	IQ4_NL	dense	4059	4169	4488	1.03x	0.93x
Llama-3.1-8B	Q3_K_S	dense	2882	3228	3321	1.12x	0.97x
Llama-3.1-8B	Q6_K	dense	3208	3259	2165	1.02x	1.51x
Llama-3.1-8B	Q8_0	dense	4230	4378	4473	1.03x	0.98x
Mistral-Nemo-12B	Q4_0	dense	2925	2882	2876	0.99x	1.00x
gemma-4-12B	Q4_0	dense	2761	2711	2429	0.98x	1.12x
Qwen3.8-27B	Q6_K	dense	955	1040	764	1.09x	1.36x
gpt-oss-20B	MXFP4	MoE	3936	5631	4231	1.43x	1.33x
Qwen3.6-35B-A3B	Q4_0	MoE	3196	4046	2638	1.27x	1.53x
Qwen3.6-35B-A3B	Q4_K_M	MoE	2760	3365	2240	1.22x	1.50x
Muse-Glimmer-30B	Q6_K	MoE	995	1065	792	1.07x	1.34x

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, AI was used for optimization loops. Code and performance was manually checked/verified.

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

This reverts commit fbaefe0.

This reverts commit 7fabc25.

… set BK_STEP back to 4

jeffbolznv · 2026-08-29T19:14:13Z

+#define ACC_BIAS_F    12582912.0f
+const bool USE_MAGIC_BIAS = WARP != 32;
+
+// Accumulator row for element e: RDNA4 blocked, RDNA3/3.5 interleaved.


Are these layouts entirely based on the hardware definition, or do they depend partly on the compiler. For example, at some point I changed the NVIDIA compiler so the layout of a 16x16 matrix in four registers was permuted to 0,2,1,3 vs earlier compiler versions.

I think it's a little bit of both. I have confirmed that Linux' RADV driver based this layout on what the hardware requires, so it should be fixed. I have also checked that at least RDNA3.5 uses the same layout on the Windows driver, and this branch does improve performance there too.

I think we can replace this with a generic solution once enough drivers support coopmat maintenance1.

jeffbolznv · 2026-08-29T19:21:38Z

+        return;
+    }
+#else
+    // L2-friendly workgroup scheduling


Would be interesting to separate out what perf gain is from this vs using int8. I've been tempted to do something like this for the other shaders, but with ubatch=512 still being the default it usually doesn't matter much (probably helps more for stable diffusion kind of workloads)

Very small, hard to measure. Maybe 2% on RDNA3.5, but my Strix Halo laptop has a lot of thermal noise in benchmarks, so not sure. Might not be worth it. I didn't measure a difference on RDNA4.

edt-xx · 2026-09-02T16:01:45Z

Built b10760 + this pr. Using a RX7900XT (RDNA3) with RADV. Running usloth/Qwen3.8-27B-Q4_0.gguf before applying this PR I was seeing:

[35963] 0.10.092.723 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   4096, progress = 0.05, t =   4.25 s / 963.84 tokens per second
...
[35963] 2.42.829.150 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  75995, progress = 1.00, t = 156.13 s / 486.73 tokens per second

after the same sort of prefill gets (+8% at start to +3% at end of prefill)

[40189] 181.52.418.736 I slot print_timing: id  0 | task 2568 | prompt processing, n_tokens =   4096, progress = 0.05, t =   3.92 s / 1046.10 tokens per second
...
[40189] 184.24.107.728 I slot print_timing: id  0 | task 2568 | prompt processing, n_tokens =  77865, progress = 1.00, t = 156.04 s / 498.99 tokens per second

There is too much variance in token gen to say anything. This has positive effects in a real use case.

NickM-27 · 2026-09-02T17:51:29Z

Apologies if this is not helpful / spam, but I have been running this on my 7900XTX for a few days and it has been a big improvement, so wanted to share a before / after llama bench run:

master:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so
| model                          |       size |     params | backend    | ngl | n_ubatch |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |           pp512 |      3410.53 ± 22.72 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |           tg128 |        135.64 ± 0.80 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |   pp512 @ d8192 |      2477.27 ± 76.78 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |   tg128 @ d8192 |        120.42 ± 0.46 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |  pp512 @ d16384 |      2130.36 ± 35.16 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |  tg128 @ d16384 |        118.42 ± 0.14 |

this PR:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: dot2 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so
| model                          |       size |     params | backend    | ngl | n_ubatch |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |           pp512 |     4331.21 ± 132.67 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |           tg128 |        142.76 ± 0.92 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |   pp512 @ d8192 |      2885.67 ± 80.24 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |   tg128 @ d8192 |        125.29 ± 0.14 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |  pp512 @ d16384 |      2402.13 ± 44.16 |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |     1024 |   1 |  tg128 @ d16384 |        120.82 ± 0.18 |

0cc4m and others added 30 commits August 29, 2026 11:30

vulkan: add int8 coopmat quantized matmul shader

13f6d8e

apply scales inline

e086b9c

use scalar sums

5f034d6

probe and directly access coopmat values instead of going through shmem

1625608

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

add q8_0 support

513f436

add BK_STEP to shader, default to 2

b121ef3

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

use larger workgroups

2a878d9

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

double buffering

5553b89

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

preload scales

b453be3

coopmat load first, then wmma

14a6080

use float for scales

95d14c9

add faster RDNA int->float conversion

ad015d6

revert load reordering and scale pre-loading

31b9bc6

workgroup scheduling for cache proximity

2335676

clean up

9f069f4

Revert "revert load reordering and scale pre-loading"

15b3ce0

This reverts commit fbaefe0.

use wave32

57b4276

increase large tile size

a9e0e89

Revert "increase large tile size"

ef5c629

This reverts commit 7fabc25.

restructure for vgpr use

2b9c533

skip computation for inactive tiles

57dd3f8

only force subgroup size 32 on AMD RDNA

ab6e84b

use BK_STEP 4

42a6269

fix compilation

1825270

move quant-specific prefetch function out of main file

eb46b8c

add q4_1, q5_0, q5_1 support

d2a5ac6

restructure mmq cm1 functions

ed294e8

enable mul_mat_id support

fa37db9

fix segfault

d8ed372

fix mul_mat_id bug

3cd7643

0cc4m added 18 commits August 29, 2026 11:51

use shmem arrays for LUTs

5037f02

use 4-byte loads where possible

ab94ffd

add q3_k, q4_k, q5_k, q6_k and nvfp4 support

4695f10

fix l warptile

22f588c

improve performance

3985f64

improve performance

092661d

improvements

eeeaced

dedup b scales

2ba1d22

merge shmem arrays

9ef5996

undo uint8_t, gate to RDNA3/4

f252045

add RDNA4 architecture, use for hardcoded coopmat elem thread access,…

0eab519

… set BK_STEP back to 4

improve offset application

c30a874

clean up

6421fbe

fix iq4_nl and nvfp4 performance

04d7ef7

rdna4 tuning

584ae05

use BK_STEP 2 on MUL_MAT_ID

e85ff84

adapt to upstream changes

8b194a1

fix shmem support function, clean up comments

965e571

github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 29, 2026

0cc4m marked this pull request as ready for review August 29, 2026 12:10

0cc4m requested a review from a team as a code owner August 29, 2026 12:10

jeffbolznv reviewed Aug 29, 2026

View reviewed changes

nr23730 mentioned this pull request Aug 30, 2026

Speculative prefill #27692

Open

SimonTeixidor mentioned this pull request Aug 30, 2026

Batch decode tuning Nathanw1014/llama.cpp#7

Closed

This was referenced Aug 30, 2026

vulkan: use spec constant for matrix matrix multiplication A-type #25773

Open

vulkan: tune mat-vec rows for batched inference on Strix Halo #27909

Merged

pwilkin mentioned this pull request Sep 1, 2026

Optimize Krea Vulkan - MUL_MAT #27493

Closed

Navigation Menu

Uh oh!

vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 - #27952

vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4#27952
0cc4m wants to merge 50 commits into
masterfrom
0cc4m/vulkan-coopmat-int8

0cc4m commented Aug 29, 2026

Uh oh!

jeffbolznv Aug 29, 2026

Uh oh!

0cc4m Aug 30, 2026

Uh oh!

jeffbolznv Aug 29, 2026

Uh oh!

0cc4m Aug 30, 2026

Uh oh!

edt-xx commented Sep 2, 2026

Uh oh!

NickM-27 commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

0cc4m commented Aug 29, 2026

Overview

Benchmarks

MUL_MAT test-backend-ops perf geomean us (lower = better)

MUL_MAT_ID test-backend-ops perf geomean us (lower = better)

llama-bench pp512 t/s (higher = better)

MUL_MAT test-backend-ops perf geomean us (lower = better)

MUL_MAT_ID test-backend-ops perf geomean us (lower = better)

llama-bench pp512 t/s (higher = better)

Requirements

Uh oh!

jeffbolznv Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

0cc4m Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

jeffbolznv Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

0cc4m Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

edt-xx commented Sep 2, 2026

Uh oh!

NickM-27 commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants