vulkan: use spec constant for matrix matrix multiplication A-type by 0cc4m · Pull Request #25773 · ggml-org/llama.cpp

0cc4m · 2026-07-16T12:21:23Z

Overview

This is an attempt to reduce the amount of matrix multiplication shaders we need by using a spec constant for the type of the A input, similar to how it works for KV cache types in Flash Attention. Additionally, it replaces the current static s, m and l shapes with a map and allows improved tuning through more shapes and a custom selection function, which should help with work like #24407.

It's still draft because currently it regresses Nvidia, because for dynamic type selection all quants with shared memory requirements must also be available. I thought it would eliminate those that are not used, this works on AMD, but not on Nvidia. @jeffbolznv Do you know why? There are other ways to resolve this, hopefully I'll find something that works.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, AI was used to draft the code, I reviewed, tested and corrected it afterwards.

jeffbolznv · 2026-07-16T15:29:03Z

I had experimented with something like this and ended up with one spirv for all the legacy and K quants and then separate spirvs for all the iq quants. That resolves the shared memory issue and probably isn't a ton of bloat because those spirvs are dominated by the table size anyway.

Fwiw, even if our compiler dead code eliminated the shared memory arrays, they would still count against the spec limit.

jeffbolznv · 2026-07-17T08:31:24Z

I think the shared memory arrays with length 1 could run into this compiler bug again #24924.

0cc4m · 2026-07-24T06:11:55Z

I don't see these CI failures locally, not sure what is causing them. Is that the compiler bug?

jeffbolznv · 2026-07-24T15:59:05Z

I don't see these CI failures locally, not sure what is causing them. Is that the compiler bug?

I also can't reproduce the cm1 failures locally. It's conceivable it's the Turing-only bug. I think it's more likely to happen if you use more fp16 math, is there any place you've changed fp32 math to fp16?

0cc4m · 2026-07-24T16:15:04Z

No, the PR should not contain any functional changes, it's all just restructuring. The running code in the end should be basically identical.

0cc4m · 2026-08-20T06:35:18Z

@ggerganov The Nvidia-CM pipeline is timing out in test-backend-ops, I haven't seen that before. Is that expected? Is it possible to increase the timeout?

ggerganov · 2026-08-20T07:49:54Z

Hm, not sure - I can't see a reason for the increased time. It almost doubled than usual for this job (~15 mins). Also, looking at the most recent runs in https://github.com/ggml-org/llama.cpp/actions/workflows/build-self-hosted.yml it seems to be fluctuating between slow/fast without a clear point where it started happening.

Let's keep monitoring. These workflows are currently running on DGX Spark that I self-host. We are already looking for ways to migrate this to some more stable and scalable infrastructure, but don't have ETA atm.

0cc4m · 2026-08-20T07:51:43Z

Did it recently switch from T4 to the Spark? That would explain why it now passed some tests that were failing earlier due (likely to the Turing driver bug.

ggerganov · 2026-08-20T07:55:33Z

The DGX Spark has been running these and other jobs for about 3 months now (added on May 26). The T4 runners are also running - whoever runner picks of the job first will execute it.

jeffbolznv · 2026-08-20T13:47:02Z

@ggerganov The Nvidia-CM pipeline is timing out in test-backend-ops, I haven't seen that before. Is that expected? Is it possible to increase the timeout?

Maybe this change effects enough shaders that it get no benefit from the shader disk cache from previous runs? If so, a rerun might be enough to make it complete.

0cc4m · 2026-08-24T06:48:15Z

I reran it multiple times and it keeps failing, so I don't think that's it. I'll try to reproduce locally.

0cc4m · 2026-08-24T08:30:09Z

It consistently happens in the CI, but I cannot reproduce it locally. I get worst case 7 minutes on my DGX Spark, with caching just a little bit more than 3 minutes. I'm not sure what is going on there.

ggerganov · 2026-08-24T08:45:41Z

Here are the logs of one slow and one fast run:

slow
fast

One thing that draws attention is that in the slow run, the cmake builds at the start are quite a bit slower. F.ex:

# slow
2026-08-24T07:26:39.7092878Z real	5m8.856s
2026-08-24T07:26:39.7093100Z user	16m45.155s
2026-08-24T07:26:39.7093300Z sys	1m53.851s
2026-08-24T07:26:39.7095059Z + '[' -z ']'
2026-08-24T07:26:39.7098364Z + tee -a /home/ggml/results/llama.cpp/ctest_release-ctest.log

# fast
2026-08-24T05:27:37.5154950Z real	1m18.797s
2026-08-24T05:27:37.5155099Z user	3m29.091s
2026-08-24T05:27:37.5155197Z sys	0m47.416s
2026-08-24T05:27:37.5157269Z + '[' -z ']'
2026-08-24T05:27:37.5159526Z + tee -a /home/ggml/results/llama.cpp/ctest_release-ctest.log

ggerganov · 2026-08-24T08:46:39Z

It consistently happens in the CI, but I cannot reproduce it locally. I get worst case 7 minutes on my DGX Spark, with caching just a little bit more than 3 minutes. I'm not sure what is going on there.

Is this 7 minutes for the entire ci/run.sh script?

0cc4m · 2026-08-24T08:50:52Z

Just for test-backend-ops

0cc4m · 2026-08-24T10:19:31Z

The entire CI script went from 22:46 on master uncached to 12:09 cached, with the PR from 13:14 uncached to 7:44 cached, on my Spark. That isn't consistent with the CI either.

ggerganov · 2026-08-24T10:32:22Z

I just tried manually - on both my DGX Sparks, this command deadlocks/blocks with Vulkan on this branch:

GGML_VK_DISABLE_COOPMAT2=1 ./bin/test-backend-ops -o MUL_MAT

There is a suspicious error though at the start about incompatible driver:

ggml@spark-17ed:~/development/llama.cpp/build-vulkan$ GGML_VK_DISABLE_COOPMAT2=1 ./bin/test-backend-ops -o MUL_MAT
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA Tegra NVIDIA GB10 (NVIDIA) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 0 | matrix cores: KHR_coopmat
Testing 2 devices

Backend 1/2: Vulkan0
  Device description: NVIDIA Tegra NVIDIA GB10
  Device memory: 91929 MB (91923 MB free)

  MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=bf16,type_b=f32,m=16,n=9,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=1,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=3,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=4,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=5,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=6,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=7,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=8,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK

Does this command complete on your end?
Any suggestions about what could be wrong causing this error?

Edit: the same command works fine on master. The error is still displayed.

0cc4m · 2026-08-24T10:54:32Z

Yes, it completes without issue. It's suspicious that your driver reports the device as NVIDIA Tegra NVIDIA GB10, mine shows NVIDIA GB10. What version are you on? vulkaninfo --summary shows it, for example for me it's:

GPU0:
        apiVersion         = 1.4.312
        driverVersion      = 580.159.3.0
        vendorID           = 0x10de
        deviceID           = 0x2e12
        deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
        deviceName         = NVIDIA GB10
        driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
        driverName         = NVIDIA
        driverInfo         = 580.159.03
        conformanceVersion = 1.4.1.3
        deviceUUID         = 89b1d82b-72d4-cfde-55b7-4adf19b0e8b3
        driverUUID         = 02d1943b-b572-5849-bc0a-9110d99e29a2

The driver warning is a mesa adreno Vulkan driver artifact, you can ignore it.

ggerganov · 2026-08-24T11:06:45Z

Here is spark 1 (the one from my last command) that shows "tegra":

Details

ggml@spark-17ed:~$ vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.335


Instance Extensions: count = 25
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 13
---------------------------
VK_LAYER_KHRONOS_profiles         Khronos Profiles layer                                                                                            1.4.335  version 1
VK_LAYER_KHRONOS_shader_object    Khronos Shader object layer                                                                                       1.4.335  version 1
VK_LAYER_KHRONOS_synchronization2 Khronos Synchronization2 layer                                                                                    1.4.335  version 1
VK_LAYER_KHRONOS_validation       Khronos Validation Layer                                                                                          1.4.335  version 1
VK_LAYER_LUNARG_api_dump          LunarG API dump layer                                                                                             1.4.335  version 2
VK_LAYER_LUNARG_crash_diagnostic  Crash Diagnostic Layer is a crash/hang debugging tool that helps determines GPU progress in a Vulkan application. 1.4.335  version 1
VK_LAYER_LUNARG_gfxreconstruct    GFXReconstruct Capture Layer Version 1.0.5                                                                        1.4.335  version 4194309
VK_LAYER_LUNARG_monitor           Execution Monitoring Layer                                                                                        1.4.335  version 1
VK_LAYER_LUNARG_screenshot        LunarG image capture layer                                                                                        1.4.335  version 1
VK_LAYER_MESA_device_select       Linux device selection layer                                                                                      1.4.303  version 1
VK_LAYER_MESA_overlay             Mesa Overlay layer                                                                                                1.4.303  version 1
VK_LAYER_NV_optimus               NVIDIA Optimus layer                                                                                              1.4.312  version 1
VK_LAYER_NV_present               NVIDIA GR2608 layer                                                                                               1.4.312  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.312
	driverVersion      = 580.95.5.0
	vendorID           = 0x10de
	deviceID           = 0x2e12
	deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
	deviceName         = NVIDIA Tegra NVIDIA GB10
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 580.95.05
	conformanceVersion = 1.4.1.3
	deviceUUID         = e13a03d3-eb3d-950a-7ca1-bfb87613a30c
	driverUUID         = b92269a1-b525-5615-ab8a-e2095ee37192
GPU1:
	apiVersion         = 1.4.305
	driverVersion      = 0.0.1
	vendorID           = 0x10005
	deviceID           = 0x0000
	deviceType         = PHYSICAL_DEVICE_TYPE_CPU
	deviceName         = llvmpipe (LLVM 20.1.2, 128 bits)
	driverID           = DRIVER_ID_MESA_LLVMPIPE
	driverName         = llvmpipe
	driverInfo         = Mesa 25.0.7-0ubuntu0.24.04.2 (LLVM 20.1.2)
	conformanceVersion = 1.3.1.1
	deviceUUID         = 6d657361-3235-2e30-2e37-2d3075627500
	driverUUID         = 6c6c766d-7069-7065-5555-494400000000

Here is the spark 2, which actually runs the CI - it does not show "tegra" for some reason:

Details

ggml@spark-a163:~$ vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
TU: error: ../src/freedreno/vulkan/tu_knl.cc:387: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
WARNING: [Loader Message] Code 0 : ICD for selected physical device does not export vkGetPhysicalDeviceDisplayPlanePropertiesKHR!
WARNING: [Loader Message] Code 0 : ICD for selected physical device does not export vkGetPhysicalDeviceDisplayPropertiesKHR!
==========
VULKANINFO
==========

Vulkan Instance Version: 1.4.341


Instance Extensions: count = 25
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1
VK_NV_display_stereo                   : extension revision 1

Instance Layers: count = 14
---------------------------
VK_LAYER_INTEL_nullhw             INTEL NULL HW                                                                                                     1.1.73   version 1
VK_LAYER_KHRONOS_profiles         Khronos Profiles layer                                                                                            1.4.341  version 1
VK_LAYER_KHRONOS_shader_object    Khronos Shader object layer                                                                                       1.4.341  version 1
VK_LAYER_KHRONOS_synchronization2 Khronos Synchronization2 layer                                                                                    1.4.341  version 1
VK_LAYER_KHRONOS_validation       Khronos Validation Layer                                                                                          1.4.341  version 1
VK_LAYER_LUNARG_api_dump          LunarG API dump layer                                                                                             1.4.341  version 2
VK_LAYER_LUNARG_crash_diagnostic  Crash Diagnostic Layer is a crash/hang debugging tool that helps determines GPU progress in a Vulkan application. 1.4.341  version 1
VK_LAYER_LUNARG_gfxreconstruct    GFXReconstruct Capture Layer Version 1.0.5                                                                        1.4.341  version 4194309
VK_LAYER_LUNARG_monitor           Execution Monitoring Layer                                                                                        1.4.341  version 1
VK_LAYER_LUNARG_screenshot        LunarG image capture layer                                                                                        1.4.341  version 1
VK_LAYER_MESA_device_select       Linux device selection layer                                                                                      1.4.303  version 1
VK_LAYER_MESA_overlay             Mesa Overlay layer                                                                                                1.4.303  version 1
VK_LAYER_NV_optimus               NVIDIA Optimus layer                                                                                              1.4.312  version 1
VK_LAYER_NV_present               NVIDIA GR2608 layer                                                                                               1.4.312  version 1

Devices:
========
GPU0:
	apiVersion         = 1.4.312
	driverVersion      = 580.126.9.0
	vendorID           = 0x10de
	deviceID           = 0x2e12
	deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
	deviceName         = NVIDIA GB10
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 580.126.09
	conformanceVersion = 1.4.1.3
	deviceUUID         = 032999c3-c6e9-ce30-47c7-b69dc4aff29e
	driverUUID         = 9756eed7-117e-56eb-8928-8e249b39bb59
GPU1:
	apiVersion         = 1.4.318
	driverVersion      = 25.2.8
	vendorID           = 0x10005
	deviceID           = 0x0000
	deviceType         = PHYSICAL_DEVICE_TYPE_CPU
	deviceName         = llvmpipe (LLVM 20.1.2, 128 bits)
	driverID           = DRIVER_ID_MESA_LLVMPIPE
	driverName         = llvmpipe
	driverInfo         = Mesa 25.2.8-0ubuntu0.24.04.1 (LLVM 20.1.2)
	conformanceVersion = 1.3.1.1
	deviceUUID         = 6d657361-3235-2e32-2e38-2d3075627500
	driverUUID         = 6c6c766d-7069-7065-5555-494400000000

The jobs also have this info at the start of the logs: https://github.com/ggml-org/llama.cpp/actions/runs/32224960472/job/97381646994?pr=25773#step:3:54. For example this job, runs on Runner name: 'ggml-dgx-spark-2-runner-4' - this is a docker container inside the spark 2.

0cc4m · 2026-08-24T11:43:38Z

Sorry, I made a mistake testing, my environment wasn't set up correctly and I didn't notice. I can actually reproduce it, I'll look into it.

0cc4m · 2026-08-26T05:20:19Z

I excluded LUT quants from the new unified shader, which seems to avoid whatever bug Nvidia was hitting with coopmat1. I'll clean up the code, then this should be ready for review.

vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types

0cc4m · 2026-08-30T20:27:31Z

The CI does indeed time out if I change something and it has recompile the matmul shaders. But otherwise it seems stable now. This PR is gonna clash with #27952, but I can rebase whichever comes last.

@jeffbolznv Let me know if you have concerns or more ideas for this branch, I think overall it's a good change to get rid of the static s, m, l * un/aligned values, and should allow device-specific tuning without complicating the code further.

I'm not sure why I couldn't get coopmat2 q4_k/q5_k unified, would be cleaner, but regressed on my 3090, while my GB10 was unaffected. But even without that getting rid of most non-LUT quant copies of mul_mm should be good.

Master ggml-vulkan.so library is 55MB, with this PR it becomes 42MB.

jeffbolznv · 2026-08-30T20:36:59Z

    const uint sign8 = sign7 | (bitCount(sign7) << 7);
    const uint sign = sign8 >> (iqs % 8);
-    const u8vec4 grid = unpack8(iq2xxs_grid[qs][(iqs % 8) / 4] >> (8 * (iqs % 4)));
+    const u8vec4 grid = unpack8(iq2_grid[qs][(iqs % 8) / 4] >> (8 * (iqs % 4)));


What's the motivation for renaming the LUTs?

Originally it was to consolidate them as much as possible for the unified shader. Now it serves no purpose anymore, I think. I'll clean it up.

jeffbolznv · 2026-08-30T20:46:44Z

 #define DECODEFUNCA , dequantFuncA
 #endif
 #else
+// Dispatch coopMatLoadTensorNV to the right decode function


I think it should be possible to avoid all this complexity and load from a uint8_t array, you would just need to scale pos_a.

True, I missed that. Done.

jeffbolznv · 2026-08-30T20:51:15Z

+              create_mm_pipelines({TYPE, GGML_TYPE_F16, false, true},  tc, "matmul_" #tstr "_f16_f16acc", matmul_##tstr##_f16_f16acc_cm2_len, matmul_##tstr##_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true); \
+              create_mm_pipelines({TYPE, GGML_TYPE_F16, false, false}, tc, "matmul_" #tstr "_f16",        matmul_##tstr##_f16_cm2_len,        matmul_##tstr##_f16_cm2_data,        sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true); \
+          } }
+        FOR_EACH_LUT_TYPE(X_CM2)


An issue reported by codex:

- [P1] Generic FP4 shaders claim the OCP pipeline keys first - ggml/src/ggml-vulkan/ggml-vulkan.cpp:4774. FOR_EACH_LUT_TYPE registers MXFP4/NVFP4 before the OCP-specific block registers the same keys. Because pipeline objects are reused and the lazy loader retains the first SPIR-V assigned, OCP-capable devices compile the generic shader and never use the accelerated OCP shader. This affects CM1 and CM2, including MUL_MAT_ID. Restore the previous if (device->ocp_fp4) ... else ... selection or exclude FP4 from the generic registration on OCP devices.

I have not attempted to review this file yet, but I read through most of the shader changes.

Yes, that was correct. Should be fixed.

0cc4m · 2026-08-31T06:18:16Z

This sometimes happens on Nvidia coopmat2/coopmat1, which is worrying:

$ build/bin/test-backend-ops -o MUL_MAT
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GB10 (NVIDIA) | uma: 1 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
Testing 2 devices

[New LWP 3628809]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000f454e9827b74 in __GI___wait4 (pid=3628812, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x0000f454e9827b74 in __GI___wait4 (pid=3628812, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x0000f454e9cb93fc in ggml_print_backtrace () from /home/testrunner/llama.cpp/build/bin/libggml-base.so.0
#2  0x0000f454e9cccc6c in ggml_uncaught_exception() () from /home/testrunner/llama.cpp/build/bin/libggml-ase.so.0
#3  0x0000f454e9aaa570 in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000f454e9aa0e04 [PAC] in std::terminate() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x0000f454e9aaa908 [PAC] in __cxa_throw () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000f454e6f2be24 [PAC] in ggml_vk_get_device(unsigned long) () from /home/testrunner/llama.cpp/build/bin/libggml-vulkan.so.0
#7  0x0000f454e6f30450 in ggml_vk_init(ggml_backend_vk_context*, unsigned long) () from /home/testrunner/llama.cpp/build/bin/libggml-vulkan.so.0
#8  0x0000f454e6f34a48 in ggml_backend_vk_init () from /home/testrunner/llama.cpp/build/bin/libggml-vulkan.so.0
#9  0x0000b68b7f64bbb0 in main ()
[Inferior 1 (process 3628807) detached]
terminate called after throwing an instance of 'vk::InitializationFailedError'
  what():  vk::PhysicalDevice::createDevice: ErrorInitializationFailed
Aborted (core dumped)

I can't reproduce it reliably.

jeffbolznv

I read through ggml-vulkan.cpp this time but not in enough detail to fully follow it. It's pretty dense, but then again the previous code was, too. I've run the backend tests, but haven't done any perf testing yet.

jeffbolznv · 2026-08-31T15:10:38Z

    }
-#if !defined(NEEDS_INIT_IQ_SHMEM)
+#if !defined(NEEDS_INIT_IQ_SHMEM) && !defined(MULMAT_QUANT)
    barrier();


codex says this barrier shouldn't be skipped for MULMAT_QUANT, since there isn't one in the stub iq_shmem_init implementation

jeffbolznv · 2026-08-31T15:13:51Z

+            string_to_spv(shader_name + "_quant_f32" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, quant_float_type_dict), {{"MULMAT_QUANT", "1"}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f32}, {"B_TYPE_SCALAR", "float"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
+        }
+
+#if defined(GGML_VULKAN_FLOAT_E2M1_GLSLC_SUPPORT) && defined(GGML_VULKAN_FLOAT_E4M3_GLSLC_SUPPORT)


codex claims these are no longer used.

jeffbolznv · 2026-08-31T15:15:27Z

+
+#ifdef MULMAT_QUANT
+
+uint mm_quant_k() {


Seems like we should have a general function that maps GGML_TYPE to the QUANT_K value.

jeffbolznv · 2026-08-31T15:17:21Z

-    uint stride_a = p.stride_a / QUANT_K;
+#ifdef MULMAT_QUANT
+    // pos_a is a byte offset into the raw buffer; strides stay in block units
+    pos_a *= MmABlockBytes;


It might now be more important to make this 64b for the 64bit indexing version of these shaders. But I'm OK with leaving that until somebody complains.

jeffbolznv · 2026-08-31T15:18:19Z

 #endif
    tensorLayoutNV<2, gl_CooperativeMatrixClampModeConstantNV> tensorLayoutD = createTensorLayoutNV(2, gl_CooperativeMatrixClampModeConstantNV);

+#ifndef MULMAT_QUANT


I wonder if we could collapse this down and just unconditionally set the block size. But I'm fine with leaving it as-is.

jeffbolznv · 2026-08-31T15:19:52Z

+struct vk_matmul_pipeline_pair {
+    vk_pipeline unaligned;
+    vk_pipeline aligned;
+    uint32_t align;


FWIW it's always been unclear to me what the "align" value is supposed to mean for these shaders.

It is the alignment requirement of the shader, what the K dimension needs to be divisible by to run the shader without bounds checking.

github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jul 16, 2026

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from a619971 to 496b099 Compare July 17, 2026 07:18

jeffbolznv reviewed Jul 24, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_cm2.comp Outdated

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from cbf3468 to 001600c Compare July 28, 2026 13:09

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from 001600c to 197898e Compare August 19, 2026 06:48

This was referenced Aug 26, 2026

vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps #27726

Merged

Vulkan: wrong matmul results on Adreno (subgroupSize=128) — warptile WM exceeds BM (implicit WM <= BM invariant violated for subgroupSize > 64) #25734

Closed

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from df88488 to 19e4a29 Compare August 27, 2026 06:34

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from 19e4a29 to ce8cadf Compare August 28, 2026 07:12

0cc4m added 5 commits August 30, 2026 11:02

remove LUT quants from unified shader

f8a74a4

clean up

7847f59

restore coopmat2 q4_k/q5_k optimization

d38a4e4

split out q4_k/q5_k cm2 shader to fix Ampere regression

a8f38a9

0cc4m force-pushed the 0cc4m/vulkan-mul-mm-refactor branch from c212193 to a8f38a9 Compare August 30, 2026 09:17

0cc4m marked this pull request as ready for review August 30, 2026 20:27

0cc4m requested a review from a team as a code owner August 30, 2026 20:27

jeffbolznv reviewed Aug 30, 2026

View reviewed changes

revert iq shmem table renames

a233dda

0cc4m added 2 commits August 31, 2026 08:42

simplify cm2 code with single uint8_t buffer

1f9690e

fix fp4 extension use switch being overwritten by generic shader

9d03578

jeffbolznv reviewed Aug 31, 2026

View reviewed changes

Uh oh!

Conversation

0cc4m commented Jul 16, 2026

Overview

Requirements

Uh oh!

jeffbolznv commented Jul 16, 2026

Uh oh!

jeffbolznv commented Jul 17, 2026

Uh oh!

0cc4m commented Jul 24, 2026

Uh oh!

Uh oh!

jeffbolznv commented Jul 24, 2026

Uh oh!

0cc4m commented Jul 24, 2026

Uh oh!

0cc4m commented Aug 20, 2026

Uh oh!

ggerganov commented Aug 20, 2026

Uh oh!

0cc4m commented Aug 20, 2026

Uh oh!

ggerganov commented Aug 20, 2026

Uh oh!

jeffbolznv commented Aug 20, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 24, 2026

Uh oh!

0cc4m commented Aug 26, 2026

Uh oh!

0cc4m commented Aug 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0cc4m commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Aug 24, 2026 •

edited

Loading

0cc4m commented Aug 31, 2026 •

edited

Loading