HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR

[yanjs]

Overview

__vsub4 and __vcmpne4 are CUDA intrinsic fallback implementations in HIP backends. This optimization improves the efficiency of these implementations using SWAR. The wrong implementation of __vsub4 is fixed.

These functions are used by IQ2/IQ3 paths only on HIP backends. These paths gain decent improvements (Qwen 3.8 27B IQ3_S tg128 +~20% on my device).

__vsub4: ~10% speed up. Fix the wrong implementation.
__vcmpne4: ~10% speed up. Less instructions. No dependency on VCC. See https://godbolt.org/z/Mf8Yv73Y1 for details.

test-backend-ops test passed.

Additional information

Tested on Arch Linux, RX 9070, gfx1201, ROCm 7.14, compared to base 77f132c . (up: master, mid: with only __vcmpne4 optimization, down: with 2 commits from this pr)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-base2/bin/llama-bench -m Qwen3.8-27B-UD-IQ3_S.gguf
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           pp512 |       801.30 ± 20.27 |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           tg128 |         27.11 ± 0.04 |

build: 77f132c (50)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-mid/bin/llama-bench -m Qwen3.8-27B-UD-IQ3_S.gguf
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           pp512 |       823.09 ± 21.53 |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           tg128 |         29.90 ± 0.03 |

build: f5725ef (50)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-swar/bin/llama-bench -m Qwen3.8-27B-UD-IQ3_S.gguf
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           pp512 |       853.85 ± 23.18 |
| qwen35 27B IQ3_S - 3.4375 bpw  |  11.20 GiB |    27.32 B | ROCm       |  -1 |           tg128 |         32.97 ± 0.06 |

build: a2731fb (51)

__vcmpeq4 is dead code. I did not change it as it is not benched. It can be removed or implemented using bitwise not of __vcmpne4 if required.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: AI assistance was used during investigation and analysis of this optimization.

[yanjs]

(up: master, down: this pr)

$ HIP_VISIBLE_DEVICES=0 /tmp/build-base2/bin/test-backend-ops perf -o MUL_MAT -p "type_a=iq3_s"
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
Testing 2 devices

Backend 1/2: ROCm0
  Device description: AMD Radeon RX 9070
  Device memory: 16304 MB (16222 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 10224 runs -   102.92 us/run - 117.44 MFLOP/run -   1.14 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  9372 runs -   107.91 us/run - 234.88 MFLOP/run -   2.18 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  9088 runs -   112.71 us/run - 352.32 MFLOP/run -   3.13 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  8307 runs -   120.44 us/run - 469.76 MFLOP/run -   3.90 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  7866 runs -   127.91 us/run - 587.20 MFLOP/run -   4.59 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  6099 runs -   165.61 us/run - 939.52 MFLOP/run -   5.67 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 706 runs -  1417.98 us/run -  60.13 GFLOP/run -  42.40 TFLOPS
  Backend ROCm0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

$ HIP_VISIBLE_DEVICES=0 /tmp/build-swar/bin/test-backend-ops perf -o MUL_MAT -p "type_a=iq3_s"
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16304 MiB):
  Device 0: AMD Radeon RX 9070, gfx1201 (0x1201), VMM: no, Wave Size: 32, VRAM: 16304 MiB
Testing 2 devices

Backend 1/2: ROCm0
  Device description: AMD Radeon RX 9070
  Device memory: 16304 MB (16222 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 15336 runs -    65.32 us/run - 117.44 MFLOP/run -   1.80 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 13206 runs -    75.92 us/run - 234.88 MFLOP/run -   3.09 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 12212 runs -    82.92 us/run - 352.32 MFLOP/run -   4.25 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 10437 runs -    96.21 us/run - 469.76 MFLOP/run -   4.88 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  9234 runs -   109.23 us/run - 587.20 MFLOP/run -   5.38 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                  6848 runs -   146.60 us/run - 939.52 MFLOP/run -   6.41 TFLOPS
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  MUL_MAT(type_a=iq3_s,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                 828 runs -  1209.49 us/run -  60.13 GFLOP/run -  49.71 TFLOPS
  Backend ROCm0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK