Repository navigation
# 70B 4-bit LLM decode bottlenecked by HIP kernel (kgemm_4bit_inference_naive) efficiency — 49% vs 91% memory bandwidth on ROCm/gfx1151 #1842
Description
Summary
On AMD gfx1151 (RDNA 3.5, Strix Point UMA), the kgemm_4bit_inference_naive HIP kernel achieves only 49% of memory bandwidth, while rocBLAS bf16 GEMM achieves 91% on the same hardware.
This results in ~2.1 tok/s for 70B 4-bit decode instead of the projected ~3.9 tok/s if the kernel matched bf16 efficiency.
Environment
- GPU: gfx1151 (Strix Point / Ryzen AI MAX 395), 40 CUs, LPDDR5X-8000
- Measured peak memory bandwidth: 210 GB/s (copy test)
- ROCm: 7.11
- PyTorch: 2.9.1+rocm7.11
- bitsandbytes: rocm_enabled branch
Benchmark Data
M=1 GEMV Bandwidth Efficiency (70B MLP dimensions: 8192×28672)
| Kernel | Data Size | Time | Achieved Read BW | % of Peak (210 GB/s) |
|---|---|---|---|---|
kgemm_4bit_inference_naive<hip_bfloat16, 128, 1> |
117 MB | 1128 µs | 102 GB/s | 49% |
rocBLAS bf16 Cijk_Ailk_Bljk_BBS_BH_...MT96x96 |
469 MB | 2450 µs | 191 GB/s | 91% |
The 4-bit kernel reads 4× less data but is only 2.1× faster — it should be closer to 4×.
Path Comparison (confirming dispatch is correct)
| Path | Time | Notes |
|---|---|---|
Fused gemv_4bit |
1147 µs | ✅ Selected |
| Dequant + bf16 mm | 7574 µs | 6.6× slower (not selected) |
| Pure bf16 mm | 2459 µs | Baseline |
We are hitting the correct fused kernel path. The issue is kernel efficiency, not dispatch.
Profiler Output
=== Fused gemv_4bit ===
void kgemm_4bit_inference_naive<hip_bfloat16, 128, 1...> 22.564ms (20 calls) 1.128ms/call
=== bf16 mm ===
Cijk_Ailk_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT96x96... 48.990ms (20 calls) 2.450ms/call
Impact on Real Workloads
70B 4-bit LLM decode (Llama-3.3-70B-Instruct, NF4):
| Scenario | Projected tok/s |
|---|---|
| Current (49% BW efficiency) | ~2.1 |
| If 4-bit matched bf16 efficiency (91%) | ~3.9 |
| Theoretical limit (100% BW) | ~4.3 |
This represents a ~1.85× optimization opportunity.
Analysis
The kernel name kgemm_4bit_inference_naive suggests this implementation has not yet been aggressively tuned for RDNA 3.5. Possible causes for the 49% bandwidth efficiency:
- Memory access not fully coalesced — 4-bit packing may cause suboptimal load patterns
- Dequantization not pipelined with memory loads — compute may be stalling fetches
- Not tuned for RDNA 3.5 wave scheduling — occupancy, wavefront sizes, LDS usage
- Missing HIP-specific optimizations — intrinsics, cache hints, vector widths
The bf16 baseline demonstrates that decode-scale GEMV on gfx1151 can achieve near-optimal memory bandwidth (91%); the gap is therefore specific to this kernel implementation rather than a platform limitation.
Questions
-
Is there a more optimized kernel path for RDNA 3.5 / gfx11 that should be selected?
-
Are there plans to optimize the HIP 4-bit GEMV kernels for recent AMD architectures?
-
Would a PR with gfx1151-specific tuning be welcome? If so, are there guidelines for the expected kernel structure?
-
Is the
naivesuffix intentional or is it the only implementation for HIP — i.e., is there a non-naive implementation planned or in progress?
Minimal Repro
import torch
import time
import bitsandbytes.functional as F
M, K, N = 1, 8192, 28672 # 70B MLP dimensions
# 4-bit path
w_fp = torch.randn(N, K, dtype=torch.bfloat16, device='cuda')
w_4bit, quant_state = F.quantize_4bit(w_fp, quant_type='nf4')
x = torch.randn(M, K, dtype=torch.bfloat16, device='cuda')
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(100):
# Note: F.gemv_4bit invokes the same kgemm_4bit_inference_naive kernel
# used by bnb.nn.Linear4bit during M=1 decode
y = F.gemv_4bit(x, w_4bit.T, state=quant_state)
torch.cuda.synchronize()
print(f"4-bit: {(time.perf_counter()-start)/100*1e6:.0f} µs")
# bf16 baseline
w_bf16 = torch.randn(K, N, dtype=torch.bfloat16, device='cuda')
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(100):
y = x @ w_bf16
torch.cuda.synchronize()
print(f"bf16: {(time.perf_counter()-start)/100*1e6:.0f} µs")Expected output on gfx1151:
4-bit: 1147 µs
bf16: 2459 µs
Profiling Script
import torch
from torch.profiler import profile, ProfilerActivity
import bitsandbytes.functional as F
M, K, N = 1, 8192, 28672
w_fp = torch.randn(N, K, dtype=torch.bfloat16, device='cuda')
w_4bit, quant_state = F.quantize_4bit(w_fp, quant_type='nf4')
x = torch.randn(M, K, dtype=torch.bfloat16, device='cuda')
for _ in range(10):
y = F.gemv_4bit(x, w_4bit.T, state=quant_state)
torch.cuda.synchronize()
with profile(activities=[ProfilerActivity.CUDA]) as prof:
for _ in range(20):
y = F.gemv_4bit(x, w_4bit.T, state=quant_state)
torch.cuda.synchronize()
print(prof.key_averages().table(sort_by="cuda_time_total"))Related
- Filed parallel issue with AMD/PyTorch ROCm team requesting guidance on RDNA 3.5 kernel optimization patterns
- Happy to collaborate on optimization efforts or provide additional profiling data
Thanks for your work on ROCm support!
Activity
BellaDoggie commentedon Jan 20, 2026
Update: Root Cause Found + 15× Speedup Achieved
After extensive profiling, I identified the bottlenecks in
kgemm_4bit_inference_naiveon RDNA3 (gfx1151 / Strix Point).Results
Optimization Journey
Phase 1: Float compute path (~102 → ~132 GB/s)
float quant_map[16]instead ofT— avoids bf16↔float conversionsfloat local_B[8]andfloat local_A[8]— all compute in float#if BNB_BF16_AVAILABLEbranches in hot path — always use float mathPhase 2: ILP and vectorization (~132 → ~154 GB/s)
__builtin_expect— branch hints for fast pathclz_blocksizeout of loopint4loads for all 32 values at oncePhase 3: Bank conflicts and accumulators (no improvement, but kept)
quant_map[32]— two copies, even/odd lanes use different copies to reduce bank conflictslocal_C0throughlocal_C3) — breaks FMA dependency chain for more ILPPhase 4: B prefetch (~154 → ~158 GB/s)
Phase 5: 64 threads per block (~158 → ~287 GB/s) — THE BIG WIN
What Didn't Work
Important for Users
Until this is fixed upstream:
Also ensure
torch.inference_mode()is used — without it,requires_grad=Truebypasses the fused kernel entirely.Optimized Kernel Code
Here's the modified
kgemm_4bit_inference_naivefunction that achieves these results:Click to expand kernel code
Note: Also requires template instantiation for 64 threads and updating the kernel launcher to use 64 threads instead of 128 for RDNA3.
I don't have bandwidth to maintain a PR, but maintainers are welcome to use this code.
TimDettmers commentedon Feb 21, 2026
Thanks for the incredibly thorough profiling and optimization work — the 15x speedup on RDNA 3.5 decode is a remarkable result, and the phase-by-phase breakdown is very useful.
We'd be very interested in a PR with these optimizations. A few notes on the integration path:
There's an active RFC (PR #1877) to unify the currently-duplicated CUDA and HIP kernel sources into a single codebase using a
compat.cuhportability layer. Right nowkernels.cuandkernels.hipare maintained as near-identical copies that inevitably drift apart — exactly the kind of thing that makes architecture-specific tuning hard to maintain.For a PR with your optimizations, either approach would work:
kernels.hip(simpler, can be merged sooner)One consideration: the current kernel is instantiated with
THREADS=128for all platforms. Your finding that 64 threads per block is optimal for RDNA 3 is interesting — we'd want to make sure the thread count can be architecture-dependent without regressing CDNA (MI-series) performance. A template parameter or compile-time selection based on target architecture would be the right approach.Would you be interested in submitting a PR?
JiwaniZakir commentedon Mar 25, 2026
The gap between 49% and 91% bandwidth utilization strongly suggests uncoalesced global memory accesses during nibble unpacking in
kgemm_4bit_inference_naive(incsrc/kernels.cu) — threads within a wavefront are likely reading from strided or non-contiguous addresses when extracting 4-bit weights, fragmenting cache line fetches on the LPDDR5X bus. On RDNA 3.5's 64-lane wavefronts, this is especially costly since a single poorly-aligned load pattern can halve effective bandwidth. The fix typically involves restructuring the weight layout so that consecutive threads read consecutive 128-bit aligned chunks (e.g., viauint4vectorized loads) before unpacking, which is precisely the kind of access pattern the tuned rocBLAS kernel uses to hit 91%. It's also worth checking whether the kernel is launching with enough wavefronts per CU to hide the memory latency — with M=1, there may be insufficient parallelism to keep all 40 CUs saturated if the grid is sized for larger batch dimensions.h34v3nzc0dex commentedon Jun 7, 2026
Confirming 4-bit works on gfx1151 once bnb is built from source, and flagging the packaging gap that's the actual blocker. Built 0.50.0.dev0 for gfx1151 (
-DROCM_VERSION=83 -DBNB_ROCM_ARCH=gfx1151, rocm83→rocm713 shim), Radeon 8060S, ROCm 7.13, torch 2.11. nf4 quantize+dequantize, 8192×8192:The snag for everyone else: stock PyPI bnb (0.49.2) ships ROCm binaries only up to
libbitsandbytes_rocm72.so, but a ROCm 7.13 / gfx1151 stack wantslibbitsandbytes_rocm83.so→ load fails, no gfx1151 binary present. So today it's build-from-source or nothing on Strix Halo, which is the thing that breaks an otherwise-workingunsloth[amd]QLoRA path. A published gfx1151 / rocm8x wheel would close it. Build recipe + datapoint: https://github.com/h34v3nzc0dex/strix-halo-llm-finetune-guide/tree/main/bnb-1842-gfx1151-4bit .