Repository navigation
Illegal memory access with quantize_4bit #1782
Description
Running the latest version of bitsandbytes with all recent versions of Torch is leading to the following error. I've tried bitsandbytes version from 0.45.5 to the latest (0.48.0) and all lead to the same result.
import torch
from bitsandbytes.functional import quantize_4bit
data = torch.rand(3000000, 384, device="cuda")
quantize_4bit(data)Error an illegal memory access was encountered at line 57 in file /src/csrc/ops.cu
I'm running this on Python 3.10, Linux and with a RTX 3090. It seems like there is more than enough free GPU memory.
Activity
matthewdouglas commentedon Oct 16, 2025
Hi @davidmezzetti
Thank you for reporting this. I can reproduce this with RTX 4090, Python 3.13, PyTorch 2.9.0+cu130.
Additionally, I can reproduce the same issue for the 8bit
quantize_blockwisefunctionality.This issue appears when we have a tensor with more than
2**30elements. I also see an issue with the dequantization kernel as we approach2**31elements. I'll open up a PR to fix this soon!davidmezzetti commentedon Oct 16, 2025
Thank you for the rapid response! Glad to hear it looks to be a simple fix.
matthewdouglas commentedon Oct 21, 2025
@davidmezzetti I've opened #1784 to fix this, with the caveat that the upper limit is now
2**31-1instead of2**30elements. So for fp32 data, this works on tensors up to 8GiB. Do you expect to need support for larger sizes? If so, we can do that, but it's a little more involved, so I would prefer we open up a feature request on that.davidmezzetti commentedon Oct 21, 2025
Thank you for the update! I really appreciate it.
Ideally I would like the GPU to set the memory limit vs a software imposed limit. Is there anything I can do to help?
matthewdouglas commentedon Oct 21, 2025
The main issue is that our C API uses int32 for tensor sizes. While we can change that to int64, it would be BC breaking for some users like Unsloth who use the C API directly, particularly for 4bit.
Instead I could consider adding int64 versions and dispatch to those for larger problem sizes. But I want to treat that as a separate feature request since all of the kernels are designed for at most
2**31-1with the usage of int32 everywhere. We would have to also consider if we want support for >2**31-1in any single dimension (which I assume we do).davidmezzetti commentedon Oct 22, 2025
@matthewdouglas Makes sense. I was looking at the CUDA code and wondering why that variable couldn't simply be changed to a 64-bit int but this explanation explains that!