Repository navigation
Support for Apple silicon #252
Description
Would it make sense for this library to support platforms other than cuda on x64 Linux? I am specifically looking for Apple silicon support. Currently not even cpuonly works since it assumes SSE2 support (Even without Neon. Support).
i would guess that the first step would be a full cross platform compile (arm64), then ideally support for Metal Performance Shaders as an alternative to CUDA (assuming it is at all feasible).
I could probably contribute some towards support if there is interest for bitsandbytes to be multi platform. I have some experience setting up cross platform Python libraries.
Activity
TheStoneSpain commentedon Apr 2, 2023
Hi there, I will contribute too, in order to get it to work on Metal Apple M1
this is my trace:
rickardp commentedon Apr 3, 2023
Nice to hear! It would be good to hear from the maintainers that they are at all interested in making this package cross-platform. It is very much CUDA focused at the moment.
Getting
libbitsandbytes_cpu.soto compile for macOS arm64 was not at all difficult, just an exercise in moving around some #ifdefs, but CPU support would obviously need to add Neon (SIMD) to make any sense IMHO. Then, of course the MPS support would be needed at one point (though I expect it's quite a lot more work).I've just started looking at the unit tests and the Python libraries.
The C++ code is quite nicely structured, but the Python code would need some refactoring since most of the calls assume CUDA (x.cuda() instead of x.to(device), etc). Also, since the CPU version does not cover 100% of the feature set, testing is going to be quite some work as there is no real baseline. I suppose one question is if it would make sense to make the CPU cover 100% of the API calls, even if inefficient, just to provide a baseline that the GPU implementations could compare against?
If pursuing this, I propose implementing cross-platform CPU support first, then tackling MPS. MPS is of course what makes it useful.
(I have the exact same setup BTW, 2021 MBP)
Edit: Specifically, here's how I imagine the unit tests would have to work
https://github.com/TimDettmers/bitsandbytes/pull/257/files#diff-659bad232c71219167252c1a5ccbc427b6f54925b78741df18613c3c49aaa4c1R153
So at least one CPU test pass on my M1 Mac :)
janrinze commentedon Apr 5, 2023
please have a look at Building on Jetson AGX Xavier Development Kit fails #221
It addresses the same AArch64 issue but on CUDA supported platforms like NVidia Jetson.
UserHIJ commentedon Jun 24, 2023
Wow .. not to be inflammatory , but are we saying that there's no immediate solution for this if you have any macbook in the last like .. 5 years? Yuck.
janrinze commentedon Jun 25, 2023
https://en.wikipedia.org/wiki/Apple_M1 introduced less than 3 years ago.
Things take time in the world of open-source. Specially when using hardware such as Apple.
KotlinFactory commentedon Aug 6, 2023
when will this be done?
benjaminhuo commentedon Aug 21, 2023
Looking forward to the support for this too, got the below errors when I tried to fine-tune llama2 7B with
load_in_8bit=Trueenabled on my Macbook M2, PyTorch‘s support to MPS is getting better and I hope this project could support this as well:AlexandreCassagne commentedon Sep 8, 2023
@benjaminhuo Getting the same issue as you.
id4thomas commentedon Sep 17, 2023
https://github.com/TimDettmers/bitsandbytes/blob/18e827d666fa2b70a12d539ccedc17aa51b2c97c/bitsandbytes/autograd/_functions.py#L227
This seems to be due to calling torch.cuda even if the device type isn't cuda.
One way to patch these unchecked torch.cuda calls is adding device checks like
mps returns "mps" as device.type
pechaut78 commentedon Nov 27, 2023
same issue here, MPS seems to be the problem
ProjectProgramAMark commentedon Dec 2, 2023
getting same issue with apple silicon. would love to see some support for it soon!
ivan-digital commentedon Dec 24, 2023
Same issue. Would be nice to have support for MPS.
ageorgios commentedon Dec 26, 2023
Same here, please have support for MPS
https://github.com/ml-explore/mlx
592319702 commentedon Jan 23, 2024
(torch-gpu) I542464@DY4GPKX1J0 test % python3 fine_tune_llama_2_in_google_colab.py /Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " 'NoneType' object has no attribute 'cadam32bit_grad_fp32' Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:32<00:00, 16.06s/it] /Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/peft/utils/other.py:102: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead. warnings.warn( /Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/trl/trainer/sft_trainer.py:159: UserWarning: You didn't pass amax_seq_lengthargument to the SFTTrainer, this will default to 1024 warnings.warn( 0%| | 0/250 [00:00<?, ?it/s]You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using thecallmethod is faster than using a method to encode the text followed by a call to thepadmethod to get a padded encoding.use_cache=Trueis incompatible with gradient checkpointing. Settinguse_cache=False... /Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants. warnings.warn( FP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first. Traceback (most recent call last): File "/Users/I542464/test/fine_tune_llama_2_in_google_colab.py", line 229, in <module> trainer.train() File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/trainer.py", line 1539, in train return inner_training_loop( ^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/trainer.py", line 2654, in training_step loss = self.compute_loss(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/trainer.py", line 2679, in compute_loss outputs = model(**inputs) ^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/peft/peft_model.py", line 922, in forward return self.base_model( ^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 806, in forward outputs = self.model( ^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 685, in forward layer_outputs = torch.utils.checkpoint.checkpoint( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/_dynamo/external_utils.py", line 17, in inner return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 451, in checkpoint return CheckpointFunction.apply(function, preserve, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 230, in forward outputs = run_function(*args) ^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 681, in custom_forward return module(*inputs, output_attentions, None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 408, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 305, in forward query_states = self.q_proj(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/peft/tuners/lora.py", line 1123, in forward result = super().forward(x) ^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 221, in forward out = bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/I542464/miniconda3/envs/torch-gpu/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py", line 567, in matmul_4bit assert quant_state is not None ^^^^^^^^^^^^^^^^^^^^^^^ AssertionError 0%| | 0/250 [00:01<?, ?it/s]mbtre commentedon Feb 15, 2024
+1 MPS support would be absolutely great!
59 remaining items
rickardp commentedon Apr 8, 2025
Most of this was actually merged in separate PRs to make it more manageable. This PR was mainly about making the library portable at all.
Before contributing, I think the architectural direction for this library has to be established. I was at the time arguing that we need a 100% test covered CPU implementation to start from so MPS support can be added gradually. If my understanding is correct, the latest architectural direction is to unify backend initialization with PyTorch. Once we get to a level where kernels can be added one by one and there are unit tests to verify correctness , I think a lot more can be done by the community. Right now IMHO there is quite a dependence on the core maintainers
matthewdouglas commentedon Apr 8, 2025
Hi folks,
We're not quite there yet, but after merging #1544 we've started to pave the path forward. I want to make it clear that we haven't abandoned Apple silicon support, but instead have had competing priorities to shift through. We're still working toward this though.
In fact, we've now got a CI test suite that is stable and can be reliably deterministic on our
mainbranch for the CUDA implementation. The CUDA implementation is always going to be the "gold standard" reference implementation, but this is a step forward toward being able to validate implementations for new hardware.We have some new PyTorch native fallback implementations of some custom operators which can be used for reference/fallback on CPU as well as other devices. To @rickardp's point, this will allow incremental support for more optimal implementations on additional platforms, as well as serve as a secondary reference to evaluate against.
I have a branch which is a WIP that I have not pushed yet, but it has an implementation of the NF4 quantization/dequantization ops which can run on CPU and even on MPS with a little bit of work. I've also acquired an M4 MacBook Pro for future development/validation. Soon we should be able to build out enough plumbing to better enable community contributions for kernel implementations.
foggyghost0 commentedon Jun 11, 2025
Have just spent 6 hours trying to implement a custom quant solution on mac, still hope to see bits and bytes support!
Add comprehensive Apple Silicon MPS support to bitsandbytes
Add comprehensive implementation summary documentation
feynon commentedon Aug 5, 2025
Any updates on this issue?
bhupesh-sf commentedon Sep 1, 2025
any update???
luvwinnie commentedon Sep 4, 2025
any update???
davidxia commentedon Sep 4, 2025
The fork https://github.com/yhsung/bitsandbytes seems to support Apple silicon since yhsung#1. People can try forking. Not sure why the PR wasn't made to upstream here too.
reneleonhardt commentedon Sep 4, 2025
#1699 had been created, but even it would've not been closed you can't know if it would've been reviewed and merged.
Besides that, CI has been disabled in that repo and not enabled in this repo, so everyone would have to try it themselves locally to see if any old or new tests would fail.
JINO-ROHIT commentedon Nov 27, 2025
hi everyone, just checking if we have updates on this?
thanks
ZerocoolZa commentedon Dec 4, 2025
just convert bit and bytes to c. and dou it your self