Support for BitNet (Microsoft's native 1 bit LLM)

[DrMurx]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

---

Feature request

Microsoft has released Bitnet B1.58-2B-4T under the MIT license. This is a 1.58 bit model with 2 billion parameters, natively trained for that precision (instead of being created by quantization of larger models).

The implication is that BitNet is fairly small, yet powerful, and can run solely on a CPU with acceptable inference speed.

Furthermore, Microsoft provides an own inference framework named BitNet.cpp, based on llama.cpp. My understanding is that this would allow a rather straightforward integration in existing codebase.

[marcelklehr]

Hello @DrMurx
Thank you for raising this feature request! We're also excited about 1-bit LLMs and the potential they hold! In this app, we're unfortunately not using llama.cpp directly but via its python bindings -- using bitnet.cpp directly thus might not be as straightforward as it seems. It is somewhat unfortunate that Microsoft forked llama.cpp instead of contributing back to the project. Since bitnet.cpp is still young, perhaps we will see an eventual convergence of the two projects soon. For now, I will leave this open until more time has passed and we have more clarity on the situation.