Feature Request: SYCL backend support for newly added Q1 quantization

[vanshnawander]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

SYCL backend support for newly added Q1 quantization. This would allow higher token/sec for normal igpu variants. the Bonsai 1.7 B model gives like 1 tok/sec for this with normal cpu, adding igpu support can make this 10 tok/sec i believe. which would be great. @khosravipasha

Motivation

Q1 quantization has been introduced and adopted in Bonsai models (by Prism ML), offering extremely aggressive compression with minimal memory footprint. The performance of these models is pretty great and i believe I can run this for daily tasks on my desktop which doesn't have dedicated gpu.

Possible Implementation

No response

[NeoZhangJianyu]

@vanshnawander
What's the target Intel GPU for your requirement?
Could you share the detail model of the iGPU or CPU?

Is it the performance target in your case: get 10 t/s on 1.7 B model?

Thank you!

[vanshnawander]

Component	Details
CPU	Intel Core i5-13500 (13th Gen)
Cores/Threads	14 cores / 20 threads
GPU (Integrated)	Intel UHD Graphics 770
VRAM (iGPU)	~15 GB shared system RAM
Linux	Ubuntu (detected by clinfo)
oneAPI Version	2025.3
Model	Bonsai-1.7B (Q1_0 quantization)
llama.cpp Build	GGML_SYCL=ON with Intel oneAPI

---

Detected Devices (from sycl-ls)

Device ID	Type	Name	Compute Units	Max Work Group	Global Mem	Driver Version
0	[opencl:gpu:0]	Intel UHD Graphics 770	32	512	14963 MB	23.43.027642

---

I want something faster, becuase it's currently only cpu based it is giving 1 tok/sec, i believe sycl would give a significant speed up boost.
Thank you!

[khosravipasha]

The CPU optimization for x86 is not merged yet, right now only fast on Metal and Mac NEON path.
After this PR is merged probably will get much better CPU speeds.
#21636

Not too familiar with SYCL.

There is some works on Vulkan backend from the community as well in our fork after its stable I can ask them to send a PR here. PrismML-Eng#9

And had some partial success with OpenCL (this is old fork so have not tested it recently):
https://github.com/PrismML-Eng/llama.cpp/tree/prism-android

[vanshnawander]

Thanks for the update, I will check that out. I would also love to contribute to this, I will check if i can help in the Vulkan/OpenCL backend part.

[NeoZhangJianyu]

It's clear to us.

It's interesting to support Q1 type.
I can't estimate the workload to support this Q1 model.
If there is reference like CUDA implementation, it will help estimate the workload.

Thank you!

[khosravipasha]

We have a PR for cuda here
#21629

What kind of hardware SYCL supports? Happy to help with testing if I have the hardware.

Just saw a vulkan backend was merged as well recently.

[NeoZhangJianyu]

SYCL backend support all Intel dGPU and iGPU since 10th Core CPU.

We will discuss this feature later.

Thank you!

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[arthw]

@vanshnawander @khosravipasha
We are planning to handle this feature.