quantize: add IQ2_NL and IQ3_NL types (CPU + Metal)

[EAddario]

Overview

K-quants and I-quants operate on 256-element super-blocks. A tensor can use them only if its row length (ncols) is an exact multiple of 256. When it isn't, the quantizer falls back to a different (usually sub-optimal) 32-element block type per tensor.

Mainline 32-block types stop at 4.50 bpw (Q4_0/IQ4_NL) creating a hard floor on every misaligned tensor and therefore on the whole file wherever misaligned tensors dominate because there is nothing better to substitute to.

This PR introduces two new CPU & Metal only types IQ2_NL (2.50 bpw) and IQ3_NL (3.50 bpw) extending the existing non-linear family downward. Being 32-block types, they apply wherever ncols % 32 == 0.

Additional information

Test models and results including PPL, KLD and inference benchmarks available at HuggingFace.

PR for CPU backend: #27322
PR for CPU + Metal + CUDA backend: #27325
PR for CPU + Metal + CUDA + Vulkan backend: #27983

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. AI used to code review and to generate Metal and CUDA implementations.

[ggml-gh-bot]

Hi @EAddario, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.