quantize: add IQ2_NL and IQ3_NL types (CPU)

[EAddario]

Overview

K-quants and I-quants operate on 256-element super-blocks. A tensor can use them only if its row length (ncols) is an exact multiple of 256. When it isn't, the quantizer falls back to a different (usually sub-optimal) 32-element block type per tensor.

Mainline 32-block types stop at 4.50 bpw (Q4_0/IQ4_NL) creating a hard floor on every misaligned tensor and therefore on the whole file wherever misaligned tensors dominate because there is nothing better to substitute to.

This PR introduces two new CPU-only types IQ2_NL (2.50 bpw) and IQ3_NL (3.50 bpw) extending the existing non-linear family downward. Being 32-block types, they apply wherever ncols % 32 == 0.

Additional information

Test models and results including PPL, KLD and inference benchmarks available at HuggingFace.

PR for CPU + Metal backend: #27324
PR for CPU + Metal + CUDA backend: #27325
PR for CPU + Metal + CUDA + Vulkan backend: #27983

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. AI used to code review and to generate Metal and CUDA implementations.

[CISC]

Seems like very reasonable quants to have, but who can/will review?

[Green-Sky]

BTW we also have q2_0 with a group size of 64. A comparison might be neat.

[EAddario]

At the risk of sounding like Claude, you are absolutely right @Green-Sky!
Forgot it got merged recently. I'll add to the test results

[EAddario]

Not sure if there's something wrong on my side but like-for-like test results with Q2_0 are terrible:

====== Perplexity statistics ======
Mean PPL(Q)                   : 1791393.810329 ± 10600.941002
Mean PPL(base)                :   7.740272 ±   0.051484
Cor(ln(PPL(Q)), ln(PPL(base))):   3.64%
Mean ln(PPL(Q)/PPL(base))     :  12.352068 ±   0.008741
Mean PPL(Q)/PPL(base)         : 231438.093611 ± 2022.897691
Mean PPL(Q)-PPL(base)         : 1791386.070057 ± 10600.939130

====== KL divergence statistics ======
Mean    KLD:  12.589577 ±   0.005998
Maximum KLD:  24.119755
99.9%   KLD:  19.458214
99.0%   KLD:  17.880480
95.0%   KLD:  16.360634
90.0%   KLD:  15.541903
Median  KLD:  12.619534
10.0%   KLD:   9.645974
 5.0%   KLD:   8.658699
 1.0%   KLD:   6.787046
 0.1%   KLD:   5.274840
Minimum KLD:   2.753649

====== Token probability statistics ======
Mean    Δp: -45.117 ± 0.101 %
Maximum Δp:  0.061%
99.9%   Δp:  0.000%
99.0%   Δp: -0.001%
95.0%   Δp: -0.049%
90.0%   Δp: -0.327%
75.0%   Δp: -4.782%
Median  Δp: -36.710%
25.0%   Δp: -89.473%
10.0%   Δp: -99.357%
 5.0%   Δp: -99.839%
 1.0%   Δp: -99.975%
 0.1%   Δp: -99.996%
Minimum Δp: -100.000%
RMS Δp    : 59.791 ± 0.086 %
Same top p:  0.023 ± 0.004 %

Full logs and scores here

[EAddario]

but who can/will review?

@CISC, if I got it right it will be either @JohannesGaessler or @ggerganov although there may be others I don't know about.

[Green-Sky]

Not sure if there's something wrong on my side but like-for-like test results with Q2_0 are terrible:

The quantizer for q2_0 might not be very smart. The format is mostly for ternary trained models. (or quaternary since each weight can have 4 states).