Skip to content

Strix rebased (SHOULD NOT BE MERGED - SHOULD BE FORCED PUSHED ON MASTER) - #181

Merged
gaetan-puleo merged 0 commit into
masterfrom
strix-rebased
Oct 16, 2026
Merged

Strix rebased (SHOULD NOT BE MERGED - SHOULD BE FORCED PUSHED ON MASTER)#181
gaetan-puleo merged 0 commit into
masterfrom
strix-rebased

Conversation

@gaetan-puleo

@gaetan-puleo gaetan-puleo commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Overview

strix/master rebuilt as a linear history on top of upstream ggml-org/llama.cpp master e467e5b5e: 313 commits, no merge commits. This branch is meant to replace master by force-push, not to be merged.

How it was built:

Fixes made while porting, applied to the commit that introduced each problem:

Main adaptations to upstream refactors:

  • Vulkan mul_mat_id. Upstream's tile configs and row-map are used. The fork's MMID probes (GGML_VK_MMID_*, enabled by default) were ported onto tc_mmq_id.

  • Vulkan flash attention. The FA flag bits are renumbered so that upstream's sparse FA fits next to the fork's flags. Selected-key attention, NaN-safe masked V, and multi-row cm1 with V^T are merged with upstream USE_SPARSE; V^T is excluded for sparse.

    Flag Bit
    USE_SPARSE 16
    DYNAMIC_KV 32
    NAN_SAFE_V 64
    V_TRANSPOSED 128
    MR_* 256–2048

  • Vulkan int8 MMQ cm1 MUL_MAT_ID epilogue. It uses the existing FusedScale binding 5 and fusion_flags.

  • HIP.

    • MMQ uses upstream's prec_src1.
    • mmb_* / hc_gate_mix take the backend context.
    • The chunked GDN path runs first; upstream's GDN MMA is the fallback.
    • The stable-tie radix top-k and the one-pass workgroup top-k are ported onto upstream's chunked radix.

  • Graph / memory.

  • Server. The persistent prompt cache (server : add --cache-disk for a persistent prompt cache #81) and its revert are both applied, so it nets to no change, as in the fork.

Intentional difference from strix/master: the Vulkan MMID experiments (GGML_VK_MMID_*) are kept. The fork dropped them in its 2026-09-29 sync in favour of upstream's matmul.

Measurements

Device:     Ryzen AI Max+ 395 / BeyondMax Series (strix-002)
Memory:     128 GB LPDDR5X
Kernel:     7.2.4-200.fc44.x86_64, amdgpu.gttsize=126976 ttm.pages_limit=32505856
Backend:    ROCm 7.16 (HIP, gfx1151) ; Vulkan RADV, Mesa 26.0.3
Build:      -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
Baseline:   strix/master deb5e2f50 (HIP build), same session
Change:     strix-rebased 93e6d0285, -dev ROCm0
Model:      Qwen3.8-Flash-Next UD-Q4_K_XL (+ Unsloth MTP head, shared Q8_0)

llama-bench was run with -fa on -ctk f16 -ctv f16 -lm none -b 2048 -ub 2048 -r 2; pp4096 used -b/-ub 4096. Values are t/s, mean of 2 interleaved rounds.

test strix/master strix-rebased
pp2048 1498.7 1504.6 ± 34.7
pp2048 @ d12000 1376.3 1355.8 ± 31.7
pp2048 @ d32000 1297.9 1268.5 ± 53.1
pp4096 1630.6 1606.4
tg64 26.8 26.7 ± 0.2
tg64 @ d12000 24.8 24.1 ± 0.3
tg64 @ d32000 24.9 24.5 ± 0.2
MTP draft (llama-server, n_max 2, 6 prompts) 42.9 (accept 80.5%) 42.5 (accept 80.7%)

Prefill and MTP are at parity. Decode at depth is 1–3% lower; the cause has not been isolated.

Correctness:

  • test-backend-ops:

    Backend Result Notes
    ROCm0 33200/33200
    Vulkan0 35987/35990 3 failures, listed below

  • The 3 Vulkan failures are MUL_MAT exact-batch cases (bf16 n=9, bf16 n=16, q8_0 n=4, k=5120). The fork implements GGML_HINT_EXACT_BATCH only for CUDA/HIP; these failures are on record since the first checkpoint of the rebase but were not re-run on strix/master in this session.

  • test-llama-context (all generated models, including qwen35 / qwen35moe rollback, replay and shared), test-chat, test-arg-parser, test-qsa-prefix, test-ple-disk: pass.

  • Perplexity, wikitext-2:

    Run PPL
    c512, 64 chunks, default (ROCm) 3.3399
    c512, 64 chunks, Vulkan0 3.3414
    c16384, 4 chunks 2.7152
    c16384, 4 chunks, fork at the same QSA commit (earlier check) 2.7174

  • MTP greedy output is byte-identical to strix/master and to the run without MTP.

Additional information

  • Per-commit decision log (109 entries): skips, ports and measurements. It is a local file, not part of the branch, and I can attach it to this PR.
  • Backup branches exist locally only. master stays untouched until the force-push.

Requirements

  • I have read and agree with the contributing guidelines
  • This change is Strix Halo specific, or justified by measurements on Strix Halo. General llama.cpp improvements
    belong in halo-box/llama.cpp instead
  • AI usage disclosure: AGENT-AUTHORED. Claude (Opus 5.5, Claude Code) did the rebase, conflict resolution, history surgery, builds, tests and benchmarks on strix-002, under the direction of @gaetan-puleo, who made the open calls: keep the fork QSA, MTP and GDN; port the MMID probes; reimplement the in-place GDN.
  • Additional verification (after the first push):

    • Per-commit build: 310/310 OK. Every commit from the first history rewrite to the tip was built with a CPU-only build of llama, common, test-chat and llama-server. These are the parts the rewrites touched; the HIP/Vulkan code of each commit was already built during the rebase.

    • Vulkan exact-batch. The 3 MUL_MAT exact-batch failures reproduce on strix/master (16/19 cases pass on both), so they are not a regression.

    • Server pytest (-m "not slow"): 417 passed, 0 failed. The slow tests (test_tool_call, test_completion_stream_with_openai_library_stops) were not run: they start the server --offline and expect 7-8B HF models to be pre-downloaded into the local test cache. They fail the same way with the strix/master binary.

    • UI:

      Check Result
      npm run check 0 errors
      npm run build OK
      Vitest unit 735/735

      With a non-C locale, 2 formatContextLength cases fail on upstream too.

    • Converter.

      • gguf-py tests: 7/7.
      • convert_hf_to_gguf output is byte-identical to upstream's converter for tiny Qwen2, Qwen3, Qwen3-Next MoE and two Qwen3.5 MoE models, and for --mtp.
      • --mtp-shared-embd produces the smaller draft (11 MB vs 27 MB).
      • The converted models load and run with this build. A full MTP sidecar drafts through llama-server.
      • A shared-embedding sidecar for qwen35moe fails to load (token_embd.weight not found), identically on strix/master: only qwen4exp borrows the target's embeddings.
  • What was NOT verified:
    • Qwen4exp conversion: no HF checkpoint was available.
    • The UI client and e2e tests: they need a browser.
    • The slow server tests: see above.
    • CUDA (NVIDIA), Metal and the other backends.

@gaetan-puleo
gaetan-puleo requested a review from pwilkin as a code owner October 16, 2026 05:12
@gaetan-puleo gaetan-puleo changed the title Strix rebased Strix rebased (SHOULD NOT BE MERGED - SHOULD BE FORCED PUSHED ON MASTER) Oct 16, 2026
@gaetan-puleo
gaetan-puleo merged commit 93e6d02 into master Oct 16, 2026
5 checks passed
@gaetan-puleo
gaetan-puleo deleted the strix-rebased branch October 16, 2026 09:42
@Nathanw1014 Nathanw1014 mentioned this pull request Oct 16, 2026
Merged
@gaetan-puleo

gaetan-puleo commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Verification of 93e6d02 (now master)

validate.sh from master before the rebase (deb5e2f50), base deb5e2f50, candidate 93e6d0285, HIP and Vulkan, with qwen4exp = Qwen3.8-Flash-Next UD-Q4_K_XL.

Result: correctness passes on both backends. Every KLD is identical to the base, or below the reference, for every small-model quant, the MoE model, the quantized KV caches and qwen4exp on all five texts. Server decode is identical and tool calls work. Memory is at or below the base. The FAIL statuses come from two things, explained below.

  1. test-save-load-state: expected. Upstream removed this test; test-llama-context replaces it and passes. The validate.sh on master still builds the target, so I patched my local copy to skip it when tests/CMakeLists.txt no longer defines it. The candidate therefore has no binary, which shows as "fails on the candidate only".

  2. Vulkan speed on the 0.8B model:

    Flagged run Validation Rerun (3 alternating rounds, -r 5)
    Q2_0 pp512 −8.1% −8.3%
    Q1_0 pp512 −6.3% −8.1%
    IQ2_M tg128 −3.2% −2.0%
    kv-q8_0 pp512 @ d8192 −7.9% +4.0%
    qwen4exp pp512 (HIP, INCOMPLETE) −4.7% +1.9%

    The kv-q8_0 run overlapped with another build on the machine, which explains its validation figure. Of the five, only Q1_0 and Q2_0 pp512 reproduce.

    They come from upstream. Upstream e467e5b5e alone gives about 10,100 t/s on both, against 12,950 for the fork base and 11,900 for this branch. The branch uses upstream's Vulkan matmul tile configs, so it recovers part of upstream's loss but not all of it. A follow-up could port the fork's tuning for these two types.

Notes:

  • Not run: sync.sh verify and sync.sh summary. They need an upstream merge commit, and this branch is a linear rebase. Instead, I scanned for strix/master lines found neither in upstream history nor in the branch; the restored ones are listed in the PR body.
  • Memory setting: both sides ran with LLAMA_ARG_LAZY_MODE=on-direct. Without it, the qwen4exp KLD step was OOM-killed four times: about 81 GB of GTT plus the logits of --kl-divergence-base. This setting keeps the PLE n-gram table on disk.
Full validation results

Validation

Generated by skills/upstream-sync/scripts/validate.sh on a Strix Halo (gfx1151) machine.
Base deb5e2f50, candidate 93e6d0285, small model Qwen3.5-0.8B-Base-BF16.gguf 8db389e421ecf9fb, ROCm core-10.1 (HIP 7.16.26385-0000000).
Scope: QUANTS="BF16 F16 Q8_0 Q6_K Q5_K_M Q5_1 Q5_0 Q4_K_M Q4_1 Q4_0 Q3_K_M Q2_K Q2_0 Q1_0 IQ4_NL IQ4_XS IQ3_S IQ3_XXS IQ2_M IQ2_XS IQ2_XXS IQ1_M IQ1_S" MOE_QUANTS="MXFP4_MOE" KV_TYPES="q8_0 q4_0" PP_SIZES=512,2048 DEPTHS=0,32000,128000, small MoE granite-3.1-1b-a400m-instruct-f16.gguf 34052b6b2ee8ea8f.
Corpus sha256 in corpus.sha256. Thresholds: Same top p >= reference - 0.5 points, Mean KLD <= 2 x reference + 0.0001, PPL(Q)/PPL(base) within 1 +/- 0.01 (a KLD above the reference is INCOMPLETE until the PR explains it); speed: a slowdown above 3% fails, or is INCOMPLETE when it is within the summed standard deviations of base and candidate (rerun or explain). The reference is the base build against itself with a smaller ubatch.

hip: FAIL

Check Result Notes
build base PASS ok
build cand PASS ok
test-backend-ops PASS no new failure (0 known in base)
test-llama-archs (the archs it supports) PASS no new failure (0 known in base)
test-save-load-state (small model) FAIL fails on the candidate only

Qwen3.5-0.8B-Base (dense), every quant type (KLD on all.txt at -c 512, llama-bench pp512 / tg128):

Quant Quantizer output Same top p (ref) Mean KLD (ref) PPL ratio pp512 tg128 Status
BF16 - 99.314% (98.873%) 0.000202 (0.000390) 1.0000 +1.0% -0.7% PASS
F16 same 98.431% (97.304%) 0.000236 (0.001058) 0.9992 +0.3% -0.6% PASS
Q8_0 same 98.578% (98.284%) 0.000207 (0.000778) 1.0006 +4.0% -1.0% PASS
Q6_K same 100.000% (98.088%) -0.000001 (0.000802) 1.0000 +4.0% -0.0% PASS
Q5_K_M same 99.951% (95.196%) -0.000001 (0.003627) 1.0000 +2.7% -0.8% PASS
Q5_1 same 100.000% (95.539%) -0.000000 (0.004575) 1.0000 +0.3% -0.9% PASS
Q5_0 same 100.000% (98.088%) -0.000000 (0.000971) 1.0000 +3.3% -0.8% PASS
Q4_K_M same 100.000% (96.422%) 0.000000 (0.003570) 1.0000 +1.7% -1.0% PASS
Q4_1 same 100.000% (95.735%) 0.000000 (0.004570) 1.0000 +1.2% -1.1% PASS
Q4_0 same 99.951% (96.961%) 0.000000 (0.000957) 1.0000 +5.9% -1.3% PASS
Q3_K_M same 100.000% (96.127%) 0.000000 (0.002893) 1.0000 +1.6% +0.4% PASS
Q2_K same 99.265% (96.618%) 0.000134 (0.002759) 1.0005 +1.8% -0.5% PASS
Q2_0 same 100.000% (25.196%) -0.000000 (0.829942) 1.1484 +1.7% -1.0% PASS
Q1_0 same 100.000% (98.186%) 0.000000 (0.000602) 10.8359 +2.2% -0.7% PASS
IQ4_NL same 98.627% (96.814%) 0.000211 (0.001038) 0.9998 -0.5% -1.2% PASS
IQ4_XS same 100.000% (97.598%) -0.000000 (0.001107) 1.0000 +0.3% -0.7% PASS
IQ3_S same 100.000% (97.206%) -0.000000 (0.001254) 1.0000 +1.1% -1.1% PASS
IQ3_XXS same 100.000% (97.696%) -0.000000 (0.001265) 1.0000 +1.0% -1.3% PASS
IQ2_M same 100.000% (97.402%) 0.000000 (0.001492) 1.0000 +1.2% -1.3% PASS
IQ2_XS same 100.000% (96.961%) -0.000000 (0.001701) 1.0000 +1.1% -1.5% PASS
IQ2_XXS same 100.000% (97.255%) 0.000000 (0.001848) 1.0000 +0.7% -1.3% PASS
IQ1_M same 98.873% (95.392%) 0.000168 (0.002612) 0.9997 +1.4% -1.2% PASS
IQ1_S same 100.000% (91.814%) -0.000000 (0.010308) 1.0000 +1.0% -1.6% PASS

Granite 3.1 1B-A400M (MoE: MXFP4_MOE changes only expert tensors) (KLD on all.txt at -c 512, llama-bench pp512 / tg128):

Quant Quantizer output Same top p (ref) Mean KLD (ref) PPL ratio pp512 tg128 Status
MXFP4_MOE same 100.000% (97.402%) -0.000000 (0.002329) 1.0000 -0.0% -1.0% PASS

Qwen3.5-0.8B-Base with a quantized KV cache, K and V of the same type (KLD on all.txt at -c 2048, llama-bench pp512 / tg128 at depth 8192):

KV cache Same top p (ref) Mean KLD (ref) PPL ratio pp512 @ d8192 tg128 @ d8192 Status
q8_0 100.000% (100.000%) -0.000000 (-0.000000) 1.0000 +0.5% -0.5% PASS
q4_0 100.000% (100.000%) 0.000000 (0.000000) 1.0000 +1.1% +0.2% PASS

Real models, logits against the base build:

Model Text Same top p (ref) Mean KLD (ref) PPL ratio Status
qwen4exp prose 99.984% (99.984%) -0.000000 (-0.000000) 1.0000 PASS
qwen4exp code 100.000% (100.000%) 0.000000 (0.000000) 1.0003 PASS
qwen4exp structured 100.000% (100.000%) 0.000000 (0.000000) 1.0000 PASS
qwen4exp numeric 100.000% (100.000%) 0.000000 (0.000000) 1.0000 PASS
qwen4exp long (-c 16384) 100.000% (98.511%) 0.000000 (0.003184) 1.0000 PASS

qwen4exp, llama-bench t/s (mean of two alternating rounds per side):

Test Base Candidate Change Status
tg128 27.00 26.93 -0.2% PASS
tg128 @ d32000 24.95 24.89 -0.2% PASS
tg128 @ d128000 24.19 23.94 -1.0% PASS
pp512 1089.52 1038.13 -4.7% INCOMPLETE
pp512 @ d32000 975.09 959.52 -1.6% PASS
pp512 @ d128000 818.76 826.50 +0.9% PASS
pp2048 1511.07 1509.78 -0.1% PASS
pp2048 @ d32000 1336.58 1313.64 -1.7% PASS
pp2048 @ d128000 1208.46 1216.50 +0.7% PASS
  • qwen4exp server: PASS. tool call: ok (base ok); decode: identical; MTP: not applicable, no drafts on either side (no MTP head)

Memory, base -> candidate: peak RSS of the process, peak GPU memory (amdgpu VRAM + GTT above the level before the run), and the llama.cpp breakdown of the candidate per device in MiB (self = model + context + compute) when the tool printed it. A rise above 5% and 256 MiB is INCOMPLETE until the PR explains it.

Run Peak RSS Peak GPU Breakdown Status
test-backend-ops 16050 -> 16008 MiB 24571 -> 20322 MiB - -
test-llama-archs 635 -> 665 MiB 267 -> 301 MiB - PASS
test-save-load-state 812 -> 0 MiB 870 -> 0 MiB - PASS
perplexity kv-q4_0 3046 -> 2584 MiB 3681 -> 3707 MiB - PASS
perplexity kv-q8_0 3046 -> 2584 MiB 3687 -> 3713 MiB - PASS
perplexity moe-MXFP4_MOE 664 -> 654 MiB 1091 -> 1098 MiB - PASS
perplexity qwen4exp-code 3508 -> 3072 MiB 81495 -> 81521 MiB - PASS
perplexity qwen4exp-long 17642 -> 13662 MiB 80301 -> 80199 MiB - PASS
perplexity qwen4exp-numeric 3507 -> 3069 MiB 81498 -> 81525 MiB - PASS
perplexity qwen4exp-prose 3508 -> 3072 MiB 81497 -> 81521 MiB - PASS
perplexity qwen4exp-structured 3507 -> 3070 MiB 81495 -> 81525 MiB - PASS
perplexity small-BF16 1553 -> 1442 MiB 2324 -> 2328 MiB - PASS
perplexity small-F16 1524 -> 1418 MiB 2309 -> 2314 MiB - PASS
perplexity small-IQ1_M 1228 -> 1117 MiB 1074 -> 1079 MiB - PASS
perplexity small-IQ1_S 1231 -> 1122 MiB 1057 -> 1062 MiB - PASS
perplexity small-IQ2_M 1208 -> 1100 MiB 1054 -> 1058 MiB - PASS
perplexity small-IQ2_XS 1226 -> 1113 MiB 1088 -> 1093 MiB - PASS
perplexity small-IQ2_XXS 1227 -> 1118 MiB 1076 -> 1081 MiB - PASS
perplexity small-IQ3_S 1230 -> 1122 MiB 1123 -> 1127 MiB - PASS
perplexity small-IQ3_XXS 1211 -> 1100 MiB 1075 -> 1079 MiB - PASS
perplexity small-IQ4_NL 1251 -> 1145 MiB 1206 -> 1211 MiB - PASS
perplexity small-IQ4_XS 1232 -> 1120 MiB 1175 -> 1179 MiB - PASS
perplexity small-Q1_0 1229 -> 1115 MiB 979 -> 984 MiB - PASS
perplexity small-Q2_0 1226 -> 1116 MiB 1038 -> 1042 MiB - PASS
perplexity small-Q2_K 1259 -> 1144 MiB 1155 -> 1159 MiB - PASS
perplexity small-Q3_K_M 1237 -> 1132 MiB 1140 -> 1144 MiB - PASS
perplexity small-Q4_0 1236 -> 1125 MiB 1173 -> 1177 MiB - PASS
perplexity small-Q4_1 1226 -> 1121 MiB 1200 -> 1204 MiB - PASS
perplexity small-Q4_K_M 1226 -> 1121 MiB 1198 -> 1202 MiB - PASS
perplexity small-Q5_0 1235 -> 1122 MiB 1231 -> 1235 MiB - PASS
perplexity small-Q5_1 1225 -> 1120 MiB 1260 -> 1264 MiB - PASS
perplexity small-Q5_K_M 1229 -> 1121 MiB 1244 -> 1248 MiB - PASS
perplexity small-Q6_K 1219 -> 1109 MiB 1293 -> 1297 MiB - PASS
perplexity small-Q8_0 1304 -> 1199 MiB 1486 -> 1491 MiB - PASS
llama-bench kv-q4_0 1093 -> 1106 MiB 2186 -> 2193 MiB - PASS
llama-bench kv-q8_0 1122 -> 1134 MiB 2212 -> 2219 MiB - PASS
llama-bench moe-MXFP4_MOE 537 -> 547 MiB 1052 -> 1058 MiB - PASS
llama-bench qwen4exp-bench 33375 -> 33264 MiB 87659 -> 87707 MiB - PASS
llama-bench small-BF16 1027 -> 1038 MiB 2156 -> 2160 MiB - PASS
llama-bench small-F16 1027 -> 1038 MiB 2178 -> 2182 MiB - PASS
llama-bench small-IQ1_M 720 -> 732 MiB 1070 -> 1074 MiB - PASS
llama-bench small-IQ1_S 726 -> 737 MiB 1052 -> 1057 MiB - PASS
llama-bench small-IQ2_M 703 -> 715 MiB 1049 -> 1054 MiB - PASS
llama-bench small-IQ2_XS 720 -> 732 MiB 1083 -> 1088 MiB - PASS
llama-bench small-IQ2_XXS 720 -> 732 MiB 1071 -> 1076 MiB - PASS
llama-bench small-IQ3_S 729 -> 741 MiB 1118 -> 1123 MiB - PASS
llama-bench small-IQ3_XXS 708 -> 720 MiB 1070 -> 1075 MiB - PASS
llama-bench small-IQ4_NL 748 -> 760 MiB 1201 -> 1206 MiB - PASS
llama-bench small-IQ4_XS 729 -> 741 MiB 1170 -> 1175 MiB - PASS
llama-bench small-Q1_0 724 -> 735 MiB 975 -> 979 MiB - PASS
llama-bench small-Q2_0 723 -> 735 MiB 1033 -> 1038 MiB - PASS
llama-bench small-Q2_K 753 -> 764 MiB 1150 -> 1154 MiB - PASS
llama-bench small-Q3_K_M 735 -> 747 MiB 1135 -> 1139 MiB - PASS
llama-bench small-Q4_0 729 -> 741 MiB 1168 -> 1172 MiB - PASS
llama-bench small-Q4_1 723 -> 735 MiB 1195 -> 1199 MiB - PASS
llama-bench small-Q4_K_M 729 -> 741 MiB 1194 -> 1199 MiB - PASS
llama-bench small-Q5_0 729 -> 741 MiB 1226 -> 1231 MiB - PASS
llama-bench small-Q5_1 724 -> 735 MiB 1255 -> 1260 MiB - PASS
llama-bench small-Q5_K_M 729 -> 741 MiB 1240 -> 1245 MiB - PASS
llama-bench small-Q6_K 723 -> 735 MiB 1289 -> 1294 MiB - PASS
llama-bench small-Q8_0 800 -> 813 MiB 1482 -> 1486 MiB - PASS
llama-server qwen4exp 2196 -> 2216 MiB 88156 -> 88180 MiB - PASS

vk: FAIL

Check Result Notes
build base PASS ok
build cand PASS ok
test-backend-ops PASS no new failure (5 known in base)
test-llama-archs (the archs it supports) PASS no new failure (0 known in base)
test-save-load-state (small model) FAIL fails on the candidate only

Qwen3.5-0.8B-Base (dense), every quant type (KLD on all.txt at -c 512, llama-bench pp512 / tg128):

Quant Quantizer output Same top p (ref) Mean KLD (ref) PPL ratio pp512 tg128 Status
BF16 - 100.000% (99.412%) 0.000000 (0.000045) 1.0000 -1.7% -0.7% PASS
F16 same 98.725% (97.696%) 0.000213 (0.001024) 1.0005 +3.3% -0.4% PASS
Q8_0 same 100.000% (97.745%) -0.000000 (0.000785) 1.0000 +3.1% -0.8% PASS
Q6_K same 100.000% (97.941%) -0.000000 (0.000783) 1.0000 +1.9% -1.0% PASS
Q5_K_M same 100.000% (97.696%) -0.000000 (0.000801) 1.0000 +1.0% -1.7% PASS
Q5_1 same 100.000% (97.696%) -0.000000 (0.000770) 1.0000 +0.4% -1.6% PASS
Q5_0 same 100.000% (97.843%) 0.000000 (0.000767) 1.0000 +2.3% -0.7% PASS
Q4_K_M same 100.000% (97.843%) 0.000000 (0.000806) 1.0000 +1.1% -1.8% PASS
Q4_1 same 100.000% (97.696%) -0.000000 (0.000817) 1.0000 +0.9% -1.9% PASS
Q4_0 same 100.000% (97.304%) -0.000000 (0.000803) 1.0000 +1.6% -1.4% PASS
Q3_K_M same 100.000% (97.598%) -0.000000 (0.000855) 1.0000 +0.4% -1.9% PASS
Q2_K same 100.000% (97.402%) -0.000000 (0.000701) 1.0000 -2.8% -2.9% PASS
Q2_0 same 99.167% (55.049%) 0.000037 (0.167949) 1.1300 -8.1% -1.7% FAIL
Q1_0 same 99.412% (98.480%) 0.000044 (0.000425) 10.6644 -6.3% -2.0% FAIL
IQ4_NL same 100.000% (97.745%) 0.000001 (0.000795) 1.0000 +1.9% -2.6% PASS
IQ4_XS same 100.000% (98.088%) -0.000000 (0.000771) 1.0000 +0.9% -1.5% PASS
IQ3_S same 98.873% (98.039%) 0.000135 (0.000560) 1.0007 +0.6% -0.6% PASS
IQ3_XXS same 99.559% (98.088%) 0.000135 (0.000549) 1.0000 -0.1% -1.4% PASS
IQ2_M same 99.412% (98.529%) 0.000144 (0.000639) 1.0000 +1.1% -3.2% FAIL
IQ2_XS same 99.020% (98.039%) 0.000161 (0.000668) 1.0000 +1.7% -2.1% PASS
IQ2_XXS same 99.118% (97.794%) 0.000175 (0.000745) 0.9999 +2.5% -1.0% PASS
IQ1_M same 98.431% (96.863%) 0.000164 (0.001064) 0.9994 +1.6% -1.7% PASS
IQ1_S same 99.363% (97.402%) 0.000155 (0.001418) 1.0003 +0.8% -1.6% PASS

Granite 3.1 1B-A400M (MoE: MXFP4_MOE changes only expert tensors) (KLD on all.txt at -c 512, llama-bench pp512 / tg128):

Quant Quantizer output Same top p (ref) Mean KLD (ref) PPL ratio pp512 tg128 Status
MXFP4_MOE same 100.000% (100.000%) -0.000000 (-0.000000) 1.0000 +3.1% +0.8% PASS

Qwen3.5-0.8B-Base with a quantized KV cache, K and V of the same type (KLD on all.txt at -c 2048, llama-bench pp512 / tg128 at depth 8192):

KV cache Same top p (ref) Mean KLD (ref) PPL ratio pp512 @ d8192 tg128 @ d8192 Status
q8_0 100.000% (100.000%) 0.000000 (0.000000) 1.0000 -7.9% -0.9% FAIL
q4_0 100.000% (100.000%) 0.000000 (0.000000) 1.0000 +1.2% -0.5% PASS

Real models, logits against the base build:

Model Text Same top p (ref) Mean KLD (ref) PPL ratio Status
qwen4exp prose 100.000% (95.161%) 0.000000 (0.010783) 1.0000 PASS
qwen4exp code 100.000% (98.495%) 0.000000 (0.005769) 1.0000 PASS
qwen4exp structured 100.000% (98.998%) -0.000000 (0.005614) 1.0000 PASS
qwen4exp numeric 100.000% (98.990%) -0.000000 (0.005665) 1.0000 PASS
qwen4exp long (-c 16384) 100.000% (98.138%) 0.000000 (0.005285) 1.0000 PASS

qwen4exp, llama-bench t/s (mean of two alternating rounds per side):

Test Base Candidate Change Status
tg128 28.26 28.17 -0.3% PASS
tg128 @ d32000 26.40 26.53 +0.5% PASS
tg128 @ d128000 25.45 25.68 +0.9% PASS
pp512 791.99 786.60 -0.7% PASS
pp512 @ d32000 664.52 660.89 -0.5% PASS
pp512 @ d128000 548.99 540.83 -1.5% PASS
pp2048 975.65 995.30 +2.0% PASS
pp2048 @ d32000 821.74 837.44 +1.9% PASS
pp2048 @ d128000 648.83 657.94 +1.4% PASS
  • qwen4exp server: PASS. tool call: ok (base ok); decode: identical; MTP: not applicable, no drafts on either side (no MTP head)

Memory, base -> candidate: peak RSS of the process, peak GPU memory (amdgpu VRAM + GTT above the level before the run), and the llama.cpp breakdown of the candidate per device in MiB (self = model + context + compute) when the tool printed it. A rise above 5% and 256 MiB is INCOMPLETE until the PR explains it.

Run Peak RSS Peak GPU Breakdown Status
test-backend-ops 15726 -> 15171 MiB 10959 -> 10959 MiB - -
test-llama-archs 250 -> 246 MiB 198 -> 112 MiB - PASS
test-save-load-state 222 -> 0 MiB 857 -> 0 MiB - PASS
perplexity kv-q4_0 1207 -> 721 MiB 5059 -> 5083 MiB - PASS
perplexity kv-q8_0 1206 -> 721 MiB 5065 -> 5089 MiB - PASS
perplexity moe-MXFP4_MOE 158 -> 133 MiB 1002 -> 1011 MiB - PASS
perplexity qwen4exp-code 1449 -> 965 MiB 82583 -> 82683 MiB - PASS
perplexity qwen4exp-long 15970 -> 12094 MiB 80386 -> 80389 MiB - PASS
perplexity qwen4exp-numeric 1448 -> 964 MiB 82583 -> 82683 MiB - PASS
perplexity qwen4exp-prose 1451 -> 966 MiB 82589 -> 82686 MiB - PASS
perplexity qwen4exp-structured 1448 -> 963 MiB 82581 -> 82683 MiB - PASS
perplexity small-BF16 425 -> 303 MiB 2722 -> 2730 MiB - PASS
perplexity small-F16 425 -> 303 MiB 2718 -> 2726 MiB - PASS
perplexity small-IQ1_M 426 -> 305 MiB 1281 -> 1289 MiB - PASS
perplexity small-IQ1_S 426 -> 304 MiB 1273 -> 1281 MiB - PASS
perplexity small-IQ2_M 426 -> 303 MiB 1318 -> 1326 MiB - PASS
perplexity small-IQ2_XS 425 -> 304 MiB 1305 -> 1313 MiB - PASS
perplexity small-IQ2_XXS 426 -> 304 MiB 1294 -> 1302 MiB - PASS
perplexity small-IQ3_S 426 -> 303 MiB 1419 -> 1428 MiB - PASS
perplexity small-IQ3_XXS 426 -> 304 MiB 1337 -> 1345 MiB - PASS
perplexity small-IQ4_NL 426 -> 303 MiB 1481 -> 1489 MiB - PASS
perplexity small-IQ4_XS 425 -> 304 MiB 1469 -> 1477 MiB - PASS
perplexity small-Q1_0 426 -> 303 MiB 1267 -> 1275 MiB - PASS
perplexity small-Q2_0 425 -> 303 MiB 1334 -> 1342 MiB - PASS
perplexity small-Q2_K 426 -> 304 MiB 1391 -> 1399 MiB - PASS
perplexity small-Q3_K_M 426 -> 304 MiB 1433 -> 1441 MiB - PASS
perplexity small-Q4_0 425 -> 303 MiB 1467 -> 1475 MiB - PASS
perplexity small-Q4_1 425 -> 303 MiB 1496 -> 1504 MiB - PASS
perplexity small-Q4_K_M 425 -> 303 MiB 1493 -> 1501 MiB - PASS
perplexity small-Q5_0 426 -> 304 MiB 1526 -> 1535 MiB - PASS
perplexity small-Q5_1 425 -> 304 MiB 1555 -> 1563 MiB - PASS
perplexity small-Q5_K_M 425 -> 303 MiB 1539 -> 1547 MiB - PASS
perplexity small-Q6_K 425 -> 303 MiB 1589 -> 1597 MiB - PASS
perplexity small-Q8_0 425 -> 303 MiB 1821 -> 1829 MiB - PASS
llama-bench kv-q4_0 232 -> 233 MiB 2523 -> 2533 MiB - PASS
llama-bench kv-q8_0 256 -> 257 MiB 2550 -> 2561 MiB - PASS
llama-bench moe-MXFP4_MOE 101 -> 102 MiB 928 -> 936 MiB - PASS
llama-bench qwen4exp-bench 32128 -> 32188 MiB 88158 -> 88333 MiB - PASS
llama-bench small-BF16 178 -> 178 MiB 2455 -> 2463 MiB - PASS
llama-bench small-F16 178 -> 178 MiB 2451 -> 2459 MiB - PASS
llama-bench small-IQ1_M 178 -> 178 MiB 1013 -> 1022 MiB - PASS
llama-bench small-IQ1_S 177 -> 178 MiB 1005 -> 1022 MiB - PASS
llama-bench small-IQ2_M 178 -> 178 MiB 1051 -> 1059 MiB - PASS
llama-bench small-IQ2_XS 177 -> 178 MiB 1041 -> 1045 MiB - PASS
llama-bench small-IQ2_XXS 177 -> 178 MiB 1026 -> 1034 MiB - PASS
llama-bench small-IQ3_S 177 -> 178 MiB 1161 -> 1161 MiB - PASS
llama-bench small-IQ3_XXS 177 -> 178 MiB 1070 -> 1078 MiB - PASS
llama-bench small-IQ4_NL 178 -> 178 MiB 1213 -> 1221 MiB - PASS
llama-bench small-IQ4_XS 177 -> 178 MiB 1202 -> 1210 MiB - PASS
llama-bench small-Q1_0 177 -> 178 MiB 1000 -> 1009 MiB - PASS
llama-bench small-Q2_0 178 -> 178 MiB 1071 -> 1075 MiB - PASS
llama-bench small-Q2_K 178 -> 178 MiB 1124 -> 1132 MiB - PASS
llama-bench small-Q3_K_M 177 -> 178 MiB 1166 -> 1174 MiB - PASS
llama-bench small-Q4_0 177 -> 178 MiB 1200 -> 1208 MiB - PASS
llama-bench small-Q4_1 178 -> 178 MiB 1229 -> 1237 MiB - PASS
llama-bench small-Q4_K_M 177 -> 178 MiB 1226 -> 1234 MiB - PASS
llama-bench small-Q5_0 178 -> 178 MiB 1259 -> 1267 MiB - PASS
llama-bench small-Q5_1 177 -> 178 MiB 1288 -> 1296 MiB - PASS
llama-bench small-Q5_K_M 177 -> 178 MiB 1272 -> 1280 MiB - PASS
llama-bench small-Q6_K 177 -> 178 MiB 1322 -> 1330 MiB - PASS
llama-bench small-Q8_0 178 -> 178 MiB 1554 -> 1562 MiB - PASS
llama-server qwen4exp 1041 -> 1042 MiB 87805 -> 87803 MiB - PASS

This was referenced Oct 20, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pwilkin pwilkin Awaiting requested review from pwilkin

Assignees

No one assigned

Labels

Apple Metal conversion CUDA devops documentation Improvements or additions to documentation ggml Hexagon IBM zDNN jinja parser model mtmd OpenCL server/ui server SYCL testing vendor Vulkan WebGPU

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gaetan-puleo