Repository navigation
Conversation
vulkan : only chunk batched mat-vec for the quant types that were measured
ROCmFPx quant types: CPU codecs + Vulkan kernels (loads the Strix Halo ROCmFP4 GGUFs)
Every check in test_rocmfpx.c is a plain assert(), including the rocmfpx_validate_row_data_*() calls and the mse ordering comparisons. The Release configuration the PR builds defines NDEBUG, so all of them compiled away and the test still exited 0 - it validated nothing. Undefine NDEBUG before including <assert.h>, the way test-quantize-fns.cpp and test-quantize-perf.cpp already do. test_rocmfp2_reference.c does not need this: it uses its own always-active check machinery rather than assert(). Assisted-by: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VcXid9TGNfxqxxNWFQGES8
rocmfpx.c is already a source of ggml-base, and everything test_rocmfpx.c calls is declared GGML_API. Compiling it into the test as well breaks MSVC shared-library builds: with BUILD_SHARED_LIBS=ON and GGML_BACKEND_DL=OFF the test inherits GGML_SHARED through llama -> ggml -> ggml-base but not GGML_BUILD, so the codec declarations become __declspec(dllimport) in the same translation unit that defines them, which MSVC rejects. Drop the extra source and let the test link the library implementation. test-rocmfp2-reference keeps compiling rocmfp2_reference.c directly: it is a standalone reference that is not part of ggml-base and carries no GGML_API. Assisted-by: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VcXid9TGNfxqxxNWFQGES8
rocmfpx_choose_scale_fp2_mse() only searched scale bytes 1..126, returning 0
solely when the block was all zeros. ROCmFP2's S40 codebook is
{-4, -1, +1, +4}, which has no zero code, so at any nonzero scale the smallest
magnitude it can emit is one scale step. A block that sits below half of the
smallest UE4M3 scale is then amplified rather than approximated.
A block filled with 2^-14 reconstructed as 2^-10, 16x too large, for a squared
error 225x greater than encoding it as zero. rocmfp2_p1_quantize_block_ref(),
which searches every legal byte 0x00..0x7e, picked scale byte 0 for the same
input - so the two encoders in this tree disagreed.
Seed the search with the zero-scale candidate so bytes 1..126 have to beat it.
Ties keep byte 0, matching both the lower-byte rule the rest of the search uses
and the standalone reference.
before: e=(1,1) sse=2.68e-05
after: e=(0,0) sse=1.19e-07 (reference: 5.96e-08 per 16-weight group)
FP3, FP6 and FP8 all have a zero code and so can already match the zero scale
by rounding; only FP2 needs this.
Assisted-by: Claude Opus 5 (1M context)
Claude-Session: https://claude.ai/code/session_01VcXid9TGNfxqxxNWFQGES8
The FP6 scale search stops descending once the unavoidable clipping error at a scale already exceeds the best error found so far. That bound was computed as max_abs - 31*scale, but rocmfpx_decode_fp6_code() maps code 32 to -32: the representable range is the asymmetric [-32, 31]. For a block whose largest magnitude is negative the bound therefore overestimates the error that cannot be avoided, and the search gives up before reaching the scale that wins. Sweeping 20000 half-blocks with a single dominant outlier: before: 324 suboptimal with a negative outlier, 0 with a positive one after: 0 suboptimal in either case worst observed case: byte 12 at sse 1.783e-04 where byte 11 gives 1.367e-04. The asymmetry between the two columns is exactly what the wrong bound predicts. The Vulkan encoder carried the same bound, so fix both together - the two encoders have to agree on the scale they choose. Assisted-by: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VcXid9TGNfxqxxNWFQGES8
rocmfpx_quantize_fp6_code(), rocmfpx_fp6_decoded_value() and rocmfpx_quantize_fp8_code() rounded to int first and clamped afterwards. The isfinite() guard keeps NaN and inf out, but a large finite weight still overflows the conversion, so the value that gets clamped is already wrong: 1e12 -> fp6 -0.03125 sign flipped 1e30 -> fp6 0, fp8 0 magnitude lost entirely Clamp the float to the representable range first, then convert. For in-range values this is identical to the old order - rounding cannot push a value past a bound it was not already past - so ordinary weights encode bit for bit as before, and the extremes now saturate: 1e12 -> fp6 31*scale, fp8 127*scale 1e30 -> fp6 31*scale, fp8 127*scale The Vulkan encoder had the same shape, and int() of an out-of-range float is undefined in GLSL, so fix it in the same commit to keep the two encoders in agreement. This is extreme-input robustness, not a defect seen on ordinary weights. Assisted-by: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VcXid9TGNfxqxxNWFQGES8
…-ring Merge: from halobox 2026 09 06 uma ring
…/q5_0/q5_1/iq4_nl) Evidence branch only - NOT for upstream. Extends the q8_0 dequant-once FA path to every KV-eligible quant type via per-type fused dequant+transpose shaders, plus a prefill-only fa_kv_ok gate for iq4_nl (no native coopmat1 path) and a GGML_VK_FA_DEQUANT env toggle. Correctness: dequant-once == native FA bit-exact for q4_0/q4_1/q5_0/q5_1; iq4_nl matches CPU. Finding: prefill is quant-type independent (all dequant to identical f16 scratch); iq4_nl is a poor KV type (ppl ~2x q4_0 at equal bits). Retained as gating evidence. Assisted-by: Claude Opus 4.8
The dequant-once path materializes a per-layer f16 K/V scratch (~2 KiB/token). On discrete devices that resident footprint can push the working set past free VRAM, at which point the driver silently pages device-local memory: measured ~15x prefill regression on an 8 GB card (RTX 3070, driver 591.86) at long context, with no error reported. Integrated/UMA devices have no separate device pool to overflow and are unaffected. Gate the path on this process's device-local usage against the physical heap size, keeping a conservative reserve for memory not observable in-process. heapBudget is deliberately not used as the signal: ggml_backend_vk_get_device_memory computes heapBudget - heapUsage in unsigned arithmetic, which wraps to a huge value exactly when the device is oversubscribed. The allocation cannot gate itself either - on WDDM vkAllocateMemory only fails at roughly physical heap size, which is above the free-VRAM level where paging begins, so a successful allocation is not evidence of a resident fit. Also fix the scratch size check: K and V are bound as one storage buffer, so their sum must fit maxStorageBufferRange, not each half independently. GGML_VK_FA_DEQUANT=0 forces the path off and =1 skips the capacity check; GGML_VK_FA_DEQUANT_RESERVE_MB overrides the reserve. Assisted-by: Claude (Opus 5)
…G, env-gated) The KV-cache view reaching FA has head-interleaved rows ([HS, NH, KV] physically), and the cm1 shader's direct-from-global coopMatLoads run ~2x slower on that stride than on per-head-contiguous K/V: a 16x16 tile touches 16 distant cache lines instead of 4. Measured on Strix Halo (RADV gfx1151), hd128/GQA8/kv10240/nb2048 f16: 29.8ms contiguous vs 63.1ms dense-permuted (the model layout; matches the in-model 59.9ms from the perf logger, where FLASH_ATTN_EXT was 72.6% of the graph at pp2048@d8192). GGML_VK_FA_KV_CONTIG=1 extends the dequant-once FA scratch to f16 K/V: dequant_f16_transpose.comp is a pure strided copy ([HS,NH,KV] -> [HS,KV,NH], same push-constant ABI and dispatch as the quant transpose shaders), engaged only when the rows are actually strided, prefill only (neq1 >= 64). FA op 63.1 -> 30.5ms (2.07x) incl. copy cost. Model-level (Qwen3-Coder-30B Q6_K_XL, ub2048, f16 KV, r=3, vs ROCm 571d0d5 nowmma): pp8192 877 -> 1199 t/s (ROCm 1216, was -28% now parity); pp2048@d4096/8192/16384: 776/513/300 -> 1120/850/580. Shallow prefill unchanged-to-better (pp2048 1542 -> 1633). Not the fix: shmem staging on AMD (loses on occupancy, 29.8 -> 54.4ms contiguous), bigger-tile/GQA-packed streaming (1-KV-head L2-resident probe runs identical -> kernel is issue-bound, not bandwidth-bound). test-backend-ops -o FLASH_ATTN_EXT green with the flag off and on (pre-existing iq4_nl+sinks failures unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
test_flash_attn_ext always built K/V as sparse views (physical dim1 doubled, half viewed), which can never satisfy the contiguize path's ggml_is_contiguously_allocated gate - so no permuted correctness case exercised it. Add a kv_view parameter (default true = unchanged) and dense-permuted eval cases matching the real KV-cache layout, including ALiBi and logit-softcap variants; all pass vs CPU with GGML_VK_FA_KV_CONTIG=1. Perf additions: Qwen3-Coder-30B prefill-at-depth shapes (hd128, 4 KV heads, GQA 8, kv up to 10240, nb 512/2048), the dense-permuted variant (model layout), and the probe set used to establish that the contiguous cm1 kernel is issue-bound: 32-distinct-KV-head MALL-spill (flat), 1-KV-head L2-resident (flat), mask=0 (-5.5%), f16 acc (-3%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ONTIG=0 opts out) Flip 442d7df from opt-in to default-on, matching the quant dequant-once path (GGML_VK_FA_DEQUANT) convention. The pass still self-gates: f16 K/V only, prefill only (neq1 >= 64), only when rows are actually strided, dense allocation, and the shared scratch-capacity check. Validated on Strix Halo (RADV gfx1151): FLASH_ATTN_EXT suite green with default env (dense-permuted cases exercise the pass) and with the opt-out; model-level pp2048@d8192 with no FA env matches the explicit GGML_VK_FA_KV_CONTIG=1 validation run (846.6 vs 847.9 t/s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ard-gate Rebased adaptation of the iq4_nl routing fix: upstream 8161641 made iq4_nl a native FA type, so the original motivation (iq4_nl had no native shader and silently read garbage outside the dequant-once path) no longer applies to any currently-admitted type. Keep the machinery as hardening: ggml_vk_fa_kv_native() is the one list, supports_op mirrors every hard condition of the dispatch-time dequant gate for any future non-native type, and dispatch asserts the invariant instead of falling back to a garbage-reading shader. Native list synced with upstream (iq4_nl in, q1_0 out to match current admission). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
iq4_nl has no native Vulkan FA shader; the dequant-once scratch path is its only route (1abdd92). The sweep's iq4_nl cases are all sparse-view, which that path correctly rejects, so iq4_nl had zero passing FA coverage. Add model-layout (dense [0,2,1,3]-permuted) cases at prefill batch size: iq4_nl/iq4_nl with sinks off and on, hd72+GQA with sinks, and mixed K=iq4_nl/V=f16. Validated on RADV gfx1151 at 146fb73: FLASH_ATTN_EXT 4765/4765 with default env and with GGML_VK_FA_KV_CONTIG=0/1; 4761/4761 with GGML_VK_FA_DEQUANT=0 (new cases correctly report unsupported); full test-backend-ops suite 15538/15538. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
The contiguize/dequant pass is dispatched inside the FLASH_ATTN_EXT node handler, and the perf logger writes one timestamp per graph node, so its cost was charged to FLASH_ATTN_EXT with no way to separate the two. That made the copy invisible and the kernel look correspondingly slower. Adds a sub-node timestamp: a handler can close an interval mid-node and have it logged under its own name. Measured on Coder-30B UD-Q6_K_XL, pp2048/ub2048 at d32768, f16 KV: graph total 4612.3 ms FLASH_ATTN_EXT 3490.6 ms 75.68% of graph FA_KV_CONTIGUIZE 33.1 ms 0.72% of graph (0.95% of FA) so the copy is under 1% of the graph and FA is 75.7% of it at that depth. Bench throughput is unchanged with the instrumentation compiled in (441.09 vs 442.14 t/s), since the marks are only emitted when the logger is on in per-op mode. Two latent bugs in the query-pool handling fall out of this and are fixed here: - The pool is created with n_nodes+100 slots but only the first n_nodes+1 were reset each graph, so anything using the headroom would read stale results. - The results buffer was sized n_nodes+1 while getQueryPoolResults was asked for query_idx entries. Equal today, but it is an overflow waiting for the first caller that writes an extra timestamp. Sub-op intervals log no flops, since they move bytes rather than doing math; attributing the node's flop count to them would corrupt the GFLOPS column for both halves. Assisted-by: Claude (Opus 5)
acc is an int32 sum of dotPacked4x8EXT results. With q8_0 both operands are full int8, so its bound is d_per_step*4*127*127, which overflows f16 when ACC_TYPE is f16 (GGML_PREC_DEFAULT): the score goes +inf, softmax is destroyed and the output comes back -FLT_MAX. Nibble types bound at ~30480 and stay in range, which is why only q8_0 tripped it. Apply the scales in fp32, then narrow. Identical arithmetic when ACC_TYPE is float, so the f32acc path is untouched. This is an upstream bug, not a fork regression, and it was fixed here once before - 61e77f4 carried it and the rebase onto b10133 dropped it. Only the scalar shader has MMQ, so the failing shape is hsk=128 + q8_0 K + prec=def + nb=1 + nr23=[1,1]; GQA>1 and nb>1 route to coopmat1 and escape. test-backend-ops -o FLASH_ATTN_EXT on gfx1151, quiet box: 13295/13295 twice, up from 13257/13295. All 38 failures were type_K=q8_0 prec=def. Both ggml_flash_attn_ext call sites in the tree force GGML_PREC_F32, so no model path reaches this - it is a landmine for the next person touching precision, not a live bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Assisted-by: GitHub Copilot Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A quantized type size describes a block rather than one tensor element. Include the block size when converting saved bytes into view dimensions in the device writer, reader and fragmented-copy fallback. Extend the existing save/load regression with fragmented-to-compact restoration. Assisted-by: GPT 6 Astra
…store With full-checkpoint rollback, a partial draft acceptance restores the pre-round state and re-decodes the accepted tokens to rebuild it. The replay went through the same verification as a fresh draft. On backends where logits change with batch shape or memory layout (Vulkan), that re-verification can reject a token the original verification accepted; the rejection restores the same checkpoint and replays again, and the slot loops on one position without emitting anything. qwen35moe with --spec-type draft-mtp stalled this way a few hundred tokens into long generations (the v0.6.8 MTP hang): the loop repeated "accepted 2/3, restore at pos 995" every 27 ms with the GPU at 90 percent. Accept the replayed tokens without re-verifying and sample only the continuation from the final position. The replayed prefix was accepted by the verification that triggered the restore; the replay exists to rebuild state. On backends with batch-shape invariant logits the re-verification always agreed, so behavior there is unchanged (verified bit-identical on CPU with and without this change, 800-token greedy pair, 118 restore rounds). Assisted-by: Claude Fable 5 (cherry picked from commit 9c5d899)
server : do not re-verify replayed draft tokens after a checkpoint restore
Implements GGML_OP_LIGHTNING_INDEXER on Vulkan (scalar subgroup shader for small batches, coopmat 16x16 tiles for prefill, dedicated decode variant) and an indexed sparse flash-attention path that consumes the indexer's top-k selection directly via a new ggml_flash_attn_ext_add_top_k() API (FA src[5] + op_param[4] = n_kv_raw dense prefix), instead of attending densely over the full compressed KV. The sparse path engages only for V4's CSA shape (hd 512, 64 heads, MQA, f16 K==V latent) when dense_kv >= 3x active_kv; everything else falls through to the dense path, which stays correct because the kq_mask still carries the top-k selection. Dropped from the original: the mul_mat_id tokens-per-expert pipeline selection, which duplicates GGML_VK_MMID_SMALLN already on this branch. Originally by Gaetan Puleo (llama-cpp-nathan-toolbox-deepseek-v4-poc, branch deepseek-v4-flash-strix-halo); cherry-picked with the mmid hunk dropped.
- flash_attn_top_k.comp: remove the dead bounds check that would skip barriers for part of the workgroup if it ever fired (barrier divergence is UB; the dispatch gate sizes the grid exactly), pin per-lane sizing to LANES=64 instead of the SUBGROUP_SIZE spec constant (array bounds fold from the spec default at compile time), name the f16 mask threshold constant - ggml.h: document ggml_flash_attn_ext_add_top_k semantics (index base, dense prefix, backends-may-ignore contract) - tests: cover the scalar indexer variant (batch 4 and boundary 15), previously only the cm and decode-cm variants had eval parity cases Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Adds test_flash_attn_ext_top_k: builds the DeepSeek V4 CSA attention shape (hd 512, 64-head MQA, V as a view of K) with a consistent per-token top-k/mask pair, one deliberately invalid index, and cases on both sides of the Vulkan engagement gates. nb >= 64 cases are the first numerical parity coverage the sparse prefill shader has had; nb < 64 and sub-3x-kv cases pin the dense-fallback contract. Perf cases sweep kv 8k/32k/64k at nb 1/8/64/512 with a fixed active set. Measured on gfx1151: the sparse shader is flat vs kv at prefill (~2.2 TFLOPS active-only) while nb < 64 falls back to dense and scales with kv (1326 us at 64k, nb=1) - the gap a sparse decode path needs to close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ction The sparse prefill shader gates on q->ne[1] >= 64, so single-token decode attends densely over the whole compressed KV and its cost grows with context. This adds a gather pass (flash_attn_gather.comp): copy the active rows (dense prefix + top-k selection; MQA, so all 64 query heads share one set) plus their mask values into a compact contiguous scratch in prealloc_y, then run the ordinary dense FA over the compacted K/V/mask. V is the K latent, so one gather serves both. Invalid indices and padding get zeroed K and -inf mask. The FA function itself only has its inputs swapped: KV, mask geometry, strides and the K/V/mask bindings are overridden up front and every downstream decision (pipeline choice, split-k, workgroup sizing, use_mask_opt) sizes itself to the compact KV unchanged. Engages for the V4 CSA decode shape when kv >= 2x the padded active set; GGML_VK_FA_TOPK_GATHER=0 disables. Measured (gfx1151, test-backend-ops perf, active set 1536): kv=8192 nb=1: 255.6 us -> 55.8 us (4.6x) kv=32768 nb=1: 986.4 us -> 56.1 us (17.6x) kv=65536 nb=1: 1333.8 us -> 58.7 us (22.7x) Time is flat vs context. nb>1 still falls back to dense pending a union gather. FLASH_ATTN_EXT eval suite green incl. the top-k parity cases (the kv=4096 nb=1 case exercises this path end-to-end vs CPU). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Fold-in note for the toolbox branch: use_dequant_kv is additionally gated on !fa_compact.active (the compact scratch is already contiguous f16, and the two scratch layers must not stack), and the compact stride overrides chain through the toolbox's nb*_eff values so the contiguize/dequant path keeps its strides when the gather is inactive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Ports ggml-cuda/dsv4-hc.cu to Vulkan: hc_pre (mix the HC input streams down to one embedding), hc_comb (per-token 4x4 stream-mixing matrix - per-source softmax then eps-stabilized alternating column/row sinkhorn normalization, whole matrix in registers, one thread per token), and hc_post (redistribute the layer output back into the streams with the mixed residual). Same launch geometry as the CUDA kernels; plain f32 compute, no subgroup or coopmat requirements, so the pipelines are created unconditionally. The value at decode is dispatch-count collapse: the unfused fallback runs the decomposed graph (measured on gfx1151 config-a partial offload: SUM_ROWS alone 79 dispatches x 39.7us = 3.1ms per graph, plus DIV/MUL/ ADD shares at 4x4 shapes) where the fused form is 3 dispatches per layer. resolve_fused_ops now keeps all three fusions enabled on Vulkan instead of printing 'not supported, set to disabled'. Parity: upstream test-backend-ops DSV4_HC_PRE/COMB/POST cases green vs CPU on first build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…-ctk The fused indexer kernels read f16 keys only, so quantizing the small (128-dim) indexer cache silently disables them and falls back to the decomposed full-KV indexer path, whose contiguize cost grows superlinearly with depth (measured 0.8ms -> 97ms per dispatch by 12k context on Vulkan). Pin the indexer key cache to f16 in both DSA cache variants; the memory cost vs q8_0 is ~120 bytes per token per layer. Assisted-by: Claude Fable 5
…(DSv4) The permuted view feeding the grouped wo_a matmul gives B a 128 KiB power-of-2 token stride. Single-token decode never touches it, but small multi-token batches (speculative verify, n=2-4) hit a strided-B matmul path that runs ~11x slower than contiguous (40 vs 460 GFLOPS measured on gfx1151, ~54% of GPU time in a draft-verify window). A contiguous copy for 2..8 tokens is far cheaper than the stride tax; n=1 and large prefill batches are unaffected. Assisted-by: Claude Fable 5
…nt types The routed MMB path decoded a whole 256-value super-block with dequantize_* and then threw away 192 of 256 values per 64-wide K slice. Replace that with true per-slice decoders for Q2_0, Q5_K, Q6_K, IQ2_XXS, IQ2_XS, IQ2_S and IQ3_XXS, plus the mmb_lb grid-decoder family generalised from the existing IQ3_S path (IQ3_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS), the Q2_0 register prefetch, and the mmb_store8 / mmb_store_grid8 / mmb_grid8_bf16 / mmb_ksigns helpers. Each branch reproduces the reference dequantize_* fp32 arithmetic, so the BF16 WMMA weights are bit-identical to the whole-super-block path. This is decode arithmetic only; the routed type gate is unchanged, so the Q2_0 / IQ2_* / IQ3_XXS branches stay unreachable until commit 4 widens the gate, and Q5_K / Q6_K keep their existing routed dispatch. Verified on gfx1151 / ROCm 10.0: qwen4exp_gateup kbench 15/15 hashes identical to the pre-change path, and test-backend-ops MUL_MAT_ID,MMB_QUANT,MMB_QUANT_HC,MOE_PREFILL 7709/7709 with the full series applied. Not measured: non-gfx1151 arches. Assisted-by: Claude Opus 4.8 (1M context) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Map the routed down small tile as <128,32,16,32> (WAVES_M=8 x WAVES_N=1) instead of <128,32,32,16> (WAVES_M=4 x WAVES_N=2). The old shape split the 8 waves into two token-column groups; a tile with <=16 real tokens (the common small-expert case at typical top-k/E ratios) left the wn=1 group's 4 waves WMMA-idle. WAVES_N=1 keeps every wave issuing WMMA. The mapping is bit-identical: same output (m,n) set, same per-element K-reduction order. gfx1151 / ROCm 10.0: qwen4exp_gateup kbench 15/15 hashes unchanged, test-backend-ops 7709/7709. Measured +6.8% on the fn_iq3xxs PP512 routed+glu kernel time at the dispatch level; end-to-end pp2048 / pp4096 / tg128 stay within run-to-run noise on this shared-memory APU (routed MMB is about 37% of GPU time), so this is a kernel-level win, not a claimed end-to-end speedup. Assisted-by: Claude Opus 4.8 (1M context) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A model that stores ffn_gate_up_exps as one fused tensor has it split into gate and up by two views, so the MMB SwiGLU fusion (ggml_cuda_mmb_supported_glu) can never fire for it. MMB would then run the gate and up (and the down, whose activation is the SwiGLU of those views) as plain BF16 WMMA mul_mat_id, paying MMB's precision cost with none of the fused-kernel payoff that justifies it. Guard ggml_cuda_mmb_supported_mmid against both, exempting Q8_0 / IQ4_NL which master already routes to MMB (rejecting them would move those ops off their master dispatch and create a fresh difference). The guard matches by tensor layout, not type, so it is inert for the split-tensor low-bit models this series targets (fn_iq3xxs, Ornith: gate and up are their own mul_mat_id) until commit 4 admits their types. For a fused gate_up model it fires immediately: gemma-4-26B-A4B / gfx1151 / ROCm 10.0, four repo corpora, mean KL vs master drops from 0.19-0.31 (same-top-1 90.3-93.5%) with those experts on MMB to -0.000000..0.000000 (max 0.00005, same-top-1 100.000%) with them on MMQ, i.e. exact master parity. pp4096 on UD-Q4_K_XL gained only +0.9% from routing them to MMB, so nothing of value is lost. Hardcoded on. Assisted-by: Claude Opus 4.8 (1M context) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Widen mmb_quant_type_routed from 9 to 14 types: add Q2_0, IQ2_XXS, IQ2_XS, IQ2_S and IQ3_XXS, which commit 1 gave per-slice decoders and which measure faster than MMQ at 512-4096 routed rows on gfx1151. The commit-3 fused gate_up guard now has effect for these types: a fused gate_up model in one of them stays on MMQ, a split-tensor model (the typical low-bit MoE) takes the BF16 path. Add the production-shape test_mul_mat_id cases (the shapes and quant types the local MoE files use) so test-backend-ops covers the newly admitted types. gfx1151 / ROCm 10.0, fn_iq3xxs (exercises all five) llama-perplexity c1024, per-type attribution via a temporary allowlist: none 4.4021, +Q2_0 4.4027, +IQ2_XXS 4.4038, +IQ2_XS 4.4029, +IQ2_S 4.4055, +IQ3_XXS 4.4023, all five 4.4036 - every delta is an order of magnitude under the +/-0.0256 corpus sampling uncertainty, so no type is a quality outlier. Ornith-1.5 IQ2_XXS four-corpus KL vs master: mean 0.0047-0.0647, same-top-1 97.1-99.3% (the small BF16-vs-MMQ margin, unchanged from before this series). test-backend-ops 7709/7709. Not measured: throughput (correctness-first) and non-gfx1151 arches. Assisted-by: Claude Opus 4.8 (1M context) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ypes hip : let routed experts take every mmb weight type (restores Qwen3.8-Flash-Next prefill after #123)
hip : keep the F32 HC gate output when no BF16 xn copy is expected
github-actions
Bot
added
documentation
ggml-cuda : round growing pool requests up to a power of two
scripts : log instead of print in gather_required_libs.py
--ple FILE opens one more GGUF after the model files. Each tensor in it replaces the lookup table of the same name in the model, so a model can use a PLE / n-gram table of another precision (e.g. BF16 or Q8_0 instead of the IQ4_NL table it ships) without a rebuilt model file. The table of the model file is never read. Works for every load and lazy mode, including the qwen4exp on-direct reader, which reads the rows from the new file. - every key of the file must match the model in type and value, except split.* and general.* (general.architecture is compared); a file without such keys loads with a warning - a replaced tensor must keep its shape; only get_rows lookup tables can be replaced; a file without tensors is an error - the replaced byte ranges of the model are kept out of the mmap prefetch - the path is part of the model params in common, so a draft or prefill model made from a copy of the params does not get the file - vocab-only loads do not open the file - path_ple is the last member of llama_model_params Assisted-by: Claude Opus 5.5 Assisted-by: Claude Fable 5.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
--ple <file|none> takes several values like the other params, so tables of different precision can be compared in one run. Adds a ple output field. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copies the PLE table of a model (any shard of a split model) to its own GGUF with the arch's ple.* keys, as a byte copy or converted to f32 / f16 / bf16 / q8_0 / q5_x / q4_x. --metadata-from takes the keys from another file, for a lone shard that has none. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RowStream duck-typed the ndarray that GGUFWriter.write_tensor_data() declares, which ty rejects; the previous commit silenced that with an ignore. Let GGUFWriter write the header, keys and tensor infos only, and append the data section with its alignment from write_rows(). The output is byte-identical to before. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama : load per-layer embedding tables from a separate GGUF (--ple)
This reverts commit 0f36c46, reversing changes made to 307c50d. --cache-dir drives runaway disk writes on ROCm: the slot state is copied straight between the GPU and a MAP_SHARED file, the runtime pins the mapped pages, and writeback keeps rewriting the file until the server exits. Users reported ~1.6 TB written to NVMe in an hour, and decode stalling under 5+ GB/s of continuous writes with a 75k-token context. Reproduced on gfx1151: one 503 MiB entry became 5.7 GB of writes within 60 s of idle. Even with that fixed (#149), an agent-style workload rewrites the whole state on every turn. Remove the feature until it can come back with bounded writes; --cache-ram is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx
Brings back #81 by @storm1er (reverted in #154 after the ROCm write storm reports), so the fixes that follow can land on top of it. This reverts commit a3a9bb3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx
The disk prompt cache metadata holds the serialized prompt tokens, but it was created 0644 and the cache directory 0755, so any local user could read cached prompts (the state file itself was already 0600). - write the metadata 0600, and tighten metadata left by older builds when the cache is restored - create a missing cache directory 0700; warn when an existing one is accessible to other users, without changing it - default --cache-dir-max to 32768 MiB instead of no limit, so remote clients cannot grow the directory without bound (-1 keeps the old behaviour) Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --cache-dir the slot state was copied straight between the backend and the MAP_SHARED cache file. On ROCm the runtime pins the mapped file pages for that copy; writeback then keeps rewriting the file for as long as the server runs. On gfx1151 one 503 MiB entry turned into 5.7 GB of writes within 60 s of idle (ext4 on a RAM-backed loop device), and users saw TBs per hour on NVMe. Copy through an ordinary heap buffer in both directions instead, so the backend never touches the mapping. Diagnosis and the staging approach come from a user report against b1015 (reproduced on a RAM disk, fix staged through RAM). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx
dzannotti
force-pushed
the
fix/prompt-cache-disk-hardening
branch
from
October 8, 2026 00:24
9c8ef65 to
c99aae6
Compare
dzannotti
changed the title
This was referenced Oct 14, 2026
gaetan-puleo
pushed a commit
to rcfa/strix-llama.cpp
that referenced
this pull request
Oct 16, 2026
…t-cache" This reverts commit 0f36c46, reversing changes made to 307c50d. --cache-dir drives runaway disk writes on ROCm: the slot state is copied straight between the GPU and a MAP_SHARED file, the runtime pins the mapped pages, and writeback keeps rewriting the file until the server exits. Users reported ~1.6 TB written to NVMe in an hour, and decode stalling under 5+ GB/s of continuous writes with a 75k-token context. Reproduced on gfx1151: one 503 MiB entry became 5.7 GB of writes within 60 s of idle. Even with that fixed (halo-box#149), an agent-style workload rewrites the whole state on every turn. Remove the feature until it can come back with bounded writes; --cache-ram is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx [rebase: applied as the revert of the rebased commits 73e0238 and 3f91881, the linear-history form of PR halo-box#81] (cherry picked from commit a3a9bb3)
|
Data point from running this PR in production, a suggestion for the bounded-writes question from the #81 revert, and a rebase onto the rebuilt master. This comment was written by an AI agent (Claude Code) working for @rcfa. The numbers are from rcfa's machine and were measured as described below. Setup. Radeon 890M (gfx1150, Strix Point - not Strix Halo), Vulkan, Qwen3.8-Flash-Next IQ3_XXS with MTP, Write volume, 2026-10-15 03:33 to 2026-10-20 11:50 UTC (5.3 days):
So with idle slots kept resident, writes are bounded by evictions, not by turns: a conversation stays in its slot between turns and is written only when its slot goes to another conversation, or at shutdown (patch 1 below). Where the per-turn rewrite comes from. Suggestion. With Two small patches we run on top of this PR. I can open them as separate PRs once this one is in, one concern each:
Rebase. Since the master rebuild (#181) this PR conflicts. For our own deployment I rebased its three commits onto master ab06204, authorship unchanged: commit 3deeb10 (the branch is now https://github.com/rcfa/strix-llama.cpp/tree/pr149-rebased, kept current; see the update below). The conflicts were with master's Verified on the rebase: it builds (Vulkan); a CPU round trip with Qwen2.5-1.5B, |
|
Update: rebased again onto current master 67b998b (after #202 and #203). The three commits applied without conflicts and are patch-identical to the ab06204 version above, authorship unchanged: https://github.com/rcfa/strix-llama.cpp/tree/pr149-rebased-67b998bdd (compare: master...rcfa:strix-llama.cpp:pr149-rebased-67b998bdd, 3 ahead, 0 behind). Verified on this base: it builds (Vulkan), and the same CPU round trip as above passes (Qwen2.5-1.5B, Written by an AI agent (Claude Code) for @rcfa. |
rcfa
added a commit
to rcfa/strix-llama.cpp
that referenced
this pull request
Oct 23, 2026
Joins halo-box#149 and halo-box#180: a disk cache entry is written from the checkpoint files, and the checkpoints restored from an entry move to --ctx-checkpoints-dir again. Assisted-by: Claude Code
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR re-instates #81 back (first commit, a straight reapply of @storm1er's work) with the fixes below on top.
Changes
Owner-only metadata.
*.bin.metaholds the serialized prompt tokens but was created0644, readable by any local user (the.binstate file was already0600). It is now written0600, and metadata left world-readable by older builds is tightened when the cache is restored.Cache directory. A directory the server creates is now
0700. An existing directory is left unchanged, with a warning if other users can access it.Default disk limit.
--cache-dir-maxdefaults to 32768 MiB instead of no limit, so clients can't grow the directory without bound.-1keeps the old behaviour.Runaway disk writes on ROCm. Saves and restores copied the slot state straight between the backend and the
MAP_SHAREDcache file. On ROCm the runtime pins the mapped file pages for that copy, and writeback then keeps rewriting the file until the server exits. The state is now staged through an ordinary heap buffer in both directions, so the backend never touches the mapping. The diagnosis and the staging approach come from a user report against b1015.Repro on gfx1151 (Qwen3-1.7B Q8_0,
-c 32768 -np 1 --cache-diron ext4 over a RAM-backed loop device), one 4.5k-token prompt saved, then the server left idle:cached_tokens4566/4567Checked, no change needed
cache_n1130/1913, 1 token recomputed) and a hybrid one (dummy qwen4exp:cache_n136/140, context checkpoint restored). The miss reported on 2026-09-29 no longer reproduces.Tests
tools/server/tests/unit/test_prompt_cache_disk.py, CPU, gfx1151 box:test_disk_cache_files_are_owner_onlyThe two failures,
test_disk_cache_strict_extension_matches_predicted_reuseandtest_disk_cache_no_usable_checkpoint_skips_candidate, also fail on master in this setup and are not touched here.