Nightly 2026-08-20 (2nd run): the GGUF import stops paying for the payload twice by physics515 · Pull Request #20 · physics515/Mummu

physics515 · 2026-08-21T06:06:07Z

Second run of 2026-08-20 (the first shipped as #19 earlier the same day). Theme: the GGUF import path stops paying for the payload twice, and the one guess that fix left behind gets measured away instead of documented.

Dependency freshness

cargo update moved four transitive crates — either 1.17→1.18, h2 0.4.17→0.4.18, icu_provider 2.3.0→2.3.1, zerovec-derive 0.11.5→0.11.6. cargo upgrade --incompatible offers exactly one major, wgpu 29 → 30, held for the reason already in P0: cargo tree -i wgpu still shows a single wgpu (29.0.4) reached through cubecl-wgpu 0.10, so bumping only our direct handle would put a second, non-Burn wgpu in the graph and the startup adapter probe would stop describing the device Burn actually runs on. burn stays 0.21.0 — crates.io's newest is still 0.22.0-pre.2, so the do-not-adopt-a-pre-release rule holds another run. Cargo.lock committed.

Items shipped

`gguf.rs` has no `.expect()` left on a production path (P3)

safetensors.rs grew a fallible to_usize last run; gguf.rs kept the usize::try_from(..).expect("bounded above") pattern it was modelled on, so the no-panic rule was true in the newest import module only. Eight sites, not the four the roadmap estimated — the four header readers, dequant_to_safetensors' payload capacity, the tensor-name JSON encode (now reports through BadTensor, naming the offending index), and dequantize's two block-geometry casts plus its F32 4-byte block cast. No behaviour change by construction: every replaced site was unreachable on a 64-bit target.

The GGUF dequant streams instead of buffering twice (P3)

dequant_to_safetensors filled a data Vec of the whole f32 payload, then copied it into a second blob Vec — peak 2× the payload, before the model was even built. For OLMoE-1B-7B that is ~28 GB dequantized and ~56 GB charged.

safetensors::fuse_into was the template and this follows it: plan the whole output first, then stream header + payload into an impl Write. dequant_to_safetensors is now a thin wrapper over a shared dequant_into<W>, joined by dequant_to_safetensors_file. The double buffer is gone from both forms — the in-memory one writes header-then-payload into a single Vec (1×, not 2×), and the file one holds nothing but the current tensor's f32 values plus a fixed 1 MiB staging buffer.

Planning first also made the path stricter: plan_dequant decides every name, shape and offset before the first payload byte is read, so an unmapped name, a reshape that changes the element count, or a rename collision fails in milliseconds instead of after N tensors have been dequantized. A unit test proves the ordering by pointing a tensor's payload past the end of the file and checking the map error still surfaces.

One `gguf_store` for all four ports, and the sink choice measured (P3)

The commit above gave OLMoE the streaming sink and left qwen2 / qwen3 / lfm2 on the in-memory one, with a roadmap note admitting the split was a guess. Measured, it was the wrong guess.

A/B on Qwen2.5-1.5B Q4_K_M (~6 GB of f32), the same load test each way:

sink	runs
in-memory	33.90 / 35.93 / 42.15 s
scratch file	36.58 / 37.14 s

The scratch numbers sit inside the in-memory run's own spread — at any payload of real size the disk round-trip is free and the doubled peak was being paid for nothing. It is free because burn-store's SafetensorsStore::from_file mmaps and materializes tensors lazily, so the read-back is page faults during a load that was already reading, not a second pass. (n=2 on the scratch side: a concurrent routine held the shared cargo target's lock and the third run never started.)

So the choice is automatic: import::DequantSink is Memory / Scratch / Auto; Auto keeps a payload in RAM only up to MAX_IN_MEMORY_DEQUANT_BYTES (512 MiB — a ~1 GiB peak, free anywhere) and streams everything else. All four ports pass Auto, so no consumer sees the knob. The RAM-aware variant the roadmap floated (compare against the device inventory's free-RAM figure) is deliberately not built and the roadmap says why: once streaming is free, a size threshold answers the question and a planner dependency would be complexity bought for nothing.

The refactor that made this expressible is worth as much as the policy. import::gguf_store is the one place the four ports shared — the dequant, the adapter chain, and the comment explaining why the target float dtype comes from the TYPE (B::FloatElem) and never from a probe tensor. That reasoning existed in four copies, phrased three different ways. Each port now adds only its own key remaps. FusedTemp moves out of models/olmoe.rs and becomes import::ScratchFile — the second caller the previous commit predicted arrived the same night.

Tokenizer + chat-template import closed on evidence (P4)

The parent item's two remaining pieces have been [x] in split sub-items for a while. Rather than tick it on bookkeeping, the whole surface was re-verified on real files (results below). Also filed: imported_render reported FAILED because MUMMU_LFM2_DIR was unset while both other legs were byte-identical — the same trap already recorded for parity_gguf. The new item states the tension rather than assuming the fix: a gate that silently skips is a gate that does not exist, so the answer is a summarized "N legs skipped for missing fixtures" outcome, not a quiet skip.

Verification

Parity — all three GGUF legs vs llama-server on the identical quantized file, with qwen2 and qwen3 now taking the scratch path they did not take before:

model	max \|Δlogprob\|	greedy (24 tok)	top-5 ids
qwen2 Q4_K_M	2.6614442413586614e-1	byte-identical	exact, in order
qwen3 Q4_K_M	4.015608155114805e-1	byte-identical	exact, in order
OLMoE-1B-7B Q4_K_M	3.687691131310693e-1	byte-identical	exact, in order

qwen2 and qwen3 are bit-identical to the values recorded before this change, which is the point: the sink moved and the numbers did not.

Memory — the number this whole PR is about. Sampled while the OLMoE parity leg ran: 26.5 GB peak private commit (the model alone) against a 51.7 GB working set — i.e. ~25 GB of the load is now file-backed mmap rather than charged to commit, which is the binding memory limit on this box. The old shape was ~28 GB blob + ~28 GB model ≈ 56 GB of commit.

Real inference: four real_qwen2_gguf_* legs (40.1 s — header parse, dequant-vs-true-weights, tokenizer byte identity, GPU load + decode); four real_inference safetensors legs (34.3 s). Model dirs hold no leftover scratch files afterwards.

Tokenizer/template: real_tokenizer_config (qwen3-0.6b ids agree), real_spm both legs (Unigram + BPE-type, byte-matching each checkpoint's own tokenizer.json), the 10-case template BYTE gate (qwen2/qwen3/lfm2, plain + history + tools), and imported_render under --features jinja-template — fallback renderer byte-identical to the family renderer at 142/748/324 B (qwen3), 157/379 B (lfm2), 57 B (no-family fallback).

Budgets — no regression. budget.rs on the reference GPU: TTFT 107.7 / 104.8 ms (budget 150), decode 11.6 / 11.9 tok/s (budget 10), prefill@2048 620 / 621 ms (budget 900). The first run of the evening read 8.5 tok/s and failed; it recovered on re-run, and it is machine state rather than a regression by construction — that gate drives qwen2::load_from_dir (safetensors), a code path this PR does not touch at all.

Static: cargo fmt clean, cargo clippy --all-targets with no new warnings, cargo build clean, 236 lib tests (+4 this run) and 6 integration tests, verified by test name against the shared cargo target.

Research folded in

P0 / burn 0.22 — two of the migration note's claims did not survive reading the published crate. burn-store 0.21 has no FloatCastAdapter (its adapters are PyTorchToBurn / BurnToPyTorch / HalfPrecision / Chain / Identity, and HalfPrecision is f32↔f16 only), so replacing our CastFloatAdapter is a 0.22 task, not a missed opportunity. And wgpu 30's f16 change is wider than recorded: SHADER_F16 works in WGSL and GLSL now, not only SPIR-V passthrough — so today's f16-is-Vulkan-only shape ends at the bump. Cooperative matrix load/store also lands, but at 8×8 f32 only.
P2 / MoE decode — llama.cpp discussion #24528 is the first route that explains our own 2026-08-03 gather regression instead of ignoring it: a VRAM cache of hot experts with hybrid hit/miss execution — cached rows go to the GPU as one batched matvec while miss rows compute on the CPU as before, so a miss costs nothing extra and the path degrades to the dense baseline rather than paying an unconditional gather. That is exactly the difference from what we measured and reverted. It also needs no whole-model-in-VRAM, so it unblocks route (b) without waiting on P9.
P5 / speculative decoding — the MTP route has a merged implementation (llama.cpp #22673) and a target: ~75 % acceptance at k=3 for >2×, with k=2 often beating k=3 as acceptance falls off with the draft window.

What's next

The remaining P3 discovery: the f32 scratch file is a bridge, not the design — both streaming sinks exist because SafetensorsStore is the only checked-load pipeline we have, so every import must first become f32 safetensors on disk. A GGUF keep-quantized load (P9) skips the temp file entirely and stops OLMoE being CPU-only.
The MoE hit/miss cache above, gated on bench/BASELINE.md and the 0.76 s/token warm number.
burn 0.22 remains the highest-leverage pending item (graph capture aims straight at the dispatch-bound decode) and remains gated on a stable release.

`cargo update` moved four transitive crates (either 1.17->1.18, h2 0.4.17->0.4.18, icu_provider 2.3.0->2.3.1, zerovec-derive 0.11.5->0.11.6). `cargo upgrade --incompatible` offers exactly one major: wgpu 29 -> 30, which stays held for the reason already recorded in P0 — `cargo tree -i wgpu` still shows a single wgpu (29.0.4) reached through `cubecl-wgpu 0.10`, so bumping only our direct handle would put a second, non-Burn wgpu in the graph and the startup adapter probe would stop describing the device Burn actually runs on. It unblocks with the burn bump, and burn's newest published version is still 0.22.0-pre.2 — 0.21.0 remains the latest stable, so the do-not-adopt-a-pre-release rule holds for another run. Green on the refreshed lock: fmt clean, `clippy --all-targets` with no new warnings, 232 lib tests + 6 integration tests passing (verified by test name against the shared cargo target), `cargo build` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`safetensors.rs` grew a fallible `to_usize` on 2026-08-20; `gguf.rs` kept the `usize::try_from(..).expect("bounded above")` pattern it was modelled on, so the no-panic-on-production-paths rule was true in the newest import module only. It is now true across the whole import suite. The same helper, same reasoning: every caller has already bounded the value against a `MAX_*` ceiling or a tensor's declared size, so on a 64-bit target the conversion cannot fail — but an import path is exactly where "cannot fail" should be an `Err`, so a 32-bit build degrades to a clean `OverBound` naming the file instead of aborting mid-load. Eight sites, all in production code: - `Reader::{string,value,read_file,tensor_table}` — the four header readers, now via `to_usize`. `tensor_table` converts `count` once instead of twice (it was doing the same fallible cast for the capacity and for the loop bound). - `dequant_to_safetensors` — the payload capacity, plus the tensor-name JSON encode, which now reports through the existing `BadTensor` arm and so names the offending tensor index. - `dequantize` — the two block-geometry casts and the F32 4-byte block cast, all mapped into the `String` error it already returns. No behaviour change by construction: every replaced site was unreachable-on-64-bit. Green — fmt, `clippy --all-targets` no new warnings, 232 lib tests. Real-weights proof that the header reader and the dequant path still agree with the file: the four `real_qwen2_gguf_*` legs pass on the Q4_K_M checkpoint (header parse, dequant-vs-true-weights, tokenizer byte identity, GPU load + decode), 259.3 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three folds, each with a number or a correction attached. P0 (burn 0.22): two of the migration note's claims did not survive reading the published crate. burn-store 0.21 has no `FloatCastAdapter` - its adapters are PyTorchToBurn / BurnToPyTorch / HalfPrecision / Chain / Identity, and HalfPrecision is f32<->f16 only - so replacing our `CastFloatAdapter` is a 0.22 task, not a missed opportunity. And wgpu 30's f16 change is wider than recorded: SHADER_F16 works in WGSL and GLSL now, not only SPIR-V passthrough, which means today's f16-is-Vulkan-only shape ends at the bump. Cooperative matrix load/store also lands, but at 8x8 f32 only. P2 (MoE decode): llama.cpp discussion #24528 is the first route that explains our own 2026-08-03 regression instead of ignoring it. It caches hot experts in VRAM and runs HYBRID hit/miss - the cached rows go to the GPU as one batched matvec while the miss rows compute on the CPU exactly as before - so a miss costs nothing extra and the path degrades to the dense baseline rather than paying an unconditional gather. That is the difference from what we measured and reverted. It also needs no whole-model-in-VRAM, so it unblocks route (b) without waiting on P9. P5 (speculative decoding): the MTP route has a merged implementation (llama.cpp #22673) and a target - ~75% acceptance at k=3 for >2x, with k=2 often beating k=3 as acceptance falls off with the draft window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`dequant_to_safetensors` filled a `data` Vec of the whole f32 payload, then copied it into a second `blob` Vec of `8 + header + data` - peak 2x the payload, before the model was even built. For OLMoE-1B-7B that is ~28 GB dequantized and ~56 GB of commit charged, which is most of why that model reads as "CPU-only, and only on a big box". `safetensors::fuse_into` was the template and this follows it: plan the whole output first, then stream header + payload into an `impl Write`. `dequant_to_safetensors` is now a thin wrapper over the shared `dequant_into<W>`, joined by `dequant_to_safetensors_file`. The double buffer is gone from BOTH forms - the in-memory one writes header then payload into a single Vec (1x, not 2x), and the file one holds nothing but the current tensor's f32 values plus a fixed 1 MiB staging buffer. Planning first also made the path stricter. `plan_dequant` decides every name, shape and offset before the first payload byte is read, so an unmapped name, a reshape that changes the element count, or a rename collision fails in milliseconds rather than after N tensors have been dequantized. A unit test proves the ordering by pointing a tensor's payload past the end of the file and checking the MAP error still surfaces. `olmoe::load_from_gguf` takes the file variant, reusing the `FusedTemp` guard the HF path already uses; that guard's name now carries a counter as well as the pid, so two concurrent loads in one process cannot pick the same scratch file. REAL-WEIGHTS proof - OLMoE-1B-7B Q4_K_M against llama.cpp on the identical file: top-5 ids exact in order, the 24-token greedy sequence byte-identical, max |dlogprob| 3.687691131310693e-1 against a 7.5e-1 tolerance, 130.5 s. Sampled peak while it ran: 26.5 GB private commit - the model alone - against a 51.7 GB working set, i.e. ~25 GB of the load is file-backed mmap rather than charged to commit, which is the binding memory limit on this box. Nothing else moved. qwen2 GGUF parity 2.6614442413586614e-1 and qwen3 4.015608155114805e-1, the latter bit-identical to the recorded value, both greedy sequences byte-identical; the four `real_qwen2_gguf_*` legs pass in 44.1 s. 234 lib tests (+2), fmt clean, clippy --all-targets with no new warnings. The three OLMoE test gates said "~60 GB free RAM"; they now say ~30 GB free commit plus ~28 GB of scratch disk, which is what they actually need. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Last commit gave OLMoE's GGUF load the streaming sink and left the other three on the in-memory one, with a roadmap note admitting that split was a guess. Measured, it was the wrong guess. A/B on Qwen2.5-1.5B Q4_K_M (~6 GB of f32), the same load test each way: in-memory 33.90 / 35.93 / 42.15 s, scratch file 36.58 / 37.14 s. The scratch numbers sit inside the in-memory run's own spread, so at any payload of real size the disk round-trip is free and the doubled peak was being paid for nothing. (n=2 on the scratch side: a concurrent routine took the shared cargo target's lock and the third run never got to start.) So the choice becomes automatic. `import::DequantSink` is Memory / Scratch / Auto, and Auto keeps a payload in RAM only up to MAX_IN_MEMORY_DEQUANT_BYTES (512 MiB, i.e. a ~1 GiB peak - free anywhere). Everything else streams. All four GGUF ports now pass DequantSink::Auto, so no consumer has to know the knob exists. The refactor that made this expressible is worth as much as the policy: `import::gguf_store` is the one place the four ports shared - the dequant, the adapter chain, and the comment explaining why the target float dtype comes from the TYPE (`B::FloatElem`) and never from a probe tensor. That reasoning existed in four copies, phrased three different ways. Each port now adds only its own key remaps. `FusedTemp` moves out of models/olmoe.rs and becomes `import::ScratchFile` - the second caller the previous commit predicted arrived immediately, and two copies of a delete-on-every-exit-path guard is one too many. Proof that the new default changed nothing: all three GGUF parity legs against llama.cpp on the identical file, with qwen2 and qwen3 now taking the scratch path they did not take before - qwen2 2.6614442413586614e-1 and qwen3 4.015608155114805e-1, both bit-identical to the values recorded before this change, OLMoE 3.687691131310693e-1, and all three 24-token greedy sequences byte-identical to llama.cpp. The four `real_qwen2_gguf_*` legs pass (40.1 s), the four `real_inference` safetensors legs pass (34.3 s), and the model dirs hold no leftover scratch files afterwards. 236 lib tests (+2: the Auto boundary in both directions, and that two guards never name the same file and both delete on drop). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Ticks the two items the last commit closed. The number worth keeping is that streaming to disk is free at ~6 GB of f32 payload (36.58 / 37.14 s vs an in-memory 33.90 / 35.93 / 42.15 s), and the reason it is free - burn-store's `SafetensorsStore::from_file` mmaps and materializes tensors lazily, so the read-back is page faults during a load that was already reading, not a second pass. Also records the design that was deliberately NOT built: this item floated a RAM-aware choice against the device inventory's free-RAM figure. Once streaming is free, a size threshold answers the question, and a planner dependency would be complexity bought for nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The parent item listed two remaining pieces - SentencePiece `tokenizer.model` import, and calling the consistency validators from `load_from_dir` - and both have been `[x]` in split sub-items for a while. Rather than tick it on that, the whole surface was re-verified on real files this run: config-to-tokenizer id agreement on qwen3-0.6b, both SentencePiece legs (Unigram and BPE-type) byte-matching their own tokenizer.json, the 10-case template BYTE gate across qwen2/qwen3/lfm2, and the jinja fallback renderer byte-identical to the family renderers. Also files one thing the pass surfaced: `imported_render` reported FAILED because MUMMU_LFM2_DIR was unset, while both other legs were byte-identical - the same trap already recorded for `parity_gguf`. The item states the tension rather than assuming the fix: a gate that silently skips is a gate that does not exist, so the answer is a summarized "N legs skipped for missing fixtures" outcome, not a quiet skip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR refactors the GGUF import pipeline to avoid double-buffering the dequantized f32 payload, adds an automatic sink selection (in-memory vs scratch-file) shared across all GGUF model ports, and updates docs/tests/roadmap to reflect the new memory and verification behavior.

Changes:

Implement streaming GGUF dequantization into an impl Write (plus a file-backed variant) and plan dequant output up-front.
Centralize GGUF store construction via import::gguf_store and introduce import::DequantSink::{Memory,Scratch,Auto} with a payload-size threshold.
Update documentation, roadmap entries, and ignored-test resource notes; refresh a handful of transitive dependencies in Cargo.lock.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ROADMAP.md	Records new findings, closes shipped items, and adds measured sink-policy details.
README.md	Documents the GGUF streaming/scratch-file behavior and updated memory profile.
crates/mummu/tests/real_olmoe.rs	Updates ignored-test resource requirements (commit + scratch disk).
crates/mummu/tests/parity_gguf.rs	Updates ignored-test resource requirements (commit + scratch disk).
crates/mummu-bench/tests/budget_moe.rs	Updates ignored-test resource requirements (commit + scratch disk).
crates/mummu/src/models/qwen3.rs	Switches GGUF loading to shared `gguf_store(..., DequantSink::Auto)`.
crates/mummu/src/models/qwen2.rs	Switches GGUF loading to shared `gguf_store(..., DequantSink::Auto)`.
crates/mummu/src/models/lfm2.rs	Switches GGUF loading to shared `gguf_store(..., DequantSink::Auto)`.
crates/mummu/src/models/olmoe.rs	Switches GGUF loading to shared `gguf_store(..., DequantSink::Auto)` and moves scratch guard usage to `import::ScratchFile`.
crates/mummu/src/import.rs	Adds `DequantSink`, `ScratchFile`, `gguf_store`, and tests for sink selection + scratch-file behavior.
crates/mummu/src/gguf.rs	Adds streaming dequant (`dequant_into`, `dequant_to_safetensors_file`), up-front planning, and replaces `.expect()` conversions with fallible helpers.
Cargo.lock	Updates transitive dependencies (`either`, `h2`, `icu_provider`, `zerovec-derive`).

Suppressed comments (2)

crates/mummu/src/gguf.rs:612

len = info.element_count() * 4 and start += len are unchecked. If either multiplication/addition overflows, offsets in the planned safetensors header can become inconsistent (and the MAX_TENSOR_F32_BYTES check can be bypassed via wrap). Use checked arithmetic and surface an OverBound error instead.

            let len = info.element_count() * 4;
            if len > MAX_TENSOR_F32_BYTES {
                return Err(GgufError::OverBound {
                    path: self.path.display().to_string(),
                    what: "single dequantized tensor bytes",

crates/mummu/src/import.rs:376

This assert!(bytes > 0, ...) can panic on a syntactically valid GGUF with zero tensors (or other edge cases where the dequant writes no payload). Since this is on a production import path, prefer returning an ImportError::Parse instead of panicking.

            let bytes = f
                .dequant_to_safetensors_file(map, scratch.path())
                .map_err(|e| parse(e.to_string()))?;
            assert!(bytes > 0, "a parsed GGUF yields a non-empty payload");

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        &self,
+        map: &dyn Fn(&GgufTensorInfo) -> Option<GgufMap>,
+    ) -> Result<Vec<PlannedTensor>, GgufError> {
        let total_f32_bytes: u64 = self.tensors.iter().map(|t| t.element_count() * 4).sum();


+        file: f.path.clone(),
+        reason,
+    };
+    let total_f32_bytes: u64 = f.tensors.iter().map(|t| t.element_count() * 4).sum();


+        // And a stale file at the same name is cleared, not adopted: write
+        // one at b's path, then re-create a guard for it.
+        let stale = b.path().to_path_buf();
+        std::fs::write(&stale, b"stale").expect("writes");
+        drop(b);
+        assert!(!stale.exists());


Justin Icenhour and others added 7 commits August 20, 2026 23:01

Copilot AI lite review requested due to automatic review settings August 21, 2026 06:06

physics515 merged commit fc01b2a into main Aug 21, 2026
1 check passed

Copilot started reviewing on behalf of physics515 August 21, 2026 06:06 View session

Copilot AI reviewed Aug 21, 2026

View reviewed changes

Navigation Menu

Nightly 2026-08-20 (2nd run): the GGUF import stops paying for the payload twice - #20

Nightly 2026-08-20 (2nd run): the GGUF import stops paying for the payload twice#20
physics515 merged 7 commits into
mainfrom
mummu-nightly-2026-08-20-2

physics515 commented Aug 21, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

physics515 commented Aug 21, 2026

Dependency freshness

Items shipped

gguf.rs has no .expect() left on a production path (P3)

The GGUF dequant streams instead of buffering twice (P3)

One gguf_store for all four ports, and the sink choice measured (P3)

Tokenizer + chat-template import closed on evidence (P4)

Verification

Research folded in

What's next

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`gguf.rs` has no `.expect()` left on a production path (P3)

One `gguf_store` for all four ports, and the sink choice measured (P3)