Nightly 2026-08-20 — OLMoE from HF safetensors, import-gate install fix, dependency refresh by physics515 · Pull Request #19 · physics515/Mummu

physics515 · 2026-08-20T17:04:30Z

Autonomous nightly run for 2026-08-20. Four increments, each verified before the next.

This run also had cleanup to do first: the 2026-08-15 run died mid-flight — it left a stale
lock (5 days old), a live worktree, ~1 370 lines of uncommitted work, and a half-downloaded
13.8 GB checkpoint, and it had never opened a PR. Its work is carried forward here, but every
number it had written into the roadmap was re-derived from scratch rather than trusted; two
of its claims turned out not to hold (below).

Dependency freshness

cargo update moved 36 transitive packages (minijinja 2.23→2.24, the icu_* 2.2→2.3 family,
futures 0.3.33→0.3.34, h2, quinn-proto, rustls-webpki, zerovec). One major taken:
hf-chat-template 0.2.1 → 1.0.0.

That crate is load-bearing twice over — the reference renderer behind the template BYTE gate, and
the fallback renderer behind the non-default jinja-template feature — so it got the gate rather
than a build check: 10/10 byte-identical across Qwen3, Qwen2 and LFM2 (plain / multi-turn /
tools / tool-history / no-system). Green both ways the crate can compile in: 216 tests default,
223 with jinja-template.

Two incompatible upgrades deliberately not taken, now recorded in the roadmap:

burn stays 0.21 — crates.io serves 0.22.0-pre.2; P0 says do not adopt a pre-release.
wgpu stays 29 — cargo tree -i wgpu shows a single wgpu (29.0.4) reached via cubecl-wgpu.
Bumping only our direct handle would put a second, non-Burn wgpu in the tree and the startup
adapter probe would stop describing the device Burn actually runs on.

P2 — OLMoE from HF safetensors (item ticked)

The port could previously read GGUF only: a GGUF ships the expert bank pre-fused (ffn_*_exps)
while the HF checkpoint stores all 64 experts separately across three shards, and burn-store
bridges neither gap (it finds only a single model.safetensors, and its remapping is 1:1).

New safetensors.rs: a sharded reader plus an N:1 fusing rewriter that plans the whole output
before reading one payload byte, so an incomplete expert group is a loud error rather than a short
bank that loads clean and computes wrong. Members are ordered numerically, not
lexicographically (experts.10 sorts before experts.2 as text — the entire risk of this item).

Real-weights proof on the 3-shard 13.84 GB bf16 checkpoint:

[64, 1024, 2048] banks over 16 layers, checked-load in 136.1 s on the CPU backend, zero
missing params;
live distribution (sanity top 194, spread 20.6), 8 tokens greedy-decoded. This leg uses an
arbitrary token probe, not a prompt — read it as liveness, not coherence;
the check a mis-ordering could not survive: layer 5 / expert 37's gate_proj, read independently
out of the raw shard bytes, is bit-identical to slot 37 of the fused bank — 0 mismatches over
2 097 152 values (bf16→f32 is an exact 16-bit widening, so bit-equality, not a tolerance).

Two defects the real run caught that unit tests had not — and that the dead run had already
written up as passing:

checkpoint_shards planned off the index without checking the shards were on disk, so a
half-finished download reported as a complete checkpoint: the resume was skipped and the load
died later with a bare os error 2. Now fails at planning time naming the missing shard, and says
so when a .part sibling shows the download was interrupted.
The fuse materialized the payload in RAM twice (a data buffer, then a blob copy) and the
second 13.8 GB allocation genuinely failed on this 128 GB box:
memory allocation of 13838346237 bytes failed. fuse_checkpoint_to_file now streams header +
payload to a temp file through one reusable ~4 MiB per-part buffer, dropping peak from ~42 GB
(13.8 blob + ~28 model) to the model alone. A FusedTemp guard removes the scratch file on every
exit path (verified gone), and a unit test pins the two fuse paths as byte-identical.

Also hardened while carrying it: every .expect() on a production path in the new module replaced
with a fallible to_usize returning OverBound; assertions 5 → 9 across 9 functions.

P3 — `fetch_model` installs the files the import gates read (new item, shipped)

Found by installing the OLMoE checkpoint through Mummu's own registry. fetch_model_with fetched
exactly config.json, tokenizer.json and the weights — so a registry-installed checkpoint had no
tokenizer_config.json, and every gate that reads it fails open: validate_checkpoint_dir
returns Ok(None), so the EOS-agreement, added-token-id and tool-call-convention checks silently do
not run. They had been passing only because the local fixtures were populated by hand.

tokenizer_config.json and chat_template.jinja are now fetched as optional files. Only a 404
counts as absent — a 403 on a gated repo or a 5xx still reaches the real fetch and is reported —
config.json/tokenizer.json stay required, and they are fetched before the weights because the
single-file branch returns early.

Proof is a clean-dir install, so no hand-populated fixture can satisfy it: the catalog model
arrives with tokenizer_config.json AND validate_checkpoint_dir returns Some — the gate has
something to check instead of quietly passing.

Research folded

Four dated notes, each attached to the item it changes:

P2 MoE decode — llama.cpp PR #25294 turns route (c) into an implementation with numbers:
bounded per-layer expert-slab cache, async demand-load, eviction by decaying route hotness with an
LRU tiebreak, O_DIRECT; 5.3x prefill / 2.4x decode over mmap+--n-cpu-moe at a 79 % hit
rate. And our own gather regression looks like a batch-size regime, not a refutation:
ik_llama.cpp switches to the copy-experts path at 32 * total_experts / active_experts = 256
tokens for OLMoE's 64/8, while we measured batch-1 decode — the worst possible point.
P6 NVMe streaming — same PR as the reference implementation of the design already written there.
P2 Qwen3.5 — the linear-attention half is Gated DeltaNet at ~75 % linear / 25 % full, so
full_attention_interval should read 4; MTP variants ship as their own GGUF repos (the concrete
draft artifact P5 needs).
P0 burn 0.22 — associated element types move onto a new BackendTypes trait with callers
steered to type aliases; every hand-written B::FloatElem is where the 0.22 diff will land.

Two new [ ] items discovered from this run's own work:

gguf::dequant_to_safetensors has the identical double-buffer (peak = 2× the dequantized f32
payload, ~56 GB for OLMoE) — fuse_into is the template for fixing it.
gguf.rs still carries .expect() on production paths (4 sites).

Cleanup: recovering the 2026-08-15 run

That run left .git/mummu-nightly.lock 5 days stale, a live ../mummu-nightly worktree, ~1 370
lines uncommitted, a shard-3 .part, and one unpushed commit — no PR. Its diff was preserved as a
patch, its worktree removed, and its code carried forward; its prose was not. Two of the roadmap
claims it had already written turned out to be false (it had written the OLMoE proof up as passing
while its third shard was still downloading), which is exactly what re-running from scratch is for.

Verification

cargo fmt --all --check, cargo clippy --all-targets, cargo build --all-targets, cargo test
— all clean. 232 lib tests (216 at branch point; +16 this run), every test binary ok.
The one pre-existing clippy line is upstream (burn-cubecl future-incompat), not ours; no
#![allow] was added anywhere.

Parity, re-run as a regression gate because olmoe.rs was touched — dev profile, which the
2026-07-16 opt-level fix made valid for real-model suites:

leg	top-5 ids	max abs Δlogprob	vs recorded
olmoe Q4_K_M	exact in order	3.687691131310693e-1	3.7e-1 ✓
qwen2 Q4_K_M	exact in order	2.6614442413586614e-1	2.7e-1 ✓
qwen3 Q4_K_M	exact in order	4.015608155114805e-1	character-identical ✓

The lfm2 leg did not run: MUMMU_LFM2_GGUF_PATH is not cached on this machine (a known local
fixture gap, not a regression) and the harness panics rather than skipping on a missing env var.

Not run this run: the criterion benches and the budget gates. Nothing in this branch touches a
decode kernel or a hot path — the changes are import-side and docs — and the machine was not quiet
(a second tenant held ~40 GB of commit throughout), which per the roadmap's own operational note
reads as a fake regression. bench/BASELINE.md is therefore unchanged and unclaimed.

Honest caveats

The OLMoE safetensors decode leg asserts only that tokens come out; on an arbitrary token probe it
emits [194, 200, 200, ...]. Liveness, not coherence. The weight-correctness claim rests entirely
on the bit-exact expert check, which is the stronger evidence anyway.
plan_output's split brought the worst function from ~150 to 93 lines; fuse_into (94) and the
inherited parse_header (95) are still over the ~70 guideline.

Dependency-freshness increment. `cargo update` moves 36 transitive packages (minijinja 2.23->2.24, the icu_* 2.2->2.3 family, futures 0.3.33->0.3.34, h2, quinn-proto, rustls-webpki, zerovec, ...). The one major available to us is hf-chat-template 0.2.1 -> 1.0.0, and it is taken. That crate is load-bearing in two roles at once, so it gets the gate that matters rather than a build check: it is the reference renderer behind the template BYTE gate AND, behind the non-default `jinja-template` feature, the fallback renderer for checkpoints with no hardcoded family renderer. Proof it is a non-event: the byte gate is 10/10 byte-identical across Qwen3, Qwen2 and LFM2 (plain / multi-turn / tools / tool-history / no-system), i.e. every rendered prompt is the same bytes under 1.0.0 as under 0.2.1. Suites green both ways the crate can be compiled in — 216 tests default, 223 with `jinja-template` on, clippy --all-targets clean on both. Two incompatible upgrades are deliberately NOT taken, and the roadmap now records why: - burn stays 0.21 — crates.io serves 0.22.0-pre.2 and the P0 migration item says do not adopt a pre-release. - wgpu stays 29 — `cargo tree -i wgpu` shows a single wgpu (29.0.4) reached through cubecl-wgpu 0.10, so bumping only our direct handle would put a second, non-Burn wgpu in the tree and the startup adapter probe would stop describing the device Burn actually runs on. It unblocks with the burn bump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Four dated notes, each attached to the item it changes rather than to a log. P2 MoE decode — the two findings that most change that item's shape. llama.cpp PR #25294 turns route (c) from a slogan into an implementation with numbers: bounded per-layer device-side expert-slab cache, CPU-side top-k id remap to cache slots, async demand-load on a miss, eviction by DECAYING ROUTE HOTNESS with an LRU tiebreak, O_DIRECT to keep the page cache from thrashing — 5.3x prefill / 2.4x decode over mmap+--n-cpu-moe at a 79 % hit rate. That hit rate is the load-bearing number for us: it says a small resident expert set covers most tokens. Separately, our own 2026-08-03 gather regression looks like a BATCH-SIZE REGIME rather than a refutation — llama.cpp only switches to the copy-experts path above a batch threshold, and ik_llama.cpp sets it at 32 * total_experts / active_experts, i.e. 256 tokens for OLMoE's 64/8. We measured batch-1 decode, the worst possible point for a materializing gather; prefill is where it should win, and that leg was never run. P6 NVMe streaming — the same PR is the reference implementation of the colibri-shaped design already written there, so the note records the two details worth stealing (hotness-with-LRU-tiebreak over plain LRU, since MoE routing is skewed; unbuffered reads once the model exceeds RAM) and its single-context limitation, which is a constraint on how we own the pool. P2 Qwen3.5 — the linear-attention half has a name, Gated DeltaNet, at roughly 75 % linear / 25 % full, so full_attention_interval is expected to read 4. Also: the MTP variants ship as their own GGUF repos, which is the concrete draft-model artifact P5's speculative-decoding item needs. P0 burn 0.22 — the associated element types are not deleted but moved onto a new BackendTypes trait, with callers steered to the type aliases. Actionable before the bump: every hand-written B::FloatElem is a place the 0.22 diff will land. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`fetch_model_with` fetched exactly config.json, tokenizer.json, and the weights. So a checkpoint installed through Mummu's own registry arrived WITHOUT tokenizer_config.json — and every gate that reads it fails open: `validate_checkpoint_dir` returns Ok(None), so the EOS-agreement, added-token-id and tool-call-convention checks silently do not run, and the `tokenizer_config` the loaders surface is always None. The gates have been passing on hand-populated fixtures. Found by installing the OLMoE safetensors checkpoint through the registry and noticing the resulting dir has no tokenizer_config.json to check. Both tokenizer_config.json and chat_template.jinja are now fetched as OPTIONAL files. Three details that make this a fix rather than a hopeful extra request: - Only a 404 counts as absent. A 403 on a gated repo or a 5xx is treated as present so the real fetch raises it, instead of the probe swallowing a real failure as "the repo just doesn't ship this". - config.json and tokenizer.json stay REQUIRED — a 404 on either is still an error. - They are fetched BEFORE the weights, because the single-file branch returns early; after it, sharded checkpoints would have been the only ones to get them. Proof is a clean-dir install, so no hand-populated fixture can satisfy it: `a_registry_install_arrives_with_the_files_the_import_gates_read` fetches a catalog model into a fresh dir and asserts both that tokenizer_config.json is on disk and that validate_checkpoint_dir returns Some — the gate now has something to check rather than quietly passing. Green, plus a unit test pinning the optional set and its URLs (223 lib tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The OLMoE port could only read GGUF, because a GGUF ships the expert bank already fused (`ffn_*_exps`) while the HF checkpoint stores every expert as its own tensor across three shards. `burn-store` cannot bridge that: it finds only a single `model.safetensors`, and its remapping is 1:1, not N:1. `safetensors.rs` is a sharded reader plus a fusing rewriter for exactly those two gaps. It plans the entire output before reading one payload byte, so a group that is not exactly `count` members `0..count` is a loud BadGroup rather than a short expert bank that loads clean and computes wrong. Members are ordered NUMERICALLY, not lexicographically — `experts.10` sorts before `experts.2` as text, and that silent mis-ordering is the whole risk here. `olmoe::load_from_dir` then rides the ordinary SafetensorsStore + adapter-chain + load_checked pipeline, landing on the same params the GGUF path renames to (pinned by a unit test — both importers must agree or one is loading a different model). REAL-WEIGHTS proof, on the 3-shard 13.84 GB bf16 checkpoint: - fuses [64, 1024, 2048] banks over 16 layers, checked-load in 136.1 s on the CPU backend, zero missing params; - emits a live distribution (sanity top 194, spread 20.6) and greedy-decodes 8 tokens. This leg uses an arbitrary token probe rather than a prompt, so read it as liveness, not coherence; - the check a mis-ordering could NOT survive: layer 5 / expert 37's `gate_proj`, read independently out of the raw shard bytes, is bit-identical to slot 37 of the fused bank — 0 mismatches over 2 097 152 values. bf16→f32 is an exact 16-bit widening, so this is bit-equality, not a tolerance. Two defects the real run caught that unit tests had not: - `checkpoint_shards` planned off the index without checking the shards were on disk. A half-finished download therefore reported as a COMPLETE checkpoint: the fetch was skipped and the load died later with a bare `os error 2`. It now fails at planning time naming the missing shard, and says so when a `.part` sibling shows the download was interrupted. - The fuse materialized the payload in RAM twice — a `data` buffer, then a `blob` copy — and the second 13.8 GB allocation genuinely failed on this 128 GB machine (`memory allocation of 13838346237 bytes failed`). `fuse_checkpoint_to_file` now streams header + payload straight to a temp file through ONE reusable per-part buffer (~4 MiB, a single expert projection), dropping peak from ~42 GB (13.8 blob + ~28 model) to the model alone. A `FusedTemp` guard deletes the scratch file on every exit path, verified gone after the run, and a unit test pins the in-memory and to-file fuses as byte-identical so the two are one importer, not two. Also hardened while carrying it: every `.expect()` on a production path in the new module is gone (a fallible `to_usize` returning OverBound), and assertions went 5 -> 9 across 9 functions. 232 lib tests green, clippy --all-targets clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`plan_output` was ~150 lines doing two separable jobs. `collect_groups` is now the first pass — walk every shard, record in first-seen order what each target is built from, and reject a checkpoint's bad *claims* (unmapped name, member index outside 0..count, members disagreeing on group size, a duplicate member). `plan_output` keeps the second pass: lay the groups out and apply THE completeness check, which can only run once every shard has been seen, because a group's members may be split across shards in any order. Bodies now 63 and 93 lines, down from one 150-line function. Behaviour is unchanged and the existing suite says so — the group-validation tests (short group, duplicate member, layout disagreement, numeric-vs-lexicographic ordering) all cover the seam and still pass. Also two doc comments that my own change had made wrong: `models::olmoe` and `MAX_FUSED_BYTES` still pointed at `fuse_checkpoint` after `load_from_dir` moved to `fuse_checkpoint_to_file`, and the module header described the result as necessarily "in-memory". A doc that names the wrong function is worse than no doc. 232 lib tests green, clippy --all-targets clean, fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

This nightly PR extends Mummu’s import pipeline to support OLMoE HuggingFace sharded safetensors checkpoints (including expert-bank fusion), hardens Hub installs so validation gates have the sibling files they require, and refreshes dependencies/documentation accordingly.

Changes:

Add a new sharded-safetensors reader + N:1 tensor fusing rewriter, and wire OLMoE’s HF safetensors import through it.
Fix Hub fetch_model_with to optionally fetch tokenizer_config.json and chat_template.jinja (when present) so import-validation gates don’t fail open on registry installs.
Update docs/roadmap/tests and refresh dependencies (including hf-chat-template 1.0.0).

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ROADMAP.md	Nightly notes: dependency holds, shipped items, and new research notes.
README.md	Document OLMoE HF safetensors import + fusion proof points.
crates/mummu/src/safetensors.rs	New sharded safetensors header reader and fusion-to-blob/file implementation.
crates/mummu/src/hub.rs	Fetch optional sibling files (gate inputs) before weights; add HEAD presence probe.
crates/mummu/src/models/olmoe.rs	Add HF safetensors import path (fuse-to-temp-file) and surface `tokenizer_config`.
crates/mummu/src/registry.rs	Add OLMoE HF safetensors entry to the catalog and update architecture docs.
crates/mummu/src/lib.rs	Export new `safetensors` module.
crates/mummu/tests/real_olmoe_safetensors.rs	New ignored real-weights proof tests for OLMoE safetensors fusion correctness.
crates/mummu/tests/real_hub.rs	New ignored test ensuring registry installs include gate-read sibling files.
Cargo.toml	Bump `hf-chat-template` to 1.0.0.
Cargo.lock	Dependency refresh from `cargo update`.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        assert!(!dir.as_os_str().is_empty(), "checkpoint dir must be named");
+        let path = dir.join(format!(
+            "mummu-fused-{}.safetensors.tmp",
+            std::process::id()
+        ));


+    // The sibling tokenizer_config.json is surfaced when the dir has one, and
+    // is legitimately absent otherwise (a registry fetch pulls config.json +
+    // tokenizer.json + weights only) — the loader must reflect the dir, not
+    // invent a config.


+        // File order (by payload offset) makes the copy pass a sequential read.
+        tensors.sort_by_key(|(_, t)| t.offsets.0);
+        Ok(tensors)


Justin Icenhour and others added 5 commits August 20, 2026 10:08

Copilot AI lite review requested due to automatic review settings August 20, 2026 17:04

physics515 merged commit efcc8ea into main Aug 20, 2026
1 check passed

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Copilot started reviewing on behalf of physics515 August 20, 2026 19:55 View session

physics515 mentioned this pull request Aug 21, 2026

Nightly 2026-08-20 (2nd run): the GGUF import stops paying for the payload twice #20

Merged

Navigation Menu

Nightly 2026-08-20 — OLMoE from HF safetensors, import-gate install fix, dependency refresh - #19

Nightly 2026-08-20 — OLMoE from HF safetensors, import-gate install fix, dependency refresh#19
physics515 merged 5 commits into
mainfrom
mummu-nightly-2026-08-20

physics515 commented Aug 20, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

physics515 commented Aug 20, 2026

Dependency freshness

P2 — OLMoE from HF safetensors (item ticked)

P3 — fetch_model installs the files the import gates read (new item, shipped)

Research folded

Cleanup: recovering the 2026-08-15 run

Verification

Honest caveats

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

P3 — `fetch_model` installs the files the import gates read (new item, shipped)