Nightly 2026-07-13: GGUF end-to-end — one .gguf file is the whole model (Qwen2.5 + LFM2.5) - #5
Merged
Merged
Conversation
wgpu 30 / tokenizers 0.23 majors offered by cargo-upgrade were reverted: burn 0.21 + wgpu 29 + tokenizers 0.22 is the intentional parity-validated pin (see Cargo.toml). Gates green: 121 unit tests, GPU budget 107.4 ms / 13.3 tok/s, CPU budget 15.1 tok/s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…om_gguf - dequantize every GGUF storage dtype: adds Q4_0/Q4_1/Q5_0/Q5_1 legacy blocks and Q2_K/Q3_K/Q5_K superblocks (exact ggml-quants ports, each with hand-computed block tests); Q8_1/Q8_K stay loud errors (activation formats, never tensor storage) - GgufFile::dequant_to_safetensors: bridge a GGUF onto the exact checked-load pipeline safetensors uses (in-memory blob, dims reversed = row-major HF layout, unmapped tensor names error loudly, 48 GiB RAM bound) - Qwen2Config::from_gguf: hyperparameters from qwen2.* metadata; vocab from the embedding tensor; EOS from tokenizer.ggml.eos_token_id - qwen2::load_from_gguf: one .gguf file -> a running model - Qwen2 gains an optional untied lm_head: llama.cpp GGUFs materialize the tied head as a separate (higher-precision) output.weight; also unlocks untied safetensors checkpoints (Qwen2-7B class) Proof (real weights, 4070 Ti SUPER): Qwen2.5-1.5B Q4_K_M greedy-decodes '2+2 equals 4.'; first-token top-1 identical to the bf16 build, top-5 overlap 4/5, logit cosine 0.977. Parity gate re-passed byte-identically after the lm_head change (max dlogit 2.670e-5; Ollama greedy exact). 133 unit tests; f16, real-inference, and budget gates green (GPU 107.4 ms / 13.6 tok/s, CPU 14.8 tok/s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…model
- mummu::tokenizer::tokenizer_from_gguf rebuilds the HF tokenizers
pipeline from tokenizer.ggml.* metadata: NFC -> Split(per-family
pre-tokenizer regex, llama.cpp-style registry keyed by
tokenizer.ggml.pre; unknown ids error loudly) -> ByteLevel -> BPE
- token id = tokens-array index; CONTROL(3) -> special added tokens,
USER_DEFINED(4) -> plain added tokens, UNUSED(5) [PADn] skipped;
every added-token id is verified post-build (drift = loud error)
- GgufValue::as_i64 for the signed token_type array
- the end-to-end GPU test now sources tokenizer + config + weights
from the single .gguf file
Proof: byte-identical ids vs the checkpoint's tokenizer.json across an
8-prompt battery (ChatML with specials, unicode/CJK/emoji, whitespace
runs, contractions, numbers, empty) and identical decodes; end-to-end
Q4_K_M decode re-passed ('2+2 equals 4.'). 135 unit tests green,
clippy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed-tensors convention (P3), llama.cpp LFM2.5 parser fix upstream (P7) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Lfm2Config::from_gguf: hyperparameters from lfm2.* metadata; layer kinds derived from llama.cpp's per-layer head_count_kv array (0 = shortconv, nonzero = attention; i32 in real files, all nonzero must agree); feed_forward_length arrives pre-adjusted - lfm2 tensor-name map incl. per-head q/k norms and shortconv.*; the depthwise conv kernel is the one shape special-case: new GgufMap::Reshape un-squeezes ggml's [K, C] back to the checkpoint's [C, 1, K] (same bytes; element count validated) - tokenizer registry gains the lfm2 pre: digits-in-<=3-groups regex, no NFC, BOS post-processor from tokenizer.ggml.add_bos_token; added tokens at LOW ids (LFM2 puts 500+ specials at 0..) work by seeding the BPE vocab and letting add_tokens reuse the model id Proof (official LiquidAI Q4_K_M vs the same checkpoint's local bf16 safetensors): 5 F32 tensors incl. both conv kernels bit-exact; tokenizer byte-identical over a 6-prompt x 2-mode battery; REAL-GPU end-to-end — the one file greedy-decodes '2 + 2 equals 4.', top-1 identical to the bf16 build, logit cosine 0.9914. 138 unit tests; qwen2 GGUF suite, parity gate (2.670e-5, Ollama byte-identical), and budget gates (GPU 106.7 ms / 13.5 tok/s, CPU 13.5 tok/s) all re-passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…to-end
- ModelSpec.format: WeightFormat::Safetensors | Gguf { file }
(serde-defaulted, so manifests written before the field still parse)
- GGUF specs fetch their one file through the resumable/verified hub
path; gguf_path() names the local file; validation rejects empty,
traversal, absolute, and non-.gguf file names
- catalog: single-file Q4_K_M entries for Qwen2.5-1.5B-Instruct and
LFM2.5-1.2B-Instruct (quarter the download of the safetensors)
REAL-NETWORK proof (real_hub.rs): the LFM2.5 GGUF spec installs
end-to-end — 697 MB fetched via spec.fetch, header parses as lfm2
(148 tensors), tokenizer builds from its metadata. 140 unit tests,
clippy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This nightly PR completes the “single-file GGUF is a whole runnable model” workflow by adding (1) a safetensors-bridge dequant path for GGUF weights, (2) tokenizer reconstruction from GGUF metadata, and (3) registry support for fetching/installing GGUF models end-to-end (Qwen2.5 + LFM2.5).
Changes:
- Add
GgufFile::dequant_to_safetensorsplus additional ggml dequantizers to cover all GGUF storage dtypes needed for real models. - Introduce
mummu::tokenizer::tokenizer_from_ggufto rebuild HFtokenizerspipelines fromtokenizer.ggml.*metadata. - Extend Qwen2 and LFM2 loaders/config parsing to support GGUF inputs; update the registry to fetch either safetensors or single-file GGUF specs.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| ROADMAP.md | Marks GGUF and tokenizer-from-metadata work as shipped; adds follow-up parity/VRAM items. |
| README.md | Updates documentation to reflect GGUF end-to-end model + tokenizer support. |
| crates/mummu/tests/real_hub.rs | Adds a real-network GGUF spec install test (ignored by default). |
| crates/mummu/tests/real_gguf.rs | Adds real-file tokenizer parity tests and end-to-end GPU decode parity checks for Qwen2/LFM2. |
| crates/mummu/src/tokenizer.rs | New module to reconstruct a byte-level BPE tokenizer from GGUF metadata. |
| crates/mummu/src/registry.rs | Adds WeightFormat and GGUF fetch/path support in ModelSpec. |
| crates/mummu/src/models/qwen2.rs | Adds GGUF config parsing + GGUF loader; supports optional untied lm_head. |
| crates/mummu/src/models/lfm2.rs | Adds GGUF config parsing + GGUF loader; adds conv-kernel reshape mapping. |
| crates/mummu/src/lib.rs | Exposes the new tokenizer module. |
| crates/mummu/src/gguf.rs | Adds dequant-to-safetensors bridge, GgufMap::Reshape, more dequantizers, and tests. |
| Cargo.lock | Lockfile-only dependency updates (uuid, zmij). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+162
to
+164
| let byte_level = ByteLevel::new(false, false, false); | ||
| tok.with_pre_tokenizer(Some(Sequence::new(vec![split.into(), byte_level.into()]))); | ||
| tok.with_decoder(Some(byte_level)); |
Comment on lines
+101
to
+105
| let hidden_size = gguf_usize(f, "qwen2.embedding_length")?; | ||
| let num_attention_heads = gguf_usize(f, "qwen2.attention.head_count")?; | ||
| let embd = f | ||
| .tensor("token_embd.weight") | ||
| .ok_or("GGUF has no token_embd.weight tensor")?; |
Comment on lines
+74
to
+77
| let safe = !file.is_empty() | ||
| && !file.contains("..") | ||
| && !file.starts_with('/') | ||
| && file.ends_with(".gguf"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated nightly run (~4.5 h wall-clock, 6 commits). Theme: the P3 GGUF import path went from "container + partial dequant" to a single .gguf file is a complete, running model — config, weights, AND tokenizer — for both zoo LLM architectures, wired through the registry so apps can install GGUF models end-to-end.
Dependency freshness (standalone increment)
cargo update: uuid 1.23.4→1.23.5, zmij 1.0.21→1.0.22 (lock-only).cargo upgrade --incompatibleoffered wgpu 29→30 and tokenizers 0.22→0.23 — both REVERTED per the intentional parity-validated pin (burn 0.21 + wgpu 29 + tokenizers 0.22, documented in Cargo.toml). No migration item needed: the pins are deliberate, not blocked.1) GGUF → running model (Qwen2.5)
GgufFile::dequant_to_safetensors: bridges a GGUF onto the exact checked-load pipeline the safetensors path trusts (in-memory blob; dims reversed = row-major HF layout; unmapped tensor names are loud errors; 48 GiB RAM bound).Qwen2Config::from_gguf(hyperparameters fromqwen2.*metadata) +qwen2::load_from_gguf.lm_head— llama.cpp GGUFs materialize the tied head as a separate higher-precisionoutput.weight(Q6_K in the real file); also unlocks untied safetensors checkpoints (7B class).\"2+2 equals 4.\"; first-token top-1 identical to the bf16 safetensors build, top-5 overlap 4/5, logit cosine 0.977 (quantization drift; a layout bug reads ≈ 0). Parity gate re-passed byte-identically after the lm_head change (max |Δlogit| 2.670e-5 vs Candle; Ollama fp16 greedy leg exact).2) Tokenizer from GGUF metadata
mummu::tokenizer::tokenizer_from_gguf: rebuilds the HFtokenizerspipeline fromtokenizer.ggml.*— per-family pre-tokenizer regex registry (llama.cpp-style, unknown ids error loudly), ByteLevel BPE, token id = array index, CONTROL/USER_DEFINED → special/plain added tokens with post-build id verification, UNUSED padding skipped.tokenizer.jsonon an 8-prompt battery (ChatML + specials, unicode/CJK/emoji, whitespace runs, contractions, empty) and identical decodes.3) LFM2.5 hybrid GGUF import
Lfm2Config::from_gguf: layer kinds derived from llama.cpp's per-layerhead_count_kvarray (0 = shortconv, nonzero = attention; i32 in real files);feed_forward_lengtharrives pre-adjusted.shortconv.*; newGgufMap::Reshapeun-squeezes the depthwise conv kernel (ggml[K,C]→ checkpoint[C,1,K], same bytes, element count validated).lfm2pre (digits-≤3 regex, no NFC, BOS post-processor fromadd_bos_token; LFM2's 500+ low-id specials work via BPE-vocab id reuse).\"2 + 2 equals 4.\", top-1 identical to bf16, logit cosine 0.9914. (Not the strict P2/P7 parity gate — that still needs the llama.cpp same-quant reference leg, tracked.)4) Registry learns GGUF
ModelSpec.format:Safetensors | Gguf { file }(serde-defaulted — old manifests parse); GGUF specs fetch their one file through the resumable/sha256-verified hub path; catalog gains Q4_K_M entries for both models (¼ the download).spec.fetch, header parses (lfm2, 148 tensors), tokenizer builds from metadata.Verification summary
cargo fmt/clippy --all-targetsclean; release build green.bench/BASELINE.mdunchanged.Research folded into ROADMAP
[ ]to evaluate as the keep-quantized matmul substrate (the actual VRAM lever; today's GGUF path dequants to f32).quantization_config, group-128 int4 standard; vLLM compressed-tensors is the unified on-disk convention to target (P3 note).What's next
/completionlogprobs on the official F16 GGUF (now that the file + fixed upstream exist).[ ]): assert our dequant matches ggml's compute path token-for-token.🤖 Generated with Claude Code