Nightly 2026-07-12: SPIR-V decode +30%, LFM2.5 function calling, GGUF import (container + K-quant dequant) by physics515 · Pull Request #4 · physics515/Mummu

physics515 · 2026-07-12T13:17:39Z

Automated nightly run — 6 commits, all gates green. Do not merge without review.

Dependency freshness

cargo update: lockfile already at latest compatible (0 changes).
cargo upgrade --incompatible: sha2 0.10 → 0.11 taken (Digest API unchanged; the hub sha256 stream-verify suite re-passed). wgpu 30 and tokenizers 0.23 left on their intentional pins — burn 0.21 / wgpu 29 / tokenizers 0.22 is the parity-validated combo, and our direct wgpu handle must stay the version Burn resolves.

Shipped

1. perf(P6): CubeCL SPIR-V compiler on Vulkan — decode 14.1 → 18.4 tok/s (+30%)

Enabled burn's vulkan feature; runtime now reports fusion<cubecl<wgpu<spirv>>> (verified live). Zero code changes — DX12/Metal keep WGSL in the same binary.

Decode 32 tokens: 2.263 s → 1.737 s (70.7 → 54.3 ms/token) on both f32 and f16 (criterion).
TTFT 88.4 → 96.7 ms — a real cost, but ⅔ under its 150 ms ceiling; decode dominates task throughput.
Parity byte-identical: top-5 ids exact, max |Δlogit| 2.670288e-5 (unchanged), 24-token greedy = Ollama fp16 exactly.
VRAM peak unchanged (11.5 GiB whole-card, ≤ 13 budget); GPU budget gate 108.4 ms / 11.7 tok/s; CPU flex gate 13.2 tok/s. bench/BASELINE.md re-baselined.

2. feat(P4): LFM2.5 Pythonic tool-calling — template + renderer + bounded parser

Implemented to LFM2.5's actual wire format (byte-verified against its chat_template.jinja + model card — 2.5 dropped gen-1's <|tool_list_start|> wrapping the roadmap item assumed):

Tools as bare JSON on a List of tools: […] system line (no default preamble); results as real tool role turns; </think> stripped from all but the last assistant history turn.
parse_tool_calls_lfm: bounded recursive-descent parser for the <|tool_call_start|>[name(k=v)]<|tool_call_end|> call list (depth ≤ 8, ≤ 64 calls, Python AND JSON literal spellings, byte-offset error taxonomy).
REAL-GPU proof: LFM2.5-1.2B greedy-emitted exactly <|tool_call_start|>[get_weather(city="Paris")]<|tool_call_end|> from our rendered prompt and the parser round-tripped it (tests/real_toolcall_lfm.rs). Qwen2 parity gate re-passed after the template refactor. 16 new unit tests.

3. feat(P3): GGUF container reader

mummu::gguf (no new deps): v2/v3 LE header, typed+bounded metadata, tensor table validated per entry (known dtype, aligned offsets, whole blocks, unique names). Real-file proof: the local Qwen2.5-1.5B Q4_K_M parses — 26 kvs, 339 tensors, Q4_K token_embd [1536, 151936] matching the 152k vocab, ~1.04 GiB payload located; 3B cross-checked (435 tensors).

4. feat(P3): GGUF dequantization — Q8_0 + Q4_K/Q6_K superblocks

Exact ports of ggml-quants' reference dequantizers + read_tensor_f32. Proof against the TRUE weights (same checkpoint on disk as bf16 safetensors AND Q4_K_M GGUF):

GGUF's F32 output_norm.weight == the bf16 originals bit-exact.
Q4_K embedding rows dequantize at cosine 0.99756 / 0.99746 vs truth — quantization-error territory; a wrong superblock decode reads ~0.

Verification (final tree)

cargo fmt / clippy --all-targets clean (no new warnings) / cargo build --release green.
121 unit tests pass (93 → 121, +28) — verified by name in an isolated CARGO_TARGET_DIR.
Real-GPU sweep on the final tree: Qwen2 parity both legs, f16 island decode, LFM2.5 tool-call emission, GGUF header + dequant cross-checks — all green.

Research folded into ROADMAP

Qwen3.5 small tier (0.8B–9B, GGUF re-releases with tool-calling template fixes) as the next zoo port; Ollama qwen3.5:9b already local as parity ref.
LFM2.5 llama-server parity route caveat: llama.cpp's own chat layer rejects LFM2.5's documented tool format (Eval bug: LFM2.5-8B-A1B tool parser rejects documented <|tool_call_start|>[...]<|tool_call_end|> format ggml-org/llama.cpp#23838) — drive RAW /completion with our byte-verified template.
Dispatch-bound decode note updated post-SPIR-V: still ~114 GB/s effective vs ~672 — deeper decode-step fusion is the next lever.

What's next

GGUF → running model: remaining dequants (Q4_0/Q5*/Q2_K/Q3_K/Q5_K), name remap + ggml dim-order transpose into the checked-load path, tokenizer-from-GGUF-metadata.
LFM2.5 parity reference via raw llama-server logprobs (P7).
P6 placement plan + per-adapter true VRAM (wgpu-hal).

🤖 Generated with Claude Code

…nizers 0.22) cargo update: lockfile already at latest compatible (0 changes). cargo upgrade --incompatible: sha2 0.11 taken (Digest API unchanged, hub stream-hashing compiles + 93 unit tests green incl. the sha256 etag/hash suite); wgpu 30 and tokenizers 0.23 left on their intentional pins — the burn 0.21 / wgpu 29 / tokenizers 0.22 combo is the parity-validated set, and our direct wgpu handle must stay the version Burn resolves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…parser LFM2.5's real wire format (verified against its chat_template.jinja + model card, which dropped gen-1's <|tool_list_start|> wrapping): - render_with_tools style-splits per template: Hermes (Qwen) unchanged; LFM appends bare tool JSON on a 'List of tools: […]' system line with no default preamble. - Tool results render as real 'tool' role turns (no Hermes user-turn merging); Turn::assistant_tool_calls_lfm re-renders histories in the <|tool_call_start|>[name(k=v)]<|tool_call_end|> shape the model emits. - LFM render strips '</think>'-prefixed reasoning from every assistant history turn but the last (the template's keep_past_thinking=false). - parse_tool_calls_lfm: bounded recursive-descent parser for the Pythonic call list (depth <= 8, <= 64 calls, single/double-quoted strings with \u escapes, Python AND JSON literal spellings, trailing commas) with a byte-offset Syntax error taxonomy. Verification: 109 unit tests green (16 new); REAL-GPU proof (tests/real_toolcall_lfm.rs) — LFM2.5-1.2B greedy-emitted exactly <|tool_call_start|>[get_weather(city="Paris")]<|tool_call_end|> from our rendered prompt and the parser round-tripped it; Qwen2 parity gate re-passed both legs after the refactor (max |dlogit| 2.670e-5, Ollama fp16 greedy byte-identical). fmt + clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…-> 18.4 tok/s) Enable burn's 'vulkan' feature: CubeCL compiles kernels straight to SPIR-V on Vulkan adapters (runtime reports fusion<cubecl<wgpu<spirv>>>); DX12/Metal transparently keep WGSL in the same binary. Zero code changes. Gates (RTX 4070 Ti SUPER, Qwen2.5-1.5B): - Parity byte-identical: top-5 ids exact, max |dlogit| 2.670288e-5 (unchanged), 24-token greedy leg matches Ollama fp16 exactly. - Decode 32 tokens: 2.263 s -> 1.737 s (70.7 -> 54.3 ms/token, +30% tok/s) on f32 AND f16 (criterion, 10 samples). - TTFT 88.4 -> 96.7 ms — inside the 150 ms ceiling; the decode win dominates the governing task-throughput metric. - VRAM peak unchanged: 11.5 GiB whole-card during the real-inference suite (<= 13 GiB budget). - Budget gates re-passed: GPU 108.4 ms / 11.7 tok/s; CPU flex path untouched at 13.2 tok/s. f16 island decode still coherent. - 109 unit tests green; fmt + clippy clean. bench/BASELINE.md re-baselined; decode stays dispatch-bound (~114 GB/s effective vs ~672) — deeper decode-step fusion is the next lever. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…able mummu::gguf (no new deps) reads the llama.cpp header: magic/version (v2/v3 little-endian; big-endian rejected by magic), typed metadata KVs bounded at every step (strings <= 1 MiB, arrays <= 4M entries, nesting <= 2, <= 4096 kvs), and a tensor table validated per entry — known ggml dtype only (F32/F16/BF16, Q4_0..Q8_1, Q2_K..Q8_K), alignment-multiple offsets, whole quantization blocks, unique names. K-quant block layouts (block_size / bytes_per_block) are recorded for the dequant slice. Payloads are located (data_offset + per-tensor offset/byte_len), never loaded here. Verification: 7 unit tests over an in-memory GGUF builder (round-trip, bad magic, unsupported version, truncation, oversize counts, unknown value/tensor types, misaligned/partial/duplicate tensors); REAL-FILE proof (tests/real_gguf.rs) — the local Qwen2.5-1.5B Q4_K_M parses: v3, 26 kvs, 339 tensors, Q4_K token_embd [1536, 151936] matching the 152k vocab array, 198 K-quant tensors, ~1.04 GiB payload; 3B cross-checked (435 tensors). 116 unit tests green; fmt + clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…bit-exact/0.9975 vs true weights Exact ports of ggml-quants' reference dequantizers for everything a Q4_K_M file carries: F32/F16/BF16 widths, Q8_0 (f16 scale + 32 i8), and the 256-element K-quant superblocks — Q4_K (packed 6-bit scale/min pairs, x = d*sc*q - dmin*m) and Q6_K (low-4 + high-2 reassembly, x = d*sc*(q-32)). GgufFile::read_tensor_f32 seeks the located payload and dequantizes whole tensors; unknown types stay a loud error. Verification (the same checkpoint exists locally as bf16 safetensors AND as a llama.cpp Q4_K_M GGUF, so the truth is on disk): - GGUF's F32 output_norm.weight == the bf16 originals BIT-EXACT (bf16->f32 widening is lossless; any layout slip would break it). - Dequantized Q4_K token_embd rows 9707/100000 hit cosine 0.99756/0.99746 vs the bf16 truth — quantization-error territory; a wrong superblock decode reads as ~0. - 5 hand-computed-block unit tests (partial blocks + unimplemented types rejected); 121 unit tests green; fmt + clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… reference caveat - P2: Qwen3.5 (0.8B/2B/4B/9B, Feb 2026) as the next zoo port — 2026 GGUF re-releases fixed tool-calling templates; unsloth GGUFs feed the P3 import path; Ollama qwen3.5:9b (already local) is the parity ref. - P7: the LFM2.5 llama-server logits route must use RAW /completion — llama.cpp's own chat/tool layer rejects LFM2.5's documented <|tool_call_start|> format (ggml-org/llama.cpp#23838), independently confirming the wire format this run implemented in mummu::chat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Automated nightly update that improves Vulkan decode throughput via Burn’s SPIR-V backend, adds LFM2.5 tool-calling support (render + bounded parser), and introduces a new GGUF container reader with initial dequantizers plus real-file proof tests.

Changes:

Enable Burn’s Vulkan SPIR-V compiler path and re-baseline perf numbers/documentation accordingly.
Add LFM2.5 “Pythonic” function-calling rendering + parsing alongside existing Hermes tooling.
Add mummu::gguf (GGUF header parsing + Q8_0/Q4_K/Q6_K dequant + validation) and real-file verification tests.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
ROADMAP.md	Marks SPIR-V + LFM2.5 tool calling + GGUF import/dequant as shipped and records follow-ups.
README.md	Updates feature list and benchmarks to reflect SPIR-V speedup, LFM2.5 support, and GGUF import/dequant.
crates/mummu/tests/real_toolcall_lfm.rs	Adds ignored real-GPU LFM2.5 tool-call emission + parse round-trip proof.
crates/mummu/tests/real_gguf.rs	Adds ignored real-file GGUF header + dequant cross-check against safetensors truth.
crates/mummu/src/lib.rs	Exposes new `gguf` module.
crates/mummu/src/gguf.rs	Implements GGUF header parsing/validation and initial dequantizers (Q8_0, Q4_K, Q6_K).
crates/mummu/src/chat.rs	Adds LFM tool-call style support, Pythonic renderer, and bounded recursive-descent parser.
Cargo.toml	Enables Burn `vulkan` feature and bumps `sha2` to 0.11.
Cargo.lock	Updates lockfile for `sha2` bump and new transitive deps from `burn` Vulkan path.
bench/BASELINE.md	Re-baselines TTFT/decode numbers for SPIR-V path and updates narrative notes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    fn skip_ws(&mut self) {
+        while self.src[self.pos..].starts_with(|c: char| c.is_ascii_whitespace()) {
+            self.pos += 1;
+        }
+    }


+            if i > 0 {
+                system.push_str(", ");
+            }
+            let json = serde_json::to_string(tool).unwrap_or_default();


+            let elements: u64 = dims.iter().product();
+            if elements == 0 || !elements.is_multiple_of(dtype.block_size()) {
+                return Err(bad(
+                    format!("{elements} elements is not whole {dtype:?} blocks"),
+                    &self.path,
+                ));
+            }


+            if tensors.iter().any(|t| t.name == name) {
+                return Err(bad(format!("duplicate tensor name '{name}'"), &self.path));
+            }


+            })?;
+        let out = dequantize(info.dtype, &bytes).map_err(|reason| GgufError::BadTensor {
+            path: self.path.display().to_string(),
+            index: 0,


physics515 and others added 6 commits July 12, 2026 07:03

Copilot AI review requested due to automatic review settings July 12, 2026 13:17

Copilot started reviewing on behalf of physics515 July 12, 2026 13:18 View session

Copilot AI reviewed Jul 12, 2026

View reviewed changes

physics515 merged commit f7aaa7f into main Jul 13, 2026
1 check passed

physics515 deleted the mummu-nightly-2026-07-12 branch July 13, 2026 04:50

Navigation Menu

Nightly 2026-07-12: SPIR-V decode +30%, LFM2.5 function calling, GGUF import (container + K-quant dequant) - #4

Nightly 2026-07-12: SPIR-V decode +30%, LFM2.5 function calling, GGUF import (container + K-quant dequant)#4
physics515 merged 6 commits into
mainfrom
mummu-nightly-2026-07-12

physics515 commented Jul 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

physics515 commented Jul 12, 2026

Dependency freshness

Shipped

1. perf(P6): CubeCL SPIR-V compiler on Vulkan — decode 14.1 → 18.4 tok/s (+30%)

2. feat(P4): LFM2.5 Pythonic tool-calling — template + renderer + bounded parser

3. feat(P3): GGUF container reader

4. feat(P3): GGUF dequantization — Q8_0 + Q4_K/Q6_K superblocks

Verification (final tree)

Research folded into ROADMAP

What's next

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants