Nightly 2026-07-10: Qwen2.5 + MiniLM parity gates PASS, decode engine, Hub import, registry, benchmarks, burn-flex CPU - #2
Merged
Conversation
…rity-validated pins Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…llama fp16 greedy leg - tools/candle-probe (out-of-workspace, Candle =0.9.1 CPU f32): same-weights top-k logits reference; output committed as a test fixture - tests/parity_qwen2.rs: logits leg (top-5 ids exact, max |dlogit| 2.7e-5, bound 1e-3) + greedy leg (24 tokens byte-identical vs qwen2.5:1.5b-instruct-fp16) - ROADMAP: P2 Qwen2 and both P7 reference items ticked; LFM2.5 reference routes captured as a new [ ] Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…diff 1.2e-7 - minilm-probe bin in tools/candle-probe: Candle BERT + masked-mean-pool + L2-normalize reference embedding, committed as a fixture - tests/real_minilm.rs: parity leg (tokenization drift check, cosine >= 0.99999, per-component bound 1e-4) - ROADMAP: P2 all-MiniLM ticked; README parity claim updated Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- decode::sample_id — temperature/top-k/top-p over an O(vocab) partial select; deterministic via an in-house PCG32 (no rand dependency) - decode::generate_loop — shared driver for Qwen2 + LFM2; on_token ControlFlow callback gives streaming and between-token cancellation in one mechanism; greedy stays on-device argmax - greedy_generate now delegates; Qwen2 parity gate re-run green - real-GPU proof: seeded sampled stream is replay-identical and cancels after 8 tokens (qwen2_sampled_streaming_... in real_inference) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gression gate - benches/runner.rs: TTFT (fresh-cache prefill + first token) 100.5 ms; decode 32 warm-cache tokens 2.414 s -> 13.3 tok/s (f32, criterion) - bench/BASELINE.md: recorded numbers + budgets (TTFT <= 150 ms, >= 10 tok/s, <= 13 GiB whole-card; peak measured 11.9 GiB with ~4 GiB desktop ambient) - tests/budget.rs: opt-in perf gate, passing at 110.4 ms / 11.8 tok/s - decode found dispatch-bound (~83 GB/s effective vs ~672 GB/s card): SPIR-V + f16 levers noted in ROADMAP Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…y gate - mummu::chat — Turn/Role + ChatMl renderer; per-model constructors (qwen2 plain ChatML, lfm2 with <|startoftext|> BOS); bounded render with negative-space asserts (empty history, trailing assistant turn) - parity tests now render their prompts through the template; the fixture prompt equality assert byte-verifies it against the Candle reference (re-run green: top-5 ids + 24-token greedy both hold) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…, clear to free VRAM - cache::ModelSlot<T>: Mutex'd one-model slot (Param isn't Sync); with(key, load, f) reuses on key hit, drop-then-load on switch (the P8 active-model-switch primitive); failed loads never cache - real-GPU proof: two greedy decodes through a static slot did exactly one 3.1 GB Qwen2.5 load Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ache + EOS - models::CausalLm<B>: associated Cache type; required new_cache/forward/ is_eos; provided generate/greedy_generate/first_token over the shared generate_loop (static dispatch, one code path); Qwen2 + LFM2 moved onto it - real_inference tests now share one ModelSlot static: one 3.1 GB load for the suite (was one per test — two concurrent loads could exceed the 16 GB card; suite runtime 56s -> 20s), slot-reuse invariant reworked to be order-independent - parity gate re-run green through the trait Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… progress - mummu::hub (ureq 3, https-only): fetch_file streams via .part with HTTP-Range resume, Content-Length verification, cache-first hits, and bounded chunks; fetch_model pulls config/tokenizer/weights with the safetensors index shard fallback; Progress callback feeds the future P8 settings surface - real-network proofs: all-MiniLM (90.8 MB) download -> checked load -> unit-norm embed; half-seeded .part resumed mid-file (first event past the seed) and finished byte-identical (6 unit + 2 network tests) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tests/real_f16.rs: the standing f16 milestone gate (load Qwen2.5 on GpuF16, decode, assert coherence). Findings: shaders compile + run on Vulkan SHADER_F16 and VRAM halves (11.9 -> 8.7 GiB whole-card), but logits NaN out — the GPU argmax returns the out-of-vocab sentinel 151936; gate stays red-when-run until mixed-precision islands land (new ROADMAP item with the f32-reduction plan) - generate_loop now rejects out-of-vocab ids loudly instead of emitting garbage the tokenizer silently drops; f32 parity gate re-run green Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- mummu::registry: ModelSpec (name/repo/revision/architecture/disk estimate) with traversal-safe validation and serde round-trip; spec.fetch() rides the hub downloader into models_root/<name> - built-in catalog: Qwen2.5-1.5B/0.5B, LFM2.5-1.2B, all-MiniLM — repo ids match laurelane's validated download constants - real_hub network proof is now spec-driven (re-run green: 90.8 MB fetch -> checked load -> unit-norm embedding) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- manage::ModelManager composes the catalog (registry), resumable downloads with per-chunk progress (hub), is_installed, disk_report, and traversal-safe remove behind one root-owning object - active-model switch stays a consumer ModelSlot keyed by manager.model_dir(name); 4 unit tests (state, removal isolation, loud unknown-name errors, empty-catalog rejection) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e parity - Cpu = burn_flex::Flex<f32, i32> (burn-ndarray's designated successor, SIMD + gemm, built-in quantized ops P9 can ride); burn's ndarray feature dropped - gate results: MiniLM embedding parity vs Candle holds on Flex (cosine 0.99999994, max component diff 1.3e-7, equivalent to ndarray) and all 80 unit tests pass incl. the CPU cache-equivalence proofs - new [ ] item: CPU decode tok/s baseline row via the 0.5B catalog tier Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tok/s - 0.5B pulled end-to-end through the registry -> hub path (988 MB, ~13 s) - budget_cpu.rs gate: 8-token warm-cache greedy decode, coherent output required, budget >= 6 tok/s (measured 11.7 on the 7950X3D, f32) - BASELINE.md gets its first CPU row Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR advances Mummu’s nightly roadmap by adding same-weights reference tooling (Candle probes) and formal parity/perf gates, while shipping a fuller runtime surface: decode sampling/streaming, a HuggingFace Hub downloader + registry/manager APIs, and a CPU backend migration to burn-flex.
Changes:
- Add Candle-based probe binaries + committed fixtures to support deterministic parity gating for Qwen2.5 logits and MiniLM embeddings.
- Implement decode sampling/streaming/cancellation via a shared
CausalLmtrait +generate_loop, plusModelSlotprocess-lifetime caching. - Add Hub download + registry/catalog +
ModelManager, plus benchmark baselines and budget regression tests; migrate CPU backend toburn-flex.
Reviewed changes
Copilot reviewed 31 out of 33 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/candle-probe/src/minilm.rs | Candle MiniLM embedding probe emitting JSON fixture |
| tools/candle-probe/src/main.rs | Candle Qwen2-family logits probe emitting JSON fixture |
| tools/candle-probe/Cargo.toml | Standalone (out-of-workspace) Candle probe crate definition |
| tools/candle-probe/Cargo.lock | Lockfile for standalone Candle probe tool |
| ROADMAP.md | Update roadmap status for shipped parity/perf/decode/registry items |
| README.md | Refresh README to reflect new features, parity gates, and CPU backend |
| crates/mummu/tests/real_minilm.rs | Add Candle-fixture parity test for MiniLM embeddings |
| crates/mummu/tests/real_inference.rs | Share a GPU model slot; add sampled/streaming determinism tests |
| crates/mummu/tests/real_hub.rs | Add real-network Hub download + resume proofs |
| crates/mummu/tests/real_f16.rs | Add f16 on-GPU validation test (ignored by default) |
| crates/mummu/tests/parity_qwen2.rs | Add two-leg Qwen2.5 parity gate (Candle logits + Ollama greedy) |
| crates/mummu/tests/parity_lfm2.rs | Route LFM2 prompt wrapping through chat::ChatMl |
| crates/mummu/tests/fixtures/qwen2_5_1_5b_first_logits.json | Commit Qwen2.5 Candle logits fixture |
| crates/mummu/tests/fixtures/minilm_embedding.json | Commit MiniLM Candle embedding fixture |
| crates/mummu/src/registry.rs | Add model registry (ModelSpec, catalog) with validation |
| crates/mummu/src/models/qwen2.rs | Port Qwen2 loader to CausalLm trait contract |
| crates/mummu/src/models/mod.rs | Introduce CausalLm trait w/ shared generate/first_token helpers |
| crates/mummu/src/models/lfm2.rs | Port LFM2 loader to CausalLm trait contract |
| crates/mummu/src/manage.rs | Add ModelManager composing registry + hub + disk/remove |
| crates/mummu/src/lib.rs | Export new modules (cache, chat, hub, registry) |
| crates/mummu/src/hub.rs | Add streaming/resumable Hub downloader with shard-index support |
| crates/mummu/src/decode.rs | Implement sampling + streaming/cancellation driver + deterministic RNG |
| crates/mummu/src/chat.rs | Add explicit ChatML prompt rendering helpers |
| crates/mummu/src/cache.rs | Add ModelSlot process-lifetime model cache (mutex-guarded) |
| crates/mummu/src/backend.rs | Switch CPU backend alias to burn-flex; keep GPU probe policy |
| crates/mummu/Cargo.toml | Add ureq + burn-flex dependencies |
| crates/mummu-bench/tests/budget.rs | Add GPU perf budget regression test (ignored by default) |
| crates/mummu-bench/tests/budget_cpu.rs | Add CPU perf budget regression test (ignored by default) |
| crates/mummu-bench/Cargo.toml | Add bench deps needed for perf gates |
| crates/mummu-bench/benches/runner.rs | Add real-model Criterion benches for TTFT + decode |
| Cargo.toml | Workspace dep updates: pollster 1.0, add ureq and burn-flex |
| Cargo.lock | Lock updates for new deps (burn-flex, ureq, etc.) |
| bench/BASELINE.md | Add recorded baseline metrics + budgets (GPU + CPU rows) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+76
to
+88
| let mut shards: Vec<String> = map | ||
| .values() | ||
| .filter_map(|s| s.as_str().map(str::to_string)) | ||
| .collect(); | ||
| shards.sort_unstable(); | ||
| shards.dedup(); | ||
| if shards.is_empty() || shards.len() > MAX_SHARDS { | ||
| return Err(HubError::BadIndex { | ||
| path: index_path.to_path_buf(), | ||
| reason: format!("{} shards (expected 1..={MAX_SHARDS})", shards.len()), | ||
| }); | ||
| } | ||
| Ok(shards) |
| dest: &Path, | ||
| mut on_progress: impl FnMut(Progress<'_>), | ||
| ) -> Result<(), HubError> { | ||
| assert!(url.starts_with("https://"), "refusing non-https url: {url}"); |
Comment on lines
+67
to
+68
| /// Max |Δcomponent| tolerated between Burn (ndarray f32) and Candle (CPU f32) | ||
| /// on the same weights — both CPU f32, so only reduction-order noise remains. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nightly dev routine — 2026-07-10 (14 increments)
Every commit is individually green (
cargo fmt/clippy --all-targetswith zero warnings /cargo test/cargo build). Final tree: 80 unit tests + 11 real-weights/network tests, all passing (verified by test name against the shared target dir).Dependency freshness
pollster0.4 → 1.0 (the one non-pinned upgrade available).burn/wgpu/tokenizersstay on the parity-validated pins;wgpu30 andtokenizers0.23 exist but are behind the pins by design.ureq3.3 (hub downloader),burn-flex0.21 (CPU backend, see below).Parity gates — the headline
tests/parity_qwen2.rs): top-5 logits match a Candle f32 reference exactly by id with max |Δlogit| 2.7e-5 (bound 1e-3), and a 24-token greedy sequence matchesollama qwen2.5:1.5b-instruct-fp16byte-for-byte on the 4070 Ti SUPER. P2 Qwen2 is now[x].[x].tools/candle-probe(out-of-workspace Candle =0.9.1 side harness) emits committed fixtures; the fp16 Ollama tag is pulled. LFM2.5 still lacks a same-weights reference (Ollama tag is the 8.5B MoE; candle-transformers has no LFM2) — routes captured as a[ ].Features shipped
ControlFlowcallback, sharedgenerate_loopdriver; greedy keeps the on-GPU argmax and re-passed the parity gate. Real-GPU proof: seeded sampled stream replays identically and cancels at 8 tokens.ModelSlot: process-lifetime model cache (load once, switch by key,clear()to free VRAM); real-GPU proof: two decodes, one 3.1 GB load. Also fixed the real-test suite to share one slot (two concurrent 6 GB loads could blow the 16 GB card; suite runtime 56s → 20s).chat::ChatMl(Qwen2 plain / LFM2 BOS-prefixed), byte-verified — the parity gate renders its prompt through it and still matches the Candle fixture.CausalLmtrait: new architectures supply cache/forward/EOS and inherit generate/greedy/first_token; both LLMs moved onto it (static dispatch, one code path).mummu::hub): streaming, resumable (.part+ HTTP Range), length-verified, shard-index aware, per-chunk progress. Network proofs: all-MiniLM (90.8 MB) downloaded → checked-load → unit-norm embedding; interrupted transfer resumed mid-file and finished byte-identical.mummu::registry): declarativeModelSpec+ built-in catalog (Qwen2.5-1.5B/0.5B, LFM2.5-1.2B, MiniLM; repo ids match laurelane's validated constants).ModelManager: catalog + install-with-progress +is_installed+ disk report + traversal-safe remove in one settings-facing surface.Cpu=burn_flex::Flex<f32,i32>(burn-ndarray's designated successor), gated on the MiniLM Candle parity re-passing on Flex (cosine 0.99999994) + all 80 unit tests; ndarray feature dropped.Benchmarks (new:
bench/BASELINE.md+ regression gates)Gates:
mummu-bench/tests/budget.rs(GPU, passed 110.4 ms / 11.8 tok/s) andbudget_cpu.rs(CPU, passed). Finding: GPU decode is dispatch-bound (~83 GB/s effective vs ~672 GB/s card) — SPIR-V compiler + f16 are the levers, noted in the roadmap.f16 validation (P6) — 2 of 3 claims hold
tests/real_f16.rs(standing gate): shaders compile + run on Vulkan SHADER_F16, VRAM drops 11.9 → 8.7 GiB whole-card — but logits NaN out (GPU argmax returns the out-of-vocab sentinel 151936). Added a loud out-of-vocab decode guard; the fix (mixed-precision islands: f32 softmax/RmsNorm/logit reductions) is a new[ ].Research folded into ROADMAP (with links)
spirvcompiler feature (TensorCores + f16 matmul) as the decode-throughput lever.What's next
LFM2.5 same-weights reference (llama.cpp logprobs or HF dump) → its parity gate; f16 mixed-precision islands → re-gate + f16 baseline row; GGUF container import; SPIR-V compiler evaluation; download sha256 integrity.
🤖 Generated with Claude Code