Nightly 2026-07-11: f16 inference lands (half the VRAM), sha256 downloads, Hermes function calling, PyTorch import, device inventory - #3
Merged
Conversation
…nd, sha1, tinyvec) wgpu 30 / tokenizers 0.23 majors were offered by cargo-upgrade and reverted: both are intentional pins (burn 0.21 + wgpu 29 + tokenizers 0.22 is the parity-validated combination; burn 0.21 itself resolves wgpu 29, a direct wgpu 30 handle would split the dependency tree). All 80 unit tests green, clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s X-Linked-ETag (P3) A redirect-stopped HEAD reads the LFS sha256 from the resolve/ endpoint before the CDN handoff would replace the headers; the GET then stream-hashes every chunk (sha2, SHA-NI), folding a resumed .part prefix in first so a resumed download verifies as one whole file. A wrong-hash .part is deleted, never resumed. FetchOptions::verify_cached re-hashes an existing destination and self-heals exactly once on mismatch (delete + verified refetch); files with no announced sha256 (non-LFS, other hosts) stay length-verified. Real-network proof (tests/real_hub.rs, ignored): fresh 90.8 MB MiniLM fetch stream-verified; clean cache re-verified untouched; a flipped byte mid-file was caught + healed with a full re-stream; a 45.4 MB-seeded resume verified whole. 83 unit tests green (3 new), clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… the VRAM One island sufficed: Qwen-class q·k^T attention scores overflow f16's 65504 max, collapsing softmax to NaN. Scores + mask + softmax now compute in f32 (per-tensor cast, Burn 0.21 multi-dtype — llama.cpp pins the same matmul to f32), and the probabilities return to the ambient dtype for the value matmul. Burn's RmsNorm already reduces in f32 upstream; the logit path needed nothing. Verified on the 4070 Ti SUPER: - f16 gate (tests/real_f16.rs) passes all 3 claims: no crash, 6.75 GiB whole-card / ~3.6 GiB runner (vs ~7.9 GiB f32), coherent greedy output. - Parity gate re-passed both legs on f32, where the casts are no-ops: top-5 logits vs Candle max |dlogit| 2.670e-5 (unchanged), 24-token greedy byte-identical to ollama qwen2.5:1.5b-instruct-fp16. - Benches (criterion, new gpu-f16 group): f16 88.0 ms TTFT / 14.1 tok/s — speed parity with f32, VRAM halved. f32 improved: TTFT 100.5 -> 88.4 ms, decode 13.3 -> 14.1 tok/s; budget gates re-passed (GPU 96.8 ms / 10.2 tok/s, CPU 8.66 tok/s). Confirms decode is dispatch-bound, not bandwidth-bound (f16 == f32 speed); SPIR-V is the remaining lever. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se, real-GPU proof
chat gains the tool-use half of the Qwen2.5/Qwen3 chat template:
- ToolSpec + render_with_tools: advertises function signatures in the
system turn's '# Tools' section, byte-matching the trained template
wording ({"type":"function","function":{...}} per line in <tools>).
- Turn::assistant_tool_calls re-renders emitted calls faithfully into
history; Turn::tool_response results merge consecutive tool turns into
ONE user turn of <tool_response> blocks, exactly as the template does.
- parse_tool_calls extracts every <tool_call> block (bounded at 64) plus
the surrounding prose, with a loud error taxonomy (Unclosed, BadJson,
TooMany); plain text parses as (no calls, text), not an error.
REAL-GPU proof (tests/real_toolcall.rs, ignored): Qwen2.5-1.5B, given a
rendered get_weather prompt, greedy-emitted a well-formed
<tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call>
and the parser round-tripped it. 90 unit tests green (10 new), clippy clean.
LFM2.5's bracket notation stays a P4 [ ] item.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hecked-load path burn-store's PytorchStore (pickle/zip reader, built-in PyTorchToBurnAdapter) joins safetensors behind one import surface: - import::weights_file picks model.safetensors first, falls back to pytorch_model.bin, and reports the safetensors name when neither exists. - load_checked generalizes over any ModuleStore, keeping the fail-loud missing/errored-param contract for every format. - MiniLM loads either format through one shared KEY_REMAPS table (the .bin path notes its no-cast limitation: .bin-era checkpoints are f32). - Direct burn-store workspace dep (same 0.21 crate burn re-exports, feature-unified) turns on its pytorch feature. REAL-WEIGHTS proof (tests/real_pytorch.rs, ignored): the Hub's actual MiniLM pytorch_model.bin embeds byte-identically (max |delta| = 0e0, cosine 1.0) to the safetensors copy of the same weights on the flex CPU backend. 91 unit tests green (1 new), clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The P6 planner's device set fills out: DeviceInventory gains CpuInfo (logical cores via available_parallelism; total RAM via GlobalMemoryStatusEx on Windows and /proc/meminfo on Linux, one syscall at inventory time) and each GpuAdapter now records max_buffer_bytes (adapter.limits) — the hard per-tensor allocation bound placement must respect. True per-adapter VRAM capacity is not portably exposed by wgpu; per-API wgpu-hal queries stay a noted P6 follow-up. Dev box readout: 4070 Ti SUPER 4 GiB max buffer on Vulkan / 2 GiB on DX12 (SHADER_F16 true/false respectively), integrated Radeon 2 GiB, CPU 32 logical cores / 127 GiB RAM. 93 unit tests green (2 new), clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR advances Mummu’s real-model “nightly” milestones by landing validated f16 inference (with an f32 attention-score island for numerical stability), adding sha256 integrity verification for Hub downloads (including cache re-verify/self-heal), introducing Hermes/Qwen-style function calling utilities, wiring PyTorch state-dict loading into the shared checked-import path, and expanding device inventory details and benchmarking.
Changes:
- Add streamed sha256 verification for Hub downloads (and optional cache re-verification) plus real-network proof tests.
- Implement Hermes/Qwen tool-use rendering/parsing + a real-GPU proof test.
- Validate f16 inference via an f32 “attention score” island, add PyTorch
.binfallback for MiniLM import, extend device inventory, and update benchmarks/baselines/docs.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| ROADMAP.md | Marks several P3/P4/P6 items complete with empirical proofs and follow-ups. |
| README.md | Updates feature list to reflect validated f16, tool calling, PyTorch import, and richer inventory. |
| crates/mummu/tests/real_toolcall.rs | Adds an ignored real-GPU tool-calling end-to-end proof test. |
| crates/mummu/tests/real_pytorch.rs | Adds an ignored real-network proof that PyTorch .bin matches safetensors embeddings. |
| crates/mummu/tests/real_hub.rs | Adds an ignored real-network proof for sha256 cache verification and resume hashing. |
| crates/mummu/src/nn/attention.rs | Introduces an f32 precision “island” for attention-score computation to prevent f16 overflow. |
| crates/mummu/src/models/minilm.rs | Adds shared key-remaps + weights-file selection and PyTorch .bin loading path. |
| crates/mummu/src/import.rs | Generalizes checked-load over ModuleStore and adds weights_file() selection logic + unit test. |
| crates/mummu/src/hub.rs | Implements sha256 verification (X-Linked-ETag), optional cache re-verify, and resume-prefix hashing. |
| crates/mummu/src/chat.rs | Adds Hermes/Qwen tool-use template rendering, tool-call parsing, and related unit tests. |
| crates/mummu/src/backend.rs | Expands device inventory with max buffer size and CPU cores/RAM reporting + tests/logging. |
| crates/mummu/Cargo.toml | Adds sha2, burn-store, and Windows-only windows-sys dependency for RAM query. |
| crates/mummu-bench/benches/runner.rs | Bench harness now runs both f32 and f16 (when available) sequentially. |
| Cargo.toml | Adds workspace burn-store (pytorch feature) and sha2 dependencies. |
| Cargo.lock | Locks new dependencies/feature resolutions for sha2 and burn-store pytorch support. |
| bench/BASELINE.md | Updates recorded perf numbers and adds an f16 baseline/budget section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+144
to
+150
| fn announced_sha256(url: &str) -> Result<Option<String>, HubError> { | ||
| assert!(url.starts_with("https://"), "refusing non-https url: {url}"); | ||
| let agent: ureq::Agent = ureq::Agent::config_builder() | ||
| .max_redirects(0) | ||
| .http_status_as_error(false) | ||
| .build() | ||
| .into(); |
| opts: FetchOptions, | ||
| mut on_progress: impl FnMut(Progress<'_>), | ||
| ) -> Result<(), HubError> { | ||
| assert!(url.starts_with("https://"), "refusing non-https url: {url}"); |
|
|
||
| #[test] | ||
| fn weights_file_prefers_safetensors_falls_back_to_pytorch() { | ||
| let dir = std::env::temp_dir().join("mummu_weights_file_test"); |
physics515
pushed a commit
that referenced
this pull request
Aug 27, 2026
…gap to ollama 3.4x (was ~18x) Flamegraph + protocol + token anatomy recorded; the live-found dispatch mis-calibration and its 1172->750 ms/token fix; the open in-situ host GEMV inflation; the 8-vs-16 thread null; lm_head as the new #3 item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@-