Nightly 2026-08-09: f16 becomes a verified path - and immediately kills flash-attention prefill

[physics515]

Nightly run 2026-08-09. Seven increments, all green, all landed on this branch.

The through-line: the f16 path went from live to verified, and that verification immediately
earned its keep by killing a change the previous run had wanted to ship.

1. Dependency freshness

cargo update moved 24 transitive packages to their newest Rust-1.99-nightly-compatible versions
(notably tracel-ash 0.39.3 → 0.39.5+sdk1.4.357 on the Vulkan path, ureq 3.3 → 3.4, minijinja
2.22 → 2.23 under the template gate, thiserror 2.0.19 → 2.0.20, zerocopy 0.8.55 → 0.8.56). No
manifest edit was needed: cargo upgrade --incompatible offers only wgpu 29 → 30, the standing
intentional pin (burn 0.21 resolves wgpu 29 transitively, so 30 unblocks with a burn bump, not a
cargo upgrade), and every other direct dep is already at its newest stable.

burn stays 0.21.0 — 0.22.0-pre.1 (2026-07-29) is still the only 0.22 tag, so the P0 migration
item stays gated. cubecl likewise still 0.11.0-pre.1.

Both GPU budget gates read low on their first post-bump run (9.1 and 6.2 tok/s) and recovered on
re-run — the documented autotune-invalidation transient, and, as increment 2 discovered, not
actually a transient at all.

2. The f16 cold-start tax is exactly one 32-token burst

Closes "close the f16 autotune warm-up gap: 16.9 tok/s cold vs 48.8 steady" — and the route the
item proposed rested on a wrong premise.

CubeCL already persists autotune across processes by default ([cubecl.autotune] cache defaults
to target; this repo's live cache is crates/mummu-bench/target/autotune/), so a global/file
location changes where the cache lives, never whether tuning carries. What a cold process still
pays is kernel compilation + pipeline creation, which the wgpu runtime caches nowhere — cubecl
0.10 wires CompilationCache for the CUDA and HIP runtimes only (read from source), so
[cubecl.compilation] cache is inert on our path.

New mummu-bench/tests/warmup_f16.rs measures the whole curve in one process — 8 bursts × 32 decode
steps, each burst a fresh cache + untimed prefill so KV length stays constant (decoding 256
consecutive tokens instead confounds warm-up with attention length; measured, it reads ~20 % low by
the last burst):

cumulative tokens	32	64	96	128	160	192	224	256
run A, tok/s	12.5	37.4	38.8	36.7	38.1	37.4	36.8	37.6
run B, tok/s	16.3	41.1	39.7	39.3	40.3	40.6	41.3	40.3

The first 32 tokens cost 2.5–3.0×, and the curve is flat from token 33 on. So budget_f16.rs's
16.9 tok/s is not a mystery — it is the first burst.

Shipped CausalLm::warm_up(probe_ids, steps, device): one prefill plus steps greedy decode
steps on a throwaway cache, every step's argmax read back (an unsynchronized warm-up returns before
the GPU runs anything), bounded by MAX_WARM_UP_STEPS = 256, default-implemented so every zoo model
inherits it beside sanity_check. REAL-GPU proof in its own binary (warmup_api_f16.rs — warm-up is
a once-per-process effect, so a second test in one binary would prove nothing): after a 4.21 s
warm_up(&ids, 32), a cold process's first burst runs at 41.9 tok/s against the next burst's
41.0 — ratio 1.02× where un-warmed it is 0.33×.

Two findings folded into bench/BASELINE.md:

1. A stale autotune cache silently cost 21–27 % of f16 decode. The day's first budget run
   happened on a contended machine; autotune tuned under that contention, wrote its picks, and every
   later process loaded them and never re-tuned. Deleting the cache: f16 decode_32_tokens
   1.0279 s → 0.8109 / 0.8371 s, while f32 was unmoved (1.9646 → 1.9797 s).
2. The gap's other half — "48.8 tok/s steady" — is not reproducible. Criterion measures 36.8
   today, and the same numbers come back on the pre-cargo update lockfile, so it is machine state,
   not a regression. Measured the mechanism directly: +22 pp of host CPU load costs f32 +3.2 % and
   f16 +9.8 %, because f16 does ~3× less GPU work per dispatch.

The README's retracted f16 claims are corrected while here: "identical speed" (an f32 run mislabelled
f16, retracted 2026-08-06 in BASELINE.md but never fixed in the README) becomes the measured ~2.3×.

3. f16 is now a parity-verified path

Every strict gate in the repo ran f32 (or GGUF-dequantized-to-f32). The f16 path had liveness only.
New tests/parity_f16.rs closes that with the honest shape — llama.cpp on the SAME Q4_K_M file, our
side on GpuF16 — and both legs passed on the first run:

leg	top-5 ids	greedy	max |Δlogprob|	f32 twin
qwen2-f16	exact in order	byte-identical	2.5284926197284596e-1	2.66e-1
qwen3-f16	exact in order	byte-identical (incl. <think>)	3.938617118639698e-1	4.02e-1

Both f16 numbers come in below their f32 twins, so f16 adds nothing measurable on top of the
reference's own Q8_K activation-quantization noise — the f32-softmax attention island is doing exactly
what it was added for. Own test binary (one dtype alias per process). The only refactor: the
comparator moved from parity_gguf.rs into a shared tests/gguf_compare/ module with explicit
port + tolerance parameters; f32 legs re-ran unchanged (qwen3 bit-identical at
4.015608155114805e-1).

Fixture note: the Qwen2.5-1.5B Q4_K_M GGUF (1.07 GB) was not cached locally and was fetched.

4. Flash attention for f16 prefill — rejected on correctness

Closes "adopt flash attention for f16 prefill only, once f16 has parity coverage". With (a) in hand,
(b) was implemented exactly as scoped: use_fused_attention(t, ambient, masked) gating on
t > 1 && f16 && masked, picking between a fused attention(.., is_causal: true) and the existing
chain, plus a CPU-backend unit test holding the two formulations to each other at four (past, t)
pairs. The win reproduced — two runs per arm, f32 as an untouched control:

metric	explicit (control)	fused f16 prefill	delta
f16 TTFT (36 tok)	24.9 / 24.6 ms	21.0 / 20.8 ms	−15.6 %
f16 prefill @ 2048	240.6 / 239.5 ms	224.2 / 221.3 ms	−7.2 %
f16 decode	868.7 / 792.8 ms	822.3 / 791.1 ms	noise (path not taken)
f32 TTFT / prefill / decode	98.2 ms / 597.3 ms / 1.9787 s	98.0 ms / 598.5 ms / 1.9877 s	control, unmoved

Then the parity gate killed it. Qwen2.5-1.5B on GpuF16 returns non-finite logits through the
fused kernel, while the identical weights through the explicit chain are llama.cpp-identical. The
2026-08-06 evaluation assumed the f32 score island survives inside the kernel
(AccumulatorPrecision::Strict(F32)); it does not, for the very model whose q·kᵀ overflow motivated
that island. It is also model-dependent — Qwen3-0.6B passed the same fused path — which is
precisely what makes it unshippable in a shared leaf function: correct for narrow models, silently
NaN for wide ones. Reverted; f16 parity re-passed green after the revert.

Standing lesson recorded: run the parity leg before the A/B. An evaluation without a parity gate
behind it would have shipped this.

5. mummu::tune — throw away a bad autotune cache

Route (a) of the hazard measured in increment 2. autotune_cache_dir() reports where CubeCL will
persist picks, read out of the very config CubeCL itself discovers
(CubeClRuntimeConfig::get().autotune.cache.root() + the autotune segment CacheOption::name
adds) rather than a hardcoded copy of the discovery rule; autotune_cache_report() measures it;
clear_autotune_cache() removes it and returns what it removed — the "re-tune GPU kernels" action a
settings UI needs.

Bounded and fail-loud: a tree deeper than 8 levels or wider than 65 536 files is
TuneError::Implausible rather than a long walk or a wide delete, and every path the module touches
ends in the autotune segment by construction (asserted before any removal). Documented honestly —
it takes effect on the next process.

One new direct dependency, cubecl-runtime 0.10, the same version burn 0.21 already resolves through
burn-cubecl (feature-unified, no compile-time cost; the burn-store/wgpu precedent). REAL-GPU proof
(tests/real_autotune_cache.rs): clears the cache, runs four 512² matmuls with readbacks, finds
3 files / 8 303 bytes written to exactly the reported directory, clears them, confirms empty. It
touches crates/mummu/target/autotune and never the bench crate's, so recorded benchmark numbers keep
their own tuning.

6. Research folded

• Pin watch: burn 0.22 still pre-release, cubecl still 0.11.0-pre.1, tokenizers 0.23.1 current.
• cubecl 0.11 adds CUBECL_AUTOTUNE_CACHE (PR #1423, "Disable persistent tune cache option") — an
  env var that bypasses persistent read and write. That is a cleaner route (c) for the hazard above
  than anything we would build, and worth adopting for mummu-bench the moment it lands: this run's
  21–27 % f16 swing came from exactly the inheritance it disables. cubecl 0.11 also moves autotune
  scoring from latency to throughput (#1422, #1408), so re-measure the hazard's magnitude after the
  bump rather than assuming it survives. CubeCL also documents the persisted caches as a shipping
  artifact ("ship a warm cache with your binary when you know the deployment target") — the mirror
  image of the hazard, and the right shape for a consumer with a fixed build target.
• wgpu ships Device::create_pipeline_cache (gfx-rs/wgpu #5293) explicitly for startup time, so a
  cubecl-wgpu compilation cache is buildable upstream — the thing to look for on the 0.22 / wgpu-30
  bump, and the route that would remove the warm-up cost rather than relocate it.
• P9: a third independent corroboration of the fused-dequant-into-shared-memory/registers split,
  this time from a shipped WebGPU product (PrismML's 1-bit 27B) rather than a paper.

7. mummu::plan — pick the highest float precision that fits

The float half of P6's precision selection. pick_precision(&ModelShape, &DeviceBudget) -> Option<Fit>
returns the highest precision that fits one adapter, or None — the honest "no float tier fits;
this needs quantization or several devices", never a silently-worse tier.
ModelShape::from_decoder takes the config.json numbers and derives KV geometry;
DeviceBudget::from_adapter reads off backend::inventory() and returns None when vram_bytes is
unknown rather than guessing; Fit carries projected/usable bytes and headroom_bytes() — already
the shape the plan/doctor introspection item will render. OVERHEAD_BYTES (1 GiB) and
USABLE_VRAM_FRACTION (0.75) are calibrated against bench/BASELINE.md, not first principles.

7 unit tests pin the decisions to real hardware and real models: Qwen2.5-1.5B projects 7.0 GiB f32 /
3.9 GiB f16 against the measured 8.0 / 3.6; the 15.7 GiB reference card takes f32 and an 8 GiB card
f16; the dev box's own DX12 rows (no SHADER_F16) never get an f16 plan; a 64k context pushes a
12 GiB card down to f16; and OLMoE-1B-7B on a 16 GiB card returns None, matching what BASELINE.md
records as "out of reach until keep-quantized VRAM".

Verification — everything green on the reference GPU

cargo fmt clean · cargo clippy --all-targets no warnings (no #![allow] added) · 216 unit
tests (14 new) · cargo build green.

Parity gates — parity_qwen2 both legs (Candle max |Δlogit| 1.907e-5, Ollama greedy
byte-identical) · parity_gguf qwen2 2.6614442413586614e-1 + qwen3 4.015608155114805e-1
(bit-identical to the recorded value) · parity_f16 qwen2 2.5284926197284596e-1 + qwen3
3.938617118639698e-1 · template_gate 10/10 byte-identical.

Budgets — GPU 106.5 ms TTFT / 11.8 tok/s / 594 ms prefill@2048 · f16 26.1 ms / 15.2 tok/s · CPU
15.5 tok/s · warm-up curve 15.5 → 40.0 tok/s · warm_up API 40.1 vs 40.3 tok/s (1.00×).

Real-model suites — real_qwen3 2/2 · real_inference 4/4 · real_mixed_dtype 1/1 ·
real_toolcall_qwen3 1/1 · real_autotune_cache 1/1.

Not run (fixtures absent locally): parity_lfm2 and the parity_gguf lfm2/olmoe legs need the LFM2
GGUFs and ~60 GB RAM respectively.

What's next

• burn 0.22 when it stabilizes — the single highest-value pending item: graph capture (the
  dispatch-bound decode lever), LoRA/QLoRA in-framework, the remote multi-device backend, plus
  CUBECL_AUTOTUNE_CACHE and possibly a wgpu compilation cache. Re-run every gate; expect the backend
  aliases and dtype helpers to change shape.
• Autotune-cache hazard routes (b) pin-the-location and (c) automatic detection — both likely satisfied
  upstream by 0.22, so re-scope rather than hand-roll.
• P6 placement plan (the multi-device half), P9 keep-quantized VRAM, P2 OLMoE-from-safetensors.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This nightly PR strengthens the f16 execution path by adding parity coverage and a warm-up API, introduces public utilities for managing CubeCL’s autotune cache, and adds an initial VRAM-based precision planner. It also updates documentation/roadmap to reflect the new verification story and measured performance behavior.

Changes:

• Added f16 parity gates and shared GGUF-vs-llama.cpp comparison harness, plus updated docs/roadmap to reflect verified f16 behavior and the rejected fused-attention attempt.
• Introduced CausalLm::warm_up plus real-GPU and benchmark harnesses to quantify and amortize the f16 cold-start cost.
• Added mummu::tune (report/clear autotune cache) and mummu::plan (pick highest float precision that fits VRAM), along with tests.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ROADMAP.md	Records nightly findings (f16 parity, warm-up, autotune hazard, planning) and updates item statuses.
README.md	Updates public claims with measured f16 throughput, parity verification, warm-up API, tuning and planning utilities.
crates/mummu/tests/real_autotune_cache.rs	Adds a real-GPU ignored test validating autotune cache persistence and clearing behavior.
crates/mummu/tests/parity_gguf.rs	Refactors GGUF parity tests to use a shared comparator and explicit port allocation.
crates/mummu/tests/parity_f16.rs	Adds a new f16 parity test binary comparing GpuF16 against llama.cpp on the same GGUF.
crates/mummu/tests/gguf_compare/mod.rs	Introduces shared GGUF-vs-llama.cpp comparator and port allocator.
crates/mummu/src/tune.rs	Adds public API to locate, measure, and clear CubeCL autotune cache with safety bounds and tests.
crates/mummu/src/plan.rs	Adds VRAM-based float precision selection (F32 vs F16) with sizing model and unit tests.
crates/mummu/src/models/qwen2.rs	Adds unit tests covering the new warm_up behavior and step bound enforcement.
crates/mummu/src/models/mod.rs	Adds MAX_WARM_UP_STEPS and default CausalLm::warm_up implementation.
crates/mummu/src/lib.rs	Exposes new plan and tune modules.
crates/mummu/Cargo.toml	Adds cubecl-runtime as a direct dependency for reading CubeCL runtime config.
crates/mummu-bench/tests/warmup_f16.rs	Adds a benchmark-style ignored test measuring f16 warm-up curve over multiple bursts.
crates/mummu-bench/tests/warmup_api_f16.rs	Adds a benchmark-style ignored test proving warm_up eliminates the cold first-burst penalty.
Cargo.toml	Adds workspace dependency on cubecl-runtime = "0.10".
Cargo.lock	Updates transitive dependencies from cargo update and adds cubecl-runtime to mummu’s resolved deps.
bench/BASELINE.md	Documents warm-up curve measurements, stale autotune-cache hazard, and the fused-attention rejection.

Suppressed comments (1)

crates/mummu/src/tune.rs:210

• This test uses a fixed directory name under the global temp directory, which can collide with parallel test runs or stale data from prior runs. Using a unique temp directory (and cleaning it up via a guard) will make the test deterministic and avoid cross-run interference.

    #[test]
    fn measuring_counts_files_and_bytes_across_nested_dirs() {
        let root = std::env::temp_dir().join("mummu-tune-measure-a41c");
        let nested = root.join("0.10.0").join("device-4-0");
        std::fs::create_dir_all(&nested).expect("fixture dirs");
        std::fs::write(nested.join("matmul.json.log"), b"12345").expect("fixture file");

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.