Nightly 2026-08-06: flash attention measured and rejected, fallback chat renderer, and the f16 numbers were f32

[physics515]

Autonomous nightly run, 2026-08-06. Five increments; every one green on cargo fmt, cargo clippy --all-targets (no warnings from our crates), cargo test, cargo build, and the perf budgets.

The headline is not a feature: the f16 performance numbers this repo has recorded since 2026-07-11 were f32 runs wearing an f16 label. Real f16 decode is 2.9x faster than f32, not equal to it, and three conclusions built on that measurement are retracted below.

1. Dependency freshness (35da9ea)

cargo update moved 8 transitive crates: aho-corasick 1.1.4→1.1.5, android_system_properties 0.1.5→0.1.6, macerator 0.3.3→0.3.4, macerator-macros 0.1.5→0.2.0, minijinja + minijinja-contrib 2.21.0→2.22.0, regex-automata 0.4.16→0.4.18, zlib-rs 0.6.6→0.6.7. macerator is burn-flex's SIMD layer and minijinja backs the template byte gate, so both touch load-bearing paths.

cargo upgrade --incompatible offers exactly one bump — wgpu 29 → 30, held for the documented reason: wgpu is not ours to pick, burn 0.21 resolves 29 transitively, and our direct handle exists only so the startup adapter probe speaks the same wgpu the backend does. Taking 30 alone would compile two wgpu copies and probe a different one than burn uses. It unblocks with a burn bump.

Pin watch: burn's newest tag is still 0.22.0-pre.1 (0.21.0 latest stable) → the P0 migration item stays gated. tokenizers 0.23.1 is current.

2. CubeCL flash attention: evaluated, measured, rejected (7d06519)

Closes the ROADMAP perf item. burn 0.21 exposes it as tensor::module::attention, which the wgpu backend routes through an autotune set of cubek flash variants, and it is a genuine drop-in:

• scale: None asks for the op's own 1/sqrt(head_dim) default — the same factor, and passing it explicitly would silently disqualify the flash kernel (burn-cubecl routes any custom scale to the unfused fallback);
• is_causal: true reproduces our additive mask exactly — the boundary aligns bottom-right, col > row + (seq_k − seq_q), which is the KV cache's rule at every past;
• the f32 island survives inside the kernel (AccumulatorPrecision::Strict(F32)), so the reason f16 attention doesn't NaN is preserved rather than discarded.

Implemented, proven equivalent by a unit test against the explicit formulation written out longhand, then A/B'd (criterion, idle card, two runs per arm) — and reverted:

metric	explicit (shipping)	fused SDPA	delta
f32 TTFT (36 tok)	90.9 ms	91.9 ms	+1.1 %
f32 prefill @ 2048	593 ms	629 ms	+6.0 %
f32 decode	60.0 ms/token	61.1 ms/token	+1.8 %
f16 TTFT (36 tok)	20.4 ms	16.7 ms	−18 %
f16 prefill @ 2048	210 ms	164 ms	−22 %
f16 decode	20.5 ms/token	22.9 ms/token	+11 %

The one real win is f16 prefill, where the accelerated plane matmuls have tiles to fill. Decode is seq_q = 1 — a matvec with no tile reuse, where flash is overhead and (leading hypothesis) an opaque node the Fusion backend cannot absorb the way it absorbs the explicit chain. Taking only the winning quadrant would fork the hottest leaf function on dtype, on a path no strict parity gate covers; that is now its own ROADMAP item with the f16 parity leg it needs listed as a prerequisite.

Kept, because it is what made the evaluation decisive: a permanent ttft_prefill_2048 row in the criterion bench and in the budget gate. The ~36-token prompt materializes a 62 KiB scores tensor; 2048 tokens materializes 201 MiB — only the second length can see an attention change at all. Recorded 593 ms f32 / 210 ms f16, budget 900 ms.

3. Fallback chat renderer for un-ported models (f75c2ec)

Closes the P3 item "General fallback chat renderer via hf-chat-template" — payoff (2) of the evaluation whose payoff (1) (the byte gate) proved the crate reproduces transformers.apply_chat_template byte for byte. New mummu::template, behind the non-default feature jinja-template. Both open questions decided:

• The dependency is promoted from dev-only to an optional runtime dependency (same crate, same 0.2.1 the gate already trusts). A default build still carries no Jinja engine, so the from-scratch ethos holds for the zoo; a consumer that must run an un-ported checkpoint opts in.
• The selection rule ships as a value: Renderer::for_checkpoint(family: Option<ChatMl>, dir) takes the family renderer when the caller has one, falls back to ImportedTemplate otherwise, and deliberately does not second-guess a family renderer by reading the template — the gate pins those bytes, and a checkpoint repackaged with a foreign template is caught at load by the tokenizer.rs consistency gate.

API mirrors ChatMl (render / render_with_tools) plus render_with_tools_json, because the tools shape is genuinely open: mainstream templates unpack the transformers {"type":"function","function":{...}} wrapper, LFM2.5's wants it bare. Bounded and fail-loud throughout (Absent / Jinja / TooLarge at 8 MiB / BadTool) — a runaway template trips the byte bound rather than returning a prompt nothing can tokenize.

One additive model change: chat::Turn gains tool_calls: Vec<ToolCall>, populated by assistant_tool_calls{,_lfm} beside the rendered content. The family renderers never read it (prompt bytes unchanged by construction and by the gate); the imported path passes calls as data so an arbitrary template writes its own markers instead of inheriting Hermes' <tool_call> wrapping, which would double-wrap under any other convention.

Proof: 7 unit tests over a toy Jinja template (roles, eos_token from the config, tool wire shape, structural calls, absent/broken template, and a render bomb hitting the byte bound) — no fixture needed — plus tests/imported_render.rs on real checkpoints: byte-identical to ChatMl::qwen3() on plain 142 B / tools 748 B / FC history 324 B, and to ChatMl::lfm2() on plain 157 B / tools 379 B, the LFM leg also exercising the standalone chat_template.jinja fallback and the bos_token injection the gate previously had to hand-inject.

4. Research folded (28c5d7e)

• burn 0.22 ships graph capture, explicitly to cut CPU-side launch overhead — the exact bottleneck the dispatch-bound item has chased since July, and the one lever that attacks it generically rather than one kernel at a time. Industry framing matches our numbers: ~5–10 µs CPU per launch, hundreds per forward, 20–40 % of batch-1 decode. Also arriving: LoRA/QLoRA in-framework (P10 becomes wiring), remote backend multi-device + client-side operation-graph caching (P6), dequant→op→quant fallbacks and BitNet b1.58 calibration (P9).
• Speculative decoding, recalibrated — on consumer GPUs a small-draft speculation is frequently a net loss: 7B target + 0.5B draft measured 0.27x on an RTX 5060 Ti (flipping to 1.4x only at a 14B target), and a public 19-configuration llama.cpp study on Qwen3.6-35B-A3B found no variant achieving net speedup on an RTX 3090 while vLLM got +27.5 % on the same hardware. So: build the MTP-head route, gate the draft-model route on a per-(target, draft, hardware) measurement, and expect the batched verify to be where winners differ from losers.
• Keep-quantized kernels — LlamaWeb's split is corroborated from CUDA (Marlin-class kernels unpack int4 to f16 in the register file, no intermediate), so it is the general answer, not a WebGPU workaround; its interface (a scheme = an unpacker + a dequant routine, downstream kernels format-agnostic) is the shape to copy for our GGUF reader.

5. The f16 rows were f32 — bisected, retracted, gated (aeaee7a)

The control runs in increment 2 came back far from the 2026-07-12 record, so two old trees were checked out and benched on the same idle machine the same day:

tree	f32 decode	f16 decode
e44debf (2026-07-12, recorded 54.3 / 54.5)	60.1 ms/token	60.2 ms/token
c1826e7 (just before the 2026-07-30 dtype pinning)	60.0 ms/token	panics DTypeMismatch
HEAD	60.0 ms/token	20.5 ms/token

(1) f32 never regressed. The same 2026-07-12 code reads 60.1 ms/token today, so the 54.3 recorded then was machine state, not a Mummu change. f32 decode has been ~60 ms/token on this card the whole time.

(2) The old f16 rows were f32 runs wearing an f16 label. The criterion bench builds Gpu then GpuF16 in ONE process; before the dtype pinning the f32 leg locked the per-device default dtype policy and the "f16" model ran in f32 — which is precisely why those rows matched f32 to a tenth of a millisecond (70.9 vs 70.7, then 54.5 vs 54.3) and were believed. Once the loaders took target_float from the TYPE (2026-07-23) the mismatch turned loud, which is the DTypeMismatch at c1826e7; the 2026-07-30 pinning fixed it. Real f16 decode is 2.9x faster and always was — the harness could not see it.

Retracted as a result: "f16 buys VRAM, not speed", the f16 half of "decode is dispatch-bound", and SPIR-V's "+30 % on BOTH dtypes" (only their f32 halves were ever measured). The f32 dispatch-bound reading stands on its own evidence — the ~30 % swing from host CPU load on an idle GPU. The f16 VRAM figures are unaffected: they come from real_f16.rs, one alias per process, always genuinely f16.

Guard so this cannot recur silently: mummu-bench/tests/budget_f16.rs, an f16 budget gate in its own test binary, which asserts logits.dtype() == F16 before believing a single number. A gate that cannot tell which precision it measured is not a gate. Its budgets are set against what it measures (21 ms TTFT / 16.9 tok/s, budgets 60 ms / 12 tok/s) — and that gap is itself a new finding: a cold f16 session's first 32 tokens run at roughly f32 speed, because a single burst never leaves autotune. Both numbers are recorded together; closing the gap with a persisted CubeCL autotune cache is a new ROADMAP item.

Verification

• cargo fmt --check clean; cargo clippy --all-targets clean in both feature configurations (no #![allow] added anywhere).
• 202 lib unit tests by name on a default build, 209 with jinja-template; 3 load-gate, 10 template-gate legs (all byte-identical, unchanged), 3 imported-render legs.
• Parity re-passed, bit-identical: Qwen3-0.6B Q4_K_M vs llama.cpp on the same file — top-5 ids exact in order [151667, 151644, 151645, 99966, 131545], 24-token greedy byte-identical, max |Δlogprob| 4.015608155114805e-1 (unchanged to the last digit).
• Real GPU: Qwen3-0.6B greedy-emitted a clean parseable <tool_call> for get_weather(city=Paris) on the 4070 Ti SUPER after the Turn change.
• Budgets, idle card: f32 99.1 ms TTFT / 12.7 tok/s / 592 ms prefill@2048; f16 21 ms / 16.9 tok/s (new gate); CPU 16.6 tok/s. bench/BASELINE.md re-baselined with the three-tree bisect table.

What's next

Ranked by what this run learned: (1) burn 0.22 graph capture the moment 0.22.0 stabilizes — it is now the named lever for the f32 dispatch problem, and it reorders the rest of the perf section if it lands; (2) the persisted autotune cache, which is what stands between a cold f16 session and its 2.9x; (3) flash attention for f16 prefill, once an f16 parity leg exists to cover it; (4) OLMoE from HF safetensors, and the Qwen3.5 hybrid (qwen35) architecture port.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

There are a few correctness/robustness issues in the new template/test code paths (including a panic-prone UTF-8 diff helper and non-erroring/opaque failure handling) that should be addressed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds an optional, runtime Jinja-based fallback prompt renderer for checkpoints without a hardcoded family renderer, and tightens performance measurement/gating (including a new long-context prefill metric and an f16 budget gate that cannot silently measure the wrong dtype). Documentation and baselines are updated to reflect the flash-attention evaluation and the corrected f16 numbers.

Changes:

• Introduce mummu::template (feature jinja-template) for rendering a checkpoint’s imported chat_template via hf-chat-template.
• Extend chat turn representation with structural tool_calls to support template-driven tool-call marker rendering.
• Add new perf coverage: ttft_prefill_2048 benchmark/budget row and a dedicated f16 budget test binary; refresh baselines and nightly research notes; update transitive deps.

File summaries

File	Description
ROADMAP.md	Documents flash-attention evaluation outcome, f16 measurement correction, and next perf levers (graph capture, autotune cache).
README.md	Adds user-facing description of the optional fallback chat renderer and its guarantees/bounds.
crates/mummu/src/template.rs	New optional template renderer module (ImportedTemplate + Renderer) using hf-chat-template.
crates/mummu/src/chat.rs	Adds Turn::tool_calls and exposes MAX bounds within crate for template renderer reuse.
crates/mummu/src/lib.rs	Conditionally exports template module behind jinja-template.
crates/mummu/Cargo.toml	Adds jinja-template feature and optional runtime dep on hf-chat-template (keeps dev dep for gates).
crates/mummu/tests/imported_render.rs	New ignored integration test validating imported-template rendering against family renderers on real checkpoints.
crates/mummu-bench/benches/runner.rs	Adds criterion row ttft_prefill_2048 to make attention formulation changes measurable.
crates/mummu-bench/tests/budget.rs	Adds long-context prefill budget assertion and logging.
crates/mummu-bench/tests/budget_f16.rs	New dedicated f16 perf gate in its own test binary with an explicit dtype assertion.
bench/BASELINE.md	Re-baselines f32/f16 and records the corrected dtype findings + flash-attention A/B results.
Cargo.toml	Adds workspace dep for hf-chat-template.
Cargo.lock	Updates transitive dependencies from cargo update.

Review details

• Files reviewed: 12/13 changed files
• Comments generated: 4
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.