From e4a6c24c47ac53abf070276cd014c58fa1e534c5 Mon Sep 17 00:00:00 2001 From: Justin Icenhour Date: Fri, 28 Aug 2026 12:13:41 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20record=20the=20merge-day=20measurements?= =?UTF-8?q?=20=E2=80=94=20legs=20green,=203.3x=20same-session=20gap,=20pin?= =?UTF-8?q?=20lifted,=20bounded=20head=20retired=20by=20its=20own=20ledger?= =?UTF-8?q?,=20the=20512MiB=20dead-zone=20OOM=20decoded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- ROADMAP.md | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 5cde202..2cefee8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1169,6 +1169,70 @@ a benchmark holds/improves its budget; README perf claims link an artifact. (P5.5): the chunk cost model (launch + linear + quadratic + Neumann-cubic·log terms) and its argmin — feed it three timed chunk sizes and let it replace the fixed C=64 when the measurement disagrees. +- [x] **Parity legs re-run on the merged optimization branch: all six runnable legs green, + byte-for-byte.** *(2026-08-28, gating PR #42's merge.)* `parity_qwen35` ×2 (top-5 exact in + order, 24-token greedy byte-identical, max |Δlogprob| 5.018e-2 — the recorded 5.02e-2 to + the digit), `parity_f16` ×2 (ids exact, greedy byte-identical, 2.677e-1 / 3.782e-1), + `parity_gguf` qwen2+qwen3 (2.6616e-1 / 4.0151e-1 vs recorded 2.6618e-1 / 4.0156e-1 — the + shared attention/decode refactors moved nothing). lfm2 leg: no local fixture (standing); + OLMoE leg skipped (30 GB commit, untouched paths). All `--no-default-features --features + vulkan-spirv` per the fusion-cannot-parity rule. +- [x] **Same-session 27B benchmark, merged build vs ollama: ~3.3×.** *(2026-08-28, back-to-back + on the box, ambient ~3.0-3.4 GiB, 47/64 layers, warm, 64-token greedy.)* mummu warm + 0.71–0.76 s/token best runs (elapsed incl. a ~25-token prefill; three runs per config, + third-request warm — the first burst after a load still runs 2-3× slow, the standing + warm-up curve). ollama 0.32.15 same GGUF, same session: **205–212 ms/token decode** at a + 32%/68% CPU/GPU split (its ctx 65536 config inflates KV and spills more than the recorded + 13%/87% @ 4096, whose quiet record is 121-128 ms). Prefill: ollama ~200 prompt-tok/s; + mummu's host prefill rides the new packed GEMM (m=21 rows at 4.7-5.5 GB/s effective + per the in-situ ledger — one weight stream per batch where the row loop paid one per row). +- [x] **The RAYON=8 pin is lifted: 16 threads, by the re-sweep the pin's own comment asked + for.** *(2026-08-28)* The ANOVA (kernel innocent at 96% of roofline; priority and memory + contention within noise) measured the packed GEMV at 26.3/~30/37.4 GB/s at 4/8/16 threads — + the 8-pin was a ~20% host-bandwidth tax whose SMT rationale described the dead pre-VNNI + kernel. End-to-end 8-vs-16 on the 27B: within noise at n=3 (host work is ~1/3 of the token; + +20% host bandwidth predicts ~7% end-to-end — below this box's session variance), so the + launcher ships 16: microbench-clear, end-to-end neutral-or-better. `RUST_BACKTRACE=1` is + now standing in the launcher (see the OOM entry below — it names the asker for free). +- [x] **The bounded head is OFF in serve, by its own ledger — the instrument caught its first + regression and its own fix's limits in one day.** *(2026-08-28)* First live run: 469 + ms/call, ~91% of rows evaluated (serial walk + k=1024 threshold too deep to prune). PR #43 + fixed both mechanisms (parallel batched bound walk; per-request k — greedy k=1 where the + threshold is the running max)… and the ledger then showed **~92% of rows STILL evaluated at + k=1** (0.67 GiB, ~98 ms/call vs the dense head's 68): this model's near-uniform row norms + swamp the logit margins, so Cauchy–Schwarz tile bounds prune nothing on real head geometry + at any k. Serve keeps the dense head; the machinery stays behind `MUMMU_HEAD_BOUND=1` for + cooperative geometries; the real head win on this model remains GPU pinning when VRAM opens + (`choices::admit_head`). Also fixed en route: `warm_host_twins` now warms the head twin — + it was the one host projection the warm skipped, so every fresh process lazily repacked + ~700 MiB on its FIRST token. +- [ ] **The 512 MiB pool-dead-zone OOM: fully decoded, shape-conditional, self-healing — + requester still unnamed.** *(2026-08-28, found during the benchmark; a 14-agent source hunt + decoded every constant.)* Post-merge builds intermittently panic-loop on `DSD-4-0`: + `failed to reserve 536870912 bytes ... allocating 4210294784`. Decoded: 536,870,912 = 2^29 + is the TRUE byte size of one recurring client-side tensor create (no rounding — cubecl's + ContiguousMemoryLayoutPolicy sums descriptor sizes); it exceeds the largest sliced-pool + max_slice (526,286,848 = heap/4/8) by 10.1 MiB, and `reserve` tries exactly ONE pool, so + the ask can only be served by the terminal pool — whose page = heap/4 = 4,210,294,784 + (this card's device-local heap is exactly 4× that: 16,841,179,136). A 3.92 GiB page next + to ~10.5 GiB of resident weights can never fit → panic per attempt, recovered each time at + ~10× decode cost. Both constants are structural, which is why they never varied with + placement. The trigger is SHAPE-CONDITIONAL: fresh prompt lengths re-fire it, runs on a + warm autotune cache stay clean — consistent with an autotune-workspace candidate paying + the dead zone once per new (kernel-checksum, shape) key, then persisting a winner (the + autotune-cache-is-first-suspect lesson, in a new costume). Falsified along the way, each + by measurement or source: the kv_append same-dtype cast (burn short-circuits it above the + backend — cast.rs:26-31), the chunked-GDN threshold (panics with it restored to t>64), the + bounded head (panics with it off), per-thread pool init (constants don't fit), and CWD/ + cache-discovery (all runs shared the pinned cache). Release binaries strip panic frames, + so naming the asker needs the client-side capture: `RUST_BACKTRACE=1` is now standing in + the launcher (cubecl embeds the alloc-site backtrace in the error Display), and the next + cold-tune firing names the op in the log. Follow-ups: (a) a `[patch.crates-io]` + cubecl-runtime capture at the four `initialize_memory` callers for sizes in the dead zone + (526,286,849..=3.4G) — one debug-build repro names it deterministically; (b) upstream: the + pool ladder has a 10 MiB dead zone at exactly 2^29 on 16 GiB cards — worth an issue with + today's arithmetic; (c) deploy protocol: warm every fresh binary through one request per + shape class before calling it live (the panics self-heal once tuned). - [ ] **What the specs asked for and this box refuses, with reasons.** *(2026-08-28)* NUMA first-touch/interleave and transparent huge pages (P1.4): the reference machine is a single-socket 7950X3D on Windows — no second NUMA domain to interleave across, no THP (and