Roadmap: plan the async/multithreaded runtime transition

[physics515]

Adds a P5 item for moving Mummu off its fully-sync library surface, per request.

Written to the roadmap's usual standard — grounded in what is actually sync today rather than stated as an aspiration:

• decode::generate_loop is a blocking driver handing tokens to an FnMut(u32) -> ControlFlow<()> callback
• ureq is blocking HTTP ("sync like the rest of the library surface", per its own Cargo.toml note)
• backend.rs:126 wraps wgpu's async adapter enumeration in pollster::block_on
• ModelSlot::with serializes inference behind a Mutex
• the crate contains zero async fn, zero .await, and one thread::spawn — in a test

Two facts probed this session (burn 0.21)

Verified with a compile probe carrying a deliberate Rc<u8> control, so a silently-passing check would have been caught — only the control failed.

1. Every model (Qwen2/Qwen3/Lfm2/Olmoe, Gpu and Cpu) and the KV cache Vec<LayerKv<Gpu>> are Send and Sync. Inference state can cross threads, and one owning worker thread per device is available today.
2. burn_store::TensorSnapshot holds an Rc<dyn Fn() -> Result<TensorData, _>> (burn-store-0.21.0/src/tensor_snapshot.rs:53), so the whole import pipeline is !Send — it cannot cross a thread boundary or be held across an .await. A load must run to completion on one thread.

This inverts the naive expectation: the inference side is thread-mobile, the import side is pinned.

It also means cache.rs's header comment is stale — it says "Burn's Param is not Sync, so the loaded value lives behind a Mutex", and the probe says Param is Sync at 0.21. Serializing one GPU is still fine as policy, but the comment currently reads as a type-level prohibition that does not exist. Correcting that, and pinning the real facts as compile-time assertions so a burn bump that revokes them fails the build, is slice (a).

Constraints the item records

App-agnostic. Mummu is shared by laurelane and Nanna. A library that picks tokio and spawns its own tasks imposes that runtime on both — the app-coupling the North Star forbids, and unfixable downstream. So: return impl Future/impl Stream, never spawn, keep sync as the base (sync-over-async deadlocks; async-over-sync does not), runtime deps behind a non-default feature per the jinja-template precedent.

Honest counterweight. Decode is dispatch-bound, not bandwidth-bound (established three ways, including a ~30 % swing from host CPU load alone), so moving work off the submit thread attacks the measured bottleneck — unusual for an async change. But async makes no single decode step faster, and a naive multithreaded executor makes it worse. The item says both.

Slices

(a) fix the stale premise + pin Send/Sync as assertions · (b) pull-based streaming shape beside the callback, no runtime adopted · (c) one owning worker thread per device with a channel API — shared with P6's multi-GPU execution item · (d) async I/O for the downloader · (e) concurrent serving / continuous batching, its own item once (c) lands.

Gate: parity byte-identical, bench/BASELINE.md held, and specifically that qwen2_sampled_streaming_is_seeded_deterministic_and_cancellable does not become flaky — a multithreaded executor is exactly where determinism dies.

---

Docs-only; no code change. Not merged — that's your call.

[Copilot]

Pull request overview

Adds a new P5 roadmap entry that documents Mummu’s current fully-synchronous surface area and lays out a staged plan (with constraints and gating criteria) for transitioning toward an async/multithreaded architecture without coupling the library to an application runtime.

Changes:

• Adds a detailed P5 checklist item describing today’s blocking decode/import/runtime touchpoints and why async/multithreading is a lever (and where it isn’t).
• Records key type-system constraints (thread-mobile inference vs !Send import pipeline) and calls out a stale premise to fix early.
• Proposes shippable slices (a–e) plus a determinism/parity gate for the transition.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.