From 2364ca5b08643642a92b7f489f3de38b7d689bb1 Mon Sep 17 00:00:00 2001 From: David Ackerman Date: Mon, 19 Oct 2026 11:02:58 -0400 Subject: [PATCH] The roadmap's first item, a real inference consumer, has shipped: cellmap-flow serves every model through chunkmirage (cellmap-flow#110 made it the default, #116 deleted its Flask server), each model an Op of its own package with the model's context as its halo, slots for the GPU and a cache_token for the weights, so the item is removed and the rest renumbered; design.md says where it landed (what the port took and what stayed in cellmap-flow) and points at item 1, no longer item 2, for materialize-on-browse --- docs/design.md | 14 ++++++++++---- docs/roadmap.md | 45 +++++++++------------------------------------ 2 files changed, 19 insertions(+), 40 deletions(-) diff --git a/docs/design.md b/docs/design.md index 902ba9f..8257f83 100644 --- a/docs/design.md +++ b/docs/design.md @@ -39,8 +39,14 @@ computation: cellmap-flow is the first consumer, and a realistic test of the design: its server mode serves a model's predictions through chunkmirage's server, and its browser mode is a page on the browser engine ([another page on the engine](#another-page-on-the-engine)), the -last card of the [gallery](demos.md), the one not built here. But chunkmirage is not built around it or any one -project. +last card of the [gallery](demos.md), the one not built here. Its own inference server is +gone: each model is an `Op` of its own package, with the model's context as its `halo`, +`slots` for the GPU and a `cache_token` for the weights, its output cached so that changing +the post-processing after it does not run the model again; model configs, warmup, weight +swaps, job launching and the dashboard stayed in cellmap-flow. What chunkmirage gained for +it is stated without models (ops that shave their halo or add axes, ops at one resolution, +`kind`, sources and routes from other packages, resolvers, `--ready-file`, `cache_token`, +`slots`). But chunkmirage is not built around it or any one project. ### Where it does *not* add value @@ -217,7 +223,7 @@ Three layers, from most to least structured: Planned: distance transform, a `Combine` op taking another pipeline as input. Registration is a source, not an op (see below). 2. **Plugins**: subclass `Op`, declare `name`, `halo`, `cache`, register via the - `chunkmirage.ops` entry point. cellmap-flow's models become one plugin package. + `chunkmirage.ops` entry point. cellmap-flow's models are one plugin package. Params are pydantic fields, so every op ships a JSON schema the UI/MCP can render. Source schemes come through `chunkmirage.sources`, and endpoints of a plugin's own through `chunkmirage.routes` ([API](reference/api.md#routes-of-your-own)); the app can @@ -564,7 +570,7 @@ as text). What such a page gets, in the order it uses it: those waiting for their turn, lets the writes under way end and resolves at once with `aborted`; a chunk that fails to compute or be written rejects it with the chunk's key and why, once the writes under way have ended. This is a first piece of - [roadmap](roadmap.md) item 2: an explicit write of a box, asked for by the page, not yet + [roadmap](roadmap.md) item 1: an explicit write of a box, asked for by the page, not yet the cache on disk that browsing fills. The parts that need no engine (the chunks of a box, the files, gzip, the stitching, the loop over any producer, the stores) run in `npm test`, the result read back by zarrita (`web/test/materialize.test.ts`). diff --git a/docs/roadmap.md b/docs/roadmap.md index 22fbfd6..d88b61e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -3,34 +3,7 @@ Ordered by impact. Items move to the docs proper when they land; this page must not describe shipped features as future work. -1. **A real inference consumer.** Port an existing live-inference server (cellmap-flow is - the obvious candidate) onto an `Op` with `halo` and `cache=True` on a GPU node. Not - because the tool is for that project, but because real models, real data and a real user - are the fastest way to expose the gaps. Reading that server's code against this - one gave the list; the rule for what goes here is that a second consumer with no notion - of models would want it too, stated without model vocabulary. Everything else (model - configs, device slots, warmup, weight swaps, job launching, dashboards) stays in the - consumer, as a plugin package of ops and a source scheme, which the two entry points - make possible with no change here. - - The five changes this needed have shipped: ops that return their block shaved by the - halo and add leading axes ([pipelines](concepts/pipelines.md#ops); `gradient` and the - Gulf Stream fronts demo use both), a `chunkmirage.sources` entry point - ([formats](concepts/formats.md#your-own-schemes-sources-from-other-packages)), `kind` - on `ArrayInfo` (`output_kind`), ops at one resolution with the pyramid downsampled from - them ([pipelines](concepts/pipelines.md#ops-at-one-resolution)), and a token on `/api/*` - ([API](reference/api.md)). So have the hooks an application needs to build on - chunkmirage as an ordinary dependency: datasets built from their name on first request - ([resolvers](reference/api.md#datasets-resolved-by-name)), routes of its own and the app - mounted inside another ([API](reference/api.md#routes-of-your-own)), `serve --ready-file` - for launchers ([CLI](reference/cli.md)), ops whose output depends on state outside their - parameters ([`cache_token`](concepts/caching.md#state-outside-the-parameters)), a limit - on how many calls of an op run at once (`slots`, [pipelines](concepts/pipelines.md#ops)), - reading the nearest level as it is (`input_level`), zero padding, labels told apart from - images by dtype, and remote stores read anonymously first. What is left is the - consumer's own port: its ops as `Op` subclasses, its scripts that read the data - themselves as a source scheme, its launcher starting `chunkmirage serve`. -2. **Materialize-on-browse with a disk cache.** Opt-in, off by default. Back the cache with +1. **Materialize-on-browse with a disk cache.** Opt-in, off by default. Back the cache with a real zarr on local or shared storage so it persists, is shared across workers, and doubles as a partially computed output. Add a background filler that expands outward from requested chunks: the viewer becomes the job scheduler. (Ordering the work by what clients ask for, and @@ -45,11 +18,11 @@ describe shipped features as future work. ([formats](concepts/formats.md#a-region-written-to-a-store-browser-engine)). The cache on disk itself, in either engine, its sharing across workers and the filler that grows it from what is browsed are still to do.) -3. **Lazy-compute story for non-viewer clients.** Demonstrate dask/tensorstore reading a +2. **Lazy-compute story for non-viewer clients.** Demonstrate dask/tensorstore reading a served pipeline and running downstream analysis with no intermediate written. (The read itself is shown in [getting started](getting-started.md#read-the-result-from-python-instead-of-a-viewer); the analysis on top of it is what remains.) -4. **DAG pipelines, multi-source ops.** Named stages with fan-out (one model, many +3. **DAG pipelines, multi-source ops.** Named stages with fan-out (one model, many post-processors) and a `Combine` op taking another pipeline as input. Unlocks masking, the difference of any two sources (two models' outputs, two conditions), registration overlays. (Two pieces have shipped: `stack://` serves images on one grid as channels and @@ -66,7 +39,7 @@ describe shipped features as future work. browser engine, an MCP server or an agent build specs instead of strings. The URL schemes (`register://`, `scene://`, ...) stay as shorthand for the command line and links; `register://` becomes a two-input node rather than a source. -5. **More registration inputs, live landmarks, fusion.** Readers for bigstream/EASI-FISH +4. **More registration inputs, live landmarks, fusion.** Readers for bigstream/EASI-FISH (`affine.mat` plus a deform zarr), BigWarp landmarks and ANTs/ITK fields into the same transform model; landmark pairs drawn in Neuroglancer, fitted to a thin-plate spline and served live; several scene images fused into one volume (for example through @@ -76,19 +49,19 @@ describe shipped features as future work. [`stitch://` sources](concepts/formats.md#stitch-sources-tiles-stitched-by-interest-points-and-ransac) (BigStitcher projects of OME-Zarr tiles; other loaders, multi-view registration and saving the result back to the project are still to do). -6. **Adaptive caching.** Measure stage compute time at runtime and cache automatically when +5. **Adaptive caching.** Measure stage compute time at runtime and cache automatically when it exceeds a threshold, removing the manual `cache` flag. -7. **MCP surface.** `list_ops`, `set_pipeline`, `try` (a spec on one block: its shape, +6. **MCP surface.** `list_ops`, `set_pipeline`, `try` (a spec on one block: its shape, dtype, value range and time, or the traceback), `neuroglancer_link`. Off by default, local only. (The REST API, `/api/events` stream, control page and python-neuroglancer viewer it would wrap already exist; see [Interactivity](concepts/interactivity.md). Ops from source need no tool of their own: the [`python` op](reference/ops.md#your-own-python-the-python-op) carries its code in the spec.) -8. **Deployment hardening.** Shared-cache multi-worker mode. (`--https` with an +7. **Deployment hardening.** Shared-cache multi-worker mode. (`--https` with an auto-generated self-signed certificate, and `--token` on `/api/*`, have shipped; see the [CLI reference](reference/cli.md).) -9. **Own-hosted Neuroglancer with a service worker.** The zero-install browser demo with +8. **Own-hosted Neuroglancer with a service worker.** The zero-install browser demo with WebGPU ops and ONNX Runtime Web inference, sharing the JSON pipeline spec with the Python server. See [FAQ](faq.md#does-this-work-with-neuroglancer-demoappspotcom) for why it cannot target the hosted appspot viewer. A first piece exists: `web/`, TypeScript @@ -127,7 +100,7 @@ describe shipped features as future work. engines cannot drift. Whether the page's WebGPU shaders can also serve Python (wgpu-py), as one implementation of the solver for both, is still to be measured against PyTorch. -10. **Stages on the GPU.** Heavy work gains on a GPU (block matching, models, large +9. **Stages on the GPU.** Heavy work gains on a GPU (block matching, models, large kernels); cheap ops do not: a chunk is read and encoded on the CPU, so an op moved on its own would copy its block there and back for work that takes as long as the copies. So a stage would move its block to the device once, run its ops there and move it back