[davidackerman]

Why

cellmap-flow's browser app serves ONNX models as engine views (through serveStages). Its users want Run on a region: compute a served view over a box, or over a whole level, and save the result as a real OME-Zarr in a folder they pick with showDirectoryPicker({mode: "readwrite"}). They can then open it again or copy it to a cluster filesystem. Today the engine computes only the chunks a client asks for, and nothing is saved. Walking the chunk grid and writing zarr is engine work, so it belongs here rather than in cellmap-flow.

This is a first, explicit piece of roadmap item 2. The disk cache, sharing across workers and the filler driven by browsing are not part of it, and the roadmap says so.

The API

The new code is in web/src/materialize.ts, plus one method on Engine. engine.ts re-exports the store helpers and the types.

interface WritableStore { put(path: string, bytes: Uint8Array): Promise<void> }
function directoryStore(dir: FileSystemDirectoryHandle, root?: string): WritableStore
function memoryStore(): WritableStore & { files: Map<string, Uint8Array> }
Engine.materialize(view: string, store: WritableStore, options?: MaterializeOptions): Promise<Materialized>

• directoryStore writes into a picked folder, or into the origin's private file system (navigator.storage.getDirectory()). Each subdirectory is made once. Each file is written whole through createWritable() and aborted if the write fails. Paths with . or .. are refused.
• memoryStore writes into a Map, for tests and for zipping a small result.
• The options:
  • level: the finest by default.
  • lo, hi: a voxel box on that level, clipped to the level and rounded out to whole chunks. The whole level by default.
  • format: "zarr2" (the default) or "zarr3".
  • compression: "gzip" (the default) or null.
  • skipEmpty: true by default.
  • concurrency: the queue's slot count by default.
  • signal: an AbortSignal.
  • onProgress: called with {done, total, written, bytes}.
  • attributes: merged into the group's attributes. These cannot replace ome or multiscales.
• The result is {chunks, written, skipped, bytes, aborted}.
• On abort, no more chunks start, the ones still waiting are given up, and the call resolves at once with aborted: true. The chunks already written stay.
• If a chunk fails to compute or write, the call rejects with the chunk's key and the reason, once the writes under way have finished.

What it writes

It writes the layout the engine already serves for the view, as files:

• Format. zarr v2 with OME-Zarr 0.4 by default, since more tools read it, or zarr v3 with OME-Zarr 0.5.
  • zarr2 writes .zgroup, .zattrs, 0/.zarray (compressor: {"id": "gzip", "level": 6}, dimension_separator: "/"), 0/.zattrs, and the chunks at 0/i/j/k.
  • zarr3 writes zarr.json, 0/zarr.json (codecs bytes then gzip), and the chunks at 0/c/i/j/k.
• Metadata first. If writing stops part way, the store is still a valid OME-Zarr, reading as zero where no chunk was written.
• One level, at path "0". The level is the group's only dataset, with its own scale and translation and its whole shape. The box therefore lands where the view has it, and the chunks outside the box are simply missing, which zarr reads as the fill value.
• Compression. gzip through CompressionStream, or none.
• Empty chunks. With skipEmpty, a chunk that is all zeros is not written.
• Channel axes. Channel axes the ops add come first, one chunk each, as the routes serve them.

Scheduling in web/src/demand.ts

This is the part most worth reviewing. submit, claim and release take an optional background flag. It defaults to false, so existing callers (the routes, contrib/blocks.ts) are unchanged. The rules:

• Never dropped as abandoned. A client giving up never releases a background claim. Only materialize's own abort gives up its waiting chunks, and those are dropped only if no client wants them either.
• Clients first. A job held only by background claims waits behind every job a client waits for, whatever its level. Among themselves, background jobs keep the usual order (finest level first, then first asked for).
• A slot kept free. Background jobs use at most max(1, slots - 1) slots, so a client's request finds a free slot, or gets the next one that frees.
• Promotion. When a client claims a background job that is still waiting, it moves ahead to the client queue. If that client gives up, it goes back.
• Bounded queue. Materialize asks for concurrency chunks at a time, never the whole region. The next chunk is waiting while one is being written.

I measured this in headless Chromium, with 4 slots and a fake producer taking 60 ms per chunk:

• With 3 background chunks running and 9 waiting, a viewer fetch through the service worker took 65 ms.
• A whole level of 36 chunks was written to OPFS: 32 written, 4 skipped as empty.
• A chunk the viewer had already asked for was computed once and shared.
• An abort resolved in about 200 ms with nothing left waiting.

All of these ran before the second commit. With the first commit alone, a second materialize of the same level also came entirely from kept chunks, with 0 recomputed. With the second commit, only chunks a client asked for are kept, so a repeat run recomputes the rest.

Second commit: background-only chunks are not kept

If only a background claim asked for a chunk, the chunk is not put in the 256 MB kept-chunk cache once it is written. That cache is for clients that fetch a chunk again. Without this change, a large region pushed what the viewer was looking at out of the cache, in favour of chunks no client might ask for.

The engine records the keys of in-flight chunks that a client asked for, and keeps only those. So a chunk a viewer asked for while materialize had it on its way is kept as before. A staged view's stage outputs (512 MB) are kept as usual, so a large region still cycles through those. caching.md says this.

layout.ts refactor

The builders for the group, array and chunk-key metadata moved from engine.ts into a new web/src/layout.ts, so the routes and the writer share one copy. The served JSON is unchanged. TO_SECONDS and isTime moved with them and are still exported from engine.ts.

Docs

• docs/concepts/formats.md: a new section, "A region written to a store (browser engine)".
• docs/concepts/caching.md: the background class, after the demand.gather paragraph.
• docs/design.md: a bullet under "Another page on the engine".
• docs/roadmap.md: item 2 now names this as a first piece.
• CHANGELOG.md: a line for it.

Known limits and open questions

• gzip runs on the page's main thread, through CompressionStream, at about 10 ms per MB chunk. Should it move to a worker?
• Only one level is written, never a pyramid, and always at path "0". Is a downsampled pyramid worth adding later? Should the level keep its own index instead?
• Engine.materialize has no automated test of its own. Node can't import engine.ts because of its JSON import. The pure parts and the write loop are tested over a fake producer, and the output is read back with zarrita. The method itself was checked on a throwaway Chromium page.

Checks, after rebasing on main with #37 and #38:

• npm run check in web/ passes: both typechecks, 56/56 tests, and the build.
• mkdocs build --strict passes.
• pytest -q tests/test_docs.py passes.
• tests/test_demand.py and tests/test_chained_passes.py pass.