Browser engine: write a region of a view into a store as an OME-Zarr (Engine.materialize)

CHANGELOG.md

@@ -146,3 +146,14 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
   every session in one WebAssembly instance that its WebGPU build suspends while it waits.
   Two models' passes, or a second model built while the first computed, could hang the page
   and crash the tab. A page's own passes on the same ORT take the turn through `exclusive`.
+- `Engine.materialize(view, store, options)` writes a box of a served view (or a whole
+  level) into a store as an OME-Zarr of its own, zarr v2 with OME 0.4 or zarr v3 with OME
+  0.5, gzipped, its empty chunks left out, the metadata the engine serves for the view
+  (`web/src/layout.ts`, shared with the routes) with the one level as its one dataset;
+  `directoryStore()` writes into a folder picked with the File System Access API (or the
+  origin's private file system), `memoryStore()` into memory
+  ([formats](docs/concepts/formats.md#a-region-written-to-a-store-browser-engine)). Its
+  chunks share the queue and the kept chunks with the viewer's but run in the background:
+  behind every chunk a client waits for, in all the queue's slots but one, and never dropped
+  as abandoned (`submit`, `claim` and `release` take `background`;
+  [caching](docs/concepts/caching.md#order-of-work-and-requests-given-up-on)).

docs/concepts/caching.md

@@ -208,6 +208,32 @@ one thread, so the first view of a deep chain used one core. The
 browser page keeps the same rules (`web/src/demand.ts`); it learns of a given-up request
 from its streamed reply being cancelled, a service worker being told of nothing else.
 
+The browser page has one class of work more: chunks it computes for itself, in the
+background, to write a region of a view into a store (`Engine.materialize`; see
+[formats](formats.md#a-region-written-to-a-store-browser-engine)). They go through the same
+queue as the viewer's chunks, so a chunk the viewer has computed is taken from the kept
+chunks and one on its way for the viewer is shared. Three rules keep browsing usable while
+a region is written:
+
+* A background claim is never released because a client gave up, so nothing materialize
+  asked for is dropped as abandoned; only its own abort gives its waiting chunks up (and
+  they are dropped if no client wants them either).
+* A job only background claims hold waits behind every job a client waits for, whatever
+  their levels; among themselves, background jobs keep the usual order.
+* Background jobs run in all the queue's slots but one (in the one, when there is only
+  one), so a viewer's request finds a slot free, or the next one to free, rather than
+  waiting for the region's chunks in line before it.
+
+A viewer asking for a chunk that waits in the background brings it forward, and it goes
+back if the viewer gives up on it. Materialize asks for a few chunks at once (as many as
+the queue has slots, by default: the next one is waiting while one is written), not the
+whole region, so the queue holds a few of its chunks at a time however large the region.
+A chunk only materialize asked for is not kept once written (the kept chunks are for
+clients that fetch a chunk again), so a large region does not push what the viewer is
+looking at out of the 256 MB; one a client asked for too, while it was on its way, is kept
+as any other. A staged view's stage outputs are kept as usual (512 MB), so writing a large
+region does cycle through those.
+
 ## What has been computed
 
 Zoomed out, a dataset shows its ops run on the coarse level, not a downsample of the

docs/concepts/formats.md

@@ -73,6 +73,50 @@ view as zarr v2 with OME-Zarr 0.4 (`virtual/<page>/zarr2/<view>/`), the same chu
 readers that predate zarr v3's final spec: GDAL 3.8, as the fires demo runs it in the page
 (gdal3.js), opens a level as a zarr array and translates it to a GeoTIFF.
 
+## A region written to a store (browser engine)
+
+The browser engine also writes a view, or a box of it, into a store as files
+(`Engine.materialize`; see [the design notes](../design.md#another-page-on-the-engine)): a
+folder the visitor picks, the origin's private file system or memory. What it writes is
+the layout it serves for the view, one level of it, so the result opens wherever the view
+did and lands in the same place: the group lists the level written as its one dataset,
+`0`, with that level's scale and translation; the level's array has the level's whole
+shape (a box keeps its place in it), the chunk shape and dtype served, and chunks
+compressed with gzip unless asked not to be; and the chunks of the box, rounded out to
+whole chunks. A chunk of nothing but zeros is left out (zarr reads a missing chunk as the
+fill value, 0), unless asked to be written. A channel axis the view's ops add is the
+first axis and one chunk, as served. As zarr v2 with OME-Zarr 0.4 (the default, which
+more tools read):
+
+```text
+.zgroup        {"zarr_format": 2}
+.zattrs        {...extra attributes, "multiscales": [{"name": <view>, "axes": [...],
+                "datasets": [{"path": "0", "coordinateTransformations": [scale, translation]}],
+                "version": "0.4"}]}
+0/.zarray      {"zarr_format": 2, "shape": <the level's>, "chunks": <as served>, "dtype": "<u2",
+                "compressor": {"id": "gzip", "level": 6}, "fill_value": 0, "order": "C",
+                "filters": null, "dimension_separator": "/"}
+0/.zattrs      {"_ARRAY_DIMENSIONS": ["z", "y", "x"]}
+0/<i>/<j>/<k>  each chunk, gzipped
+```
+
+As zarr v3 with OME-Zarr 0.5:
+
+```text
+zarr.json        {"zarr_format": 3, "node_type": "group", "attributes": {...extra attributes,
+                  "ome": {"version": "0.5", "multiscales": [{... "datasets": [{"path": "0", ...}]}]}}}
+0/zarr.json      the level's array as served, its codecs [{"name": "bytes", ...},
+                  {"name": "gzip", "configuration": {"level": 6}}]
+0/c/<i>/<j>/<k>  each chunk, gzipped
+```
+
+The metadata is written first, so a store whose writing was stopped part way is still a
+valid OME-Zarr, zero where no chunk was written. Extra attributes (a consumer's
+provenance) sit beside the OME ones in the group's attributes, which they cannot replace.
+gzip is what a browser compresses with on its own (`CompressionStream`; level 6 is the
+level it writes at, which no reader needs); the Python server's default, blosc, would need
+an encoder of its own in the page.
+
 ## Meshes, computed when fetched
 
 The `mesh` frontend serves a dataset's surface in Neuroglancer's precomputed mesh formats,

docs/design.md

@@ -516,6 +516,39 @@ as text). What such a page gets, in the order it uses it:
   so the stage is typed with the rest and run in `npm test` through the WebAssembly backend,
   which runs in Node as in a browser without WebGPU, on two models of a hundred bytes
   (`web/test/onnx.test.ts`, the models written by `web/test/fixtures/make_onnx.py`).
+* `Engine.materialize(view, store, options)` writes a region of a served view into a store
+  as an OME-Zarr of its own (`web/src/materialize.ts`), to open again later or copy where
+  other tools read it: a model's output over a box, computed once in the page and kept.
+  The store is anything with `put(path, bytes)`; `directoryStore(dir, root)` makes one of
+  a `FileSystemDirectoryHandle` (a folder the visitor picks with
+  `showDirectoryPicker({mode: "readwrite"})`, in Chrome and Edge, or the origin's private
+  file system, `navigator.storage.getDirectory()`), its subdirectories made as needed and
+  each file written whole through `createWritable()`, and `memoryStore()` one of a `Map`,
+  for tests and for zipping a small result. The options: the view's `level` (the finest by
+  default), a voxel box `lo`, `hi` on it (the whole level by default; rounded out to whole
+  chunks and clipped to the level), `format` (`"zarr2"`, OME-Zarr 0.4, the default, since
+  more tools read it; or `"zarr3"`, OME-Zarr 0.5), `compression` (`"gzip"`, the default, or
+  `null`), `skipEmpty` (a chunk of nothing but zeros left out, by default), `concurrency`,
+  `signal` (an `AbortSignal`), `onProgress` (`done`, `total`, `written`, `bytes`) and
+  `attributes` (the consumer's provenance, beside the OME metadata). It resolves with
+  `{chunks, written, skipped, bytes, aborted}`. The layout is the one the engine serves for
+  the view, from one module (`web/src/layout.ts`) for both, with the one level written as
+  the group's one dataset at `0` with its own scale and translation, so it lands where the
+  view does ([formats](concepts/formats.md#a-region-written-to-a-store-browser-engine)). A
+  single level rather than the pyramid: the engine downsamples nothing itself, and every
+  level would be the model run again on its own grid. The metadata is written first, so a
+  store whose writing stopped is still an OME-Zarr, zero where no chunk was written. The
+  chunks come from the same producer the routes use, through the same queue, but in the
+  background: they wait behind the viewer's, in all its slots but one, and are never dropped
+  as abandoned ([caching](concepts/caching.md#order-of-work-and-requests-given-up-on)), so
+  browsing goes on while a region is written. An abort starts no more chunks, gives up
+  those waiting for their turn, lets the writes under way end and resolves at once with
+  `aborted`; a chunk that fails to compute or be written rejects it with the chunk's key
+  and why, once the writes under way have ended. This is a first piece of
+  [roadmap](roadmap.md) item 2: an explicit write of a box, asked for by the page, not yet
+  the cache on disk that browsing fills. The parts that need no engine (the chunks of a
+  box, the files, gzip, the loop over any producer, the stores) run in `npm test`, the
+  result read back by zarrita (`web/test/materialize.test.ts`).
 * `Engine.unserve(view)` stops serving a view: its infos, producer and kept chunks are
   forgotten and chunks of it waiting to be computed dropped, so a page that serves a model
   again under a new name (viewers cache by URL, so an edit is a new name) does not keep the

docs/roadmap.md

@@ -36,7 +36,12 @@ describe shipped features as future work.
    viewer becomes the job scheduler. (Ordering the work by what clients ask for, and
    dropping what they stop waiting for, has shipped; see
    [caching](concepts/caching.md#order-of-work-and-requests-given-up-on). Inference ops
-   would join its queues.)
+   would join its queues. A first piece has shipped in the browser engine: an explicit
+   write of a box of a view into a store, a folder the visitor picks or memory, as an
+   OME-Zarr, its chunks computed in the background behind the viewer's
+   ([formats](concepts/formats.md#a-region-written-to-a-store-browser-engine)). The cache on
+   disk itself, in either engine, its sharing across workers and the filler that grows it
+   from what is browsed are still to do.)
 3. **Lazy-compute story for non-viewer clients.** Demonstrate dask/tensorstore reading a
    served pipeline and running downstream analysis with no intermediate written. (The
    read itself is shown in [getting started](getting-started.md#read-the-result-from-python-instead-of-a-viewer);

web/src/demand.ts

@@ -8,6 +8,14 @@
 // placeholder while the fine chunk computes) wants the finer. Within a level the first asked
 // for goes first. A few run at once, so the GPU works on one while the next ones read, and the
 // rest cannot flood the network with reads.
+//
+// One rule more than chunkmirage.demand's: work asked for in the background (a page writing a
+// region to a store, Engine.materialize), not for a client waiting on it. A background claim
+// is never released by a client giving up, so the work is not dropped, but a job only
+// background claims hold waits behind every other, whatever its level, and such jobs take at
+// most all the slots but one (one, if there is only one): a viewer's request finds a slot free
+// or the next one to free, while the background work goes on in the rest. A client asking for
+// a job waiting in the background brings it forward, and it goes back if the client gives up.
 
 const SLOTS = 3;  // jobs run at once
 
@@ -26,16 +34,21 @@ export class Cancelled extends Error { constructor(why = "no request wants this
 
 interface Job<T> {
   run: () => Promise<T>; level: number; seq: number;  // seq: when first asked for
-  demand: number;  // requests holding a claim on it
+  demand: number;      // requests holding a claim on it
+  background: number;  // of them, background claims: a job only they hold waits behind the rest
   resolve: (v: T) => void; reject: (e: unknown) => void; onDrop: () => void;
 }
 
 /** Jobs waiting, in the order described at the top; `slots` of them run at once. */
 export class Queue<T = Float32Array> {
-  private slots: number;
-  constructor(slots = SLOTS) { this.slots = slots; }
+  /** Jobs run at once. */
+  readonly slots: number;
+  /** Of them, jobs only background claims hold (`submit`'s `background`): all but one, so a
+   * client's request finds a slot free; one, if there is only one. */
+  readonly backgroundSlots: number;
+  constructor(slots = SLOTS) { this.slots = slots; this.backgroundSlots = Math.max(1, slots - 1); }
   private waiting = new Map<string, Job<T>>();
-  private active = new Map<string, number>();  // running: key -> level
+  private active = new Map<string, { level: number; background: boolean }>();  // running, by key
   dropped = 0;  // waiting fits no request wanted any more, never run
   onChange: () => void = () => {};  // a fit waiting, started, done or dropped
   private seq = 0;
@@ -44,26 +57,38 @@ export class Queue<T = Float32Array> {
   /** Fits running and waiting, per level. */
   byLevel(): { running: Map<number, number>; waiting: Map<number, number> } {
     const count = (levels: Iterable<number>) => { const m = new Map<number, number>(); for (const l of levels) m.set(l, (m.get(l) ?? 0) + 1); return m; };
-    return { running: count(this.active.values()), waiting: count([...this.waiting.values()].map((j) => j.level)) };
+    return { running: count([...this.active.values()].map((a) => a.level)), waiting: count([...this.waiting.values()].map((j) => j.level)) };
   }
 
-  /** A new fit, wanted by one request; `onDrop` runs, synchronously, if it is dropped unrun. */
-  submit(key: string, level: number, run: () => Promise<T>, onDrop: () => void): Promise<T> {
+  /** A new fit, wanted by one request (one in the background, with `background`); `onDrop`
+   * runs, synchronously, if it is dropped unrun. */
+  submit(key: string, level: number, run: () => Promise<T>, onDrop: () => void, background = false): Promise<T> {
     return new Promise((resolve, reject) => {
-      const job: Job<T> = { run, level, resolve, reject, onDrop, seq: ++this.seq, demand: 1 };
+      const job: Job<T> = { run, level, resolve, reject, onDrop, seq: ++this.seq, demand: 1, background: background ? 1 : 0 };
       this.waiting.set(key, job);
       this.pump();
       this.onChange();
     });
   }
 
-  /** Another request wants a fit: if it is still waiting, it counts one more. */
-  claim(key: string) { const job = this.waiting.get(key); if (job) job.demand++; }
+  /** Another request wants a fit (one in the background, with `background`): if it is still
+   * waiting, it counts one more, and a client's claim brings a background job forward. */
+  claim(key: string, background = false) {
+    const job = this.waiting.get(key);
+    if (!job) return;
+    job.demand++;
+    if (background) job.background++;
+    else this.pump();  // in the background before: it may take a slot kept for clients
+  }
 
-  /** A request no longer wants a fit: waiting and wanted by no one, it is dropped unrun. */
-  release(key: string) {
+  /** A request no longer wants a fit (`background`: one that claimed it in the background):
+   * waiting and wanted by no one, it is dropped unrun; wanted in the background alone, it goes
+   * back behind the rest. */
+  release(key: string, background = false) {
     const job = this.waiting.get(key);
-    if (!job || --job.demand > 0) return;
+    if (!job) return;
+    if (background && job.background > 0) job.background--;
+    if (--job.demand > 0) return;
     this.drop(key);
   }
 
@@ -84,18 +109,26 @@ export class Queue<T = Float32Array> {
     while (this.active.size < this.slots && this.waiting.size) {
       let best: [string, Job<T>] | null = null;
       for (const e of this.waiting) if (!best || before(e[1], best[1])) best = e;
-      const [key, job] = best!;
+      const [key, job] = best!, background = inBackground(job);
+      // the best in the background: so is every job waiting, and they have slots of their own
+      if (background && [...this.active.values()].filter((a) => a.background).length >= this.backgroundSlots) return;
       this.waiting.delete(key);
-      this.active.set(key, job.level);
+      this.active.set(key, { level: job.level, background });
       job.run().then(job.resolve, job.reject).finally(() => { this.active.delete(key); this.pump(); this.onChange(); });
     }
   }
 }
 
-/** Which of two waiting fits goes first: the finer level (a client asking for one place at
- * two levels, as a viewer does to show a coarse placeholder while the fine chunk computes,
- * wants the finer one; coarser fits run when nothing finer waits), then the first asked for. */
+/** Whether only background claims hold a job. */
+const inBackground = <T>(job: Job<T>) => job.demand === job.background;
+
+/** Which of two waiting fits goes first: one a client waits for before one wanted in the
+ * background alone; then the finer level (a client asking for one place at two levels, as a
+ * viewer does to show a coarse placeholder while the fine chunk computes, wants the finer
+ * one; coarser fits run when nothing finer waits), then the first asked for. */
 function before<T>(a: Job<T>, b: Job<T>): boolean {
+  const ba = inBackground(a), bb = inBackground(b);
+  if (ba !== bb) return bb;
   return a.level !== b.level ? a.level < b.level : a.seq < b.seq;
 }