Browser engine: Engine.materialize writes exactly the box, stitched from the served chunks (extent: "box", the default)

CHANGELOG.md

@@ -153,7 +153,17 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
 - `Engine.materialize(view, store, options)` writes a box of a served view (or a whole
   level) into a store as an OME-Zarr of its own, zarr v2 with OME 0.4 or zarr v3 with OME
   0.5, gzipped, its empty chunks left out, the metadata the engine serves for the view
-  (`web/src/layout.ts`, shared with the routes) with the one level as its one dataset;
+  (`web/src/layout.ts`, shared with the routes) with the one level as its one dataset:
+  exactly the box (`extent: "box"`, the default), `hi − lo` voxels with the level's
+  translation moved by `lo` voxels, its chunks the served chunk shape counted from `lo`,
+  each stitched from the served chunks it overlaps (each computed once, walked with the
+  split axis of the most chunks outermost, so a slab across the other split axes is held
+  at a time), those at its far edge stored whole with the fill value past the box; or the
+  level's whole shape, the box's chunks in it rounded out to whole chunks (`extent:
+  "level"`, for several boxes in one store); a box of the whole level is the same store
+  either way; progress counts the array's chunks done and the served chunks computed
+  (`computed`, `needed`); the Python side (`chunkmirage materialize`) writes the level's
+  extent only, for now;
   `directoryStore()` writes into a folder picked with the File System Access API (or the
   origin's private file system), `memoryStore()` into memory
   ([formats](docs/concepts/formats.md#a-region-written-to-a-store-browser-engine)). Its

docs/concepts/caching.md

@@ -231,8 +231,11 @@ whole region, so the queue holds a few of its chunks at a time however large the
 A chunk only materialize asked for is not kept once written (the kept chunks are for
 clients that fetch a chunk again), so a large region does not push what the viewer is
 looking at out of the 256 MB; one a client asked for too, while it was on its way, is kept
-as any other. A staged view's stage outputs are kept as usual (512 MB), so writing a large
-region does cycle through those.
+as any other. So materialize asks for each chunk once: a box written as exactly the box
+(`extent: "box"`, the default) that does not start on chunk edges has each of its chunks
+stitched from the served chunks it overlaps as they come, rather than asking a served
+chunk again for each of the box's chunks it overlaps. A staged view's stage outputs are kept as usual
+(512 MB), so writing a large region does cycle through those.
 
 ## What has been computed
 

docs/concepts/formats.md

@@ -90,7 +90,9 @@ part way is still the dataset, zero where no chunk was written yet; then the chu
 levels asked for (all by default), several at once, each of nothing but zeros left out
 (`--keep-empty` writes them). `--box` takes level 0's voxels along the last axes, `lo:hi`
 each comma separated in C order (`0,0,0:64,512,512`), and every level writes the same
-region on its own voxels, rounded out to whole chunks. From Python,
+region on its own voxels, rounded out to whole chunks, in an array of the level's whole
+shape: what the browser engine writes with `extent: "level"`. Writing exactly the box
+(the browser engine's default, `extent: "box"`) is not supported here yet. From Python,
 `chunkmirage.materialize(pipeline, where, format=, levels=, box=, compressor=,
 skip_empty=, progress=)` does the same for any pipeline, another dataset's chain read
 through `dataset://` among them (`registry.get(name)`), and returns the counts
@@ -105,20 +107,50 @@ The browser engine also writes a view, or a box of it, into a store as files
 folder the visitor picks, the origin's private file system or memory. What it writes is
 the layout it serves for the view, one level of it, so the result opens wherever the view
 did and lands in the same place: the group lists the level written as its one dataset,
-`0`, with that level's scale and translation; the level's array has the level's whole
-shape (a box keeps its place in it), the chunk shape and dtype served, and chunks
-compressed with gzip unless asked not to be; and the chunks of the box, rounded out to
-whole chunks. A chunk of nothing but zeros is left out (zarr reads a missing chunk as the
-fill value, 0), unless asked to be written. A channel axis the view's ops add is the
-first axis and one chunk, as served. As zarr v2 with OME-Zarr 0.4 (the default, which
-more tools read):
+`0`, with that level's scale; the array has the chunk shape and dtype served, and chunks
+compressed with gzip unless asked not to be. What the array covers is the `extent`:
+
+* `"box"` (the default): exactly the box, `hi − lo` voxels (clipped to the level, rounded
+  out to whole voxels, not to chunks), with the level's translation moved by `lo` voxels
+  (the translation is where voxel 0's centre is, so it lands where the view's voxel `lo`
+  does). Its chunks have the served chunk shape, counted from `lo`. The chunks at the box's
+  far edge overhang it and are stored whole, zero (the fill value) past the box, as zarr
+  stores a chunk at an array's edge (zarr v3's regular grid: border chunks have the full
+  chunk size, the fill value recommended outside the array; zarr v2: every chunk has the
+  same shape).
+* `"level"`: the level's whole shape, with the level's translation, so a box keeps its
+  place in it, as several boxes written into one store need; the chunks of the box are
+  written, rounded out to whole chunks, each the chunk the engine serves.
+
+A box of the whole level is the same store either way, byte for byte. Where a box's `lo` is
+off the served chunks' edges along an axis, the box splits the served chunks there: each of
+its chunks is stitched from the served chunks it overlaps, up to two along each such axis,
+and each served chunk is computed once. The served chunks are walked axis by axis, the axes
+the box does not split outermost, then the split ones, the one with the most of the box's
+chunks first, and each is copied into the box's chunks it overlaps, a box chunk written
+once the last of them is in. Only the box's chunks begun and not yet complete are held: a
+slab of them across the split axes but the outermost, a row and one more. With the box's
+chunks n₁ ≥ n₂ ≥ n₃ along three split axes that is n₂ × n₃ + n₃ + 1; along two, n₂ + 1;
+along one, 1; none where `lo` is on the chunk edges on every axis, the box's chunks then
+the served chunks as they are. A box chunk none of whose served chunks holds a value but
+zero takes no memory. So a 512 × 2048 × 2048 box of 64³ chunks off the chunk edges
+everywhere (8 × 32 × 32 chunks) holds 32 × 8 + 8 + 1 = 265 chunks, which for 3-channel
+float32 chunks (3 MiB) is about 800 MiB; with `lo` on a chunk edge along z, 32 + 1. No
+order of the axes holds fewer; putting `lo` on the chunk edges along an axis is what
+shrinks it further.
+
+A chunk of nothing but zeros is left out (zarr reads a missing chunk as the fill value,
+0), unless asked to be written. A channel axis the view's ops add is the first axis and
+one chunk, as served, whole whatever the extent. Progress counts the array's chunks done
+(written or skipped) of its total, and the served chunks computed of those the box needs;
+each only grows. As zarr v2 with OME-Zarr 0.4 (the default, which more tools read):
 
 ```text
 .zgroup        {"zarr_format": 2}
 .zattrs        {...extra attributes, "multiscales": [{"name": <view>, "axes": [...],
                 "datasets": [{"path": "0", "coordinateTransformations": [scale, translation]}],
                 "version": "0.4"}]}
-0/.zarray      {"zarr_format": 2, "shape": <the level's>, "chunks": <as served>, "dtype": "<u2",
+0/.zarray      {"zarr_format": 2, "shape": <the level's, or the box's>, "chunks": <as served>, "dtype": "<u2",
                 "compressor": {"id": "gzip", "level": 6}, "fill_value": 0, "order": "C",
                 "filters": null, "dimension_separator": "/"}
 0/.zattrs      {"_ARRAY_DIMENSIONS": ["z", "y", "x"]}
@@ -130,8 +162,8 @@ As zarr v3 with OME-Zarr 0.5:
 ```text
 zarr.json        {"zarr_format": 3, "node_type": "group", "attributes": {...extra attributes,
                   "ome": {"version": "0.5", "multiscales": [{... "datasets": [{"path": "0", ...}]}]}}}
-0/zarr.json      the level's array as served, its codecs [{"name": "bytes", ...},
-                  {"name": "gzip", "configuration": {"level": 6}}]
+0/zarr.json      the level's array as served (the box's shape, with the box's extent),
+                  its codecs [{"name": "bytes", ...}, {"name": "gzip", "configuration": {"level": 6}}]
 0/c/<i>/<j>/<k>  each chunk, gzipped
 ```
 

docs/design.md

@@ -525,18 +525,32 @@ as text). What such a page gets, in the order it uses it:
   file system, `navigator.storage.getDirectory()`), its subdirectories made as needed and
   each file written whole through `createWritable()`, and `memoryStore()` one of a `Map`,
   for tests and for zipping a small result. The options: the view's `level` (the finest by
-  default), a voxel box `lo`, `hi` on it (the whole level by default; rounded out to whole
-  chunks and clipped to the level), `format` (`"zarr2"`, OME-Zarr 0.4, the default, since
-  more tools read it; or `"zarr3"`, OME-Zarr 0.5), `compression` (`"gzip"`, the default, or
-  `null`), `skipEmpty` (a chunk of nothing but zeros left out, by default), `concurrency`,
-  `signal` (an `AbortSignal`), `onProgress` (`done`, `total`, `written`, `bytes`) and
+  default), a voxel box `lo`, `hi` on it (the whole level by default; clipped to the level),
+  `extent` (`"box"`, the default: the array is exactly the box, translated to its first
+  voxel, its chunks the served chunk shape counted from `lo`, each stitched from the served
+  chunks it overlaps, since a page passing a box expects an array of that box; `"level"`:
+  the array is the level's whole shape and the box's chunks are written, rounded out to
+  whole chunks, for several boxes written into one store), `format` (`"zarr2"`, OME-Zarr 0.4, the
+  default, since more tools read it; or `"zarr3"`, OME-Zarr 0.5), `compression` (`"gzip"`,
+  the default, or `null`), `skipEmpty` (a chunk of nothing but zeros left out, by default),
+  `concurrency`, `signal` (an `AbortSignal`), `onProgress` (`done`, `total`, `written`,
+  `bytes` for the array's chunks; `computed`, `needed` for the served chunks) and
   `attributes` (the consumer's provenance, beside the OME metadata). It resolves with
   `{chunks, written, skipped, bytes, aborted}`. The layout is the one the engine serves for
   the view, from one module (`web/src/layout.ts`) for both, with the one level written as
-  the group's one dataset at `0` with its own scale and translation, so it lands where the
-  view does ([formats](concepts/formats.md#a-region-written-to-a-store-browser-engine)). A
-  single level rather than the pyramid: the engine downsamples nothing itself, and every
-  level would be the model run again on its own grid. The metadata is written first, so a
+  the group's one dataset at `0` with its own scale and translation (moved to the box's
+  first voxel, with the box's extent), so it lands where the view does
+  ([formats](concepts/formats.md#a-region-written-to-a-store-browser-engine)). A box that
+  does not start on the served chunks' edges is not computed again on the box's own grid:
+  the engine keeps no chunk only materialize asked for, so asking a served chunk again for
+  each of the box's chunks it overlaps would compute it up to eight times; instead the
+  served chunks are each asked for once and copied into the box's chunks they overlap, a
+  box chunk written once complete, so about a slab of the box's chunks is held at once. The
+  walk puts the axes the box does not split outermost and the split axis with the most of
+  the box's chunks next, so the slab is the one across the other split axes, the smallest
+  any order of the axes gives (a wide, shallow box walked in C order would hold a slab
+  across its two wide axes). A single level rather than the pyramid: the engine downsamples nothing itself,
+  and every level would be the model run again on its own grid. The metadata is written first, so a
   store whose writing stopped is still an OME-Zarr, zero where no chunk was written. The
   chunks come from the same producer the routes use, through the same queue, but in the
   background: they wait behind the viewer's, in all its slots but one, and are never dropped
@@ -547,8 +561,8 @@ as text). What such a page gets, in the order it uses it:
   and why, once the writes under way have ended. This is a first piece of
   [roadmap](roadmap.md) item 2: an explicit write of a box, asked for by the page, not yet
   the cache on disk that browsing fills. The parts that need no engine (the chunks of a
-  box, the files, gzip, the loop over any producer, the stores) run in `npm test`, the
-  result read back by zarrita (`web/test/materialize.test.ts`).
+  box, the files, gzip, the stitching, the loop over any producer, the stores) run in
+  `npm test`, the result read back by zarrita (`web/test/materialize.test.ts`).
 * `Engine.unserve(view)` stops serving a view: its infos, producer and kept chunks are
   forgotten and chunks of it waiting to be computed dropped, so a page that serves a model
   again under a new name (viewers cache by URL, so an edit is a new name) does not keep the

docs/reference/cli.md

@@ -66,7 +66,7 @@ batch's progress on stderr.
 | `--op`, `-o` | none | op spec, as for `serve` (repeatable) |
 | `--format` | `zarr3` | `zarr3` (OME-Zarr 0.5), `zarr` (zarr v2, OME-Zarr 0.4, consolidated) or `n5` |
 | `--level`, `-l` | every level | a level whose chunks to write (repeatable); every level's metadata is written |
-| `--box` | all of it | level 0's voxels along the last axes, `lo:hi` each comma separated in C order; every level writes the same region, rounded out to whole chunks |
+| `--box` | all of it | level 0's voxels along the last axes, `lo:hi` each comma separated in C order; every level writes the same region, rounded out to whole chunks, in an array of the level's whole shape (the browser engine's exact box, `extent: "box"`, is not supported here yet) |
 | `--chunk`, `--select` | as for `serve` | output chunk shape; non-spatial axes pinned |
 | `--compressor` | blosc (zarr), gzip (n5) | as the format takes it |
 | `--keep-empty` | off | also write chunks of nothing but zeros (left out otherwise: a reader takes a missing chunk as the fill value) |

web/src/engine.ts

@@ -501,14 +501,17 @@ export class Engine {
 
   /** Write a region of `view` into `store` as an OME-Zarr of its own (materialize.ts): one
    * level of the view (`options.level`, the finest by default) over a box of its voxels
-   * (`lo`, `hi`, rounded out to whole chunks; the whole level by default), as zarr v2 with
-   * OME-Zarr 0.4 or zarr v3 with OME-Zarr 0.5, gzipped, its chunks of nothing but zeros left
-   * out. The group and the array are written first, so the store reads as the view from the
-   * start, zero where no chunk is yet; then the chunks, each computed as a client's request
-   * for it is (kept, or shared with one on its way, and kept after) but in the background:
-   * behind every chunk a client waits for, in all the queue's slots but one, and never
-   * dropped because a client gave up on it (demand.ts). `concurrency` chunks are asked for
-   * at once, as many as the queue has slots by default. Resolves with what was written, or
+   * (`lo`, `hi`; the whole level by default), as zarr v2 with OME-Zarr 0.4 or zarr v3 with
+   * OME-Zarr 0.5, gzipped, its chunks of nothing but zeros left out. With `extent` "box"
+   * (the default) the array is exactly the box, translated to its first voxel, each of its
+   * chunks stitched from the view's it overlaps; with "level" it is the level's whole shape
+   * and the box's chunks are written, rounded out to whole chunks. The group and the array
+   * are written first, so the store reads as the view from the start, zero where no chunk is
+   * yet; then the chunks, each of the view's computed once, as a client's request for it is
+   * (kept, or shared with one on its way, and kept after) but in the background: behind
+   * every chunk a client waits for, in all the queue's slots but one, and never dropped
+   * because a client gave up on it (demand.ts). `concurrency` chunks are asked for at once,
+   * as many as the queue has slots by default. Resolves with what was written, or
    * on `signal`'s abort with `aborted` (chunks waiting for their turn are dropped, the ones
    * written stay); rejects naming the chunk that failed to compute or be written, once the
    * writes under way have ended. */