Skip to content

Engine.materializePlan: what materialize will write, and its peak memory, before it starts - #43

Merged
davidackerman merged 1 commit into
mainfrom
feature/materialize-box
Oct 16, 2026
Merged

davidackerman merged 1 commit into
mainfrom
feature/materialize-box

Conversation

@davidackerman

Copy link
Copy Markdown
Member

Adds Engine.materializePlan(view, options), which reports what materialize will write and the most memory it will hold, without computing anything. This is the follow-up the second review of #41 asked for. #41 merged before this commit was pushed, so it comes as its own PR.

With it, a page can warn before a large box starts, or offer to snap the box to the chunk grid along one axis.

MaterializePlan has these fields:

  • shape and chunk: the array's shape and chunk shape, channel axes first.
  • total: the array's chunks.
  • needed: the served chunks it computes, each once.
  • split: the axes the box splits.
  • walk: the walk's axis order, outermost first.
  • chunkBytes.
  • peakChunks: the most box chunks held at once.
  • peakBytes: peakChunks × chunkBytes. This leaves out the up to concurrency served chunks in flight, as the docs say.

The same plan is available without an Engine, as plan(region(...)). materialize and materializePlan share how the region is built (regionOf).

peakChunks is the walk's exact count, not just the documented bound.

  • The bound is n2·n3 + n3 + 1 with three split axes, n2 + 1 with two, 1 with one and 0 with none. It is exact whenever every split axis has three or more of the box's chunks.
  • Where one axis has fewer, every axis inside it has at most two, and those are counted directly.
  • So the plan never says more than the walk holds. The bound alone overcounted 20 of 3,000 random boxes.
  • It takes about 0.1 ms for a 10k³ box.

Tests. The projected peak equals the walk's measured count:

  • for each alignment, with three, two, one and no split axes (13, 4, 1 and 0 chunks);
  • for the wide, shallow box (23) and a box on a chunk edge along y (3);
  • for 400 seeded random boxes.

There are also two plan checks: the full plan of a 2-channel float32 box, and that the level's extent plans a peak of 0.

Docs.

Checks: npm run check in web/ passes (typecheck, 71/71 tests, build), as do mkdocs build --strict, pytest -q tests/test_docs.py and ruff.

…ed, what materialize will write and the most bytes of the box's chunks it will hold, as the chunkmirage session asked in its second review of #41, so a page can warn, or offer to put the box on the chunk grid along an axis, before it starts: plan(region(...)) in materialize.ts gives the array's shape and chunk shape (the channel axes the ops add first), its chunks, the served chunks it computes, the axes it splits and the walk's order, and peakChunks with its bytes, peakBytes (chunkBytes each, besides the served chunks on their way); peakChunks is exactly what the walk holds, the documented bound (a slab across the split axes inside the outermost, n2 × n3 + n3 + 1 along three, n2 + 1 along two, 1 along one, none along none) where each split axis has three chunks or more, and where one has fewer (one, or two the last of which stays within one served chunk), every axis inside it has two at most and they are counted one by one, so it is never more than the walk holds (the bound alone overcounted 20 of 3000 random boxes), and it takes a tenth of a millisecond for a 10k³ box; the Region keeps the channel axes it adds (added); Engine.materialize and materializePlan share the region's making (regionOf); the tests check the plan of a box with channels and of the level's extent, that the projected peak is what the walk holds for each alignment (three, two, one and no split axes), for the wide box and for 400 random boxes; formats, design and the CHANGELOG say so, and that a smaller peak, should it ever matter, comes from computing the plane of served chunks two slabs share twice along the outermost split axis, not from compressing chunks while they wait or refusing a box; two over-long lines the commit before left in caching.md and design.md are rewrapped
@davidackerman
davidackerman force-pushed the feature/materialize-box branch from 821f22f to f2fa984 Compare October 16, 2026 20:12
@davidackerman
davidackerman marked this pull request as ready for review October 16, 2026 20:28
@davidackerman
davidackerman merged commit d6fd94a into main Oct 16, 2026
2 checks passed
@davidackerman
davidackerman deleted the feature/materialize-box branch October 16, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant