Batched calls for ops on a GPU, and a plan of what a read computes

[davidackerman]

Asked for by registration-station after timing a live last pass of its registration. A warm block-matching call costs about 20 ms of fixed overhead (launches, copies), whatever its size. And a view at s1 needs 80–96 small lattice chunks, which ran one call at a time.

batch on an op

• Declaring batch puts the blocks of one stage and shape that are waiting in the op's queue, when a slot frees, into one call of apply_batch, up to batch of them, first in line first.
• So the chunks a read gathers, or a viewer's concurrent requests, wait together while a call runs and go into the next one. In the test, 16 chunks run in calls of 1, 8 and 7.
• Each block is still a job of its own: shared, ordered and dropped as one. A call gives up through demand.cancelled() only once none of its blocks is wanted.
• An op with batch is a stage of its own and goes through its queue as with slots (one call at a time if it sets none). apply_batch defaults to one apply_at after another.
• Mechanism: demand.Batch. When a slot frees, the chosen job takes the others of its group into its call. fused.run_batch runs it, and fused.run takes the first op's result already computed.
• cellmap-flow's models gain the same way on a GPU.

Pipeline.plan(level, box)

• Says what a read would compute, computing nothing. For each stage, from the first to the output, it gives the chunks the read asks of it, those held and those to compute; an overlapping op's blocks are a stage of their own.
• It follows dataset:// into the named dataset's stages. It stops at a stage that holds all it is asked for, or at a source that is not a stage, such as a warp:// whose reads depend on its field's values.
• ChunkedSource.reads says what a chunk's computation reads, and covering says which chunks a read takes.

/api/ops reports batch. The docs (pipelines, caching, contributing, API, roadmap) and the CHANGELOG are updated.

Not included: registration-station's materialize --verify-chunks. They can compare two pipelines with different chunkings on the public API.