Skip to content

fix: own the on-disk annotation layer: zarr decode, atomic fetch, right level - #3

Closed
tabedzki wants to merge 1 commit into
mainfrom
fix/atlas-decode-peak
Closed

tabedzki wants to merge 1 commit into
mainfrom
fix/atlas-decode-peak

Conversation

@tabedzki

@tabedzki tabedzki commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

🤖 Human guided, AI assisted PR (using this skill). AI text below. 🤖

Problem. brainglobe decodes via dask.compute(), holding every chunk plus the result: the 1.07 GB rat atlas peaks at 2,305 MB. Two related bugs: a download killed mid-way leaves a c dir that is trusted forever (missing chunks read as 0; chunk counts can't detect it since all-zero chunks are omitted), and is_downloaded accepted any pyramid level, so with 25 µm cached the 4.6 GB 10 µm atlas reported as downloaded and fetched inline on the event loop.

Change. load_annotation reads through zarr into one preallocated array (peak 1,263 MB, 2× faster, byte-identical). Downloads stage in <level>.partial and rename into place. _annotation_is_cached checks the atlas's own level. ensure_downloaded loads via canonical_annotation.

Tests use a tiny real OME-Zarr pyramid with an annotation property that raises. Independent of #2; second to land needs a small rebase.

Assisted-by: Claude-Code:Opus-5

@tabedzki

tabedzki commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner Author

🤖 Human guided, AI assisted comment (using this skill). AI text below. 🤖

Findings

Same process, rat atlas (1,074 MB), both decode paths:

zarr open_array()[:]      0.4 s   peak 1259 MB
brainglobe .annotation    0.9 s   peak 2305 MB   (dask.compute: chunks + result coexist)

Results are np.array_equal.

Partial downloads: the only "downloaded" signal is the c dir, created before fs.get finishes. A complete rat download has 217 of 512 chunks (zeros omitted), so counting can't detect a truncated one; atomic rename is the only fix.

Wrong level: annotation-sets/allen_mouse-annotation/…/s0 (10 µm) has no chunks, s1 (25 µm) does. _annotation_is_cached accepted any level → is_downloaded("allen_mouse_10um") was True on main, False here. On main that sends the 4.6 GB fetch through the non-worker branch of _on_compute_anatomy_click, i.e. inline on the event loop.

Assisted-by: Claude-Code:Opus-5

@tabedzki
tabedzki force-pushed the fix/atlas-decode-peak branch from 8d358ae to 8912dde Compare September 18, 2026 15:47
…ht level

brainglobe's `annotation` property materialises the volume with
`dask.compute()`: every chunk is decompressed into its own buffer and only
then concatenated, so the process briefly holds two copies. Measured on
whs_sd_rat_39um (1.07 GB array): 2,305 MB peak RSS. On a 3 GB container
that peak, on top of the Bokeh baseline and whatever atlas is already
resident, is what OOM-kills the server when a user computes an overlay
with a second large atlas.

`load_annotation` reads the same chunks straight through zarr indexing
into one preallocated array. Peak drops to 1,263 MB (the array plus
baseline), the decode is ~2x faster, and the result is byte-identical.
The pyramid level is chosen from the standard OME `multiscales` metadata
by matching the atlas resolution. Nothing in this module touches
`atlas.annotation` any more.

Owning that step exposes two more problems in the same layer:

* A killed download passed for a finished one. `fs.get` writes chunks
  one at a time, and the `c` directory it creates is the only "already
  downloaded" signal both this loader and brainglobe use; an OOM kill
  mid-transfer (which a large atlas is exactly the thing to cause) left a
  partial `c` that was trusted forever and read back with the missing
  chunks as zeros. Counting chunks can't detect this -- a complete
  download omits all-zero chunks (the rat atlas ships 217 of 512) -- so
  the fetch now lands in a staging directory and is renamed into place.

* `is_downloaded` accepted *any* cached pyramid level. One annotation set
  serves a whole atlas family: allen_mouse_25um is level s1, allen_mouse_10um
  is s0. With 25 µm baked into the image, the 10 µm atlas (4.8 GB, nothing
  local) reported as downloaded, so the GUI took its "this is free" path
  and the download ran inline on the Bokeh event loop. The check now
  resolves the atlas's own level.

`ensure_downloaded` now loads via `canonical_annotation` so the volume
decoded on the worker thread is the one the overlay compute reads, not a
throwaway copy.

Assisted-by: Claude-Code:Opus-5
@tabedzki
tabedzki force-pushed the fix/atlas-decode-peak branch from 8912dde to c902868 Compare September 18, 2026 15:49
@tabedzki tabedzki closed this Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant