Skip to content

fix: keep atlas downloads and anatomy overlay computes off the event loop - #85

Closed
tabedzki wants to merge 4 commits into
m-beau:devfrom
tabedzki:tabedzki/fix/atlas-switch-download-off-loop
Closed

tabedzki wants to merge 4 commits into
m-beau:devfrom
tabedzki:tabedzki/fix/atlas-switch-download-off-loop

Conversation

@tabedzki

@tabedzki tabedzki commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Problem

On the deployed server the watchdog kept reporting the event loop blocked for 5–10 s, freezing every session and the healthcheck:

[watchdog] event loop blocked for 5.1 s (threshold 5.0 s); all sessions and the healthcheck are frozen.
  pixelmap/gui/gui.py, in _on_atlas_change
    self._recompute_anatomy_if_active()
  pixelmap/gui/gui.py, in _recompute_anatomy_if_active
    self.compute_anatomy_overlay()
  ...
  pixelmap/anatomy/atlas.py, in get_atlas
    return BrainGlobeAtlas(name, check_latest=False)
  brainglobe_atlasapi/bg_atlas.py, in download

This happened for azba_zfish_4um, demba_allen_seg_dev_mouse_p50_20um, and kim_mouse_100um, each time an atlas was picked from the dropdown while an anatomy overlay was showing.

Everything that blocked the loop

The Compute button already sent cold downloads to a worker thread. The other paths didn't:

Path Cost on the event loop (before)
Atlas switch with an overlay showing, uncached atlas the whole download (5–10 s in the logs above)
Every live pose edit, cached atlas 0.74 s (allen_mouse_25um) to 3.3 s (allen_mouse_10um), per keystroke
Compute click, cached atlas the same 0.7–3.3 s
Switching to a cached human or fish atlas (AC origin) derive_origin_from_ac scans the whole volume: 0.8 s (25 µm) to 7.7 s (10 µm)
Any on-loop annotation read while another session downloads waits on _resident_lock for that whole download

Fix

Overlay compute runs in three steps (pixelmap/gui/gui.py):

  • _anatomy_request(): snapshots every widget input on the loop. It's cheap and never reads the atlas.
  • _compute_anatomy(): runs in a worker thread and does every atlas read: region bands, the locator render and PNG encode, and the tip depth. It never touches widgets.
  • _apply_anatomy(): writes the results to Bokeh models on the loop, under the document lock. This takes about 3 ms.

_drain_anatomy_requests (@pn.io.with_lock) runs at most one compute per session. Edits made while it runs only mark the overlay dirty, so a burst of keystrokes costs one extra compute instead of one per keystroke. Each pass re-snapshots the inputs, so an atlas picked mid-download or mid-compute is the one that gets drawn. _anatomy_overlay_active is set on Compute rather than on the first draw, so edits made during the first compute aren't lost. Clear bumps a generation counter, which drops any result still in flight. A failed download clears the stale overlay instead of retrying on every later edit.

AC origin (pixelmap/anatomy/atlas.py): derive_origin_from_ac gets a plain-dict memo plus peek_origin_from_ac, and is single-flight per atlas: concurrent callers wait for the running scan instead of starting another. _origin_spec runs on the loop several times per switch, so it now only peeks at the memo. Until the origin is known it reports ac_pending; that banner already said "located from the atlas on the first Compute". One coroutine, _locate_ac_origin, derives it in a worker and then fills in the fields. The drain awaits it before taking its snapshot, so the first draw already includes the origin. With no overlay showing, switching to a cached AC atlas schedules the same coroutine.

Render speed (pixelmap/anatomy/schematic.py): _region_fill_rgba used to build one full-slice mask per region, which was 2.4 s of a 3.5 s render on a 10 µm atlas. It now builds one palette entry per distinct region and does a single gather through np.unique's inverse index.

Measured after the fix

total compute work on the event loop
allen_mouse_25um 0.74 → 0.41 s 0.74 s → 2 ms
whs_sd_rat_39um 1.0 → 0.54 s 1.0 s → 2 ms
allen_mouse_10um 3.3 → 1.16 s 3.3 s → 3 ms

Wider cleanup of everything since v1.4.1

A second cleanup pass covered every change since v1.4.1, not just this PR's:

  • Wiring maps: a cold rebuild (no or stale wiring_maps.pkl) re-indexed the wiring table once per electrode, about 55 s inside the first session's GUI constructor, on the loop. It now indexes each table once: about 1.1 s, with output identical to the shipped pickle.
  • PDF export: electrodes are drawn as one PolyCollection built from a vertex array with precomputed colours, and single- and multi-shank probes share that code. Renders are pixel-identical; plot_probe_layout is 2–3× faster (QuadBase with a survey: 1.14 → 0.46 s).
  • import pixelmap no longer loads Panel or Bokeh (1.28 → 0.55 s), because the IMRO parser imports the notification helper lazily.
  • One pipeline for the overlay: compute_anatomy_overlay() now runs the same drain as the Compute button (synchronous when no event loop is running), so tests exercise the path users actually hit. The worker no longer writes session state: the locator figure travels in the request and result.
  • Smaller cleanups: one notify() helper in place of gui._notify and error_reporting._toast; anatomy flags set in __init__ instead of getattr fallbacks; _selected_atlas_name(); dead readout branch and the test-only _tip_depth_below_surface removed; stdlib traceback helpers; one-pass survey column conversion.

Not changed

The first time a cached atlas is used, is_downloaded() constructs its BrainGlobeAtlas to check the zarr cache: up to 0.34 s on the loop for a 10 µm atlas, after which it is cached per atlas. That's well under the watchdog threshold, so I left it alone.

Tests

I wrote these before the fixes and confirmed the GUI and atlas tests fail against the old code:

  • TestAtlasSwitchNeverDownloadsOnTheEventLoop: cold switch with an overlay, a second switch mid-download, a failed download, a cold switch without an overlay, a warm switch.
  • TestOverlayComputeRunsOffTheEventLoop: a live pose edit and a Compute click on a cached atlas run bands, locator and tip depth off the main thread; edits during a compute are coalesced into one more; Clear during a compute drops the result; a lookup failure leaves the button usable.
  • TestAnteriorCommissureOriginIsLocatedOffTheEventLoop: switching with and without an overlay derives the AC origin off the main thread and fills in the fields; _origin_spec never scans by itself.
  • peek_origin_from_ac / memo: unknown vs. derived vs. "no AC" (None), failures aren't memoised, concurrent callers share one scan, and cache_clear works.
  • TestRegionFillMatchesThePerRegionLoop: the vectorised fill matches the original loop for int32, uint32, int64, and uint16 IDs, including unknown IDs (grey), all-zero slices, and empty slices.

New regression tests: a rebuilt wiring map matches the shipped one, a rebuild indexes each table once, and import pixelmap loads neither Panel nor Bokeh; the last two fail on the old code.

Full suite: 959 passed, 1 xfailed. ruff check . is clean.

🤖 Generated with Claude Code

With an anatomy overlay on screen, picking a not-yet-cached atlas from
the dropdown live-recomputed the overlay synchronously
(_on_atlas_change -> _recompute_anatomy_if_active ->
compute_anatomy_overlay), so the atlas download ran on the Bokeh event
loop and froze every session and the healthcheck for 5-10 s per switch.

_recompute_anatomy_if_active now only draws inline when the atlas is
already local; otherwise it schedules the Compute button's async handler,
which downloads in a worker thread and redraws under the document lock.

The handler also keeps fetching until the currently selected atlas is
local, since the user can switch again mid-download and the subsequent
draw reads the dropdown, and it skips starting a second download while
one is in flight.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Oct 30, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.74312% with 18 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev@206230c). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pixelmap/gui/gui.py 91.15% 13 Missing ⚠️
pixelmap/utils/imro.py 16.66% 5 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@          Coverage Diff           @@
##             dev      #85   +/-   ##
======================================
  Coverage       ?   84.56%           
======================================
  Files          ?       21           
  Lines          ?     3440           
  Branches       ?        0           
======================================
  Hits           ?     2909           
  Misses         ?      531           
  Partials       ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The cold-atlas download was not the only thing blocking the Bokeh event
loop. Measured on cached atlases:

- Every live pose edit recomputed the overlay inline: 0.7 s on
  allen_mouse_25um, 3.3 s on allen_mouse_10um, per keystroke.
- Switching to a human or fish atlas derived its anterior-commissure
  origin inline, a whole-volume scan taking 0.8-7.7 s.
- Any on-loop annotation read could wait on _resident_lock, which
  another session's download holds for its whole duration.

The overlay is now three steps: snapshot the widget inputs on the loop
(_anatomy_request), do every atlas read in a worker thread
(_compute_anatomy: bands, AC origin, locator render, tip depth), and
write Bokeh models back on the loop under the document lock
(_apply_anatomy, ~3 ms). _drain_anatomy_requests runs one compute per
session at a time and coalesces edits made during it. Clear bumps a
generation counter so a result that outlives its overlay is dropped,
and a failed download clears the stale overlay instead of retrying on
every later edit.

derive_origin_from_ac gets a hand-rolled memo plus
peek_origin_from_ac, so _origin_spec on the loop only reads the memo.
When no overlay is showing, switching to a cached AC atlas derives the
origin in a worker and fills the fields afterwards.

_region_fill_rgba builds one palette entry per distinct region and
gathers through np.unique's inverse index, instead of a full-slice mask
per region. That was 2.4 s of the 3.5 s render on a 10 um atlas; total
compute work is now 0.41 s / 1.16 s on allen_mouse_25um / 10um.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tabedzki tabedzki changed the title fix: download cold atlases off the event loop on atlas switch fix: keep atlas downloads and anatomy overlay computes off the event loop Oct 30, 2026
tabedzki and others added 2 commits October 29, 2026 23:21
- Locate a pending AC origin in the drain before the snapshot, so a lap
  draws with the origin. This drops origin_was_pending, the derive in
  _compute_anatomy, and the extra fill-and-recompute lap in
  _apply_anatomy, which cost a full 0.7-3.3 s recompute on the first
  overlay for a human or fish atlas.
- Merge the two AC paths into one _locate_ac_origin coroutine and a
  _fill_reference_params helper.
- Make derive_origin_from_ac single-flight per atlas name, so concurrent
  callers wait on the running scan instead of starting another.
- Fold _anatomy_overlay_wanted into _anatomy_overlay_active, now set on
  Compute rather than on the first draw.
- Pass the tip depth into one _update_tip_depth_readout(depth) and drop
  the _UNSET sentinel, _set_tip_depth_readout, and the readout's on-loop
  atlas read. Share the lookup and error handling via
  _measure_tip_depth.
- Add button-label constants, _refresh_compute_button, and an
  _anatomy_recompute_suppressed() context manager. Drop dead button
  writes from the sync compute_anatomy_overlay.
- Tests: reset the AC memo in the layered_atlas fixture, share one
  _assert_off_loop helper, and add a single-flight test.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second /simplify pass, widened from this PR's diff to everything since
v1.4.1.

- compute_anatomy_overlay() now runs the same off-loop drain as the
  Compute button (synchronous when no event loop is running), so the
  separate inline pipeline and its duplicate AC-origin handling are gone.
  _request_anatomy_recompute drops its redundant "already running" check;
  the drain's own guard is the one that matters.
- The overlay worker no longer writes session state: the locator figure
  travels in the request/result and is adopted (or closed if stale) on
  the loop. Label width/offset are computed at draw time instead of
  being snapshotted.
- Anatomy flags are initialised in __init__, so the getattr fallbacks go;
  _selected_atlas_name() replaces eight copies of the name lookup; the
  unreachable "available once downloaded" readout branch and the
  test-only _tip_depth_below_surface are removed.
- AC-origin memo is a plain dict (no hand-rolled eviction).
- Wiring maps: a cold rebuild indexed the wiring table once per
  electrode (~55 s in the first session's constructor); it now indexes
  each table once (~1.1 s), with identical output.
- PDF electrodes are drawn as one PolyCollection built from a vertex
  array with precomputed colours, sharing one path between single- and
  multi-shank probes: pixel-identical renders, 2-3x faster.
- `import pixelmap` no longer loads Panel/Bokeh (1.28 s -> 0.55 s): the
  IMRO parser imports the notification helper lazily.
- One notify() helper in error_reporting replaces gui._notify and
  error_reporting._toast; traceback helpers use the stdlib directly;
  survey parsing converts all columns in one pass.

Assisted-by: ClaudeCode:claude-opus-5-5
@tabedzki

tabedzki commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

Closing as replaced by #91, which merged the rebased version of this work into dev.

Assisted-by: ClaudeCode:claude-opus-5-5

@tabedzki tabedzki closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants