Skip to content

fix: keep anatomy overlay computes and atlas downloads off the event loop - #91

Merged
tabedzki merged 4 commits into
m-beau:devfrom
tabedzki:tabedzki/fix/overlay-off-event-loop
Nov 7, 2026
Merged

tabedzki merged 4 commits into
m-beau:devfrom
tabedzki:tabedzki/fix/overlay-off-event-loop

Conversation

@tabedzki

@tabedzki tabedzki commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Fixes the production [watchdog] event loop blocked alert from 2026-11-06, whose stack ends in the locator's PNG render. This is the #85 work on the current dev (#85 has merge conflicts), with the Co-Authored-By lines removed from its commits. The code is otherwise the same as #85's rebased branch; #85 has more detail on the design and the wider cleanup.

Summary

Every overlay compute ran on the Bokeh event loop that serves all sessions and /liveness. The stack in the alert is only where the sampler landed: np.isnan is cheap, and the blocking cost is the whole handler.

 _on_compute_anatomy_click / pose edit
-  compute_anatomy_overlay            # on the loop
-    compute_region_bands
-    render_locator
-    _fig_to_locator_html             # savefig, frame in the alert
+  _drain_anatomy_requests            # one per session, coalesces edits
+    _anatomy_request                 # loop: snapshot widgets
+    to_thread(_compute_anatomy)      # worker: bands, render, PNG, depth
+    _apply_anatomy                   # loop: write Bokeh models

Also included from #85: cold atlas downloads and AC-origin scans off the loop on atlas switch, a faster region fill, and the cleanup pass (wiring maps, PDF electrode drawing, lighter import pixelmap).

Evidence

Python 3.14, allen_mouse_25um, real Panel server, headless browser clicking Compute, polling /liveness:

dev (f3e0c69) this branch
Thread running savefig MainThread asyncio_0
Largest loop gap, in-process (click and 3 pose edits) 0.71–0.93 s 0.02–0.18 s
Slowest /liveness, 8 computes 0.931 s 0.036 s
Slowest /liveness, 4 computes, server pinned to a core shared with 7 busy loops 5.565 s 0.104 s
Watchdog at the default 5 s, same pinned run compute stall through _fig_to_locator_html (5.9 s) no compute stall

The pinned run reproduces the alert's path. It shows contention alone can push a sub-second stall past 5 s, but which atlas and load caused the production alert isn't known. #85 measured 3.3 s on allen_mouse_10um uncontended, which leaves less headroom.

  • Overlay rendered and the button recovered in the real session on both trees, with no document-lock errors in the server log.
  • test_a_live_pose_edit_computes_off_the_event_loop on dev code: fails with compute_region_bands ran on the event loop thread. Passes here.
  • Full suite here: 980 passed, 1 xfailed. ruff check pixelmap is clean.

Not fixed: the first session's constructor still blocks the loop: create_app → __init__ → _atlas_tip_center → is_downloaded → get_atlas. It logged a 5.7 s stall in the pinned run on both trees (recovered after about 13 s). This is a separate source and needs its own fix.

Merge Danger

Door: two-way

Revert the merge. No data or format changes.

Blast Radius: overlay

Overlay computes now run concurrently with widget edits, so a bug would show as a stale or missing overlay rather than a stall. The wider cleanup also touches wiring map rebuilds and PDF export. This reaches production only once dev is promoted to main.

Assisted-by: ClaudeCode:claude-opus-5-5

With an anatomy overlay on screen, picking a not-yet-cached atlas from
the dropdown live-recomputed the overlay synchronously
(_on_atlas_change -> _recompute_anatomy_if_active ->
compute_anatomy_overlay), so the atlas download ran on the Bokeh event
loop and froze every session and the healthcheck for 5-10 s per switch.

_recompute_anatomy_if_active now only draws inline when the atlas is
already local; otherwise it schedules the Compute button's async handler,
which downloads in a worker thread and redraws under the document lock.

The handler also keeps fetching until the currently selected atlas is
local, since the user can switch again mid-download and the subsequent
draw reads the dropdown, and it skips starting a second download while
one is in flight.

Assisted-by: ClaudeCode:claude-opus-5-5
The cold-atlas download was not the only thing blocking the Bokeh event
loop. Measured on cached atlases:

- Every live pose edit recomputed the overlay inline: 0.7 s on
  allen_mouse_25um, 3.3 s on allen_mouse_10um, per keystroke.
- Switching to a human or fish atlas derived its anterior-commissure
  origin inline, a whole-volume scan taking 0.8-7.7 s.
- Any on-loop annotation read could wait on _resident_lock, which
  another session's download holds for its whole duration.

The overlay is now three steps: snapshot the widget inputs on the loop
(_anatomy_request), do every atlas read in a worker thread
(_compute_anatomy: bands, AC origin, locator render, tip depth), and
write Bokeh models back on the loop under the document lock
(_apply_anatomy, ~3 ms). _drain_anatomy_requests runs one compute per
session at a time and coalesces edits made during it. Clear bumps a
generation counter so a result that outlives its overlay is dropped,
and a failed download clears the stale overlay instead of retrying on
every later edit.

derive_origin_from_ac gets a hand-rolled memo plus
peek_origin_from_ac, so _origin_spec on the loop only reads the memo.
When no overlay is showing, switching to a cached AC atlas derives the
origin in a worker and fills the fields afterwards.

_region_fill_rgba builds one palette entry per distinct region and
gathers through np.unique's inverse index, instead of a full-slice mask
per region. That was 2.4 s of the 3.5 s render on a 10 um atlas; total
compute work is now 0.41 s / 1.16 s on allen_mouse_25um / 10um.

Assisted-by: ClaudeCode:claude-opus-5-5
- Locate a pending AC origin in the drain before the snapshot, so a lap
  draws with the origin. This drops origin_was_pending, the derive in
  _compute_anatomy, and the extra fill-and-recompute lap in
  _apply_anatomy, which cost a full 0.7-3.3 s recompute on the first
  overlay for a human or fish atlas.
- Merge the two AC paths into one _locate_ac_origin coroutine and a
  _fill_reference_params helper.
- Make derive_origin_from_ac single-flight per atlas name, so concurrent
  callers wait on the running scan instead of starting another.
- Fold _anatomy_overlay_wanted into _anatomy_overlay_active, now set on
  Compute rather than on the first draw.
- Pass the tip depth into one _update_tip_depth_readout(depth) and drop
  the _UNSET sentinel, _set_tip_depth_readout, and the readout's on-loop
  atlas read. Share the lookup and error handling via
  _measure_tip_depth.
- Add button-label constants, _refresh_compute_button, and an
  _anatomy_recompute_suppressed() context manager. Drop dead button
  writes from the sync compute_anatomy_overlay.
- Tests: reset the AC memo in the layered_atlas fixture, share one
  _assert_off_loop helper, and add a single-flight test.

Assisted-by: ClaudeCode:claude-opus-5-5
Second /simplify pass, widened from this PR's diff to everything since
v1.4.1.

- compute_anatomy_overlay() now runs the same off-loop drain as the
  Compute button (synchronous when no event loop is running), so the
  separate inline pipeline and its duplicate AC-origin handling are gone.
  _request_anatomy_recompute drops its redundant "already running" check;
  the drain's own guard is the one that matters.
- The overlay worker no longer writes session state: the locator figure
  travels in the request/result and is adopted (or closed if stale) on
  the loop. Label width/offset are computed at draw time instead of
  being snapshotted.
- Anatomy flags are initialised in __init__, so the getattr fallbacks go;
  _selected_atlas_name() replaces eight copies of the name lookup; the
  unreachable "available once downloaded" readout branch and the
  test-only _tip_depth_below_surface are removed.
- AC-origin memo is a plain dict (no hand-rolled eviction).
- Wiring maps: a cold rebuild indexed the wiring table once per
  electrode (~55 s in the first session's constructor); it now indexes
  each table once (~1.1 s), with identical output.
- PDF electrodes are drawn as one PolyCollection built from a vertex
  array with precomputed colours, sharing one path between single- and
  multi-shank probes: pixel-identical renders, 2-3x faster.
- `import pixelmap` no longer loads Panel/Bokeh (1.28 s -> 0.55 s): the
  IMRO parser imports the notification helper lazily.
- One notify() helper in error_reporting replaces gui._notify and
  error_reporting._toast; traceback helpers use the stdlib directly;
  survey parsing converts all columns in one pass.

Assisted-by: ClaudeCode:claude-opus-5-5
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.70507% with 18 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev@f3e0c69). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pixelmap/gui/gui.py 91.15% 13 Missing ⚠️
pixelmap/utils/imro.py 16.66% 5 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@          Coverage Diff           @@
##             dev      #91   +/-   ##
======================================
  Coverage       ?   85.66%           
======================================
  Files          ?       21           
  Lines          ?     3445           
  Branches       ?        0           
======================================
  Hits           ?     2951           
  Misses         ?      494           
  Partials       ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tabedzki
tabedzki merged commit faf32d9 into m-beau:dev Nov 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants