Repository navigation
fix: keep anatomy overlay computes and atlas downloads off the event loop - #91
Merged
tabedzki merged 4 commits intoNov 7, 2026
Merged
Conversation
With an anatomy overlay on screen, picking a not-yet-cached atlas from the dropdown live-recomputed the overlay synchronously (_on_atlas_change -> _recompute_anatomy_if_active -> compute_anatomy_overlay), so the atlas download ran on the Bokeh event loop and froze every session and the healthcheck for 5-10 s per switch. _recompute_anatomy_if_active now only draws inline when the atlas is already local; otherwise it schedules the Compute button's async handler, which downloads in a worker thread and redraws under the document lock. The handler also keeps fetching until the currently selected atlas is local, since the user can switch again mid-download and the subsequent draw reads the dropdown, and it skips starting a second download while one is in flight. Assisted-by: ClaudeCode:claude-opus-5-5
The cold-atlas download was not the only thing blocking the Bokeh event loop. Measured on cached atlases: - Every live pose edit recomputed the overlay inline: 0.7 s on allen_mouse_25um, 3.3 s on allen_mouse_10um, per keystroke. - Switching to a human or fish atlas derived its anterior-commissure origin inline, a whole-volume scan taking 0.8-7.7 s. - Any on-loop annotation read could wait on _resident_lock, which another session's download holds for its whole duration. The overlay is now three steps: snapshot the widget inputs on the loop (_anatomy_request), do every atlas read in a worker thread (_compute_anatomy: bands, AC origin, locator render, tip depth), and write Bokeh models back on the loop under the document lock (_apply_anatomy, ~3 ms). _drain_anatomy_requests runs one compute per session at a time and coalesces edits made during it. Clear bumps a generation counter so a result that outlives its overlay is dropped, and a failed download clears the stale overlay instead of retrying on every later edit. derive_origin_from_ac gets a hand-rolled memo plus peek_origin_from_ac, so _origin_spec on the loop only reads the memo. When no overlay is showing, switching to a cached AC atlas derives the origin in a worker and fills the fields afterwards. _region_fill_rgba builds one palette entry per distinct region and gathers through np.unique's inverse index, instead of a full-slice mask per region. That was 2.4 s of the 3.5 s render on a 10 um atlas; total compute work is now 0.41 s / 1.16 s on allen_mouse_25um / 10um. Assisted-by: ClaudeCode:claude-opus-5-5
- Locate a pending AC origin in the drain before the snapshot, so a lap draws with the origin. This drops origin_was_pending, the derive in _compute_anatomy, and the extra fill-and-recompute lap in _apply_anatomy, which cost a full 0.7-3.3 s recompute on the first overlay for a human or fish atlas. - Merge the two AC paths into one _locate_ac_origin coroutine and a _fill_reference_params helper. - Make derive_origin_from_ac single-flight per atlas name, so concurrent callers wait on the running scan instead of starting another. - Fold _anatomy_overlay_wanted into _anatomy_overlay_active, now set on Compute rather than on the first draw. - Pass the tip depth into one _update_tip_depth_readout(depth) and drop the _UNSET sentinel, _set_tip_depth_readout, and the readout's on-loop atlas read. Share the lookup and error handling via _measure_tip_depth. - Add button-label constants, _refresh_compute_button, and an _anatomy_recompute_suppressed() context manager. Drop dead button writes from the sync compute_anatomy_overlay. - Tests: reset the AC memo in the layered_atlas fixture, share one _assert_off_loop helper, and add a single-flight test. Assisted-by: ClaudeCode:claude-opus-5-5
Second /simplify pass, widened from this PR's diff to everything since v1.4.1. - compute_anatomy_overlay() now runs the same off-loop drain as the Compute button (synchronous when no event loop is running), so the separate inline pipeline and its duplicate AC-origin handling are gone. _request_anatomy_recompute drops its redundant "already running" check; the drain's own guard is the one that matters. - The overlay worker no longer writes session state: the locator figure travels in the request/result and is adopted (or closed if stale) on the loop. Label width/offset are computed at draw time instead of being snapshotted. - Anatomy flags are initialised in __init__, so the getattr fallbacks go; _selected_atlas_name() replaces eight copies of the name lookup; the unreachable "available once downloaded" readout branch and the test-only _tip_depth_below_surface are removed. - AC-origin memo is a plain dict (no hand-rolled eviction). - Wiring maps: a cold rebuild indexed the wiring table once per electrode (~55 s in the first session's constructor); it now indexes each table once (~1.1 s), with identical output. - PDF electrodes are drawn as one PolyCollection built from a vertex array with precomputed colours, sharing one path between single- and multi-shank probes: pixel-identical renders, 2-3x faster. - `import pixelmap` no longer loads Panel/Bokeh (1.28 s -> 0.55 s): the IMRO parser imports the notification helper lazily. - One notify() helper in error_reporting replaces gui._notify and error_reporting._toast; traceback helpers use the stdlib directly; survey parsing converts all columns in one pass. Assisted-by: ClaudeCode:claude-opus-5-5
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #91 +/- ##
======================================
Coverage ? 85.66%
======================================
Files ? 21
Lines ? 3445
Branches ? 0
======================================
Hits ? 2951
Misses ? 494
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced Nov 7, 2026
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Fixes the production
[watchdog] event loop blockedalert from 2026-11-06, whose stack ends in the locator's PNG render. This is the #85 work on the currentdev(#85 has merge conflicts), with theCo-Authored-Bylines removed from its commits. The code is otherwise the same as #85's rebased branch; #85 has more detail on the design and the wider cleanup.Summary
Every overlay compute ran on the Bokeh event loop that serves all sessions and
/liveness. The stack in the alert is only where the sampler landed:np.isnanis cheap, and the blocking cost is the whole handler.Also included from #85: cold atlas downloads and AC-origin scans off the loop on atlas switch, a faster region fill, and the cleanup pass (wiring maps, PDF electrode drawing, lighter
import pixelmap).Evidence
Python 3.14,
allen_mouse_25um, real Panel server, headless browser clicking Compute, polling/liveness:dev(f3e0c69)savefigMainThreadasyncio_0/liveness, 8 computes/liveness, 4 computes, server pinned to a core shared with 7 busy loops_fig_to_locator_html(5.9 s)The pinned run reproduces the alert's path. It shows contention alone can push a sub-second stall past 5 s, but which atlas and load caused the production alert isn't known. #85 measured 3.3 s on
allen_mouse_10umuncontended, which leaves less headroom.test_a_live_pose_edit_computes_off_the_event_loopondevcode: fails withcompute_region_bands ran on the event loop thread. Passes here.ruff check pixelmapis clean.Not fixed: the first session's constructor still blocks the loop:
create_app → __init__ → _atlas_tip_center → is_downloaded → get_atlas. It logged a 5.7 s stall in the pinned run on both trees (recovered after about 13 s). This is a separate source and needs its own fix.Merge Danger
Door: two-way
Revert the merge. No data or format changes.
Blast Radius: overlay
Overlay computes now run concurrently with widget edits, so a bug would show as a stale or missing overlay rather than a stall. The wider cleanup also touches wiring map rebuilds and PDF export. This reaches production only once
devis promoted tomain.Assisted-by: ClaudeCode:claude-opus-5-5