Repository navigation
Conversation
With an anatomy overlay on screen, picking a not-yet-cached atlas from the dropdown live-recomputed the overlay synchronously (_on_atlas_change -> _recompute_anatomy_if_active -> compute_anatomy_overlay), so the atlas download ran on the Bokeh event loop and froze every session and the healthcheck for 5-10 s per switch. _recompute_anatomy_if_active now only draws inline when the atlas is already local; otherwise it schedules the Compute button's async handler, which downloads in a worker thread and redraws under the document lock. The handler also keeps fetching until the currently selected atlas is local, since the user can switch again mid-download and the subsequent draw reads the dropdown, and it skips starting a second download while one is in flight. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #85 +/- ##
======================================
Coverage ? 84.56%
======================================
Files ? 21
Lines ? 3440
Branches ? 0
======================================
Hits ? 2909
Misses ? 531
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The cold-atlas download was not the only thing blocking the Bokeh event loop. Measured on cached atlases: - Every live pose edit recomputed the overlay inline: 0.7 s on allen_mouse_25um, 3.3 s on allen_mouse_10um, per keystroke. - Switching to a human or fish atlas derived its anterior-commissure origin inline, a whole-volume scan taking 0.8-7.7 s. - Any on-loop annotation read could wait on _resident_lock, which another session's download holds for its whole duration. The overlay is now three steps: snapshot the widget inputs on the loop (_anatomy_request), do every atlas read in a worker thread (_compute_anatomy: bands, AC origin, locator render, tip depth), and write Bokeh models back on the loop under the document lock (_apply_anatomy, ~3 ms). _drain_anatomy_requests runs one compute per session at a time and coalesces edits made during it. Clear bumps a generation counter so a result that outlives its overlay is dropped, and a failed download clears the stale overlay instead of retrying on every later edit. derive_origin_from_ac gets a hand-rolled memo plus peek_origin_from_ac, so _origin_spec on the loop only reads the memo. When no overlay is showing, switching to a cached AC atlas derives the origin in a worker and fills the fields afterwards. _region_fill_rgba builds one palette entry per distinct region and gathers through np.unique's inverse index, instead of a full-slice mask per region. That was 2.4 s of the 3.5 s render on a 10 um atlas; total compute work is now 0.41 s / 1.16 s on allen_mouse_25um / 10um. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Locate a pending AC origin in the drain before the snapshot, so a lap draws with the origin. This drops origin_was_pending, the derive in _compute_anatomy, and the extra fill-and-recompute lap in _apply_anatomy, which cost a full 0.7-3.3 s recompute on the first overlay for a human or fish atlas. - Merge the two AC paths into one _locate_ac_origin coroutine and a _fill_reference_params helper. - Make derive_origin_from_ac single-flight per atlas name, so concurrent callers wait on the running scan instead of starting another. - Fold _anatomy_overlay_wanted into _anatomy_overlay_active, now set on Compute rather than on the first draw. - Pass the tip depth into one _update_tip_depth_readout(depth) and drop the _UNSET sentinel, _set_tip_depth_readout, and the readout's on-loop atlas read. Share the lookup and error handling via _measure_tip_depth. - Add button-label constants, _refresh_compute_button, and an _anatomy_recompute_suppressed() context manager. Drop dead button writes from the sync compute_anatomy_overlay. - Tests: reset the AC memo in the layered_atlas fixture, share one _assert_off_loop helper, and add a single-flight test. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second /simplify pass, widened from this PR's diff to everything since v1.4.1. - compute_anatomy_overlay() now runs the same off-loop drain as the Compute button (synchronous when no event loop is running), so the separate inline pipeline and its duplicate AC-origin handling are gone. _request_anatomy_recompute drops its redundant "already running" check; the drain's own guard is the one that matters. - The overlay worker no longer writes session state: the locator figure travels in the request/result and is adopted (or closed if stale) on the loop. Label width/offset are computed at draw time instead of being snapshotted. - Anatomy flags are initialised in __init__, so the getattr fallbacks go; _selected_atlas_name() replaces eight copies of the name lookup; the unreachable "available once downloaded" readout branch and the test-only _tip_depth_below_surface are removed. - AC-origin memo is a plain dict (no hand-rolled eviction). - Wiring maps: a cold rebuild indexed the wiring table once per electrode (~55 s in the first session's constructor); it now indexes each table once (~1.1 s), with identical output. - PDF electrodes are drawn as one PolyCollection built from a vertex array with precomputed colours, sharing one path between single- and multi-shank probes: pixel-identical renders, 2-3x faster. - `import pixelmap` no longer loads Panel/Bokeh (1.28 s -> 0.55 s): the IMRO parser imports the notification helper lazily. - One notify() helper in error_reporting replaces gui._notify and error_reporting._toast; traceback helpers use the stdlib directly; survey parsing converts all columns in one pass. Assisted-by: ClaudeCode:claude-opus-5-5
Collaborator
Author
|
🤖 AI text below 🤖 Closing as replaced by #91, which merged the rebased version of this work into Assisted-by: ClaudeCode:claude-opus-5-5 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Problem
On the deployed server the watchdog kept reporting the event loop blocked for 5–10 s, freezing every session and the healthcheck:
This happened for
azba_zfish_4um,demba_allen_seg_dev_mouse_p50_20um, andkim_mouse_100um, each time an atlas was picked from the dropdown while an anatomy overlay was showing.Everything that blocked the loop
The Compute button already sent cold downloads to a worker thread. The other paths didn't:
derive_origin_from_acscans the whole volume: 0.8 s (25 µm) to 7.7 s (10 µm)_resident_lockfor that whole downloadFix
Overlay compute runs in three steps (
pixelmap/gui/gui.py):_anatomy_request(): snapshots every widget input on the loop. It's cheap and never reads the atlas._compute_anatomy(): runs in a worker thread and does every atlas read: region bands, the locator render and PNG encode, and the tip depth. It never touches widgets._apply_anatomy(): writes the results to Bokeh models on the loop, under the document lock. This takes about 3 ms._drain_anatomy_requests(@pn.io.with_lock) runs at most one compute per session. Edits made while it runs only mark the overlay dirty, so a burst of keystrokes costs one extra compute instead of one per keystroke. Each pass re-snapshots the inputs, so an atlas picked mid-download or mid-compute is the one that gets drawn._anatomy_overlay_activeis set on Compute rather than on the first draw, so edits made during the first compute aren't lost. Clear bumps a generation counter, which drops any result still in flight. A failed download clears the stale overlay instead of retrying on every later edit.AC origin (
pixelmap/anatomy/atlas.py):derive_origin_from_acgets a plain-dict memo pluspeek_origin_from_ac, and is single-flight per atlas: concurrent callers wait for the running scan instead of starting another._origin_specruns on the loop several times per switch, so it now only peeks at the memo. Until the origin is known it reportsac_pending; that banner already said "located from the atlas on the first Compute". One coroutine,_locate_ac_origin, derives it in a worker and then fills in the fields. The drain awaits it before taking its snapshot, so the first draw already includes the origin. With no overlay showing, switching to a cached AC atlas schedules the same coroutine.Render speed (
pixelmap/anatomy/schematic.py):_region_fill_rgbaused to build one full-slice mask per region, which was 2.4 s of a 3.5 s render on a 10 µm atlas. It now builds one palette entry per distinct region and does a single gather throughnp.unique's inverse index.Measured after the fix
Wider cleanup of everything since v1.4.1
A second cleanup pass covered every change since v1.4.1, not just this PR's:
wiring_maps.pkl) re-indexed the wiring table once per electrode, about 55 s inside the first session's GUI constructor, on the loop. It now indexes each table once: about 1.1 s, with output identical to the shipped pickle.PolyCollectionbuilt from a vertex array with precomputed colours, and single- and multi-shank probes share that code. Renders are pixel-identical;plot_probe_layoutis 2–3× faster (QuadBase with a survey: 1.14 → 0.46 s).import pixelmapno longer loads Panel or Bokeh (1.28 → 0.55 s), because the IMRO parser imports the notification helper lazily.compute_anatomy_overlay()now runs the same drain as the Compute button (synchronous when no event loop is running), so tests exercise the path users actually hit. The worker no longer writes session state: the locator figure travels in the request and result.notify()helper in place ofgui._notifyanderror_reporting._toast; anatomy flags set in__init__instead ofgetattrfallbacks;_selected_atlas_name(); dead readout branch and the test-only_tip_depth_below_surfaceremoved; stdlib traceback helpers; one-pass survey column conversion.Not changed
The first time a cached atlas is used,
is_downloaded()constructs itsBrainGlobeAtlasto check the zarr cache: up to 0.34 s on the loop for a 10 µm atlas, after which it is cached per atlas. That's well under the watchdog threshold, so I left it alone.Tests
I wrote these before the fixes and confirmed the GUI and atlas tests fail against the old code:
TestAtlasSwitchNeverDownloadsOnTheEventLoop: cold switch with an overlay, a second switch mid-download, a failed download, a cold switch without an overlay, a warm switch.TestOverlayComputeRunsOffTheEventLoop: a live pose edit and a Compute click on a cached atlas run bands, locator and tip depth off the main thread; edits during a compute are coalesced into one more; Clear during a compute drops the result; a lookup failure leaves the button usable.TestAnteriorCommissureOriginIsLocatedOffTheEventLoop: switching with and without an overlay derives the AC origin off the main thread and fills in the fields;_origin_specnever scans by itself.peek_origin_from_ac/ memo: unknown vs. derived vs. "no AC" (None), failures aren't memoised, concurrent callers share one scan, andcache_clearworks.TestRegionFillMatchesThePerRegionLoop: the vectorised fill matches the original loop forint32,uint32,int64, anduint16IDs, including unknown IDs (grey), all-zero slices, and empty slices.New regression tests: a rebuilt wiring map matches the shipped one, a rebuild indexes each table once, and
import pixelmaploads neither Panel nor Bokeh; the last two fail on the old code.Full suite: 959 passed, 1 xfailed.
ruff check .is clean.🤖 Generated with Claude Code