Skip to content

perf: build each session's GUI off the event loop - #95

Closed
tabedzki wants to merge 2 commits into
m-beau:devfrom
tabedzki:tabedzki/perf/session-build-off-loop
Closed

tabedzki wants to merge 2 commits into
m-beau:devfrom
tabedzki:tabedzki/perf/session-build-off-loop

Conversation

@tabedzki

@tabedzki tabedzki commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Stacks on #93, which has not merged. Until it does, this diff includes #93. Review only the top commit, eb8d7e4. It has no textual conflict with #92 (checked with git merge-tree), and #92's tests pass on the merged tree.

Summary

panel serve runs the app per session on the loop that serves every session and /liveness. This moves GUI construction to a worker. The attach and serialisation stay on the loop, so the stall is smaller, not gone.

create_app()                          on the loop
├─ pn.extension(...), read ?cfg=
├─ on_session_destroyed(_release_dead_session_views)
└─ return Column("Loading PixelMap…")    <- served at once
document_ready -> _fill_session()     on the loop, no document lock
├─ await one-worker executor          _build_session_content(cfg), no document access
├─ tab already closed? clean up the GUI and stop
├─ register the GUI and analytics destroy hooks
└─ next tick, under the lock: app_layout[:] = [cookie_bridge, layout]   <- remaining stall

With no session (scripts, tests), create_app still builds inline.

What the design had to fix:

  • validate(False) is removed. It saves and restores a process-global flag: enter A, enter B, exit A, exit B leaves validation off for the whole server (reproduced against Bokeh 3.7.3). One worker doesn't prevent this, because on_probe_type_change builds the same source on the loop. The cost is about 41 ms per electrode source (7 -> 48 ms). That runs on the worker for session builds and on the loop only for a probe switch.
  • The worker runs in a copied context. Bokeh's curdoc patch stack is process-global, so without the copy pn.state.curdoc in a worker is whichever session the loop is serving. A profile of a real build shows the worker only reads its own session's request (Panel's ?theme= lookup) and creates ColumnDataChangedEvent hints with document=None.
  • ChannelmapGUI.__init__ no longer registers its cleanup, and ?cfg= is read on the loop.
  • The build holds no document lock. A locked session can't be discarded, so a closed tab would stay pinned until its turn in the queue. A tab closed while queued now skips its build.
  • Panel's state._onload would leak sessions that never load. It is a WeakKeyDictionary whose value reaches its own key, so _release_dead_session_views now pops the entry.
  • Failures still surface. A failed build or swap shows the existing alert and calls report_error(context="session startup", notify_user=False).

Evidence

Local panel serve, 2 fresh servers per variant, 8 sessions each, range across the two runs, PIXELMAP_STALL_WARN_S=5. Contended means the server and 7 busy loops are pinned to CPU 0.

quiet: 2b6ce90 quiet: this PR contended: 2b6ce90 contended: this PR
worst /liveness per session, median 0.52–0.53 s 0.27–0.28 s 4.35–5.05 s 2.39–2.50 s
same, max 0.58 s 0.29–0.30 s 5.99–6.35 s 3.87–4.20 s
page ready, median 1.35 s 1.34–1.37 s 7.4–7.5 s 5.4–6.3 s
first session, worst /liveness 2.6 s 1.3 s 21–22 s 10–11 s
burst of 6: first / last page ready 4.3–4.6 / 5.9–6.0 s 2.3 / 5.6–5.8 s 53–59 / 66–70 s 9–12 / 27–33 s
burst of 6: worst /liveness 0.86–0.92 s 0.63–0.68 s 9.5–11.4 s 5.9–6.8 s
watchdog log lines, per run 0, 0 0, 0 12, 20 4, 2

Docker, -m 4500m --memory-swap 4500m, median of 5 rounds. Each cell is 2b6ce90 -> this PR. Rounds 3–5 alternated which image ran first.

--cpus=8 --cpus=0.25
worst /liveness per session, median 0.60 -> 0.33 s 2.49 -> 1.38 s
same, max per round 0.55–5.93 -> 0.37–5.34 s 2.17–6.39 -> 1.48–5.40 s
page ready, median 1.56 -> 1.59 s 3.75 -> 3.57 s
burst first page ready (6 at 8 CPUs, 4 at 0.25) 5.1 -> 2.9 s 11.7 -> 4.7 s
burst last page ready 7.3 -> 8.9 s 13.7 -> 13.1 s
watchdog log lines, 5 rounds total 16 -> 20 20 -> 18
memory 90 s after close 302–325 -> 302–314 MiB 283–285 -> 281–282 MiB

Results that don't favour this PR:

  • Docker shows no watchdog improvement. The medians improve, but the 5 s+ stalls remain.
  • The machine was not quiet. Other jobs (rsync, a Python process at 100%) ran during the Docker rounds, so the max rows and watchdog counts are noisy. In the noisiest round, 2, this PR was worse than base at both CPU limits, with a median of 1.79 s against 0.62 s at 8 CPUs.
  • The burst's last visitor gains little. At 8 CPUs in Docker it is slower in the median, though with noise. Builds run one at a time, so the wait grows with queue position, and that page's ready time doesn't improve quiet either.

Memory doesn't return to its starting level after close on either image. In rounds 3–5, every session in the logs was cleaned up on both.

Browser (Chromium, real server). The plot canvas matches 2b6ce90 pixel for pixel after page load, tap, box select, QuadBase switch and ?cfg=, both locally and in the Docker image.

  • The tap turned electrodes red and the overlay image appeared.
  • A ?cfg= link restored 20/384 electrodes.
  • A notification appeared in a non-first session, and the URL updated.
  • Reload worked, the page script tags matched, and there were no page errors or tracebacks.
  • Five tabs closed while the placeholder was showing all ran cleanup, and a new session worked afterwards.
  • A local=True page also loaded.

Tests. 1040 passed, 1 xfailed; ruff is clean. New coverage is in tests/test_session_off_loop.py and tests/test_session_leak.py. It covers overlapping builds, no document access from the worker, closing while queued or mid-build, sessions that never load, failed builds and swaps, bad or empty ?cfg=, and building without a session.

Merge Danger

Door: two-way. It is one commit to revert, with no data or format changes.

Blast Radius: sessions

Every visitor's page load goes through this path.

  • Visitors now see "Loading PixelMap…" before the GUI appears, typically for under a second when quiet.
  • One worker means one hung build blocks every later session's build. Those visitors stay on the placeholder, with no timeout, while the server stays up. Before this PR the same hang froze the whole loop. I found no network call on the default build path, but that comes from reading the code, not from a failure test.
  • A request that never connects a websocket (a bot, curl /app) now costs a placeholder and builds no GUI.
  • A tab closed on the placeholder records no analytics visit.
  • ChannelmapGUI() constructed directly in someone's own served app no longer registers _cleanup_session; create_app does that now.
  • Property validation is back on for the electrode source.
  • _release_dead_session_views now also touches the private state._onload, so a Panel upgrade could break it.

Not verified:

Assisted-by: ClaudeCode:claude-opus-5-5

Each session builds, validates, serialises and sends its Bokeh document on
the loop that serves every other session and /liveness.

- Draw shank outlines, bank lines and labels with one renderer per style.
  A 4-shank document drops from 33 renderers and 485 models to 9 and 269.
- Limit tap and box-select tools to the electrode and survey-bar renderers,
  so a tap on a shared outline renderer can't dim the other outlines.
- Send numeric electrode columns as arrays, compute their geometry with
  NumPy, and skip Bokeh's per-cell check of that fixed-shape source.
- Reuse the cached data URI for tool icons instead of re-encoding six PNGs.

Assisted-by: ClaudeCode:claude-opus-5-5
`panel serve` runs the app per session on the loop that serves every session
and /liveness. Serve a placeholder at once, build the GUI on one worker thread
once the page connects, and swap it in on the loop under the document lock.

Nothing on the worker touches a document: ?cfg= is read in create_app, and
destroy hooks are registered on the loop, so ChannelmapGUI.__init__ no longer
registers its own. A tab closed while queued skips the build, and one closed
mid-build has its GUI cleaned up. Any build or swap failure shows the existing
alert and is reported.

Drop validate(False) from create_electrode_data: it toggles a process-global
flag, and on_probe_type_change on the loop can overlap the worker's build,
which could leave property validation off for the whole server. This costs
about 41 ms per electrode source.

Assisted-by: ClaudeCode:claude-opus-5-5
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (dev@faf32d9). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@          Coverage Diff           @@
##             dev      #95   +/-   ##
======================================
  Coverage       ?   86.86%           
======================================
  Files          ?       21           
  Lines          ?     3463           
  Branches       ?        0           
======================================
  Hits           ?     3008           
  Misses         ?      455           
  Partials       ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tabedzki tabedzki closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants