Skip to content

fix: stop per-click gc.collect stalling the server; add stall/RSS watchdog - #5

Closed
tabedzki wants to merge 23 commits into
mainfrom
t3code/3887861a
Closed

tabedzki wants to merge 23 commits into
mainfrom
t3code/3887861a

Conversation

@tabedzki

Copy link
Copy Markdown
Owner

🤖 Human guided, AI assisted PR (using this skill). AI text below. 🤖

Problem

The production container (ghcr.io/m-beau/pixelmap) runs for hours and then dies with no traceback. In about 14 h of logs:

  • The 60 s Memory usage log fired 20–90 s late 23 times, nearly always right after Selection changed: [] -> [N]. A late log means the single Bokeh event loop was blocked that long.
  • During one of those stalls the proxy dropped a user's websocket. That produced Failed sending message as connection was closed followed by 236 AssertionError: assert self._session is not None on queued PATCH-DOC messages.
  • RSS sat at about 3.1–3.5 GB, with 50–150 MB swings on each click.

on_electrode_selection ran a full gc.collect() on every tap.

Changes

  • Remove the per-click gc.collect().
  • New pixelmap/gui/watchdog.py, which only logs and never raises or kills anything:
    • Stall watchdog: the event loop beats a heartbeat; a daemon thread logs [watchdog] event loop blocked for X s with the blocked thread's stack (PIXELMAP_STALL_WARN_S, default 5, 0 disables). It logs again with the total duration when the loop recovers.
    • Memory watch: warns once when RSS crosses PIXELMAP_RSS_WARN_MB and re-arms below 90%, so RSS hovering at the limit doesn't flood Slack.
    • Installed once per process from app_util.main(local=False); thresholds set in docker-compose.yml.
  • Regression tests: tests/test_selection_responsiveness.py (no gc on tap, box select or repeated taps; toggle and empty-selection behaviour kept) and tests/test_watchdog.py (threshold edges, report-once-per-stall, recovery and re-arm, stack capture, disabled/0/negative/None configs, idempotent install).

Testing

  • The 3 gc regression tests failed before the fix and pass after.
  • Full suite: 557 passed.
  • Smoke-tested with a real panel serve (very low thresholds): the stall warning with stack, the RSS warning and the recovery line all appear in the server log.

Not addressed here

  • update_electrode_colors still rebuilds every column in Python on each click.
  • What makes up the ~3 GB baseline still needs a memray/tracemalloc snapshot.
  • Whether the last crash was an OOM kill or a healthcheck-driven restart (check docker inspect OOMKilled and dmesg).

Assisted-by: Claude-Code:Opus-5.5

tabedzki and others added 23 commits July 8, 2026 10:41
Add [tool.pyright] venvPath/venv so pyright resolves imports against the
project's .venv, matching the installed dependencies instead of a global
environment.

Assisted-by: ClaudeCode:claude-opus-4.8
…config

chore: point pyright at the project .venv
Brainglobe atlas efficient loading + fixed memory leak
Paper fix
fixed paper figure 1 caption
fixed fig1 caption
Add machine-readable citation metadata so GitHub shows a "Cite this
repository" button. The preferred citation is the JOSS paper
(10.21105/joss.11008); the software entry points to the v1.4.0 JOSS
release archived on Zenodo (10.5281/zenodo.22850328), with the concept
DOI listed for all versions.

Builds on m-beau#45.

Assisted-by: ClaudeCode:claude-opus-5-5
…-beau#63

docs: add CITATION.cff referencing the JOSS paper and v1.4.0 release
…chdog

Production logs show the single-threaded Bokeh event loop freezing for
20-90 s after nearly every electrode click: the periodic memory log fires
late right after each "Selection changed". on_electrode_selection ran a full
gc.collect() on every tap, which walks every live object in a multi-GB
process. While frozen, healthchecks fail and the proxy drops websockets
(the burst of 236 AssertionErrors on PATCH-DOC for an already-destroyed
session), and the container eventually dies without a traceback.

- Remove the gc.collect() from on_electrode_selection.
- Add pixelmap/gui/watchdog.py: a heartbeat on the event loop plus a
  daemon thread that logs a "[watchdog]" WARNING with the blocked thread's
  stack when the loop stalls (PIXELMAP_STALL_WARN_S, default 5 s; 0
  disables), logs again on recovery, and warns once when RSS crosses
  PIXELMAP_RSS_WARN_MB (with re-arm hysteresis). It only logs; it never
  raises into the app or kills it.
- Install it once per process from app_util.main(local=False) and set
  the thresholds in docker-compose.yml.
- Regression tests for the per-click gc and for the watchdog.

Assisted-by: Claude-Code:Opus-5.5
@tabedzki

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Superseded by #8. Same commit, from the renamed branch tabedzki/fix/selection-gc-stall; GitHub closed this PR when its branch was renamed.

Assisted-by: Claude-Code:Opus-5.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants