Repository navigation
Description
🤖 AI text below 🤖
Summary
The production container (ghcr.io/m-beau/pixelmap:latest, v1.4.0, at pixelmap.pni.princeton.edu) runs for a while and then dies with no traceback. This issue documents the investigation, the reproduction, the root causes found so far, the fixes in progress, and what is still open.
Root causes identified
- OOM kills. Production RSS idled at about 3.4 GB against a 3500 MB container limit, and the highest logged RSS was 3,485.53 MB. Reproduced locally:
OOMKilled=true, exit 137. - A Panel bug leaks every closed session (Panel 1.7.5 and the latest 1.9.4). A per-session
NotificationAreais left inpn.state._viewsand pins the destroyed BokehDocument, and in PixelMap often the session's entireChannelmapGUI. Minimal reproduction below. - Event-loop freezes of 20–90 s on electrode clicks, caused by a
gc.collect()on every selection. These are separate from the deaths, but they fail healthchecks and drop websockets. - Large peaks from concurrent sessions and atlas overlays. Still open.
- The image build is broken for the next release, independent of the above: the Dockerfile doesn't pin Python, so uv picks CPython 3.15, which
dracopy2.0.0 has no wheel for.
1. Production evidence
From about 14 h of production logs (relayed through the Slack notifier):
- RSS between 3,120 and 3,485 MB the whole time; VMS only grows (4,150 → 4,360 MB). A fresh container starts at about 258 MB, so this is built up during runtime, not by the image or the brainglobe volume. The volume's 21 downloaded atlases cost nothing until an overlay is computed.
- The 60 s
--mem-log-frequencyline fired 20–90 s late 23 times, nearly always right afterSelection changed: [] -> [N], so the single event loop was blocked. - During one stall the proxy dropped a websocket:
Failed sending message as connection was closed, then 236 ×AssertionError: assert self._session is not Noneon queuedPATCH-DOCmessages. - The live-session count never dropped below 3 in 14 h.
- The production compose sets
deploy.resources.limits.memory: 3500M. Plain Docker doesn't restart unhealthy containers and the notifier only reports, so the deaths are OOM kills, not healthcheck restarts.
2. How it was reproduced
All runs used the production image (or a build from the fix branch) under a hard memory cap with swap disabled (-m <cap> --memory-swap <cap>). They were driven by headless Chromium through Playwright: N concurrent "users", each looping through:
- open the app, pick an atlas and compute the anatomy overlay
- edit the pose three times (each edit recomputes the overlay live)
- 5 clicks and a box drag on the plot, then Apply Preset (three rounds of the above)
- change the probe type, then close the tab
Memory was sampled every 5 s from docker stats and the cgroup's memory.stat (anon = process memory, file = page cache). Scripts are at the bottom.
| Run | Image | Cap | Load | Result |
|---|---|---|---|---|
| 1 | prod v1.4.0 | 1500 MB | 4 users, atlases incl. 10 µm | OOM-killed (exit 137) after ~1 min 50 s. 10 live sessions for 4 users; died while two 10 µm atlases were in use |
| A/B | prod v1.4.0 | 3500 MB | 4 users, 2–5 min, no atlases | Idle after load: ~1.3–1.5 GB, never returns |
| C | prod + debug page | 3500 MB | 4 users, 2 min, no atlases; 41 sessions, 41 cleaned up | 0 live sessions, but 16 ChannelmapGUI alive, 43 Documents, 1.48 M Electrode objects; malloc_trim frees only 57 MB, so these are live objects |
| in-proc | local, one GUI | n/a | cycle 5 atlases ×2 (incl. 10 µm) | Peak ~750 MB, settles ~430 MB, no growth on round 2: single-user atlas switching is fine |
| fix | fixed build | 3500 MB | same as A/B; 40 sessions, 40 cleaned up | Idle after load: ~660 MB (vs ~1.3–1.4 GB prod) |
| fix | fixed build | 1500 MB | same as run 1 | Still OOM-killed after ~1 min: 11 live sessions + atlas peaks exceed 1500 MB (capacity, see §5) |
| side by side | prod vs fixed | 3500 MB each | 4 users, atlases, 10 min, then idle | Both survived. Peak RSS 2,239 vs 2,248 MB. Idle after cleanup: 1,714 MB prod vs 1,135 MB fixed |
| round 2 | same containers | 3500 MB | another 5 min of the same | Idle: prod 1,714 → 1,520 MB, fixed 1,135 → 1,155 MB |
Round 2 did not show production climbing further within that window. The leaked memory is partly reused, so a 15-minute test can't distinguish linear growth from a plateau. The production numbers (3.4 GB after days) and the minimal reproduction (one leaked Document per closed session, linear) are the stronger evidence for growth. A long soak test is listed as open.
3. Root cause of the leak (Panel)
Found by serving a second "debug" app in the same server process and walking gc referrers from a leaked ChannelmapGUI/Document:
pn.state._viewsis a plaindictholding(view, root, doc, comm). After all sessions were destroyed it still had 20 entries, allpanel.io.notifications.NotificationArea, each pointing at a destroyed Document.- Removing just those entries in the live server freed all leaked GUIs: RSS 927 → 545 MB, no restart.
- Why they're never removed: the notification area's
_server_destroy(which would drop the entry) is only reached fromstate._destroy_sessionthroughstate._notifications[doc]. Tracing deletions onstate._notificationsshowed Panel's ownDocument.destroyoverride,panel/io/document.py_destroy_document, runningdel state_obj[doc]over every per-documentWeakKeyDictionary,_notificationsincluded. Bokeh callssession.destroy()before running theon_session_destroyedcallbacks, so by the timestate._destroy_sessionruns,doc in state._notificationsisFalseand the cleanup is skipped. Instrumented trace:[wkd] __delitem__ called from: bokeh/server/contexts.py:283 do_discard -> session.destroy() bokeh/server/session.py:188 destroy -> self._document.destroy(self) panel/io/document.py:254 _destroy_document -> del state_obj[self] [wkd] _destroy_session doc_in_notifications=False roots=DESTROYED - The same logic is present in Panel 1.9.4 (latest), so upgrading doesn't fix it.
In PixelMap, the orphaned Document often keeps the session's ChannelmapGUI alive (in run C, 16 of 41 sessions, about 90k Electrode objects each, about 70–100 MB per leaked session). The minimal reproduction below shows the Document leak deterministically. The extra step from the dead Document to app state only appeared with real browser traffic, and that exact path isn't fully pinned down (see open items).
Minimal reproduction (Panel only, no PixelMap)
$ uv run --no-project --python 3.13 --with "panel==1.9.4" python mre.py 50
panel 1.9.4 sessions opened+closed: 50 live sessions: 0
state._views entries for destroyed docs: 50 ['NotificationArea']
destroyed Documents still alive: 50 bokeh Models alive: 103
Same result on 1.7.5. Remove pn.extension(notifications=True) and the entries go away.
4. Fixes
- Draft PR fix: stop per-click gc.collect stalling the server; add stall/RSS watchdog #5: removes the per-click
gc.collect()(the freezes) and adds a[watchdog]that logs the blocked thread's stack when the event loop stalls, plus a one-time warning when RSS crosses a threshold. Includes regression tests. - Branch
tabedzki/fix/panel-session-leak(not yet pushed):_release_dead_session_views(), registered withon_session_destroyedincreate_app. It drops everypn.state._viewsentry still pointing at the dead session's Document, calling_cleanupfirst and tolerating failures. It works regardless of hook order.tests/test_session_leak.py: an end-to-end test with a real in-process Panel server that opens and closes sessions and asserts no_viewsentry for a destroyed Document survives. It fails before the fix and passes after, together with unit tests for the hook's edge cases. Full suite: 530 passed.Dockerfile:ENV UV_PYTHON=3.14. Without it,docker buildcurrently fails (dracopy==2.0.0 ... doesn't have a source distribution or wheel for the current platformunder CPython 3.15). The next tagged release would fail indocker-publish.yml.
5. Still open
- Long soak test (hours, realistic traffic) on the fixed image, to show production growth versus a plateau directly.
- Idle floor after atlas work is ~1.1 GB, not ~300 MB. Likely the last-used atlas volume stays resident (closing a tab doesn't call
release_atlas_memorythe way "Clear overlay" does) plus allocator high-water. Candidate fix: release the atlas andmalloc_trimwhen the last session is destroyed. - Peak capacity. About 50 MB per session (closed tabs linger 60 s under
--unused-session-lifetime 60000) plus about 400 MB per 10 µm overlay computation. Concurrent large-atlas computations can still exceed the limit. Options: serialize or limit large-atlas computations, block very fine (≤10 µm) atlases on the server, shorten the unused-session lifetime, and size the limit from measured peaks. - Exact path from the orphaned Document to
ChannelmapGUIunder real browser traffic. The fix covers it (clearing_viewsfreed every GUI), but it matters for the upstream report. - Report upstream to Panel with the minimal reproduction.
- Forward
[watchdog]log lines to Slack (the production notifier only watches Docker events), and checkdocker inspect --format '{{.State.OOMKilled}}'on future deaths. - Merge PR fix: stop per-click gc.collect stalling the server; add stall/RSS watchdog #5; push the leak-fix branch and open its PR.
Scripts
mre.py
"""Panel leaks every closed session when notifications are enabled.
Run: python mre.py -> prints how many state._views entries still point at
destroyed Documents after N sessions are opened and closed.
"""
import gc, socket, sys, time
import panel as pn
from bokeh.client import pull_session
from panel.io.state import state
N = int(sys.argv[1]) if len(sys.argv) > 1 else 5
class Payload:
"""Stands in for per-session app state (here: 20 MB)."""
def __init__(self):
self.data = bytearray(20 * 2**20)
def app():
pn.extension(notifications=True)
payload = Payload()
button = pn.widgets.Button(name="click")
button.on_click(lambda e: payload) # app state reachable from the doc's models
return pn.Column(button)
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
srv = pn.serve({"app": app}, port=port, address="127.0.0.1", show=False, threaded=True,
check_unused_sessions_milliseconds=100,
unused_session_lifetime_milliseconds=100)
while srv.server_id not in state._servers:
time.sleep(0.1)
bokeh_server = state._servers[srv.server_id][0]
for _ in range(N):
pull_session(url=f"http://127.0.0.1:{port}/app").close()
def live_sessions():
return sum(len(ctx.sessions) for ctx in bokeh_server._tornado._applications.values())
deadline = time.monotonic() + 30
while live_sessions() and time.monotonic() < deadline:
time.sleep(0.1)
time.sleep(1)
gc.collect()
dead = [(type(v).__name__) for v, _r, d, _c in state._views.values() if not hasattr(d, "_roots")]
payloads = sum(isinstance(o, Payload) for o in gc.get_objects())
from bokeh.document import Document
from bokeh.model import Model
docs = sum(isinstance(o, Document) and not hasattr(o, "_roots") for o in gc.get_objects())
models = sum(isinstance(o, Model) for o in gc.get_objects())
print(f"panel {pn.__version__} sessions opened+closed: {N} live sessions: {live_sessions()}")
print(f"state._views entries for destroyed docs: {len(dead)} {sorted(set(dead))}")
print(f"destroyed Documents still alive: {docs} bokeh Models alive: {models}")
print(f"Payload objects still alive: {payloads} (~{payloads * 20} MB)")
srv.stop()drive.py
"""Simulate N concurrent users hammering PixelMap until the container dies."""
import asyncio, os, random, sys, time
from playwright.async_api import async_playwright
URL = f"http://localhost:{os.environ.get('PORT', '5408')}/app"
CHROME = "/home/chris/.cache/ms-playwright/chromium_headless_shell-1243/chrome-headless-shell-linux64/chrome-headless-shell"
ATLASES = sys.argv[2].split(",")
PROBES = ["1.0", "2.0-1shank", "2.0-4shanks", "QuadBase"]
N_USERS = int(sys.argv[1])
DEADLINE = time.time() + float(sys.argv[3]) if len(sys.argv) > 3 else float("inf")
def log(*a): print(time.strftime("%H:%M:%S"), *a, flush=True)
async def wait_compute_done(pg, timeout=180_000):
await pg.locator("button", has_text="Compute anatomical overlay").wait_for(timeout=timeout)
async def user(browser, uid):
rng = random.Random(uid)
cycle = 0
while time.time() < DEADLINE:
cycle += 1
ctx = await browser.new_context(viewport={"width": 1600, "height": 1000})
pg = await ctx.new_page()
try:
await pg.goto(URL, timeout=60_000)
await wait_compute_done(pg, 60_000)
for _ in range(3 if os.environ.get("ANAT", "1") == "1" else 0):
atlas = rng.choice(ATLASES)
await pg.locator("select").nth(6).select_option(atlas)
await pg.locator("button", has_text="anatomical overlay").first.click()
await pg.wait_for_timeout(500)
await wait_compute_done(pg)
# Pose edits live-recompute the overlay.
for _ in range(3):
box = pg.locator("input").nth(rng.choice([11, 12, 13]))
await box.fill(str(rng.randint(-20, 20)))
await box.press("Enter")
await pg.wait_for_timeout(300)
# Plot interaction: taps + a box drag.
for _ in range(5):
await pg.mouse.click(rng.randint(450, 1150), rng.randint(60, 700))
await pg.wait_for_timeout(150)
x, y = rng.randint(450, 1000), rng.randint(60, 500)
await pg.mouse.move(x, y); await pg.mouse.down()
await pg.mouse.move(x + 150, y + 200, steps=5); await pg.mouse.up()
await pg.locator("button", has_text="Apply Preset").click()
await pg.wait_for_timeout(500)
for _ in range(5):
await pg.mouse.click(rng.randint(450, 1150), rng.randint(60, 700))
await pg.wait_for_timeout(150)
await pg.locator("select").nth(0).select_option(rng.choice(PROBES))
await pg.wait_for_timeout(1500)
log(f"user{uid} cycle {cycle} ok")
except Exception as e:
log(f"user{uid} cycle {cycle} error: {type(e).__name__}: {str(e).splitlines()[0][:120]}")
await asyncio.sleep(5)
finally:
await ctx.close()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(executable_path=CHROME)
await asyncio.gather(*(user(browser, i) for i in range(N_USERS)))
asyncio.run(main())monitor.sh
#!/bin/bash
# Log container memory every 5 s; exit when it stops running.
while true; do
st=$(docker inspect pm-repro --format '{{.State.Status}} oom={{.State.OOMKilled}} exit={{.State.ExitCode}} restarts={{.RestartCount}}' 2>&1)
mem=$(docker stats --no-stream --format '{{.MemUsage}}' pm-repro 2>/dev/null)
anon=$(docker exec pm-repro sh -c 'grep -E "^(anon|file) " /sys/fs/cgroup/memory.stat | awk "{printf \"%s=%dMB \", \$1, \$2/1048576}"' 2>/dev/null)
echo "$(date +%H:%M:%S) $st mem=[$mem] $anon"
case "$st" in running*) ;; *) exit 0;; esac
sleep 5
doneDebug page used for in-process inspection
The server was started with a second app served in the same process:
docker run -d --name pm-repro -p 5408:5008 -m 3500m --memory-swap 3500m \
-e PYTHONUNBUFFERED=1 -v pm-repro-bg:/root/.brainglobe \
-v /tmp/pmexp/dbg:/dbg:ro -v /tmp/pmexp/dbg/debug.py:/app/debug.py:ro \
--entrypoint /app/.venv/bin/python ghcr.io/m-beau/pixelmap:latest \
-m panel serve app.py debug.py --address 0.0.0.0 --port 5008 \
--allow-websocket-origin '*' --check-unused-sessions 10000 \
--unused-session-lifetime 60000 --liveness --mem-log-frequency 60000debug.py is just exec(open("/dbg/probe.py").read()), so each curl localhost:5408/debug runs whatever probe is currently in /dbg/probe.py inside the live server (Panel caches each served script's source, so the indirection allows iterating without a restart). Probes counted live ChannelmapGUI/Document/Model objects, walked gc.get_referrers, classified pn.state._views entries, removed stale entries to prove the fix, and wrapped state._notifications to log deletions.
Assisted-by: Claude-Code:Opus-5.5
Activity
fix: release Panel views left behind by destroyed sessions
fix: pin the image's Python to 3.14