Skip to content

Production container OOM-killed: Panel leaks every closed session (+ freezes, peaks, broken image build) #6

Open

Description

@tabedzki

🤖 AI text below 🤖

Summary

The production container (ghcr.io/m-beau/pixelmap:latest, v1.4.0, at pixelmap.pni.princeton.edu) runs for a while and then dies with no traceback. This issue documents the investigation, the reproduction, the root causes found so far, the fixes in progress, and what is still open.

Root causes identified

  1. OOM kills. Production RSS idled at about 3.4 GB against a 3500 MB container limit, and the highest logged RSS was 3,485.53 MB. Reproduced locally: OOMKilled=true, exit 137.
  2. A Panel bug leaks every closed session (Panel 1.7.5 and the latest 1.9.4). A per-session NotificationArea is left in pn.state._views and pins the destroyed Bokeh Document, and in PixelMap often the session's entire ChannelmapGUI. Minimal reproduction below.
  3. Event-loop freezes of 20–90 s on electrode clicks, caused by a gc.collect() on every selection. These are separate from the deaths, but they fail healthchecks and drop websockets.
  4. Large peaks from concurrent sessions and atlas overlays. Still open.
  5. The image build is broken for the next release, independent of the above: the Dockerfile doesn't pin Python, so uv picks CPython 3.15, which dracopy 2.0.0 has no wheel for.

1. Production evidence

From about 14 h of production logs (relayed through the Slack notifier):

  • RSS between 3,120 and 3,485 MB the whole time; VMS only grows (4,150 → 4,360 MB). A fresh container starts at about 258 MB, so this is built up during runtime, not by the image or the brainglobe volume. The volume's 21 downloaded atlases cost nothing until an overlay is computed.
  • The 60 s --mem-log-frequency line fired 20–90 s late 23 times, nearly always right after Selection changed: [] -> [N], so the single event loop was blocked.
  • During one stall the proxy dropped a websocket: Failed sending message as connection was closed, then 236 × AssertionError: assert self._session is not None on queued PATCH-DOC messages.
  • The live-session count never dropped below 3 in 14 h.
  • The production compose sets deploy.resources.limits.memory: 3500M. Plain Docker doesn't restart unhealthy containers and the notifier only reports, so the deaths are OOM kills, not healthcheck restarts.

2. How it was reproduced

All runs used the production image (or a build from the fix branch) under a hard memory cap with swap disabled (-m <cap> --memory-swap <cap>). They were driven by headless Chromium through Playwright: N concurrent "users", each looping through:

  • open the app, pick an atlas and compute the anatomy overlay
  • edit the pose three times (each edit recomputes the overlay live)
  • 5 clicks and a box drag on the plot, then Apply Preset (three rounds of the above)
  • change the probe type, then close the tab

Memory was sampled every 5 s from docker stats and the cgroup's memory.stat (anon = process memory, file = page cache). Scripts are at the bottom.

Run Image Cap Load Result
1 prod v1.4.0 1500 MB 4 users, atlases incl. 10 µm OOM-killed (exit 137) after ~1 min 50 s. 10 live sessions for 4 users; died while two 10 µm atlases were in use
A/B prod v1.4.0 3500 MB 4 users, 2–5 min, no atlases Idle after load: ~1.3–1.5 GB, never returns
C prod + debug page 3500 MB 4 users, 2 min, no atlases; 41 sessions, 41 cleaned up 0 live sessions, but 16 ChannelmapGUI alive, 43 Documents, 1.48 M Electrode objects; malloc_trim frees only 57 MB, so these are live objects
in-proc local, one GUI n/a cycle 5 atlases ×2 (incl. 10 µm) Peak ~750 MB, settles ~430 MB, no growth on round 2: single-user atlas switching is fine
fix fixed build 3500 MB same as A/B; 40 sessions, 40 cleaned up Idle after load: ~660 MB (vs ~1.3–1.4 GB prod)
fix fixed build 1500 MB same as run 1 Still OOM-killed after ~1 min: 11 live sessions + atlas peaks exceed 1500 MB (capacity, see §5)
side by side prod vs fixed 3500 MB each 4 users, atlases, 10 min, then idle Both survived. Peak RSS 2,239 vs 2,248 MB. Idle after cleanup: 1,714 MB prod vs 1,135 MB fixed
round 2 same containers 3500 MB another 5 min of the same Idle: prod 1,714 → 1,520 MB, fixed 1,135 → 1,155 MB

Round 2 did not show production climbing further within that window. The leaked memory is partly reused, so a 15-minute test can't distinguish linear growth from a plateau. The production numbers (3.4 GB after days) and the minimal reproduction (one leaked Document per closed session, linear) are the stronger evidence for growth. A long soak test is listed as open.

3. Root cause of the leak (Panel)

Found by serving a second "debug" app in the same server process and walking gc referrers from a leaked ChannelmapGUI/Document:

  1. pn.state._views is a plain dict holding (view, root, doc, comm). After all sessions were destroyed it still had 20 entries, all panel.io.notifications.NotificationArea, each pointing at a destroyed Document.
  2. Removing just those entries in the live server freed all leaked GUIs: RSS 927 → 545 MB, no restart.
  3. Why they're never removed: the notification area's _server_destroy (which would drop the entry) is only reached from state._destroy_session through state._notifications[doc]. Tracing deletions on state._notifications showed Panel's own Document.destroy override, panel/io/document.py _destroy_document, running del state_obj[doc] over every per-document WeakKeyDictionary, _notifications included. Bokeh calls session.destroy() before running the on_session_destroyed callbacks, so by the time state._destroy_session runs, doc in state._notifications is False and the cleanup is skipped. Instrumented trace:
    [wkd] __delitem__ called from:
      bokeh/server/contexts.py:283 do_discard -> session.destroy()
      bokeh/server/session.py:188 destroy -> self._document.destroy(self)
      panel/io/document.py:254 _destroy_document -> del state_obj[self]
    [wkd] _destroy_session doc_in_notifications=False roots=DESTROYED
    
  4. The same logic is present in Panel 1.9.4 (latest), so upgrading doesn't fix it.

In PixelMap, the orphaned Document often keeps the session's ChannelmapGUI alive (in run C, 16 of 41 sessions, about 90k Electrode objects each, about 70–100 MB per leaked session). The minimal reproduction below shows the Document leak deterministically. The extra step from the dead Document to app state only appeared with real browser traffic, and that exact path isn't fully pinned down (see open items).

Minimal reproduction (Panel only, no PixelMap)

$ uv run --no-project --python 3.13 --with "panel==1.9.4" python mre.py 50
panel 1.9.4  sessions opened+closed: 50  live sessions: 0
state._views entries for destroyed docs: 50 ['NotificationArea']
destroyed Documents still alive: 50  bokeh Models alive: 103

Same result on 1.7.5. Remove pn.extension(notifications=True) and the entries go away.

4. Fixes

  • Draft PR fix: stop per-click gc.collect stalling the server; add stall/RSS watchdog #5: removes the per-click gc.collect() (the freezes) and adds a [watchdog] that logs the blocked thread's stack when the event loop stalls, plus a one-time warning when RSS crosses a threshold. Includes regression tests.
  • Branch tabedzki/fix/panel-session-leak (not yet pushed):
    • _release_dead_session_views(), registered with on_session_destroyed in create_app. It drops every pn.state._views entry still pointing at the dead session's Document, calling _cleanup first and tolerating failures. It works regardless of hook order.
    • tests/test_session_leak.py: an end-to-end test with a real in-process Panel server that opens and closes sessions and asserts no _views entry for a destroyed Document survives. It fails before the fix and passes after, together with unit tests for the hook's edge cases. Full suite: 530 passed.
    • Dockerfile: ENV UV_PYTHON=3.14. Without it, docker build currently fails (dracopy==2.0.0 ... doesn't have a source distribution or wheel for the current platform under CPython 3.15). The next tagged release would fail in docker-publish.yml.

5. Still open

  • Long soak test (hours, realistic traffic) on the fixed image, to show production growth versus a plateau directly.
  • Idle floor after atlas work is ~1.1 GB, not ~300 MB. Likely the last-used atlas volume stays resident (closing a tab doesn't call release_atlas_memory the way "Clear overlay" does) plus allocator high-water. Candidate fix: release the atlas and malloc_trim when the last session is destroyed.
  • Peak capacity. About 50 MB per session (closed tabs linger 60 s under --unused-session-lifetime 60000) plus about 400 MB per 10 µm overlay computation. Concurrent large-atlas computations can still exceed the limit. Options: serialize or limit large-atlas computations, block very fine (≤10 µm) atlases on the server, shorten the unused-session lifetime, and size the limit from measured peaks.
  • Exact path from the orphaned Document to ChannelmapGUI under real browser traffic. The fix covers it (clearing _views freed every GUI), but it matters for the upstream report.
  • Report upstream to Panel with the minimal reproduction.
  • Forward [watchdog] log lines to Slack (the production notifier only watches Docker events), and check docker inspect --format '{{.State.OOMKilled}}' on future deaths.
  • Merge PR fix: stop per-click gc.collect stalling the server; add stall/RSS watchdog #5; push the leak-fix branch and open its PR.

Scripts

mre.py
"""Panel leaks every closed session when notifications are enabled.

Run: python mre.py   -> prints how many state._views entries still point at
destroyed Documents after N sessions are opened and closed.
"""
import gc, socket, sys, time

import panel as pn
from bokeh.client import pull_session
from panel.io.state import state

N = int(sys.argv[1]) if len(sys.argv) > 1 else 5


class Payload:
    """Stands in for per-session app state (here: 20 MB)."""
    def __init__(self):
        self.data = bytearray(20 * 2**20)


def app():
    pn.extension(notifications=True)
    payload = Payload()
    button = pn.widgets.Button(name="click")
    button.on_click(lambda e: payload)  # app state reachable from the doc's models
    return pn.Column(button)


with socket.socket() as s:
    s.bind(("127.0.0.1", 0))
    port = s.getsockname()[1]

srv = pn.serve({"app": app}, port=port, address="127.0.0.1", show=False, threaded=True,
               check_unused_sessions_milliseconds=100,
               unused_session_lifetime_milliseconds=100)
while srv.server_id not in state._servers:
    time.sleep(0.1)
bokeh_server = state._servers[srv.server_id][0]

for _ in range(N):
    pull_session(url=f"http://127.0.0.1:{port}/app").close()

def live_sessions():
    return sum(len(ctx.sessions) for ctx in bokeh_server._tornado._applications.values())

deadline = time.monotonic() + 30
while live_sessions() and time.monotonic() < deadline:
    time.sleep(0.1)
time.sleep(1)
gc.collect()

dead = [(type(v).__name__) for v, _r, d, _c in state._views.values() if not hasattr(d, "_roots")]
payloads = sum(isinstance(o, Payload) for o in gc.get_objects())
from bokeh.document import Document
from bokeh.model import Model
docs = sum(isinstance(o, Document) and not hasattr(o, "_roots") for o in gc.get_objects())
models = sum(isinstance(o, Model) for o in gc.get_objects())
print(f"panel {pn.__version__}  sessions opened+closed: {N}  live sessions: {live_sessions()}")
print(f"state._views entries for destroyed docs: {len(dead)} {sorted(set(dead))}")
print(f"destroyed Documents still alive: {docs}  bokeh Models alive: {models}")
print(f"Payload objects still alive: {payloads}  (~{payloads * 20} MB)")
srv.stop()
drive.py
"""Simulate N concurrent users hammering PixelMap until the container dies."""
import asyncio, os, random, sys, time
from playwright.async_api import async_playwright

URL = f"http://localhost:{os.environ.get('PORT', '5408')}/app"
CHROME = "/home/chris/.cache/ms-playwright/chromium_headless_shell-1243/chrome-headless-shell-linux64/chrome-headless-shell"
ATLASES = sys.argv[2].split(",")
PROBES = ["1.0", "2.0-1shank", "2.0-4shanks", "QuadBase"]
N_USERS = int(sys.argv[1])
DEADLINE = time.time() + float(sys.argv[3]) if len(sys.argv) > 3 else float("inf")

def log(*a): print(time.strftime("%H:%M:%S"), *a, flush=True)

async def wait_compute_done(pg, timeout=180_000):
    await pg.locator("button", has_text="Compute anatomical overlay").wait_for(timeout=timeout)

async def user(browser, uid):
    rng = random.Random(uid)
    cycle = 0
    while time.time() < DEADLINE:
        cycle += 1
        ctx = await browser.new_context(viewport={"width": 1600, "height": 1000})
        pg = await ctx.new_page()
        try:
            await pg.goto(URL, timeout=60_000)
            await wait_compute_done(pg, 60_000)
            for _ in range(3 if os.environ.get("ANAT", "1") == "1" else 0):
                atlas = rng.choice(ATLASES)
                await pg.locator("select").nth(6).select_option(atlas)
                await pg.locator("button", has_text="anatomical overlay").first.click()
                await pg.wait_for_timeout(500)
                await wait_compute_done(pg)
                # Pose edits live-recompute the overlay.
                for _ in range(3):
                    box = pg.locator("input").nth(rng.choice([11, 12, 13]))
                    await box.fill(str(rng.randint(-20, 20)))
                    await box.press("Enter")
                    await pg.wait_for_timeout(300)
                # Plot interaction: taps + a box drag.
                for _ in range(5):
                    await pg.mouse.click(rng.randint(450, 1150), rng.randint(60, 700))
                    await pg.wait_for_timeout(150)
                x, y = rng.randint(450, 1000), rng.randint(60, 500)
                await pg.mouse.move(x, y); await pg.mouse.down()
                await pg.mouse.move(x + 150, y + 200, steps=5); await pg.mouse.up()
                await pg.locator("button", has_text="Apply Preset").click()
                await pg.wait_for_timeout(500)
            for _ in range(5):
                await pg.mouse.click(rng.randint(450, 1150), rng.randint(60, 700))
                await pg.wait_for_timeout(150)
            await pg.locator("select").nth(0).select_option(rng.choice(PROBES))
            await pg.wait_for_timeout(1500)
            log(f"user{uid} cycle {cycle} ok")
        except Exception as e:
            log(f"user{uid} cycle {cycle} error: {type(e).__name__}: {str(e).splitlines()[0][:120]}")
            await asyncio.sleep(5)
        finally:
            await ctx.close()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(executable_path=CHROME)
        await asyncio.gather(*(user(browser, i) for i in range(N_USERS)))

asyncio.run(main())
monitor.sh
#!/bin/bash
# Log container memory every 5 s; exit when it stops running.
while true; do
  st=$(docker inspect pm-repro --format '{{.State.Status}} oom={{.State.OOMKilled}} exit={{.State.ExitCode}} restarts={{.RestartCount}}' 2>&1)
  mem=$(docker stats --no-stream --format '{{.MemUsage}}' pm-repro 2>/dev/null)
  anon=$(docker exec pm-repro sh -c 'grep -E "^(anon|file) " /sys/fs/cgroup/memory.stat | awk "{printf \"%s=%dMB \", \$1, \$2/1048576}"' 2>/dev/null)
  echo "$(date +%H:%M:%S) $st mem=[$mem] $anon"
  case "$st" in running*) ;; *) exit 0;; esac
  sleep 5
done
Debug page used for in-process inspection

The server was started with a second app served in the same process:

docker run -d --name pm-repro -p 5408:5008 -m 3500m --memory-swap 3500m \
  -e PYTHONUNBUFFERED=1 -v pm-repro-bg:/root/.brainglobe \
  -v /tmp/pmexp/dbg:/dbg:ro -v /tmp/pmexp/dbg/debug.py:/app/debug.py:ro \
  --entrypoint /app/.venv/bin/python ghcr.io/m-beau/pixelmap:latest \
  -m panel serve app.py debug.py --address 0.0.0.0 --port 5008 \
  --allow-websocket-origin '*' --check-unused-sessions 10000 \
  --unused-session-lifetime 60000 --liveness --mem-log-frequency 60000

debug.py is just exec(open("/dbg/probe.py").read()), so each curl localhost:5408/debug runs whatever probe is currently in /dbg/probe.py inside the live server (Panel caches each served script's source, so the indirection allows iterating without a restart). Probes counted live ChannelmapGUI/Document/Model objects, walked gc.get_referrers, classified pn.state._views entries, removed stale entries to prove the fix, and wrapped state._notifications to log deletions.

Assisted-by: Claude-Code:Opus-5.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions