Skip to content

fix: warm the first session before the server accepts connections - #92

Merged
tabedzki merged 2 commits into
m-beau:devfrom
tabedzki:tabedzki/fix/first-session-off-loop
Nov 11, 2026
Merged

tabedzki merged 2 commits into
m-beau:devfrom
tabedzki:tabedzki/fix/first-session-off-loop

Conversation

@tabedzki

@tabedzki tabedzki commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Removes the extra stall on the first visit after each server start. This is the separate create_app → … → is_downloaded → get_atlas stall noted in #91.

Summary

panel serve runs app.py per session on the event loop that also serves every other session and /liveness. The imports, wiring pickle and default atlas metadata are one-time costs, and the first visitor paid them on that loop. They now run before the port is bound, at the image's default NUM_PROCS=1.

 entrypoint.sh
   exec panel serve ./app.py
+    --setup ./scripts/warm_first_session.py   # builds one throwaway ChannelmapGUI
     --liveness ...

The throwaway GUI has no document, so it leaves no session behind, and analytics and session hooks don't run. A warmup failure logs a warning and the server still starts, because Panel aborts startup if a setup script raises.

Evidence

Real image, 4500M memory cap, headless browser loading three sessions while polling /liveness. "Before" is the same image with the --setup line removed. The 0.25 CPU rows show both runs of each.

Before After
First session, slowest /liveness, 8 CPUs 2.79 s 1.07 s
First session, slowest /liveness, 0.25 CPU 16.2 s, 15.6 s 4.8 s, 4.7 s
Server accepting connections, 8 CPUs 1.4 s 3.5 s
Server accepting connections, 0.25 CPU 5.4 s, 5.0 s 13.5 s, 14.1 s
Idle memory before the first visit 68 MiB 253 MiB
Memory after 3 sessions 307–316 MiB 304–314 MiB
  • With a simulated import failure in the real container, the warning was logged, /liveness returned 200 and the container kept running.
  • The overlay rendered in session 1 in all of these runs. Each session had a notification area in a separate non-container comparison with the --setup flag.
  • tests/test_startup_warmup.py covers the entrypoint flag, a fresh-process warmup, no leftover session state, failed warmups and imports, and the default atlas with and without a copy on disk. Against a mutant whose constructor raises, 4 of its 7 tests fail.
  • Full suite: 987 passed, 1 xfailed. ruff check . is clean.

Not fixed: every session, including the first, is still built on the event loop; this removes only the one-time costs. Later sessions block /liveness for about 0.8–0.9 s at 8 CPUs and 3.9–7.2 s at 0.25 CPU, so the watchdog can still fire under heavy contention, and it did at 0.25 CPU in both runs with this change. Those alerts will have the same create_app stack. Fixing that means making session construction cheaper, which is left for a separate PR.

Known limits:

  • With NUM_PROCS above 1, Panel runs the warmup on each process's event loop after the server starts, so it stalls about as much as a first session does today (from reading Panel's code, not run). Production uses 1.
  • A failed warmup only logs a warning and doesn't alert the webhook, so the first-visit stall could return unnoticed.

Merge Danger

Door: two-way

Revert, or remove the one --setup line. No data or format changes.

Blast Radius: startup

The port opens later, by about 2 s at 8 CPUs and 9 s at 0.25 CPU, so /liveness fails during that window and a reverse proxy may return errors. Idle memory is about 185 MiB higher until the first visit. The deployment's compose file needs no change as long as it keeps using the image's entrypoint.

Assisted-by: ClaudeCode:claude-opus-5-5

panel serve runs app.py per session on the event loop that serves every
session and /liveness, so the first visit after each start paid for the
imports, the wiring pickle and the default atlas's metadata on that loop.

Run one throwaway ChannelmapGUI through panel serve --setup instead, before
the port is bound. It has no document, so it leaves no session behind. A
failed warmup logs a warning and the server still starts.

Assisted-by: ClaudeCode:claude-opus-5-5
@codecov-commenter

codecov-commenter commented Nov 7, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (dev@faf32d9). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@          Coverage Diff           @@
##             dev      #92   +/-   ##
======================================
  Coverage       ?   85.71%           
======================================
  Files          ?       21           
  Lines          ?     3445           
  Branches       ?        0           
======================================
  Hits           ?     2953           
  Misses         ?      492           
  Partials       ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The script swallows every exception, so tests must also assert no failure
was logged, or they pass when the constructor raises. Cover opening the
default atlas when it is on disk, and share the run-and-collect-failures
helper.

Say that the warmup only runs before the port is bound with NUM_PROCS=1,
and drop the bytecode and "exactly what a session needs" claims.

Assisted-by: ClaudeCode:claude-opus-5-5
@tabedzki
tabedzki requested a review from m-beau November 7, 2026 21:15
@tabedzki
tabedzki merged commit e961d2e into m-beau:dev Nov 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants