Defer background-tab scrollback restoration on startup (APP-5257) - #15174
Defer background-tab scrollback restoration on startup (APP-5257)#15174warp-agent-staging[bot] wants to merge 5 commits into
Conversation
…PP-5257) Root cause (per the linear-attached heap-profile analysis, Sentry issue 7259255054): on macOS startup, restoring a window's tabs eagerly fed every tab's persisted scrollback blocks and AI conversation history into its terminal view, even for tabs the window doesn't display at launch. The text-layout/rendering work this triggers dominates the reported heap growth (39.8%/3.83 GB of the profiled sample). This change defers that specific work for any tab that isn't the window's initially-active tab, behind a new `FeatureFlag::LazyBackgroundTabScrollbackRestore` (default off): the terminal session/shell for every tab still starts eagerly as before (no product-visible timing change to shell startup), but the persisted blocks and AI conversation restoration are stashed and only applied to the live terminal view when the tab is first activated. - `PaneGroup::restore_pane_leaf` skips feeding block/conversation data into `create_session` for non-active tabs when the flag is on, and stashes the original snapshot plus deferred payload in a new `pending_lazy_terminal_restorations` map. - `PaneGroup::snapshot_for_node` echoes back the original, unmodified snapshot for any pane still pending, so a tab closed or persisted before ever being activated round-trips its history intact. - `Workspace::set_active_tab_index` calls the new `PaneGroup::materialize_lazy_tab_restorations`, which applies the deferred blocks (via a new `BlockList::apply_deferred_restored_blocks`, modeled on the existing `append_followup_shared_session_scrollback`) and conversation restoration (via `TerminalView::restore_conversations_on_view_creation`, visibility bumped to `pub(crate)`) the first time a tab is shown. CHANGELOG-IMPROVEMENT: Reduced memory usage on startup for sessions with many restored background tabs, by deferring restoration of their scrollback and AI conversation history until the tab is first opened. Co-Authored-By: Warp <agent@warp.dev>
warp-agent-staging
Bot
added
factory:wilson
area:performance:memory
|
This PR was generated with Warp.   |
…ments Addresses an independent review of the initial implementation: 1. The deferral never actually happened: configure_new_workspace's restore loop called add_tab_with_pane_layout_and_active_hint, which activated every tab as it was inserted (activate_tab_internal), immediately draining each tab's pending_lazy_terminal_restorations. Added Workspace::add_restored_tab, a non-activating insertion path used only by the restore loop; the intended active tab is activated once, after the whole loop completes. Added an end-to-end regression test that drives the real configure_new_workspace loop (not just PaneGroup in isolation) and asserts a background tab's restoration is still pending after restore. 2. Materialization could truncate a live command: a background tab's shell stays live while deferred, so it can produce genuine content (e.g. via Workspace::process_sync_event_for_all_synced_pane_groups mirroring synced input) before activation. BlockList::apply_deferred_restored_blocks now checks whether the session is still pristine (bootstrapped but no real post-bootstrap command has started) before finishing the active block, and drops the deferred restoration rather than force-finish and misorder live output. Returns whether it actually applied. Added a regression test covering a live, still-running command. 3. Pending entries were never cleaned up on close or move: PaneGroup::cleanup_closed_pane now drops the pending entry for a permanently-closed pane. Added take_pending_lazy_terminal_restoration/insert_pending_lazy_terminal_restoration so remove_pane_for_move's three call sites (child-agent re-adoption, tab-bar drop, cross-tab pane move) transfer a still-pending payload to the destination pane group instead of silently dropping it. Added a lifecycle test for the close path. 4. Trimmed doc comments that narrated mechanics or named callers (restore_pane_leaf, restore_pane_tree, new_with_panes_layout, add_restored_tab, the pending_lazy_terminal_restorations field) down to why-only rationale, per AGENTS.md. 5. Moved the CHANGELOG-IMPROVEMENT line into the PR body as well. Validation: cargo test -p warp --lib for pane_group::, workspace::, and terminal::model::blocks:: all pass (122 + 210 + 67 tests). ./script/format and cargo clippy -p warp --all-targets --tests -- -D warnings are clean. Co-Authored-By: Warp <agent@warp.dev>
![warp-agent-staging[bot]](https://avatars.githubusercontent.com/in/2196476?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
Defers restored scrollback and AI-conversation restoration for background tabs at startup behind a default-off flag, targeting the 4.29 GB the APP-5257 heap profile attributes to PaneGroup::restore_pane_tree. Two rounds of findings are already fixed in df5d53d; what remains needs a human decision, so this is neither an approval nor a rejection.
Concerns
- Needs verification on a Mac before the flag is enabled anywhere, including dogfood. This changes when a user's terminal history and agent conversations appear, and no one in the factory can run the macOS app. Please exercise four cases: launch with many restored tabs, first activation of a tab whose shell already produced output, quit before ever activating a restored tab, and closing or moving a tab that was never activated.
- Deferred conversations are invisible to cross-tab consumers until their tab is opened. They are taken out of
RestoredAgentConversationsduring restore and only enterBlocklistAIHistoryModelat materialization, so conversation navigation and listing cannot see a background tab's restored conversations beforehand. Confirm that temporary invisibility is acceptable, or the restoration should keep lightweight conversation metadata registered up front. - The follow-up this PR deliberately leaves open. Only the scrollback and conversation payload is deferred; the per-pane terminal view and session are still constructed eagerly for every restored tab, which is the remaining part of the profile's 4.29 GB. Closing that gap means dormant tabs whose shell does not start until first click — a user-visible change worth deciding on its own.
Verdict
Checks: build pass, tests pass, CI required jobs skipped while draft, visual proof missing (needs macOS)
Found: 0 critical, 0 important, 0 suggestions, 0 nits, 3 questions
Responding as wilson: Open session · View factory task
| // `started()` (e.g. the script-execution stage), which is expected | ||
| // and harmless to finish here. Only bail once bootstrapping is done | ||
| // and a real post-bootstrap command has genuinely started. | ||
| if self.is_bootstrapping_precmd_done() && self.active_block().started() { |
There was a problem hiding this comment.
When this guard fires the restored scrollback is silently dropped from the live view rather than spliced in, because BlockList.blocks is append-only and cannot interleave history ahead of live content without a reindexing change. Dropping beats corrupting a running command, and the persisted blocks on disk are untouched, but the user sees a restored tab that is missing its history with only a Sentry report to show for it. Worth confirming this is the tradeoff you want, and worth watching that report_error! after rollout to learn how often it actually fires.
Responding as wilson: Open session · View factory task
macOS verification of the previous revision confirmed the deferral itself now works correctly (background tabs' restoration stays pending until activation, materializes cleanly on click), but found a genuine flag-ON-only regression: a pending background tab's title/subtitle showed a generic "New session" placeholder instead of a command-derived label, and the real title never took over even after activation. Root cause: `TerminalView::last_completed_command_text()` (used by the vertical-tabs title/subtitle rendering) scans the pane's live block list for the last completed command. For a deferred background tab that list is empty until materialization, so it fell through to a hardcoded "New session" placeholder. And `materialize_lazy_tab_restorations` never emitted any event after applying the deferred blocks, so nothing told the workspace to re-render the tab bar with the now-current state. Fixes both, without touching TabData's optionality or the tab-bar render path: - Added `SerializedBlock::plain_text_command_preview`, a best-effort ANSI/OSC-stripping text extractor (with unit tests covering plain commands, CSI color codes, and an OSC title escape sequence). This is intentionally not the full ANSI-aware parsing `restore_block` performs; it only needs to produce a short, readable title string. - `PaneGroup::restore_pane_leaf` computes this preview once from the deferred blocks and stores it on the new `TerminalView`, via a new `pending_restoration_title_hint` field and setter. - `TerminalView::last_completed_command_text` falls back to that hint when the live block list has no match yet, so a pending tab shows a real command-derived title/subtitle instead of the generic placeholder. - `materialize_lazy_tab_restorations` clears the hint once the real blocks are applied, and now emits `Event::TerminalViewStateChanged` / `Event::AppStateChanged` (the same events every other state-changing method on `PaneGroup` emits) so the workspace actually re-renders the tab with the fresh, real title. Added coverage for both the pending-tab title and the post-materialization title recovery in a single test, plus dedicated tests for the new ANSI preview stripper. Validation: cargo test -p warp --lib for pane_group:: (123), terminal::model:: (605), and workspace:: (210) all pass. ./script/format and cargo clippy -p warp --all-targets --tests -- -D warnings are clean. Co-Authored-By: Warp <agent@warp.dev>
Two scoped fixes from review, orthogonal to the structural scrollback- ordering issue tracked separately (see PR description): - materialize_lazy_tab_restorations unconditionally cleared pending_restoration_title_hint, even when apply_deferred_restored_blocks declined to apply (the "session no longer pristine" guard). On that path the tab lost both its restored scrollback and its title fallback, with no route back to a usable label. The hint is now only cleared once has_restored_command_blocks confirms the blocks actually landed. - cleanup_closed_pane, added in the previous revision, is not on the path a real tab close takes: closing a tab goes through Workspace::remove_tab -> UndoCloseStack::handle_tab_closed, and the pane group is only permanently discarded later when the undo grace period expires, via ClosedItem::discard -> PaneGroup::clean_up_panes. That method never touched pending_lazy_terminal_restorations, so a tab closed (and eventually permanently discarded) before its background panes were ever activated retained their pending payloads until the whole PaneGroup was dropped. clean_up_panes now clears the map explicitly, matching cleanup_closed_pane's per-pane undo path. Added test_clean_up_panes_drops_pending_lazy_restorations covering the real teardown path, and removed a pane-group-level test that depended on the pristine guard reliably firing against a real TerminalModel/PTY-backed session; it does not (see PR description), so the earlier test was not a trustworthy regression check. Validation: cargo test -p warp --lib pane_group:: (124 passed) and workspace:: (210 passed). ./script/format and cargo clippy -p warp --all-targets --tests -- -D warnings clean. Co-Authored-By: Warp <agent@warp.dev>
Verified on real macOS — result: failedBuilt and ran this branch on a Warp-hosted macOS 26.3.1 aarch64 runner. Scenarios on Blocking: restored history renders below newer live output. With live output on a background pty before first activation, the live line lands as the first block, above the entire restored history — screenshot · video. The non-deferred tab in the same run ordered correctly. All three materializations logged Titles: half fixed as of What held up. Deferral is real: flag-ON logged Unverified, stated as gaps rather than passes: the AI-conversation half of the deferral (runner cannot sign in; every line read Responding as wilson: Open session · View factory task |
macOS verification showed the tab-title fix regressed at the opposite point: the label was correct while a background tab was pending, but degraded to a plain cwd label right after activation. Logs showed blocks applied and the hint cleared as designed, but the live block-list scan that was supposed to take over produced nothing on the real (PTY-backed) session that materialization exercises. Rather than chase why the live scan misses immediately after materialization on a real session, stop discarding the value that is known to work: pending_restoration_title_hint is no longer cleared once blocks land. It's one small string per restored pane, and it only ever acts as a fallback behind a live-scan match in TerminalView::last_completed_command_text, so retaining it for the pane's lifetime is cheap and can't override a real title once the live scan succeeds. Strengthened test_lazy_background_tab_reports_a_title_while_pending_and_after_materializing to assert equality against a genuine eager-restoration reference for the same block, rather than just checking for a fixed string — the previous version of this test passed against a value the deferred path had cached from before materialization, without confirming it matches what real, non-deferred restoration would show. Validation: cargo test -p warp --lib pane_group:: (124 passed) and workspace:: (210 passed). ./script/format and cargo clippy -p warp --all-targets --tests -- -D warnings clean. Co-Authored-By: Warp <agent@warp.dev>
Measured on macOS: the restore path costs ~7.8 MB per pane, and graphics allocation is flatWe built stock Pane count is the only linear driver — 7.79 MB/pane, intercept 166.5 MB, linear across a 32× range with no bend (means: 177.9 / 197.6 / 221.0 / 293.6 / 358.4 / 412.7 MB at 1 / 4 / 8 / 16 / 24 / 32 panes). Launch-to-launch variance is ~13 MB, about 1.7 panes' worth. Graphics allocation does not scale with panes. Restored content saturates. Per-pane persisted payload caps at ~2.80 MB — emitting 5× more scrollback (200k→1M lines/pane) moved restored bytes by 72 bytes and footprint not at all. Marginal cost per restored block collapses 91 KB → 13.8 KB → 0.83 KB. All per-pane growth is jemalloc heap ( This does not reach the reported footprints. At 7.8 MB/pane, the 4 GB attributed to restore needs ~510 panes and 11.6 GB needs ~1480. Nothing we could vary — pane count, scrollback volume, block count — gets within two orders of magnitude of the field reports. A hypothesis for the gap, offered as such: in these profiles Bounds, stated plainly: Apple Paravirtual GPU (hardware-backed Metal, but Evidence: restored 16-pane session · second restored pane · minimized-state sample · video: scrolling, pane switch, minimize Responding as wilson: Open session · View factory task |
|
Superseded by #15833, which bounds what AI conversation restoration materializes instead of deferring it. Deferral turned out not to be the answer here: the active tab is never deferred, so it materialized the full unbounded payload at startup regardless, and a deferred background tab materialized the identical payload later. That also makes the ordering defect documented above moot in the new approach, since nothing is deferred. Leaving this open rather than closing it — that call is the reviewer's. The measurement and findings recorded here stand on their own and are worth keeping. Responding as wilson: Open session · View factory task |
FeatureFlag::LazyBackgroundTabScrollbackRestoremust stay off in every environment. With it enabled, a background tab's restored (older) scrollback can render below live output the tab produced before it was ever activated — inverted, incorrect chronological order, visible to the user on first activation. This is not a bug awaiting a small patch: it is a structural property of this PR's approach (explained below), and closing it requires a different design, not a follow-up fix to this code. See "Known limitation" below before touching the flag or building on this code.Description
APP-5257 / Sentry issue 7259255054: on macOS, restoring a window's tabs at startup fed every tab's persisted scrollback blocks and AI conversation history into its terminal view eagerly — including tabs the window doesn't display at launch.
The premise behind this PR does not survive measurement — read this before judging the change. The heap profiles for this signature attribute 11.6 GB of a 15.9 GB sample below
PaneGroup::restore_pane_tree, fanning into AppKit/CoreGraphics/Metal frames; that was read first as per-pane block/text layout, then as per-pane view/scene construction. A direct measurement of the same restore path on macOS 26.3.1, on a stable-equivalent build (release + jemalloc), contradicts both readings:IOSurfaceis byte-identical at 40.7 MB with exactly 6 regions at 1, 4, 8, 16, 24 and 32 panes;IOAccelerator,CoreAnimation,CoreGraphicsandIOKitare likewise identical. Surfaces are allocated per window, not per pane, so there is no per-pane surface to defer.All of the per-pane growth is jemalloc heap plus ~0.33 MB of per-pane thread stack. Since this PR defers only the scrollback/conversation payload — the component measured to saturate — its expected benefit is small independently of the ordering defect below. Caveat on the negative result: the measurement ran on an Apple Paravirtual GPU (hardware-backed Metal, but
supportsFamily(.apple7)/.metal3false), and never entered the memory-pressure regime the field reports show, so a real M-series driver behaving differently is not excluded.This PR's approach: for background tabs (any tab other than the window's initially-active one), defer feeding persisted scrollback blocks and AI conversation history into the terminal view until the tab is first activated, while still starting every tab's terminal session/shell eagerly at launch exactly as before. Keeping the shell eager was deliberate — it avoids the product decision that deferring the whole pane (session included) would raise. All of it is gated behind a new, default-off
FeatureFlag::LazyBackgroundTabScrollbackRestore.CHANGELOG-IMPROVEMENT: Reduced memory usage on startup for sessions with many restored background tabs, by deferring restoration of their scrollback and AI conversation history until the tab is first opened.
Known limitation: this approach cannot be made ordering-correct
Because a background tab's shell stays live while its restoration is deferred, the tab's session can produce genuine PTY output — shell-integration chatter,
direnv, a finished background job, SSH noise — without ever starting a command block, before the tab is ever activated.apply_deferred_restored_blocks's safety guard (checkingis_bootstrapping_precmd_done() && active_block().started()) does not detect this as a live, non-pristine session, so on activation the deferred (older) restored history is appended after output that already arrived — the tab renders with live content above its entire restored session.Widening the guard would only convert misordering into more frequent silent dropping of the user's scrollback, which is worse, so that is not a path forward. The two structural alternatives:
TerminalModelfrom a dedicated background OS thread, independent of and prior to any App-side scheduling. A correct version needs real cross-thread synchronization gating PTY bytes on a hot, per-byte path.BlockList's block vector and index map are structured for append-only growth; inserting older content earlier requires reindexing every subsequent block and auditing every cached block index elsewhere in the codebase (selection, scroll position, etc.) — a core-data-structure change, not a local one.Both exceed a "smallest correct change" PR and are not implemented here. The underlying tension is structural: keeping every shell eager (to sidestep the product decision around deferring whole panes) is exactly what removes any guarantee that restored history stays behind live output, since there is always a window in which live output can reach the append-only block list first. Deferring the whole pane — including its session — has no such window, since there is no live output to interleave with; that option was set aside earlier specifically because it raises the product decision this approach was chosen to avoid. Resolving this is a decision for the feature's owner about which tradeoff to accept, not an implementation gap in this PR.
What this PR implements
PaneGroup::restore_pane_leafskips passing block-list/conversation-restoration data intoPaneGroup::create_sessionfor tabs that aren't the window's initially-active tab, stashing the original snapshot plus the deferred payload in a newpending_lazy_terminal_restorationsmap onPaneGroup.Workspace::add_restored_tabinserts a restored tab without activating it (unlikeadd_tab_with_pane_layout, which activates on insertion).configure_new_workspace's restore loop uses it for every tab, then activates the window's real active tab once, after the loop completes.PaneGroup::snapshot_for_nodereturns the original, unmodified snapshot for any pane still pending, so a background tab closed (or the app quit) before ever being activated round-trips its full history intact.Workspace::set_active_tab_indexcallsPaneGroup::materialize_lazy_tab_restorationson first activation, applying the deferred blocks (BlockList::apply_deferred_restored_blocks) and AI conversation restoration (TerminalView::restore_conversations_on_view_creation). Idempotent.apply_deferred_restored_blocksdeclines to apply (no-ops) if its (incomplete, see above) pristine check fails, rather than force-finishing and misordering live output.PaneGroup::cleanup_closed_paneandPaneGroup::clean_up_panes(the latter is the real teardown path for a closed tab, reached onceUndoCloseStack's grace period expires) both release a never-activated pane's pending payload.remove_pane_for_move's call sites transfer a still-pending payload to the destination pane group instead of dropping it.SerializedBlock::plain_text_command_preview, a best-effort ANSI/OSC-stripping extractor) instead of a generic placeholder while pending. The hint is retained for the pane's lifetime rather than cleared after materialization, so a live block-list match always takes precedence but a real title is never lost if that live scan comes up empty.Not in scope
Deferring the per-pane terminal view/session construction itself (PTY spawn, terminal model) — which the symbolicated profile above indicates is the larger share of the cost, not a residual — is a product-visible behavior change (background tabs wouldn't start their shell until opened) requiring its own decision; not attempted here.
Deferred AI conversations are temporarily invisible to cross-tab consumers of
RestoredAgentConversationsbetween restore and materialization, mirroring the existing lazy pattern for AI-document panes; not exercised on a real build (the verification runner was signed out).There is currently no way to enable this flag on a real build even if it were safe to (
override_enabledis test-only, no production path sets the preference, the Debug-menu toggle doesn't survive relaunch) — moot given the flag must stay off regardless.Linked Issue
ready-to-specorready-to-implement.Linear: https://linear.app/warpdotdev/issue/APP-5257/memory-2195-gb-35percent-resident-65percent-compressed-mac-stable
Sentry: https://sentry.io/organizations/warpdotdev/issues/7259255054/
Testing
This is a backend/data-flow change with essentially no UI surface of its own (aside from tab titles). A maintainer with macOS + computer-use access verified the restore/materialize flow directly on a real build across two rounds:
Automated tests (all pass):
workspace::view::tests::test_restoring_a_window_defers_background_tab_scrollback_through_the_real_loop(drives the real restore loop),pane_group::tests::test_lazy_background_tab_reports_a_title_while_pending_and_after_materializing,terminal::model::block::serialized_block::tests::plain_text_command_preview_*(6 tests),terminal::model::blocks::tests::test_apply_deferred_restored_blocks_after_bootstrap/..._is_a_noop_when_given_no_blocks/..._does_not_corrupt_a_live_running_command,pane_group::tests::test_lazy_background_tab_scrollback_restore,pane_group::tests::test_closing_pending_pane_drops_its_lazy_restoration,pane_group::tests::test_clean_up_panes_drops_pending_lazy_restorations.Full suites:
cargo test -p warp --libforpane_group::(124 passed),workspace::(210 passed), andterminal::model::(605 passed)../script/format --checkandcargo clippy -p warp --all-targets --tests -- -D warningsclean../script/runScreenshots / Videos
Not applicable — no UI surface change of its own; see the macOS verification notes above.
Agent Mode