Skip to content

Stop sessions that exceed 90% of host RAM - #184

Draft
t-kalinowski wants to merge 14 commits into
mainfrom
feat/session-memory-guard
Draft

t-kalinowski wants to merge 14 commits into
mainfrom
feat/session-memory-guard

Conversation

@t-kalinowski

@t-kalinowski t-kalinowski commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Stop a session when its worker process tree exceeds 90% of total physical host RAM. Console checks every five seconds, including while idle, forcibly terminates the worker, and returns a banner with the observed usage and advice to use smaller datasets or process data in chunks. Existing worker replacement rules still apply.

Internally, a controller-owned sampler requests immediate shutdown through the existing generation owner. The macOS and Linux samplers retain memory records only for the owned tree, avoiding large retained allocations for unrelated host processes. macOS reads resident memory through libproc; Linux reads /proc metadata without changing file-descriptor limits or depending on optional per-thread child-list files. There are no new configuration fields or wire protocol messages. The documentation explains that periodic RSS sampling cannot prevent every OOM and can count shared pages more than once.

Public regressions use injected host metrics to verify below-limit survival, active descendant memory, idle termination, the complete banner, cleanup, and replacement without large allocations. The idle case measures a complete sample against the existing 64 KiB allocation budget, and the active case requires the combined usage of all 17 children to exceed the limit. Shutdown assertions require a graceful server exit and report process identities if a descendant survives.

The sandbox companion is pinned to the Linux proxy-retirement fix, which waits for managed-proxy bridge children to exit before returning. The native helper retains and reaps the exact children it creates; Console continues to use the companion's cleanup boundary.

Transcript collectors preserve the final [done] marker when restart or interrupt output arrives before the submitted cell completes. A gated native R startup regression requires the split replies and verifies exact completion within one response deadline; related restart continuations use the same contract. Runtime behavior and deadline allowances are unchanged by this fixture repair.

Local macOS validation covers all eight affected public cases (13 executions), the final shared-deadline startup case, and all 23 MCP client self-tests on identical client and collector sources. Formatting and scripts/check passed; the final one-file deadline adjustment was validated by the focused startup case. Hosted checks for the current test-only repair are pending.

Apply the existing test-only DuckDB snapshot to direct and MXC SQL
scenarios. Assert the selected version before the long query so cold
caches cannot silently introduce the known Arrow cancellation limitation.
Preserve production defaults, interruption deadlines, and exact results.
Queue stdin while the replacement cell is held at its checkpoint, then
release it and collect the complete result. Preserve the exact output
and idle-state assertions without assuming one response completes the cell.
Read parent metadata across procfs passes without retaining unrelated
process records. Read resident memory only for descendants, and cancel
between entries or passes so retirement can stop a growing tree scan.
This keeps the existing procfs requirements and does not cache file
descriptors or change the caller's resource limits.
Walk a candidate's parent chain to identify owned descendants without
waiting for the process tree to stop growing. Keep unrelated ancestry
transient, read RSS only for owned processes, and stop observed PID cycles.
Cancellation remains explicit between enumeration and ancestry reads.
Preserve shutdown errors before checking descendants, and include captured and
surviving process identities when the immediate kernel exit check fails.
Wait for the native helper's host proxy children before returning from the
sandbox lifetime, preserving workload statuses and inherited signal behavior.
Keep the required completion marker when a control-and-cell response is
split across polls. Exercise the split with a gated native R startup cell
and select the same contract in related restart continuations.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant