perf(server): shed startup load so connection setup fits its budget - #89
Merged
Conversation
On a large install (28 projects, 125 threads, 72 live worktrees) the server
spent its event loop on background repository work while a client was still
connecting. Measured live on 0.0.49 over a 90s trace: trivial local HTTP GETs
took 3-8.5s, so the client's 15s CONNECTION_ESTABLISHMENT_TIMEOUT failed
repeatedly; ~25 concurrent VcsStatusBroadcaster.refreshRemoteStatus spans of
24-30s each; a 33s ThreadSettlementReactor.sweep at concurrency 8 spawning
`gh pr list` per unsettled thread; ~15k synchronous shell.isExecutableFile
stats per trace rotation behind 572 runGitCommand / 173 resolveSpawnCommand;
and loadServerConfig walking PATH x PATHEXT for 21 editors on every
subscribeServerConfig snapshot.
Four fork-owned changes shed that load without changing product behavior:
- VcsStatusBroadcaster: automatic remote refreshes take a permit from a
Semaphore.make(REMOTE_STATUS_REFRESH_CONCURRENCY = 3) and are held for
REMOTE_STATUS_STARTUP_GRACE (90s) after the broadcaster is built, with the
remaining grace returned as the poller's next delay so nothing is dropped.
Local status, getStatus/refreshStatus, and the failure backoff are untouched.
- ThreadSettlementReactor: sweep fan-out drops from 8 to
SETTLEMENT_SWEEP_CONCURRENCY (2), and claimAutomaticSweep gates the periodic
sweep behind SETTLEMENT_SWEEP_BOOT_DELAY (5 min) and
SETTLEMENT_SWEEP_MIN_INTERVAL (10 min). A settings change still sweeps at
once. Settlement semantics are unchanged.
- @t3tools/shared/shell: the explicit-path branch now shares
CommandResolutionCache under COMMAND_RESOLUTION_EXPLICIT_PATH_KEY (hits only,
so a just-written binary is never masked), the synchronous Windows spawn
resolver memoizes its scan, and isExecutableFile is no longer an Effect.fn -
one span per resolution instead of tens of thousands per connect.
- ws.loadServerConfig reads editors through the new
ExternalLauncher.availableEditorsSnapshot, which answers from the existing 60s
cache or returns [] and warms it on a detached fiber. The warm cannot be
interrupted by a client timeout, which is what previously left the cache cold
on every connect.
Registered as seam `startup-load-shedding` with SEAM.md guidance and a 0.0.50
changelog entry. Deliberately skipped: switching the desktop LAN bind from
0.0.0.0 to a dual-stack "::" - unrelated to the measured CPU starvation, and
listen("::") hard-fails on hosts with IPv6 disabled where 0.0.0.0 always works.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: unavailable · PR result: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a large install the server spends its event loop on background repository work while a client is
still connecting, so connection setup misses its budget and retries. Measured live on 0.0.49, one
backend, 28 projects / 125 threads / 72 live worktrees, over a 90 s trace:
CONNECTION_ESTABLISHMENT_TIMEOUT(packages/client-runtime/src/connection/supervisor.ts) failed repeatedlyVcsStatusBroadcaster.refreshRemoteStatusconcurrency: "unbounded", one poller per worktree, 30 s interval)ThreadSettlementReactor.sweepconcurrency: 8, onegh pr listper unsettled thread, re-run every minutecheckClaudeProviderStatus/discoverClaudeSkillsshellcommand resolutionrunGitCommand+ 173shell.resolveSpawnCommand, ~15k synchronousshell.isExecutableFilestats per trace rotationloadServerConfig(apps/server/src/ws.ts)resolveAvailableEditorson everysubscribeServerConfigsnapshot: 21 editors × PATHEXT × PATH, up to the 5 sCONFIG_DISCOVERY_TIMEOUTWhat changed
Four fork-owned changes, no product behavior change.
apps/server/src/vcs/VcsStatusBroadcaster.ts— automatic remote refreshes take a permit from aSemaphore.make(REMOTE_STATUS_REFRESH_CONCURRENCY = 3), andremainingStartupGraceholds them forREMOTE_STATUS_STARTUP_GRACE(90 s) after the broadcaster is built, returning the remaining grace asthe poller's next delay so nothing is dropped. Local status,
getStatus/refreshStatus, and theexponential failure backoff are untouched.
RemoteStatusStartupGraceis aContext.Referencesotests can zero the grace.
apps/server/src/orchestration/ThreadSettlementReactor.ts— fan-out drops from 8 toSETTLEMENT_SWEEP_CONCURRENCY(2), andclaimAutomaticSweepgates the periodic sweep behindSETTLEMENT_SWEEP_BOOT_DELAY(5 min) andSETTLEMENT_SWEEP_MIN_INTERVAL(10 min). The workerpayload is a
SweepTrigger; only"periodic"is throttled, so a settings change still sweepsimmediately.
ThreadSettlementPolicyis untouched.packages/shared/src/shell.ts— the explicit-path branch ofresolveCommandPathForPlatformnow shares
CommandResolutionCacheunderCOMMAND_RESOLUTION_EXPLICIT_PATH_KEY, storing hitsonly so a just-written binary is never masked by a stale negative;
resolveSpawnExecutableWithNodememoizes its synchronous scan (hits only, 30 s, keyed onplatform + PATH + PATHEXT + command); and
isExecutableFileis no longer anEffect.fn, so onespan per resolution replaces tens of thousands per connect.
apps/server/src/process/externalLauncher.ts+apps/server/src/ws.ts—loadServerConfigreads editors through the new
ExternalLauncher.availableEditorsSnapshot, which answers from theexisting 60 s
editorDiscoveryCachewhen fresh and otherwise returns[]at once whilewarmAvailableEditors(semaphore-deduped) fills the cache on a detached fiber. Because the warm isdetached it cannot be interrupted by a client timeout — which is exactly what previously left the
cache cold on every connect, so the scan was never actually amortized.
Trade-off
On a genuinely cold editor cache the first config snapshot advertises no editors and they appear on
the next one. There is no full-config re-emit hook on
subscribeServerConfig(only per-fielddeltas), and adding one would need a contract change across web, desktop, and mobile.
Not done
Switching the desktop LAN bind from
0.0.0.0to a dual-stack::(
apps/desktop/src/backend/DesktopServerExposure.ts). It is unrelated to the measured CPUstarvation,
listen("::")hard-fails on hosts with IPv6 disabled where0.0.0.0always works, andthe WSL backend's wildcard bind carries its own documented forwarding rationale.
Seam and docs
New seam
startup-load-shedding(31 seams,node scripts/turbo-customization-manifest.ts verifypasses), a SEAM.md section with the table above and the nightly-sync guidance, and a 0.0.50
changelog entry. Related upstream work: pingdotgg#7231 and pingdotgg#7233. Versions bumped to 0.0.50 in
apps/desktop,apps/server,apps/web,packages/contracts.🤖 Generated with Claude Code