Skip to content

perf(server): shed startup load so connection setup fits its budget - #89

Merged
gfsaaser24 merged 1 commit into
turbofrom
fix/startup-load-shedding
Sep 1, 2026
Merged

perf(server): shed startup load so connection setup fits its budget#89
gfsaaser24 merged 1 commit into
turbofrom
fix/startup-load-shedding

Conversation

@gfsaaser24

Copy link
Copy Markdown
Owner

On a large install the server spends its event loop on background repository work while a client is
still connecting, so connection setup misses its budget and retries. Measured live on 0.0.49, one
backend, 28 projects / 125 threads / 72 live worktrees, over a 90 s trace:

Symptom Measurement
Trivial local HTTP GET 3–8.5 s, so the client's 15 s CONNECTION_ESTABLISHMENT_TIMEOUT (packages/client-runtime/src/connection/supervisor.ts) failed repeatedly
VcsStatusBroadcaster.refreshRemoteStatus ~25 concurrent spans, 24–30 s each (concurrency: "unbounded", one poller per worktree, 30 s interval)
ThreadSettlementReactor.sweep 33 s, concurrency: 8, one gh pr list per unsettled thread, re-run every minute
checkClaudeProviderStatus / discoverClaudeSkills 30 s / 25 s
shell command resolution 572 runGitCommand + 173 shell.resolveSpawnCommand, ~15k synchronous shell.isExecutableFile stats per trace rotation
loadServerConfig (apps/server/src/ws.ts) resolveAvailableEditors on every subscribeServerConfig snapshot: 21 editors × PATHEXT × PATH, up to the 5 s CONFIG_DISCOVERY_TIMEOUT

What changed

Four fork-owned changes, no product behavior change.

  • apps/server/src/vcs/VcsStatusBroadcaster.ts — automatic remote refreshes take a permit from a
    Semaphore.make(REMOTE_STATUS_REFRESH_CONCURRENCY = 3), and remainingStartupGrace holds them for
    REMOTE_STATUS_STARTUP_GRACE (90 s) after the broadcaster is built, returning the remaining grace as
    the poller's next delay so nothing is dropped. Local status, getStatus/refreshStatus, and the
    exponential failure backoff are untouched. RemoteStatusStartupGrace is a Context.Reference so
    tests can zero the grace.
  • apps/server/src/orchestration/ThreadSettlementReactor.ts — fan-out drops from 8 to
    SETTLEMENT_SWEEP_CONCURRENCY (2), and claimAutomaticSweep gates the periodic sweep behind
    SETTLEMENT_SWEEP_BOOT_DELAY (5 min) and SETTLEMENT_SWEEP_MIN_INTERVAL (10 min). The worker
    payload is a SweepTrigger; only "periodic" is throttled, so a settings change still sweeps
    immediately. ThreadSettlementPolicy is untouched.
  • packages/shared/src/shell.ts — the explicit-path branch of resolveCommandPathForPlatform
    now shares CommandResolutionCache under COMMAND_RESOLUTION_EXPLICIT_PATH_KEY, storing hits
    only
    so a just-written binary is never masked by a stale negative;
    resolveSpawnExecutableWithNode memoizes its synchronous scan (hits only, 30 s, keyed on
    platform + PATH + PATHEXT + command); and isExecutableFile is no longer an Effect.fn, so one
    span per resolution replaces tens of thousands per connect.
  • apps/server/src/process/externalLauncher.ts + apps/server/src/ws.tsloadServerConfig
    reads editors through the new ExternalLauncher.availableEditorsSnapshot, which answers from the
    existing 60 s editorDiscoveryCache when fresh and otherwise returns [] at once while
    warmAvailableEditors (semaphore-deduped) fills the cache on a detached fiber. Because the warm is
    detached it cannot be interrupted by a client timeout — which is exactly what previously left the
    cache cold on every connect, so the scan was never actually amortized.

Trade-off

On a genuinely cold editor cache the first config snapshot advertises no editors and they appear on
the next one. There is no full-config re-emit hook on subscribeServerConfig (only per-field
deltas), and adding one would need a contract change across web, desktop, and mobile.

Not done

Switching the desktop LAN bind from 0.0.0.0 to a dual-stack ::
(apps/desktop/src/backend/DesktopServerExposure.ts). It is unrelated to the measured CPU
starvation, listen("::") hard-fails on hosts with IPv6 disabled where 0.0.0.0 always works, and
the WSL backend's wildcard bind carries its own documented forwarding rationale.

Seam and docs

New seam startup-load-shedding (31 seams, node scripts/turbo-customization-manifest.ts verify
passes), a SEAM.md section with the table above and the nightly-sync guidance, and a 0.0.50
changelog entry. Related upstream work: pingdotgg#7231 and pingdotgg#7233. Versions bumped to 0.0.50 in
apps/desktop, apps/server, apps/web, packages/contracts.

🤖 Generated with Claude Code

On a large install (28 projects, 125 threads, 72 live worktrees) the server
spent its event loop on background repository work while a client was still
connecting. Measured live on 0.0.49 over a 90s trace: trivial local HTTP GETs
took 3-8.5s, so the client's 15s CONNECTION_ESTABLISHMENT_TIMEOUT failed
repeatedly; ~25 concurrent VcsStatusBroadcaster.refreshRemoteStatus spans of
24-30s each; a 33s ThreadSettlementReactor.sweep at concurrency 8 spawning
`gh pr list` per unsettled thread; ~15k synchronous shell.isExecutableFile
stats per trace rotation behind 572 runGitCommand / 173 resolveSpawnCommand;
and loadServerConfig walking PATH x PATHEXT for 21 editors on every
subscribeServerConfig snapshot.

Four fork-owned changes shed that load without changing product behavior:

- VcsStatusBroadcaster: automatic remote refreshes take a permit from a
  Semaphore.make(REMOTE_STATUS_REFRESH_CONCURRENCY = 3) and are held for
  REMOTE_STATUS_STARTUP_GRACE (90s) after the broadcaster is built, with the
  remaining grace returned as the poller's next delay so nothing is dropped.
  Local status, getStatus/refreshStatus, and the failure backoff are untouched.
- ThreadSettlementReactor: sweep fan-out drops from 8 to
  SETTLEMENT_SWEEP_CONCURRENCY (2), and claimAutomaticSweep gates the periodic
  sweep behind SETTLEMENT_SWEEP_BOOT_DELAY (5 min) and
  SETTLEMENT_SWEEP_MIN_INTERVAL (10 min). A settings change still sweeps at
  once. Settlement semantics are unchanged.
- @t3tools/shared/shell: the explicit-path branch now shares
  CommandResolutionCache under COMMAND_RESOLUTION_EXPLICIT_PATH_KEY (hits only,
  so a just-written binary is never masked), the synchronous Windows spawn
  resolver memoizes its scan, and isExecutableFile is no longer an Effect.fn -
  one span per resolution instead of tens of thousands per connect.
- ws.loadServerConfig reads editors through the new
  ExternalLauncher.availableEditorsSnapshot, which answers from the existing 60s
  cache or returns [] and warms it on a detached fiber. The warm cannot be
  interrupted by a client timeout, which is what previously left the cache cold
  on every connect.

Registered as seam `startup-load-shedding` with SEAM.md guidance and a 0.0.50
changelog entry. Deliberately skipped: switching the desktop LAN bind from
0.0.0.0 to a dual-stack "::" - unrelated to the measured CPU starvation, and
listen("::") hard-fails on hosts with IPv6 disabled where 0.0.0.0 always works.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.3 KiB 15.1 KiB
Codex Thread snapshot wire 6.9 KiB 7.3 KiB
Codex Live turn WebSocket wire 6.4 KiB 7.8 KiB
Codex Live turn WebSocket decoded 55.5 KiB 66.4 KiB
Codex Live turn messages 9 21
Claude Total thread wire 13.3 KiB 15.1 KiB
Claude Thread snapshot wire 6.9 KiB 7.3 KiB
Claude Live turn WebSocket wire 6.4 KiB 7.8 KiB
Claude Live turn WebSocket decoded 56.4 KiB 66.4 KiB
Claude Live turn messages 9 21

Baseline: unavailable · PR result: ac0b04b · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 109.4 KiB
  • Claude decoded thread snapshot: 110.1 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@gfsaaser24
gfsaaser24 merged commit 8014d59 into turbo Sep 1, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant