Skip to content

fix(e2e): stop the cap-eviction scenario from stampeding cold DO starts - #1895

Open
ra-co88 wants to merge 1 commit into
UsefulSoftwareCo:mainfrom
ra-co88:fix/e2e-cap-eviction-thundering-herd
Open

fix(e2e): stop the cap-eviction scenario from stampeding cold DO starts#1895
ra-co88 wants to merge 1 commit into
UsefulSoftwareCo:mainfrom
ra-co88:fix/e2e-cap-eviction-thundering-herd

Conversation

@ra-co88

@ra-co88 ra-co88 commented Aug 30, 2026

Copy link
Copy Markdown

What

Fixes the E2E (cloud 13of16) shard failure that's been red on main: cloud/mcp-session-cap-eviction.test.ts failing with openSession (session-N): no mcp-session-id header.

Why it fails

The scenario opens cap + 10 sessions at concurrency 8 to force a resident-runtime cap eviction. Every open is a cold Durable Object start — sqlite open plus runtime construction inside the agents SDK's blockConcurrencyWhile. Under that burst the start-up block regularly outlives workerd's wall-clock budget for it, and the platform resets the object mid-initialize. The in-flight initialize then surfaces as the restart envelope (503, JSON-RPC -32001, "MCP session is restarting, please retry") instead of a response carrying mcp-session-id — and the test failed on the very first one instead of honoring it.

Server-side evidence from a local repro: the dev server's log showed concurrency_reset platform failures for nearly every session in the burst, each correctly answered with the retryable 503 envelope by agent-handler's classifier.

What changed

Two changes, root cause first:

  • Concurrency 8 → 2 for the opening Effect.forEach. Cold DO stops overlapping into reset territory. The scenario goes from a guaranteed failure (180s timeout or first-reset throw) to a ~5s pass.
  • openSession honors the restart envelope. On the documented 503 "MCP session is restarting, please retry" response it retries the same initialize after a short delay (bounded, 8 attempts) instead of treating a retryable platform blip as a setup precondition failure — the same contract a real streamable-http client follows. Kept as a backstop for the occasional reset that still slips through.

Verification

  • Local repro on current main: failed 2/2 before the change, both times with the 503 restart envelope (sessions 4–7 in one run, 5 in the other).
  • With the change: 4/4 consecutive green runs, ~5s each.
  • Sibling shard members (mcp-priming-reconnect, session-gate) still green.
  • oxlint and tsc --noEmit clean on the touched package.

The scenario opened cap+10 sessions at concurrency 8, and every open is a
cold Durable Object start (sqlite open plus runtime construction inside
the agents SDK blockConcurrencyWhile). The burst regularly made those
blocks outlive the runtime wall-clock budget, so workerd reset the object
mid-initialize and the client received the 503 restart envelope instead
of an mcp-session-id header - the scenario then failed on the very first
reset. Fails on main today.

Two changes, root cause first:

- Open at concurrency 2. Cold starts no longer overlap into reset
  territory; the scenario passes in ~5s locally, 4/4 consecutive runs.
- openSession now honors the restart envelope it can receive: on the
  documented 503 "MCP session is restarting, please retry" response it
  retries the same initialize after a short delay (bounded, 8 attempts)
  instead of treating a retryable platform blip as a setup failure -
  the same contract a real streamable-http client follows.
@ra-co88
ra-co88 force-pushed the fix/e2e-cap-eviction-thundering-herd branch from 3a0b881 to d4da0e5 Compare August 30, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants