Classify Durable Object platform failures as retryable protocol errors - #1788
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | 040aa82 | Aug 27 2026, 10:35 PM |
Contributor
Cloudflare previewTorn down — the PR is closed. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 040aa82 | Commit Preview URL Branch Preview URL |
Aug 27 2026, 10:33 PM |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
Terminate a session while requests are already in flight instead of after they drain, so the isolate abort at the end of the teardown lands on a request the handler is holding. That reaches the unhandled 500 without the fix and passes with it. Also narrow the blockConcurrencyWhile match to the runtime's own cancellation message, so a defect thrown from inside the callback is not read as a platform reset.
… run Stop each teardown stream on the observed post-wipe verdict instead of a fixed timer, which cuts enough wasted load to raise the crossing count and in-flight concurrency. Assert every teardown was straddled, and that a terminated session is never advertised as retryable.
RhysSullivan
marked this pull request as ready for review
August 27, 2026 23:32
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A request that landed on a session Durable Object the platform had just reset, destroyed, or was redeploying came back as an unhandled 500 with no body, instead of a protocol error the client could act on.
Three things caused it:
agent-handler.tsguards two of its three session-stub touchpoints —_cf_scheduleDestroyis wrapped inEffect.ignore,target.fetchhas a catch that maps a destroyed session to a 404 reconnect — but thevalidateMcpSessionOwnerRPC between them was bare. A session whose object had already been torn down rejected that call, and nothing caught it.markActivityis pure bookkeeping — it writes a timestamp and arms the idle alarm — yet it ran as the last, fatal statement ofinit(). A platform reset of that one write tore down a session that was already fully built and serving, and failed the user's request.The same failure was also reported twice internally: the Durable Object captured it through its own error seam, and the error-reporting SDK's Durable Object instrumentation captured the identical rejection again as it escaped the method.
Fix
One
classifyDurableObjectErrormodule inpackages/hosts/cloudflarerecognizes the platform's reset vocabulary (destroyed,Durable Object reset because its code was updated, storage timeout, storage internal error, a cancelledblockConcurrencyWhile,internal error; reference = …, and the runtime's ownretryableflag) and returns either "this id is dead, reconnect" or "transient, retry". Anything it does not recognize returnsnulland keeps its current behaviour — rethrown and reported — so a real defect is never silently swallowed.It is used at three places:
validateMcpSessionOwnerRPC is now guarded like its siblings. A recognized platform failure renders the existing envelopes: 404 for a dead id, or a 503 carryingRetry-Afterfor a transient reset.target.fetchcatch, which previously only understood the literal destroyed-session abort, now covers the rest of the vocabulary too.markActivity) is best-effort against recognized platform resets only. The session survives, and the idle alarm self-heals on the next request. A bookkeeping write that fails for any other reason is still a defect and still fails.Error reporting gets a single owner: the Durable Object claims a cause once it has decided what to do with it, and
beforeSenddrops the auto-instrumentation's echo of that same cause. Anything the object never claims — an alarm crash, a transport fault — is untouched and still reported. Each classified failure recordsmcp.do.reset_kindon its span and in a structured log, so the volume stays countable per cause now that it is no longer a pile of 500s.Testing
Primary verification is the new e2e scenario
e2e/cloud/mcp-destroyed-session-envelope.test.ts, which reproduces the production symptom black-box.It terminates a session while requests are already in flight — a client with an open tool loop, rather than one that politely drains first — so the isolate abort at the end of the teardown lands on a request the handler is holding. It then asserts that every answer is a well-formed JSON-RPC envelope, never a bare 500, and that any retryable verdict carries
Retry-After.With the product code reverted to its pre-fix state and this scenario in place, it fails on exactly that assertion, receiving a raw HTML 500 error page whose embedded message is
destroyed, thrown at the unguardedvalidateMcpSessionOwnercall:The race is forced rather than hoped for. Each teardown's request stream now stops on the observed post-wipe verdict rather than on a fixed timer, which removes enough dead load to afford 12 crossings at 24 concurrent requests each inside a smaller request budget than before — and the scenario checks per teardown that every crossing was actually straddled, so a run that never entered the window fails instead of passing vacuously. Pre-fix it fails 6 of 6 runs, leaking 3–9 unhandled 500s each; with the fix it passes 6 of 6. It is still tuned to sit under the request volume at which the shared org-membership lookup on the dev stack starts answering 403 — an unrelated pre-existing behaviour that otherwise drowns the signal — and it uses one pre-minted identity per teardown for the same reason.
Two properties the teardown alone cannot demonstrate are asserted alongside it, because both would otherwise regress silently while every other assertion still passed:
Retry-Afteron either.Unit tests are kept only for seams this scenario genuinely cannot observe, and the case it now covers end to end (a dead session id rendering as a 404 reconnect envelope) was removed from them:
blockConcurrencyWhile— cannot be provoked on the dev stack at all, so the classifier and the wire response it renders for them are pinned inpackages/hosts/cloudflare/src/mcp/durable-object-errors.test.ts.agent-session-durable-object.test.ts.apps/cloud/src/observability/observability.test.ts.cloud/mcp-protocol.test.tsstill passes;typecheck,lint,format:checkclean.Excluded, and left for follow-ups: restructuring the long
blockConcurrencyWhileinit, overriding_cf_scheduleDestroy, and the cold-init org-meta work.