Back off vault version-check retries and persist rotated refresh tokens first

[RhysSullivan]

Problem

Two bugs on the credential-persistence path. In production both showed up as a request that refreshed an OAuth token coming back a bare 500, and the second one left the connection permanently broken afterwards.

1. Persisting a refreshed token is a version-checked write into WorkOS Vault, and two concurrent probes of one connection write the same objects. A version conflict was retried at most 3 times with no wait at all, so a fiber racing a peer re-read and re-wrote inside the peer's own round trip and collided again deterministically — the attempts drained in microseconds and the refresh failed.
2. A refresh persists two credentials one after the other, and the access token went first. A failure in between therefore threw away the rotated refresh token, which minting had already spent, so every later refresh came back invalid_grant and the connection silently disconnected itself.

Fix

• Version-checked writes retry 5 times with half-jittered exponential backoff (50ms base, 800ms cap). A 400 on an update counts as contention too: an update carries no name and no context, so a stale version is the only thing it can mean, and a genuinely malformed write still surfaces on the final attempt.
• When every version-checked attempt loses, one last un-versioned write lands the value. Both racers hold a credential they just minted, so last-writer-wins keeps a usable token; dropping one is the only unrecoverable outcome.
• persistRefreshedToken writes the rotated refresh token before the access token.

Testing

New cloud e2e e2e/cloud/credential-write-durability.test.ts covers both halves black box. It drives a real OAuth refresh through MCP and arms the failures on the WorkOS emulator the product's own client talks to. Contention is modelled two ways, because the two halves of the write policy fail differently:

• contention by count - three, then five straight 409s on the vault write; the user-visible call still succeeds, and the connection still refreshes afterwards. This is the attempt budget and the last-resort write.
• contention by time - the vault keeps conflicting for a window that starts at the first collision the product actually suffers, rather than for a fixed number of collisions. A loop that fires its attempts back to back drains all of them, and its last-resort write, inside the window and the call fails; the shipped policy is still trying when the window closes. This is the backoff.
• ordering - one vault write fails mid-refresh (the access token's object only); the next call must still refresh, which it can only do if the rotated refresh token is what survived.

Every scenario reads the emulator's ledger back and asserts the collisions it armed were really served, so a fault whose pattern stopped matching cannot leave a test that passes without ever contending.

All three fail against the pre-fix code (Internal tool error, and a dead grant on the recovery call) and pass with it. Removing the backoff, restoring the old write order, and shrinking the attempt budget each turn one of them red.

No unit tests were added: the e2e scenarios observe everything here. The one unit change is clock mode in the existing workos-vault suite, because a conflicted write now waits before retrying and the TestClock never advances that wait.

Not in scope: refreshes are single-flight only within one request, so concurrent probes of one connection still produce concurrent writers. A cross-request refresh lease is the structural fix and is its own change.

[github-actions]

Cloudflare preview

Torn down — the PR is closed.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	executor-marketing	c576efa	Commit Preview URL Branch Preview URL	Aug 27 2026, 08:35 PM

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	executor-cloud	5a9ae3d	Aug 27 2026, 10:25 PM

[pkg-pr-new]

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1779

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1779

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1779

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1779

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1779

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1779

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1779

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1779

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1779

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1779

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1779

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1779

executor

npm i https://pkg.pr.new/executor@1779

commit: 5a9ae3d