Repository navigation
Rotate project proxy tokens when access changes - #379
Merged
Brooooooklyn merged 29 commits intoOct 8, 2026
Merged
Conversation
Brooooooklyn
added this pull request to stack #381
October 6, 2026 21:33
Brooooooklyn
changed the title
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 7, 2026 02:40
767e231 to
ed2d039
Compare
1 task
Brooooooklyn
changed the title
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 7, 2026 07:55
a7f2543 to
5a74f1e
Compare
Brooooooklyn
changed the title
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 7, 2026 08:22
5a74f1e to
6562595
Compare
Brooooooklyn
marked this pull request as ready for review
October 7, 2026 09:24
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65625950e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 7, 2026 09:33
6562595 to
8d4a14c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d4a14c30c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82852f37ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 7, 2026 12:27
82852f3 to
51c20df
Compare
Member
Author
|
All five handled:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51c20df3db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a777bcd03b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Brooooooklyn
removed this pull request from stack #381
October 7, 2026 19:58
Brooooooklyn
added this pull request to stack #385
October 7, 2026 19:58
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 8, 2026 02:59
a777bcd to
671d0a0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 671d0a021a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 8, 2026 03:28
671d0a0 to
9a164e1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a164e10b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s replaced Removing a collaborator or admin, changing one to reader, or deleting a user who was one requests a proxy token rotation in the same transaction as the member change. Readers never trigger it. While code from a user who can no longer deploy is active, it receives every new token. The deploy cutover, the recovered cutover and rollback now request one more rotation right before they replace such code, so protection starts once the code stops running. Owner suspension does not count as lost access, and rows without a recorded author never trigger it. The request writer moves to its own module so the db/ modules can call it without importing db.ts back. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
void platform system rotate-proxy-tokens [--project <id>] [--all] requests proxy token rotations and advances them within a bounded budget (150 Cloudflare calls, 60 seconds); the scheduled run finishes the rest. Without --project it covers every project still on its original token; --all covers every project. On a job already past START, --project and --all add one more generation, so the run always ends on a token minted after the command. The response is 207 until no rotation job is left in scope and lists the jobs that keep failing. --plan reads the database only and never returns a token: for one project its generation, job state and Workers; otherwise counts of legacy projects, projects a run would queue, and pending and failed jobs. The audit target is the project ID or "all". Platforms without a Cloudflare API token, or under the development auth bypass, get 409. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a project's proxy token rotation has failed four steps in a row, GET /projects/:id/deployments and GET /projects/:id/members add a fixed warning for owners and admins (never for readers, collaborators or deploy tokens). `void project status` and `void project team list` print it before their output. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Describe the rotate-proxy-tokens operator command, the one-time rotation after upgrading and the rollback limits for rotated projects. Tell owners that removing a collaborator or admin, or changing one to reader, replaces the project's platform credentials, that deploying trusted code replaces them again, and that a stuck replacement shows a warning in project status and team list. Note that deployments kept for rollback also receive the new credentials and their Durable Object alarms may keep running. For maintainers, describe the release order, the staging canary, the failure log to alert on, and running a rotation for every project if the routing namespace is recreated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Releases older than the one that accepts rotated credentials drop rotation jobs that have not finished, and Workers deployed while they run carry the legacy token. Upgrading again does not restore either, so tell operators to rerun the rotation for every project and check its progress. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A deploy of a project still on generation 0 now moves it to generation 1 and bakes tok(1) into the new Worker, so new code never receives the legacy token. The accept floor stays at 0 and no rotation is queued: older Workers of the project keep the legacy token, which stays accepted, and the proxy tries it after generation 1. The project leaves the legacy token when a rotation raises its floor. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The operator command no longer has a default scope for projects still on the legacy token. Applying needs --project <id> or --all; --plan without either shows the progress of every rotation, and --plan --all also counts the projects --all would queue. The bulk request now always covers every project, and the unused legacy-init reason is gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The resolve-worker, resolve-resource and verify-prerender-build routes now require x-void-proxy-internal and ignore x-void-internal, so a tenant project token no longer authenticates them and they never read the token floor. The proxy stops sending the project token on those calls. /internal/revalidate still checks the forwarded project token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The runbook states the release order (the release that accepts token generations first, then rotation), that a deploy issues generation 1 while older Workers keep the accepted legacy token, and that Workers deployed before the binding rename must be redeployed before their project rotates. The rollback notes cover projects deployed since the upgrade and keeping the API and proxy on the same release. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An administrator suspension ends a collaborator's or admin's deploy access, the same as deleting the account, so request a rotation for each project where the user holds one of those roles, in the suspension transaction. Usage-limit suspensions keep deploy access and request nothing; owned projects are left alone, as with account deletion. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
If Cloudflare applied a settings PATCH but the step then failed (lost response or rolled-back transaction), the retry found the Worker already on the new token, recorded no lastPatchAt, and REVOKE could raise the floor at once while the Worker's old version still served. When the step's previous attempt failed and the Worker already holds the new token, record lastPatchAt as if it was patched now. A deploy that baked the new token still skips the wait. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Deleting the project removed its accept floor, which brought back floor 0: the legacy token and every revoked generation worked again on routes that check only the token, such as /ai/run. Deletion now writes a floor above any issued generation instead, through a new routing-put cleanup job that the retry runner repeats when the KV write fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A rotation patched every retained Worker, including code from a user who can no longer deploy, so a rotation that followed the replacement of that code still handed it the new token. The walk now skips retained Workers whose author has lost deploy access; they keep the old token, which the raised floor rejects. The active Worker is always patched. A rollback to a skipped Worker is still allowed, and its platform calls fail until a redeploy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Deployments made before project teams have no recorded author, so the replacement rotation skipped them and the rotation walk patched them as trusted code. That includes a transferred project's history when the former owner is later removed. Only the owner could deploy back then, and the billing user backfilled for those rows is that owner, so the author check now falls back to it. A project that was never transferred keeps the current owner as author and does not rotate on its next deploy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Suspending a platform administrator writes the suspension before the routing update, but the rotation requests were queued only by the final suspend step. When the routing update failed, the account stayed suspended and no rotation was queued until an operator retried. The requests are now queued in the same transaction that first writes the suspension; the final step still queues them, and a pending request absorbs the repeat. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tenant Worker runs its own middleware before the generated handlers check the proxy token, so a sender must present the one token the Worker holds and never replay a rejected call with another. Each deployment now records the generation its Worker holds (deployments.proxy_token_gen, default 0: every Worker deployed before this release holds the legacy token). A deploy records the generation it bakes, and resolve-worker reports the active deployment's generation instead of the project's issued one, so the proxy sends a rolled-back gen-0 Worker the legacy token. Rollback and cutover replay give the queue entries and the cron scheduler only the token the target Worker holds. Rotation no longer stages a fallback: START goes straight to the PATCH walk. After Cloudflare confirms a PATCH, the same step records the new generation on the deployment row and, for the active Worker, points its queue entries and cron scheduler at the new token. A step that finds a Worker already holding the new token, but a row that does not say so, records it and drains it as if just patched. REVOKE walks the Workers again while any row it would cut off records an older generation. A queue relay rejected with 401/403 is retried after 120 seconds, once the queue entry has caught up, instead of at once. Cron fires stay acked with AUTH_DRIFT. The only remaining gap is the drain right after a PATCH. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A rotation recorded the new generation on the row of a Worker that has no __VOID_PROXY_TOKEN binding, although nothing was patched. Senders then presented a token that Worker does not hold. Such a Worker is now skipped like one whose author cannot deploy: its row, queue entries and cron scheduler keep the generation it holds, the step logs that it needs a redeploy, and REVOKE raises the floor without waiting for it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A rollback to a deployment that kept its old token still serves remote D1, KV and R2, because the proxy sends the token its deployment row records; only the Worker's own AI, email and ISR revalidation calls fail. Workers that hold __AI_TOKEN are never skipped by a rotation. Their projects have no active deployment, so the rotation stops at its first step and the floor never rises; deploys and normalization are refused for the same projects, so they need an operator repair. The new query lists them. Every Worker deploy now moves a project to token generation 1, so a platform rollback to 0.26.0 or earlier breaks every project deployed or rotated since the upgrade, not only rotated ones. The rollback command still allows it, so the docs tell operators not to do it and how to recover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 429 reschedules the rotation without counting a failed attempt, because owners cannot fix a platform-wide rate limit. Until now it also logged nothing, so a rotation that stayed limited was invisible to operators. Log a RATE-LIMITED line on each 429 and document the alert. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Brooooooklyn
force-pushed
the
security/1c-proxy-token-rotation
branch
from
October 8, 2026 09:45
3e4d53f to
b4a0522
Compare
When Cloudflare applies a step's PATCH but the step then fails (a copy write, the row update, the commit, or a lost PATCH response), the Worker holds the new token while its row still names the old one, and so do the queue and cron copies of the active Worker. The proxy, the queue relay and the cron consumer then send a token the Worker rejects: remote D1/KV/R2 get 403, queue messages use up their retries, and cron fires are dropped. A retained Worker in this state breaks the same way once a rollback makes it active. The job used to retry only after 15 minutes. Retry such a step on the next cron run. A scheduled run now tries each job at most once, so a step that is due again at once waits for the next run, and a lasting failure costs one attempt per run. Steps that fail before any PATCH keep the 15-minute delay. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fengmk2
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attack surface before this PR
Every tenant Worker holds a project token,
__VOID_PROXY_TOKEN. With it, the Worker uses AI, email, D1, KV and R2 through the platform proxy.1. A removed teammate keeps access forever
The previous PR added numbered tokens (
<generation>.<signature>) and a per-project floor. Nothing raised the floor yet.2. A deleted project's token still works
3. Internal routes still take a project token
What this PR changes
When a project is rotated
void platform system rotate-proxy-tokens --project <id> --yes(or--all)We assume no current user has copied a token, so there is no fleet-wide rotation on upgrade.
How one rotation works (about 6 minutes)
A project's life after this PR
What people see
void project statusandvoid project team listshow a warning.void platform system rotate-proxy-tokens.--planshows progress.--project <id> --yesor--all --yesstarts a rotation.There is no outage on upgrade.
Before merging
After it reaches staging, rotate one test project with
--project <id> --yes. Then check AI, email, queues, crons, remote D1/KV/R2, prerendering and WebSocket reconnects on that app.After deploy, alert on log lines
[proxy-token-rotation] FAILED ... attempts=4or higher.Known limits (on purpose)
Very old projects. A project whose live Worker is from before early March 2026 (named
user-<slug>, token under the old name__AI_TOKEN) cannot rotate. The rotation stops at its first step and retries every 15 minutes, the floor stays, and the app keeps working. Owners and admins see the stuck-rotation warning after about an hour. These projects already cannot deploy (project_requires_normalization), so an operator must repair them before the rotation can finish. Only AI used that old token.platform/meta/operations.mdhas a query that lists these projects. Hosted only.Code from a removed teammate keeps running while it is the live deployment, and gets each new token. The project rotates again as soon as a deploy or rollback replaces it. Kept copies of that code get no new tokens, so a rollback to one fails its AI, email and ISR revalidation calls until a redeploy. Remote D1/KV/R2 keep working.
Short switch window. While Cloudflare moves a Worker to its new version, a call can reach the old version and get a clean 403. Cron runs in that window are dropped (
[cron-fires] AUTH_DRIFT); queue messages are retried after 120 s.Failed step. If a step fails right after a token change, that Worker's proxy calls get 403, its queue messages use up retries and its cron fires are dropped until the next 2-minute run retries the step. A failure that repeats keeps this going, one retry per run.
Cache delay. The floor is in Cloudflare KV. In rare cases, the old token can work for a short time after a rotation completes.
Platform rollback. Do not roll back below this release. Each Worker deploy moves its project to token generation 1, which 0.26.0 and earlier cannot read, so every project deployed since the upgrade loses AI, email, ISR revalidation, remote D1/KV/R2 and prerendering. A release with only the token-generation PR is not a full fallback either: its proxy sends those Workers the old token, so remote D1/KV/R2 and prerendering fail.
void platform rollbackdoes not block it; the operator docs say not to. After upgrading again from any rollback, runrotate-proxy-tokens --all --yesand check with--plan.Testing
vp run checkvp run test --base security/1b-proxy-token-v2vp test run(whole stack, on maincf45d55db)vp lintvp fmt --check🤖 Generated with Claude Code