Repository navigation
Add rotatable proxy tokens and a proxy-only key for internal API routes - #378
Merged
Brooooooklyn merged 12 commits intoOct 8, 2026
Merged
Conversation
Brooooooklyn
added this pull request to stack #381
October 6, 2026 21:33
Brooooooklyn
changed the title
Brooooooklyn
force-pushed
the
security/1b-proxy-token-v2
branch
from
October 7, 2026 02:40
de445a7 to
4cfe2b9
Compare
1 of 2 tasks
Brooooooklyn
changed the title
Brooooooklyn
marked this pull request as ready for review
October 7, 2026 09:24
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 14031a87fe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Brooooooklyn
force-pushed
the
security/1b-proxy-token-v2
branch
from
October 7, 2026 09:33
14031a8 to
f01b141
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f01b141d48
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
Brooooooklyn
force-pushed
the
security/1b-proxy-token-v2
branch
from
October 7, 2026 12:27
f01b141 to
d5d58f2
Compare
Member
Author
|
Fixed. Senders (proxy, cron consumer, queue relay) now send one token and never replay a rejected request, since user middleware runs before the Worker's own token check. The next PR records the generation each deployment's Worker holds, so the one token sent is always the right one. |
Brooooooklyn
removed this pull request from stack #381
October 7, 2026 19:58
Brooooooklyn
added this pull request to stack #385
October 7, 2026 19:58
Brooooooklyn
force-pushed
the
security/1b-proxy-token-v2
branch
3 times, most recently
from
October 8, 2026 04:34
102b5b1 to
6f2bb9a
Compare
Tenant proxy tokens gain a v2 form, `v2.<gen>.<mac>`, signed with a key derived from JWT_SECRET by HKDF. The legacy token stays generation 0 and keeps its exact bytes. A per-project accept floor in ROUTING (`proxy-token:<projectId>`) decides which generations are accepted; a missing key is floor 0, so every current token keeps working. Checks run the project id and syntax checks first, then the MAC, and read the floor only for a valid MAC. A malformed or unreadable floor fails closed. The module also has an isolate cache for floor reads and a route-bound credential the proxy can use for the API metadata routes. Nothing uses the module yet. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The proxy's service-binding auth and /email/verify-receipt now check the tenant project token through the shared proxy token module: the MAC first, then the project's accept floor in ROUTING. Both the legacy token and `v2.<gen>.<mac>` tokens are accepted while their generation is at or above the floor. The proxy keeps one floor cache per isolate. A floor that cannot be read or is malformed returns 503 instead of accepting the token. /internal/revalidate checks the forwarded token against the same floor, so revalidation keeps working with v2 tokens and stops accepting a token the proxy would refuse. No floor is written yet, so every project keeps floor 0 and the legacy token works exactly as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The proxy now picks the token it presents to a tenant Worker from the project's token generation. When /internal/resolve-worker returns `proxyTokenGen`, the proxy sends that generation's token. Without it, the proxy sends the token for the project's cached accept floor, so tenant Workers keep working when the API does not report a generation. If the Worker rejects the token with its pre-handler 403 `unauthorized` response, the proxy retries once with the previous generation's token and the same body. It never retries after a thrown fetch or any other response. If the floor cannot be read, a write fails with 502 and a read takes its REST fallback, without dispatching. With no floor written, every project still receives the legacy token in a single request. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/internal/resolve-worker, /internal/resolve-resource and /internal/verify-prerender-build accepted the same HMAC that tenant Workers hold as their project token, so a tenant could call them directly on the public API host. The proxy now also sends `x-void-proxy-internal`, an HMAC bound to the route and project under a key derived from JWT_SECRET with HKDF. It never leaves the platform. The API accepts that header first and never falls through when it is present but wrong. A request without it is checked like a tenant project token: the MAC, then the project's accept floor. A legacy token therefore stops opening these routes as soon as the floor rises past it, the same moment it stops working at the proxy. The proxy keeps sending the legacy `x-void-internal` header so an API that predates the new credential keeps working while the two deploy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
provisionBindings now takes the Worker's proxy token in its platform options instead of deriving it from JWT_SECRET. The deploy upload issues it with issueProxyToken, which returns the token for the project's accept floor (the legacy token while no floor is set), so existing projects get the same bytes as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cron fire messages, the CronScheduler state and queue routing entries can now carry an optional `proxyTokenFallback`. When a tenant Worker rejects the first token with its generated `unauthorized` response, the cron-fires consumer and the dispatch queue relay try once more with the fallback token, using the same request body and the same /__void then legacy path order. Both rejects come before any cron or queue handler runs, so the second attempt cannot run a handler twice. There is no second attempt after a thrown fetch, another status or another body. If the fallback is rejected too, the existing behavior applies: cron fires ack with the AUTH_DRIFT log, and queue relays retry the batch. The CronScheduler copies the fallback into each fire message and leaves it out of /status, like `proxyToken`. Nothing writes the fallback yet, so every relay and cron fire still sends one request with one token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduled cleanup runner now claims only the job kinds it knows how to run. Rows of any other kind keep their attempts and payload instead of being claimed and deleted, so they cannot fill the retry batch and they survive a platform rollback. executeCleanupJob throws on an unknown kind as a guard. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Describe the legacy and v2 project token forms, the per-project accept floor in ROUTING, how the proxy picks tokens for tenant Workers, and the proxy-only key on the internal metadata routes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The proxy checks a 403 from the tenant Worker for the binding handler's token reject before retrying with the previous token. That check parsed the whole cloned body, so a Worker could make the shared proxy buffer an arbitrarily large response. Skip bodies that declare more than 256 bytes and stop reading a stream once it passes that size; the response returned to the caller still streams unbuffered. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tokens for generation 1 and above are now `<gen>.<mac>`. The dot still keeps them apart from the 43-character original token. The HKDF labels and signed messages carry the key's purpose instead of a version: the tenant key uses salt `void-proxy-token`, info `proxy-token`, and signs `void-proxy-token:<projectId>:<gen>`; the proxy internal key uses salt `void-proxy-internal`, info `proxy-internal`, and signs `void-proxy-internal\n<route>\n<projectId>`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tenant Worker runs its own middleware and routes before the generated /__void/* binding, cron and queue handlers check the proxy token. A request rejected with 401/403 has therefore already run user code, and sending it again with a second token would run that code twice. The proxy, the cron fire consumer and the queue relay now send exactly one token per call. The proxy sends the generation resolve-worker reports, or the floor's generation when none is reported. A rejected call is returned (proxy), acked with AUTH_DRIFT (cron) or retried by the queue as before. The fallback token fields, the 403 body sniffing and sendGenerations are removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Brooooooklyn
force-pushed
the
security/1b-proxy-token-v2
branch
from
October 8, 2026 09:45
6f2bb9a to
a1431cc
Compare
fengmk2
approved these changes
Oct 8, 2026
Brooooooklyn
added a commit
that referenced
this pull request
Oct 8, 2026
## Attack surface before this PR
Every tenant Worker holds a project token, `__VOID_PROXY_TOKEN`. With
it, the Worker uses AI, email, D1, KV and R2 through the platform proxy.
**1. A removed teammate keeps access forever**
```
collaborator deploys code that reads env.__VOID_PROXY_TOKEN ──► keeps a copy
│
▼
owner removes the collaborator ──► token still works ✗
D1 SQL, KV, R2, AI, email — forever
```
The previous PR added numbered tokens (`<generation>.<signature>`) and a
per-project floor. Nothing raised the floor yet.
**2. A deleted project's token still works**
```
owner deletes project ──► floor and ai-block removed ──► old token on /ai/run ──► accepted ✗
(Workers AI on the platform account)
```
**3. Internal routes still take a project token**
```
internet ──► public API host /internal/resolve-worker, resolve-resource, verify-prerender-build
x-void-internal: <project token> ──► accepted ✗ (proxy-only routes)
```
## What this PR changes
```
access change ──► rotation ──► every Worker gets a new token ──► floor rises ──► old token refused ✓
new deploy ──► Worker gets a numbered token
/internal/* ──► only the proxy's own key (x-void-internal no longer accepted) ✓
```
### When a project is rotated
| Event | Rotates? |
| --- | --- |
| A collaborator or admin is removed, leaves, or their account is
deleted | Yes |
| A collaborator or admin is changed to reader | Yes |
| A platform admin suspends a collaborator or admin | Yes |
| A user is suspended for a usage limit | No (they can still deploy) |
| A project is deleted | Its floor is closed: no token of that project
works again |
| A reader is removed | No (readers can't deploy code, so they can't
read the token) |
| A deploy or rollback replaces code written by someone who can no
longer deploy | Yes, again |
| An operator runs `void platform system rotate-proxy-tokens --project
<id> --yes` (or `--all`) | Yes |
| Platform upgrade | No. Old tokens stay valid until the project has one
of the events above. |
We assume no current user has copied a token, so there is no fleet-wide
rotation on upgrade.
### How one rotation works (about 6 minutes)
```
1. new number 2. each Worker 3. record 4. lock out 5. wait
gen + 1 ──► gets new token ──► which token ──► old token ──► for caches
(database) (Cloudflare setting, it holds refused to catch up
no rebuild) (+ cron/queue copies)
```
- Senders always send the token that a Worker holds (step 3 records it).
They never retry with another token.
- The floor rises only after every Worker holds the new token.
- If a step fails, the job retries after 15 minutes. If it failed after
a PATCH may have reached a Worker, it retries on the next 2-minute run,
once per run, until it commits.
- Workers kept for rollback get the new token too, except code from
people who lost access.
- Live WebSocket connections reconnect once (step 2 starts a new Worker
version).
### A project's life after this PR
```
today (old token) ──deploy──► new Worker gets a numbered token; older Workers keep the old token (still accepted)
──member removed──► rotation ──► every Worker gets a new token ──► old token refused
```
## What people see
| Who | What changes |
| --- | --- |
| App users | Nothing. |
| Project owners and admins | If a rotation fails 4 times in a row
(about 1 hour), `void project status` and `void project team list` show
a warning. |
| Operators | New command: `void platform system rotate-proxy-tokens`.
`--plan` shows progress. `--project <id> --yes` or `--all --yes` starts
a rotation. |
There is no outage on upgrade.
## Before merging
- [x] **Cloudflare check.** Done live on a test account (2026-10-07),
with the real rotation client, 6 rotations:
```
bindings: 13 others (plain_text, secret_text, d1, kv, r2, queue,
service, durable_object, assets) byte-identical
secrets: upload-metadata and PUT /secrets ones both kept
code: no upload, same etag; compat, tags, tail_consumers, observability,
placement unchanged
switch: new token at 0.4–11 s, last old token at ≤ 18 s (drain wait is
120 s)
```
- [ ] **#378 is live on prod.** The proxy must understand token
generations first.
After it reaches staging, rotate one test project with `--project <id>
--yes`. Then check AI, email, queues, crons, remote D1/KV/R2,
prerendering and WebSocket reconnects on that app.
After deploy, alert on log lines `[proxy-token-rotation] FAILED ...
attempts=4` or higher.
## Known limits (on purpose)
- **Very old projects.** A project whose live Worker is from before
early March 2026 (named `user-<slug>`, token under the old name
`__AI_TOKEN`) cannot rotate. The rotation stops at its first step and
retries every 15 minutes, the floor stays, and the app keeps working.
Owners and admins see the stuck-rotation warning after about an hour.
These projects already cannot deploy (`project_requires_normalization`),
so an operator must repair them before the rotation can finish. Only AI
used that old token. `platform/meta/operations.md` has a query that
lists these projects. Hosted only.
- **Code from a removed teammate** keeps running while it is the live
deployment, and gets each new token. The project rotates again as soon
as a deploy or rollback replaces it. Kept copies of that code get no new
tokens, so a rollback to one fails its AI, email and ISR revalidation
calls until a redeploy. Remote D1/KV/R2 keep working.
- **Short switch window.** While Cloudflare moves a Worker to its new
version, a call can reach the old version and get a clean 403. Cron runs
in that window are dropped (`[cron-fires] AUTH_DRIFT`); queue messages
are retried after 120 s.
- **Failed step.** If a step fails right after a token change, that
Worker's proxy calls get 403, its queue messages use up retries and its
cron fires are dropped until the next 2-minute run retries the step. A
failure that repeats keeps this going, one retry per run.
- **Cache delay.** The floor is in Cloudflare KV. In rare cases, the old
token can work for a short time after a rotation completes.
- **Platform rollback.** Do not roll back below this release. Each
Worker deploy moves its project to token generation 1, which 0.26.0 and
earlier cannot read, so every project deployed since the upgrade loses
AI, email, ISR revalidation, remote D1/KV/R2 and prerendering. A release
with only the token-generation PR is not a full fallback either: its
proxy sends those Workers the old token, so remote D1/KV/R2 and
prerendering fail. `void platform rollback` does not block it; the
operator docs say not to. After upgrading again from any rollback, run
`rotate-proxy-tokens --all --yes` and check with `--plan`.
```
upgrade ──► deploy ──► project at generation 1
rollback to 0.26 ──► proxy reads only the legacy token ──► AI, email,
ISR, remote bindings, prerender ✗
```
## Testing
| Command | Result |
| --- | --- |
| `vp run check` | pass |
| `vp run test --base security/1b-proxy-token-v2` | pass: 13401 tests
passed, 30 skipped |
| `vp test run` (whole stack, on main `cf45d55db`) | pass: 13691 tests
passed, 30 skipped |
| `vp lint` | pass |
| `vp fmt --check` | pass |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attack surface before this PR
Every tenant Worker holds a project token,
__VOID_PROXY_TOKEN. With it, the Worker uses AI, email, D1, KV and R2 through the platform proxy.1. Shared key: anyone could make a token
The previous PR blocks this by checking the project id format. This PR gives new tokens their own key, so a signature from any other feature can never be a token.
2. A token never expires
There was no way to change a token without breaking the project.
3. Internal routes opened by a tenant token
These routes are meant for the proxy only. They accepted the same token every tenant Worker holds.
What this PR adds
This PR lays the groundwork. It does not change behavior for any current project, and the next PR does the rotation.
Numbered tokens and a floor
Check order (a forged token never costs a KV read)
Proxy-only key on internal routes
One token per call, never replayed
The Worker runs user middleware before its own token check, so a replay could run app code twice. Senders therefore never retry with another token. The next PR makes sure they send the token each Worker holds.
Decisions
JWT_SECRETthrough HKDF. Operators have nothing to set up.cacheTtl30 s + isolate cache 15 s. This keeps KV reads low on the hot path.Known limit: a Worker kept for rollback keeps the token it was deployed with. The next PR patches those Workers during rotation, except code from people who lost access.
Rollout
Testing
vp run checkvp run test --base security/1a-quick-fixesvp lint,vp fmt --check🤖 Generated with Claude Code