Skip to content
4 changes: 4 additions & 0 deletions docs/guide/platform/installation/maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,10 @@ An upgrade validates the runtime, applies supported pending migrations, deploys

Before starting, Void checks access to the installation's Cloudflare account, Workers, and existing R2 buckets. If credentials are rejected, set `CLOUDFLARE_API_TOKEN` to a management token scoped to that account and retry. This overrides the installation's saved management token.

::: warning Projects deployed before credential rotation
Current releases no longer accept the original project credentials. After upgrading, the platform gives each project new credentials by itself. Until a project's turn comes (minutes to hours depending on how many projects you have), its pages keep working but AI, email, ISR revalidation, prerendering, and remote D1, KV, and R2 can fail. Check progress with `void platform system rotate-proxy-tokens --plan` until `legacy`, `pending` and `failed` are all 0; a project leaves `legacy` when its rotation starts, before it is finished. Redeploying a project fixes it right away; if that deploy fails, run `void platform system rotate-proxy-tokens --project <id> --yes`.
:::

::: details Migration safety, credential rotation, and interrupted upgrades

Some upgrades briefly put the platform into maintenance mode. Traffic resumes after the new version passes health checks. If an upgrade is interrupted during maintenance, rerun the same command to complete it.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/cli/platform-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ void platform system rotate-proxy-tokens [--project <id>] [--all]

`sandbox-drain` removes legacy tenant-owned Sandbox resources. Use `--plan` to preview and `--yes` to apply, then pass `nextCursor` as `--cursor` to continue. Application traffic stays paused during cleanup.

`rotate-proxy-tokens` gives projects new platform credentials on all of their Workers, without a redeploy. The platform also rotates projects that still use their original credentials by itself after an upgrade; use this command to check progress, rotate one project, or rotate every project with `--all`. Without `--project` it covers every project that still uses its original credentials; `--all` covers every project. Use `--plan` to preview and `--yes` to apply. Each run does a bounded batch and the platform finishes the rest in the background; a rotation takes about 6 minutes per project, longer while a deploy is running. Use `--plan` to check progress until nothing is pending or failed. Running `--yes` again with `--project` or `--all` while a rotation is still running starts one more rotation for that project. When a rotation is finished, the previous credentials no longer work. Each rotation starts a new version of the project's live Worker, so WebSocket connections reconnect once.
`rotate-proxy-tokens` gives projects new platform credentials on all of their Workers, without a redeploy. The platform also rotates projects that still use their original credentials by itself after an upgrade; use this command to check progress, rotate one project, or rotate every project with `--all`. Without `--project` it covers every project that still uses its original credentials; `--all` covers every project. Use `--plan` to preview and `--yes` to apply. Each run does a bounded batch and the platform finishes the rest in the background; a rotation takes about 6 minutes per project, longer while a deploy is running. Use `--plan` to check progress until nothing is pending or failed. Running `--yes` again with `--project` or `--all` while a rotation is still running starts one more rotation for that project. When a rotation is finished, the previous credentials no longer work. Each rotation starts a new version of the project's live Worker, so WebSocket connections reconnect once. After an upgrade, projects still on their original credentials can see errors until they are rotated; see [Upgrade the platform](../../guide/platform/installation/maintenance.md#upgrade-the-platform).

Use the [platform lifecycle commands](./platform-installation.md#lifecycle-commands) to maintain your installation's Workers.

Expand Down
6 changes: 5 additions & 1 deletion platform/meta/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ These deployment and smoke-test workflows target Void Cloud, from `prod` for pro

Both deploy workflows serialize deployments without cancelling an active rollout. Advancing `prod` requests a production deployment. A failed or interrupted deployment needs inspection before another rollout, particularly when database migrations already ran.

Hosted API and email gateway deploys report the release version from `packages/platform/package.json` as `PLATFORM_VERSION`. The `void` CLI refuses platforms older than its minimum platform version, so release in this order: advance `prod` and wait for the production deployment to succeed, then push the release tag that publishes the npm packages.
Hosted API and email gateway deploys report the release version from `packages/platform/package.json` as `PLATFORM_VERSION`. The `void` CLI refuses platforms older than its minimum platform version, so release in this order: advance `prod` and wait for the production deployment to succeed, then push the release tag that publishes the npm packages. Before the release that removes the legacy proxy token reaches staging or production, `rotate-proxy-tokens --plan` there must show 0 legacy, 0 pending and 0 failed; `legacy: 0` alone is not enough, because a project leaves it when its rotation starts or a deploy moves it to generation 1, while its older Workers still hold the legacy token until the rotation finishes. If that environment was rolled back below the rotation release after it shipped, run `rotate-proxy-tokens --all --yes` there first, since `--plan` cannot see a Worker redeployed with the legacy token during the rollback. See [Proxy token rotation](#proxy-token-rotation).

Consult the workflow files for their current inputs. Store credentials in the
selected GitHub environment and the deployment owner's secret manager. Keep
Expand Down Expand Up @@ -84,6 +84,10 @@ The API also requests a rotation when a collaborator or admin loses deploy acces

Accept floors live in the ROUTING namespace. If it is ever recreated, every floor reads as 0 and old tokens work again: run `rotate-proxy-tokens --all --yes`. Before rolling the platform back, let running rotations finish (`--plan` shows nothing pending); after a rollback to the release before rotation, a project in the middle of a rotation gets 403 on remote D1/KV/R2 until the platform is upgraded again. So does a project deployed since the upgrade, until it is redeployed or the platform is upgraded again. A rollback to an older release drops rotations that have not finished, and Workers deployed while it runs carry the legacy token, which a rotated project rejects. After a rollback to the release just before rotation, a project first deployed since the upgrade is on generation 1 with no floor, so a redeploy during the rollback bakes the legacy token; after upgrading again, `--plan` does not list it, and the release that stops accepting the legacy token breaks it. After upgrading again from a rollback to either release, run `rotate-proxy-tokens --all --yes` and check with `--plan` until nothing is pending or failed: every Worker gets a new token and every floor rises again.

The proxy and API no longer accept or send the legacy token. Rotation in the previous release moves every project off it by itself; the scheduled run keeps queuing any project still on it. Wait until `--plan` shows 0 legacy, 0 pending and 0 failed (see the release order above). Redeploy the Workers listed by the query above before this release. A project still on the legacy token moves to generation 1 at its next deploy, and its older Workers get a rotation. Static projects are rotated too: they can keep earlier Worker deployments for rollback.

Do not roll back only the API to a release that does not check `x-void-proxy-internal` while the proxy runs a release without the legacy token: metadata calls fail until both match. A self-hosted upgrade from such a release fails these calls for the seconds until the new proxy is active.

## Secrets

Use separate credentials for CI deployment and platform runtime access. Runtime permissions cover tenant resource provisioning; the deploy credential also needs the permissions required to upload the configured Workers and build-container images.
Expand Down
6 changes: 3 additions & 3 deletions platform/packages/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,9 @@ HMAC. See [Platform Development](../../../docs/guide/platform/development/runtim
- `POST /internal/verify-owner/:projectId` — legacy URL that live-checks the caller's project deploy capability

`resolve-resource`, `resolve-worker` and `verify-prerender-build` authenticate
the proxy with its own key (`x-void-proxy-internal`); a request without it is
checked like a project token. `revalidate` checks the project token forwarded
by the proxy against the project's accept floor (see the proxy README).
the proxy with its own key (`x-void-proxy-internal`) and do not accept project
tokens. `revalidate` checks the project token forwarded by the proxy against
the project's accept floor (see the proxy README).

### Operator API

Expand Down
15 changes: 5 additions & 10 deletions platform/packages/api/src/admin/backfill-queue-tokens.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import { hmacSignState } from '../auth/hmac.ts';
import {
getDispatchWorkerProxyToken,
type ProxyTokenLookup,
Expand All @@ -8,7 +7,6 @@ export type BackfillQueueTokensOptions = Readonly<{
accountId: string;
apiToken: string;
dispatchNamespace: string;
jwtSecret: string;
/** Validate every candidate without writing to KV. */
dryRun?: boolean;
/**
Expand Down Expand Up @@ -52,9 +50,8 @@ type QueueRoutingEntry = Readonly<{
* Iterate every `queue:*` KV entry and stamp `proxyToken` on any entry
* written before the dispatch relay started forwarding
* `x-void-internal`. The stamped token is read from the target worker's
* `__VOID_PROXY_TOKEN` binding so it survives `JWT_SECRET` rotation —
* a freshly `hmacSignState(projectId, currentSecret)` value would 401
* any worker deployed before the rotation.
* `__VOID_PROXY_TOKEN` binding: the worker accepts only the value it
* holds. A worker without that binding is skipped; it needs a redeploy.
*
* Concurrency-safe-ish: each entry is re-read immediately before the
* put; if the value changed we skip the write so a concurrent deploy
Expand Down Expand Up @@ -123,12 +120,10 @@ export async function backfillQueueTokens(
);
errored++;
continue;
} else if (entry.projectId) {
// lookup.kind === 'missing' — legacy worker, no baked binding.
// Fall back to re-derivation. Only safe while JWT_SECRET has
// not rotated since this worker was deployed.
proxyToken = await hmacSignState(entry.projectId, opts.jwtSecret);
} else {
console.warn(
`[backfill-queue-tokens] worker "${entry.workerName}" has no __VOID_PROXY_TOKEN binding. Skipping "${name}" — redeploy the project.`,
);
skipped++;
continue;
}
Expand Down
12 changes: 7 additions & 5 deletions platform/packages/api/src/deploy/recovery.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { sql as drizzleSql } from 'drizzle-orm';
import { hmacSignState } from '../auth/hmac.ts';
import { DEFAULT_DISPATCH_NAMESPACE } from '../constants.ts';
import { clearCronScheduler, syncCronScheduler } from '../cron-scheduler-client.ts';
import {
Expand Down Expand Up @@ -172,11 +171,11 @@ export function resolvePersistedDeploymentType(
return manifest.assetConfig?.not_found_handling === 'single-page-application' ? 'spa' : 'static';
}

/** The Worker's own token, or `undefined` when it has no token binding. */
async function resolveCutoverProxyToken(
env: ApiEnv['Bindings'],
workerName: string,
projectId: string,
): Promise<string> {
): Promise<string | undefined> {
const lookup = await getDispatchWorkerProxyToken(
env.CF_ACCOUNT_ID,
env.CF_API_TOKEN,
Expand All @@ -189,7 +188,10 @@ async function resolveCutoverProxyToken(
if (lookup.kind === 'error') {
throw lookup.error;
}
return hmacSignState(projectId, env.JWT_SECRET);
console.warn(
`[cutover] worker "${workerName}" has no __VOID_PROXY_TOKEN binding; its cron and queue relays get no token. Redeploy the project.`,
);
return undefined;
}

/** Make every persisted custom-domain key match one authoritative route. */
Expand Down Expand Up @@ -340,7 +342,7 @@ export async function replayDeploymentCutover(
? await proxyTokenCopies(
env.JWT_SECRET,
project,
await resolveCutoverProxyToken(env, workerName, project.id),
await resolveCutoverProxyToken(env, workerName),
)
: {};

Expand Down
7 changes: 3 additions & 4 deletions platform/packages/api/src/id.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,9 @@ export function generateId(prefix: string): string {
const PROJECT_ID = new RegExp(`^proj_[${ALPHABET}]{${ID_LENGTH}}$`);

/**
* Whether `value` has the shape `generateId('proj')` produces. The project
* token is an HMAC over the project id under the same key that signs login
* JWTs and OAuth state, so callers check this before verifying the token:
* any other string (such as a JWT's `header.body`) must not authenticate.
* Whether `value` has the shape `generateId('proj')` produces. Callers check
* this before verifying a token bound to a project id, so no other string
* reaches the MAC check.
*/
export function isProjectId(value: unknown): value is string {
return typeof value === 'string' && PROJECT_ID.test(value);
Expand Down
24 changes: 6 additions & 18 deletions platform/packages/api/src/internal-proxy-auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,7 @@ export async function checkTenantProxyToken(
/**
* Authenticates the proxy on an internal metadata route. The proxy signs
* `x-void-proxy-internal` with its own key, bound to the route and project.
* A request without that header is checked like a tenant project token in
* `x-void-internal`, so a token below the project's accept floor is refused
* here as it is at the proxy.
* Tenant project tokens are not accepted here.
*
* Returns null when the call may proceed, else the error response.
*/
Expand All @@ -56,23 +54,13 @@ export async function requireProxyInternalAuth(
projectId: string,
): Promise<Response | null> {
const proxyAuth = context.req.header(PROXY_INTERNAL_AUTH_HEADER);
const token = context.req.header('x-void-internal');
if (!proxyAuth && !token) {
if (!proxyAuth) {
return context.json({ error: 'missing internal auth' }, 401);
}
if (!isProjectId(projectId)) {
return context.json({ error: 'invalid internal auth' }, 403);
}
if (proxyAuth) {
return (await verifyProxyInternalAuth(context.env.JWT_SECRET, route, projectId, proxyAuth))
? null
: context.json({ error: 'invalid internal auth' }, 403);
}
const tokenCheck = await checkTenantProxyToken(context.env, projectId, token ?? '');
if (tokenCheck === 'unavailable') {
return context.json({ error: 'project token state unavailable' }, 503);
}
if (tokenCheck === 'invalid') {
if (
!isProjectId(projectId) ||
!(await verifyProxyInternalAuth(context.env.JWT_SECRET, route, projectId, proxyAuth))
) {
return context.json({ error: 'invalid internal auth' }, 403);
}
return null;
Expand Down
1 change: 0 additions & 1 deletion platform/packages/api/src/operator/system-operations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,6 @@ export async function backfillPlatformQueueTokens(env: ApiEnv['Bindings'], dryRu
accountId: env.CF_ACCOUNT_ID,
apiToken: env.CF_API_TOKEN,
dispatchNamespace: env.DISPATCH_NAMESPACE ?? DEFAULT_DISPATCH_NAMESPACE,
jwtSecret: env.JWT_SECRET,
dryRun,
});
}
4 changes: 2 additions & 2 deletions platform/packages/api/src/provisioning/cloudflare-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -409,8 +409,8 @@ export type ProxyTokenLookup =
* Returns a typed result so callers can distinguish:
* - `found`: use the baked token as-is.
* - `missing`: bindings list returned but no `__VOID_PROXY_TOKEN` —
* a legacy worker. Callers may fall back to re-derivation with a
* visible warning.
* a legacy worker. Callers must not re-derive a token; the Worker
* needs a redeploy.
* - `error`: CF API failure (rate limit, transport, auth, 5xx).
* Callers MUST NOT fall back silently — a transient error must
* not end up writing a mismatched token that 401s after a
Expand Down
16 changes: 10 additions & 6 deletions platform/packages/api/src/proxy-token-rotation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,12 @@ export {
* Proxy token generations per project.
*
* `projects.proxy_token_gen` is the highest generation the platform has issued
* for the project (0 = the legacy token). Deploys bake the token for that
* generation into the Worker, and the proxy sends it. A Worker deploy moves 0
* to 1 (`issueProjectProxyToken`); after that only a rotation raises it. The
* KV accept floor (`proxy-token:<projectId>`) never exceeds it.
* for the project. 0 means nothing has been issued yet: the project's Workers
* still hold the legacy token, which is no longer accepted, and the proxy
* sends nothing. From 1 up, deploys bake the token for that generation into
* the Worker and the proxy sends it. A Worker deploy moves 0 to 1
* (`issueProjectProxyToken`); after that only a rotation raises it. The KV
* accept floor (`proxy-token:<projectId>`) never exceeds it.
*/

/** How long a rotation waits for KV copies and old Worker versions to drain. */
Expand Down Expand Up @@ -139,8 +141,10 @@ export async function proxyTokenCopies(
// deployment is in flight (the same fence as deploys, rollback and secret
// writes). A wait is stored in the payload as `waitUntil`, taken from
// `clock_timestamp()` after the step's last outside write, so a kick of
// `next_attempt_at` never cuts it short. Until the floor rises both tokens are
// accepted, so a failed or unfinished rotation never breaks a tenant.
// `next_attempt_at` never cuts it short. When the old generation is 1 or
// higher, both tokens are accepted until the floor rises, so a failed or
// unfinished rotation never breaks a tenant. The legacy token (generation 0)
// is never accepted.

export type ProxyTokenRotationPhase = 'requested' | 'copies' | 'patch' | 'revoke' | 'settle';

Expand Down
Loading
Loading