diff --git a/docs/guide/platform/administration/projects.md b/docs/guide/platform/administration/projects.md index 49311504a..953784381 100644 --- a/docs/guide/platform/administration/projects.md +++ b/docs/guide/platform/administration/projects.md @@ -23,7 +23,7 @@ void platform project show Project details include resources, domains, and recent builds and deployments. The [command reference](../../../reference/cli/platform.md#operator-commands) also covers suspending and restoring users, deleting projects, and removing accounts. -In the admin dashboard, open a project to view its team. Search for a platform user and choose a role to add them immediately; an email address is not required. You can also change or remove existing members. Pending invitations created through other project workflows remain visible until accepted or revoked. If email is unavailable for a pending invitation, share its ID so the user can accept it with `void project team accept `. The owner cannot be removed from the team; transfer ownership first. +In the admin dashboard, open a project to view its team. Search for a platform user and choose a role to add them immediately; an email address is not required. You can also change or remove existing members. Pending invitations created through other project workflows remain visible until accepted or revoked. If email is unavailable for a pending invitation, share its ID so the user can accept it with `void project team accept `. The owner cannot be removed from the team; transfer ownership first. Removing a collaborator or admin, changing one to reader, or suspending or deleting their account [replaces the project's platform credentials](../../project-collaboration.md#change-or-remove-access), the same as the CLI. ### Transferring Project Ownership diff --git a/docs/guide/platform/installation/maintenance.md b/docs/guide/platform/installation/maintenance.md index 6a81515b6..7a3d19d5c 100644 --- a/docs/guide/platform/installation/maintenance.md +++ b/docs/guide/platform/installation/maintenance.md @@ -98,6 +98,10 @@ void platform rollback [id] --runtime /path/to/earlier/runtime Keep the earlier runtime files if you need rollback. If the installed version is a custom build, also pass its files with `--from-runtime /path/to/current/runtime`. Void checks that the earlier version can run against your current database and refuses incompatible rollbacks. A later `upgrade` can move forward again. See [Platform Development](/guide/platform/development/runtime#deploying-your-runtime) for working with custom builds. +Do not roll back to 0.26.0 or earlier after upgrading to a later release. Void does not refuse this rollback, but those releases reject the credentials that later releases give your apps. Every project deployed or rotated since the upgrade loses AI, email, ISR revalidation, remote D1, KV, and R2, and prerendering until you upgrade again. If you have rolled back that far, upgrade again, then run `void platform system rotate-proxy-tokens --all --yes` and check with `--plan` until nothing is pending or failed. + +Let running rotations finish before you roll back. If an older proxy runs next to a newer API, prerendering and remote D1, KV, and R2 fail until both run the same release. After a rollback to the release just before credential rotation, projects deployed since the upgrade lose prerendering and remote D1, KV, and R2 until they are deployed again or you upgrade again. A rotated project deployed during any rollback stays broken after you upgrade again, and 0.26.0 and earlier also drop rotations that have not finished. After upgrading again from any rollback, run the same `rotate-proxy-tokens` commands. + ::: ### Follow upgrade and rollback progress diff --git a/docs/guide/project-collaboration.md b/docs/guide/project-collaboration.md index 852b78128..92a910c1d 100644 --- a/docs/guide/project-collaboration.md +++ b/docs/guide/project-collaboration.md @@ -92,3 +92,7 @@ void project team revoke ``` A non-owner member can leave with `void project team leave`. The owner cannot leave; an installation administrator must [transfer ownership](./platform/administration/projects.md#transferring-project-ownership) first. + +When a collaborator or admin is removed, leaves, or is changed to reader, the platform replaces the credentials the project's Workers use to call platform services such as AI, email, and remote bindings. The old credentials stop working once every Worker of the project has the new ones, usually within about 10 minutes, later while a deploy is running. If the replacement cannot finish, the old credentials keep working until it does, and owners and admins see a warning in `void project status` and `void project team list`. WebSocket connections reconnect once. While a Worker is switched to the new credentials, its calls to platform services can fail for a moment, a scheduled run can be skipped, and queue messages are delivered again a couple of minutes later. + +Code the removed person deployed keeps running until you replace it. If you no longer trust it, deploy code you trust; the platform replaces the credentials again after that deploy. Earlier deployments of theirs kept for rollback keep the old credentials, so after a rollback to one of them, AI, email, and ISR revalidation fail until you deploy again. diff --git a/docs/reference/cli/platform-operations.md b/docs/reference/cli/platform-operations.md index 4c0af538a..f550b18b3 100644 --- a/docs/reference/cli/platform-operations.md +++ b/docs/reference/cli/platform-operations.md @@ -54,6 +54,7 @@ void platform system asset-migrations [--page ] [--limit ] void platform system events [--page ] [--limit ] void platform system backfill-queue-tokens void platform system sandbox-drain [--cursor ] +void platform system rotate-proxy-tokens [--project | --all] ``` `overview` shows platform totals and recent activity. `health` checks the configured services and database, and exits with a nonzero status if a check fails. `cli-versions` reports the CLI versions used by deployments. @@ -64,6 +65,8 @@ void platform system sandbox-drain [--cursor ] `sandbox-drain` removes legacy tenant-owned Sandbox resources. Use `--plan` to preview and `--yes` to apply, then pass `nextCursor` as `--cursor` to continue. Application traffic stays paused during cleanup. +`rotate-proxy-tokens` gives projects new platform credentials on their live Worker and the Workers kept for rollback, without a redeploy. Workers kept for rollback whose author can no longer deploy keep the old credentials. Use `--project ` for one project or `--all` for every project, with `--plan` to preview and `--yes` to apply. `--plan` alone shows every rotation that is still running. Each run does a bounded batch and the platform finishes the rest in the background; a rotation takes about 6 minutes per project, longer while a deploy is running. Use `--plan` to check progress until nothing is pending or failed. Running `--yes` again with `--project` or `--all` while a rotation is still running starts one more rotation for that project. When a rotation is finished, the previous credentials no longer work. Each rotation starts a new version of the project's live Worker, so WebSocket connections reconnect once. + Use the [platform lifecycle commands](./platform-installation.md#lifecycle-commands) to maintain your installation's Workers. ## Hosted Workers {#operator-workers} diff --git a/docs/reference/cli/project.md b/docs/reference/cli/project.md index d47afec4b..a9b525fb7 100644 --- a/docs/reference/cli/project.md +++ b/docs/reference/cli/project.md @@ -10,6 +10,7 @@ Show deployments for the configured target. - Void targets show recent hosted deployments; `[name]` looks up a project by slug and otherwise the linked project is used. - Cloudflare targets list Worker Versions, identify the active version, and show the recorded migration count. A project name is not accepted because the Worker name comes from `cloudflare.name` in `void.config.ts`. +- Owners and admins also see a warning when the platform cannot finish [replacing the project's credentials](../../guide/project-collaboration.md#change-or-remove-access). ## `void project link [name]` {#void-project-link-name} @@ -36,7 +37,7 @@ void project team accept void project team decline ``` -Project-scoped commands use `--project`, then `VOID_PROJECT`, then the linked project. Invitations can target only an email address already registered to a user on that platform; they do not create accounts or grant signup access. Only the invited account can accept or decline its invitation. +Project-scoped commands use `--project`, then `VOID_PROJECT`, then the linked project. Invitations can target only an email address already registered to a user on that platform; they do not create accounts or grant signup access. Only the invited account can accept or decline its invitation. `list` shows the same credential warning as `void project status`. Readers can view the project but cannot deploy. Collaborators can deploy and manage deploy prerequisites. Project administrators can additionally manage domains, email destinations, GitHub configuration, and the project team. The owner alone can delete the project. See [Project Collaboration](../../guide/project-collaboration.md) for the full role boundaries. diff --git a/packages/void/src/cli/client.ts b/packages/void/src/cli/client.ts index 762e1e879..5ff1ec133 100644 --- a/packages/void/src/cli/client.ts +++ b/packages/void/src/cli/client.ts @@ -221,6 +221,11 @@ export type ProjectMemberResponse = { }; }; +export type ProjectMemberListResponse = { + members: Array; + warnings?: Array; +}; + export type ProjectInvitationResponse = { id: string; target_email?: string; @@ -337,6 +342,7 @@ export type DeploymentResponse = { export type DeploymentListResponse = { deployments: Array; activeDeploymentId: string | null; + warnings?: Array; }; export type SecretResponse = { @@ -963,15 +969,15 @@ export class PlatformClient { return res.json(); } - async listProjectMembers(projectId: string): Promise> { + async listProjectMembers(projectId: string): Promise { const res = await this.request(`${this.baseUrl}/projects/${projectId}/members`, { headers: { ...this.authHeaders }, }); if (!res.ok) { await throwPlatformApiError('Failed to list project members', res, 'read'); } - const body = (await res.json()) as { members: Array }; - return body.members; + const body = (await res.json()) as ProjectMemberListResponse; + return { members: body.members, ...(body.warnings ? { warnings: body.warnings } : {}) }; } async listProjectInvitations(projectId: string): Promise> { diff --git a/packages/void/src/cli/help.ts b/packages/void/src/cli/help.ts index eef856c25..74a70f747 100644 --- a/packages/void/src/cli/help.ts +++ b/packages/void/src/cli/help.ts @@ -111,7 +111,9 @@ function operatorHelpTree(): HelpTree { ? entry.action === 'domain-remove' ? 'Read a replacement Cloudflare token for blocked project cleanup from stdin' : 'Read a scoped Cloudflare API token from stdin' - : 'Read a full API session from stdin for explicit operator session exchange', + : flag === 'all' + ? 'Rotate every project' + : 'Read a full API session from stdin for explicit operator session exchange', })), ], }, diff --git a/packages/void/src/cli/list.ts b/packages/void/src/cli/list.ts index 98c03b4de..26c8ebcee 100644 --- a/packages/void/src/cli/list.ts +++ b/packages/void/src/cli/list.ts @@ -1,7 +1,7 @@ import { outro, log } from '@voidzero-dev/vite-plus-prompts'; import { getToken } from './auth.ts'; import { PlatformClient, type DeploymentResponse } from './client.ts'; -import { dim, ensureUtc, formatCount, pc, tablePrefix } from './output.ts'; +import { dim, ensureUtc, formatCount, pc, sanitizeTerminalText, tablePrefix } from './output.ts'; import { readDeploymentPlatform, readProjectConfig } from './project.ts'; import { resolveProjectBySlug } from './resolve-project.ts'; @@ -90,7 +90,10 @@ async function showDeployments( projectId: string, slug: string, ): Promise { - const { deployments, activeDeploymentId } = await client.listDeployments(projectId); + const { deployments, activeDeploymentId, warnings } = await client.listDeployments(projectId); + for (const warning of warnings ?? []) { + log.warn(sanitizeTerminalText(warning)); + } if (deployments.length === 0) { outro(`No deployments for ${slug}.`); diff --git a/packages/void/src/cli/operator-args.ts b/packages/void/src/cli/operator-args.ts index 03951b5f3..df153a617 100644 --- a/packages/void/src/cli/operator-args.ts +++ b/packages/void/src/cli/operator-args.ts @@ -364,6 +364,15 @@ export const operatorCommands: Array = [ values: { cursor: 'Opaque cursor returned by --plan' }, method: 'POST', }, + { + group: 'system', + action: 'rotate-proxy-tokens', + description: 'Rotate project proxy tokens on tenant Workers', + path: '/system/rotate-proxy-tokens', + values: { project: 'Project ID' }, + booleans: ['all'], + method: 'POST', + }, { group: 'email', action: 'policy', diff --git a/packages/void/src/cli/project-team.ts b/packages/void/src/cli/project-team.ts index 0c9bd75c5..a2dd4ad0d 100644 --- a/packages/void/src/cli/project-team.ts +++ b/packages/void/src/cli/project-team.ts @@ -6,7 +6,7 @@ import { type ProjectInvitationResponse, type ProjectMemberResponse, } from './client.ts'; -import { printSuccess, formatCount, note } from './output.ts'; +import { printSuccess, formatCount, note, sanitizeTerminalText } from './output.ts'; import { parsePlatformErrorBody, platformErrorMessage } from './platform-errors.ts'; import type { ProjectArgs } from './project-args.ts'; import { readProjectConfig } from './project.ts'; @@ -149,7 +149,10 @@ export async function runProjectTeamCommand(root: string, args: ProjectTeamArgs) 'projectSlug' in args ? args.projectSlug : '', ); if (args.action === 'list') { - const members = await client.listProjectMembers(projectId); + const { members, warnings } = await client.listProjectMembers(projectId); + for (const warning of warnings ?? []) { + log.warn(sanitizeTerminalText(warning)); + } note(memberLines(members), 'Project Team'); outro(formatCount(members.length, 'member')); return; diff --git a/packages/void/test/unit/cli-help.test.ts b/packages/void/test/unit/cli-help.test.ts index 54caa8e6c..cddee050c 100644 --- a/packages/void/test/unit/cli-help.test.ts +++ b/packages/void/test/unit/cli-help.test.ts @@ -141,6 +141,15 @@ describe('rendered help advertises the flags the parser honours', () => { expect(page).toContain('--project '); }); + it('lists proxy token rotation under platform system with its scope flags', () => { + expect(render(['platform', 'system'])).toContain('rotate-proxy-tokens'); + const page = render(['platform', 'system', 'rotate-proxy-tokens']); + expect(page).toContain('--project '); + expect(page).toContain('--all'); + expect(page).toContain('Rotate every project'); + expect(page).toContain('--plan'); + }); + it('still shows --project on the email subcommands that take only that', () => { for (const sub of ['usage', 'destinations', 'allow', 'disallow']) { const page = render(['email', sub]); diff --git a/packages/void/test/unit/list-command.test.ts b/packages/void/test/unit/list-command.test.ts index 08a3d9e1b..f30ccb120 100644 --- a/packages/void/test/unit/list-command.test.ts +++ b/packages/void/test/unit/list-command.test.ts @@ -163,4 +163,63 @@ describe('runListCommand', () => { expect(rowFor('dep-global')).toMatch(/\b120s\b/); expect(rowFor('dep-none')).not.toMatch(/rule|\d+s\b/); }); + + it('prints platform warnings before the deployment table', async () => { + setIsolatedHome(); + mockGetToken.mockReturnValue('test-token'); + const root = mkdtempSync(join(tmpdir(), 'void-list-')); + tempHomes.push(root); + mkdirSync(join(root, '.void'), { recursive: true }); + writeFileSync( + join(root, '.void', 'project.json'), + JSON.stringify({ projectId: 'proj-1', slug: 'demo' }), + ); + const body = (warnings?: Array) => ({ + deployments: [ + { + id: 'dep-live', + status: 'live', + source: 'cli', + commit: null, + manifest: JSON.stringify({ type: 'ssr' }), + error_message: null, + retained: true, + applied_migrations: null, + created_at: '2026-01-01T00:00:00Z', + deployed_at: '2026-01-01T00:00:00Z', + }, + ], + activeDeploymentId: 'dep-live', + ...(warnings ? { warnings } : {}), + }); + + const run = async (warnings?: Array) => { + vi.stubGlobal( + 'fetch', + vi.fn(async () => Response.json(body(warnings))), + ); + const capture = captureOutput(); + const runListCommand = await importRunListCommand(); + const { log } = await import('@voidzero-dev/vite-plus-prompts'); + const warnedAt: Array = []; + const warn = vi.spyOn(log, 'warn').mockImplementation(() => { + warnedAt.push(capture.output().length); + }); + // The prompts module survives vi.resetModules(), so a second spyOn + // returns the first spy with its earlier calls. + warn.mockClear(); + await runListCommand(root); + const out = capture.output(); + capture.restore(); + return { warn, warnedAt, tableAt: out.indexOf('dep-live') }; + }; + + const warned = await run(['stuck \u001b[31mred']); + expect(warned.warn).toHaveBeenCalledTimes(1); + expect(warned.warn).toHaveBeenCalledWith('stuck [31mred'); + expect(warned.tableAt).toBeGreaterThan(warned.warnedAt[0]!); + + const quiet = await run(); + expect(quiet.warn).not.toHaveBeenCalled(); + }); }); diff --git a/packages/void/test/unit/operator-cli.test.ts b/packages/void/test/unit/operator-cli.test.ts index 69861852d..38c351220 100644 --- a/packages/void/test/unit/operator-cli.test.ts +++ b/packages/void/test/unit/operator-cli.test.ts @@ -372,6 +372,33 @@ describe('operator argument and help coverage', () => { parseOperatorArgs(['deployment', 'migrate-assets', 'id', '--plan', '--yes']), ).toThrow('--plan and --yes cannot be combined'); }); + it('routes proxy token rotation with its project scope in the body', () => { + expect( + parsePlatformArgs([ + 'system', + 'rotate-proxy-tokens', + '--project', + 'proj_000000000001', + '--plan', + ]), + ).toMatchObject({ + method: 'POST', + path: '/system/rotate-proxy-tokens', + body: { project: 'proj_000000000001' }, + plan: true, + }); + const all = parseOperatorArgs(['system', 'rotate-proxy-tokens', '--all', '--yes']); + expect(all).toMatchObject({ method: 'POST', yes: true }); + expect(all.body.all).toBe(true); + expect(parseOperatorArgs(['system', 'rotate-proxy-tokens', '--yes']).body).toEqual({}); + expect(() => parseOperatorArgs(['system', 'rotate-proxy-tokens', '--dry-run'])).toThrow( + '--dry-run requires a value', + ); + expect(() => parseOperatorArgs(['system', 'sandbox-drain', '--all'])).toThrow( + "Unknown flag '--all'", + ); + expect(hasHelpPage(['platform', 'system', 'rotate-proxy-tokens'])).toBe(true); + }); it('routes asset migration inventory with bounded pagination and nested help', () => { expect( parsePlatformArgs(['system', 'asset-migrations', '--page', '2', '--limit', '100']), diff --git a/packages/void/test/unit/platform-installer.test.ts b/packages/void/test/unit/platform-installer.test.ts index 08b2ee23d..ef5401afc 100644 --- a/packages/void/test/unit/platform-installer.test.ts +++ b/packages/void/test/unit/platform-installer.test.ts @@ -2909,6 +2909,7 @@ describe('platform installer', () => { expect(database.migrations.slice(baseline.length).map((migration) => migration.tag)).toEqual([ '0056_email_shared_recipients', '0057_github_authority_constraints', + '0058_project_proxy_token_gen', ]); const rows = baseline.map((migration, index) => ({ id: index + 1, @@ -2934,7 +2935,7 @@ describe('platform installer', () => { '0.22.0', true, ); - expect(result.appliedMigrations).toBe(2); + expect(result.appliedMigrations).toBe(3); expect( accepted.queries.some((query) => query.includes('CREATE TRIGGER "void_shared_email_recipient_teardown_guard"'), @@ -2973,7 +2974,7 @@ describe('platform installer', () => { sourceVersion, true, ); - expect(result.appliedMigrations).toBe(1); + expect(result.appliedMigrations).toBe(2); expect( accepted.queries.some((query) => query.includes('"authorized_by_user_id" SET NOT NULL')), ).toBe(true); @@ -2981,6 +2982,44 @@ describe('platform installer', () => { }, ); + it.each(['0.25.0', '0.26.0'])( + 'upgrades a %s installation through the project proxy token generation', + async (sourceVersion) => { + const database = readPlatformRuntimeManifest().database; + const baseline = database.migrations.filter((migration) => migration.index <= 57); + expect(baseline.at(-1)?.tag).toBe('0057_github_authority_constraints'); + const rows = baseline.map((migration, index) => ({ + id: index + 1, + hash: migration.sha256, + created_at: migration.timestampMillis, + })); + const rejected = migrationConnection(rows); + await expect( + preparePlatformDatabaseConnection( + rejected, + getPlatformRuntimeDirectory(), + database, + '0.24.0', + true, + ), + ).rejects.toThrow('cannot safely roll back'); + expect(rejected.queries).not.toContain('BEGIN'); + const accepted = migrationConnection(rows); + const result = await preparePlatformDatabaseConnection( + accepted, + getPlatformRuntimeDirectory(), + database, + sourceVersion, + true, + ); + expect(result.appliedMigrations).toBe(1); + expect(accepted.queries.some((query) => query.includes('ADD COLUMN "proxy_token_gen"'))).toBe( + true, + ); + expect(accepted.rows).toHaveLength(database.migrations.length); + }, + ); + it('rejects the isolated legacy email0054 fork without rewriting its history or applying canonical migrations', async () => { const database = readPlatformRuntimeManifest().database; const commonPrefix = database.migrations.filter((migration) => migration.index <= 53); diff --git a/packages/void/test/unit/project-team-client.test.ts b/packages/void/test/unit/project-team-client.test.ts index d876e2ebd..ba7de6a84 100644 --- a/packages/void/test/unit/project-team-client.test.ts +++ b/packages/void/test/unit/project-team-client.test.ts @@ -39,12 +39,22 @@ describe('PlatformClient project teams', () => { fetchMock.mockResolvedValueOnce(json({ members: [member] })); fetchMock.mockResolvedValueOnce(json({ invitations: [invitation] })); - await expect(client().listProjectMembers(projectId)).resolves.toEqual([member]); + await expect(client().listProjectMembers(projectId)).resolves.toEqual({ members: [member] }); await expect(client().listProjectInvitations(projectId)).resolves.toEqual([invitation]); expect(request(0)[0]).toBe(`${apiUrl}/projects/${projectId}/members`); expect(request(1)[0]).toBe(`${apiUrl}/projects/${projectId}/invitations`); }); + it('passes member list warnings through', async () => { + const member = { user: { id: 'user_1', login: 'alex' }, role: 'owner' }; + fetchMock.mockResolvedValueOnce(json({ members: [member], warnings: ['stuck'] })); + + await expect(client().listProjectMembers(projectId)).resolves.toEqual({ + members: [member], + warnings: ['stuck'], + }); + }); + it('invites only by email and role', async () => { const result = { invitation: { id: 'invite_1', role: 'collaborator' }, diff --git a/packages/void/test/unit/project-team-command.test.ts b/packages/void/test/unit/project-team-command.test.ts index ac8e9001a..492ab67be 100644 --- a/packages/void/test/unit/project-team-command.test.ts +++ b/packages/void/test/unit/project-team-command.test.ts @@ -1,6 +1,7 @@ import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import { log } from '@voidzero-dev/vite-plus-prompts'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vite-plus/test'; import { PlatformApiError } from '../../src/cli/client.ts'; import { @@ -298,6 +299,46 @@ describe('project team target selection', () => { ); }); + it('prints platform warnings before the member list', async () => { + const root = rootWithPlatform('void'); + vi.stubEnv('VOID_TOKEN', 'test-token'); + vi.stubEnv('VOID_API_URL', 'https://api.example.test'); + let warnings: Array | undefined = ['stuck \u001b[31mred']; + vi.stubGlobal( + 'fetch', + withPlatformDiscovery( + vi.fn(async () => + Response.json({ + members: [{ user: { id: 'user_1', login: 'alex' }, role: 'owner' }], + ...(warnings ? { warnings } : {}), + }), + ), + ), + ); + let written = ''; + vi.spyOn(process.stdout, 'write').mockImplementation((chunk) => { + written += String(chunk); + return true; + }); + vi.spyOn(console, 'log').mockImplementation(() => undefined); + const warnedAt: Array = []; + const warn = vi.spyOn(log, 'warn').mockImplementation(() => { + warnedAt.push(written.length); + }); + const list = () => + runProjectCommand(root, { subcommand: 'team', action: 'list', projectSlug: '' }); + + await list(); + expect(warn).toHaveBeenCalledTimes(1); + expect(warn).toHaveBeenCalledWith('stuck [31mred'); + expect(written.indexOf('alex')).toBeGreaterThan(warnedAt[0]!); + + warn.mockClear(); + warnings = undefined; + await list(); + expect(warn).not.toHaveBeenCalled(); + }); + it('resolves an explicit project override before acting', async () => { const root = rootWithPlatform('void'); vi.stubEnv('VOID_TOKEN', 'test-token'); diff --git a/platform/meta/operations.md b/platform/meta/operations.md index aaaa88825..0eefc17cc 100644 --- a/platform/meta/operations.md +++ b/platform/meta/operations.md @@ -66,6 +66,35 @@ Choose a version from `worker show` for a single rollback. `rollback-all` select These commands are for Void Cloud's Workers. Self-hosted installations use the [platform lifecycle commands](../../docs/guide/platform/installation/maintenance.md#resume-repair-recover-and-upgrade). +### Proxy token rotation + +`void platform system rotate-proxy-tokens` replaces the project tokens tenant Workers use to call the proxy; see the [proxy README](../packages/proxy/README.md#project-tokens) for the token format and accept floor. Release order: the release whose API, proxy, and dispatch Workers accept token generations must be live on production first; then ship the release that adds rotation. After it reaches staging, rotate one canary project with `--project --yes` and check AI, email, queues, crons, remote D1/KV/R2, prerendering, and WebSocket reconnects. + +A deploy gives a project still on generation 0 a generation 1 token. Its older Workers keep the legacy token, which stays accepted until the project rotates. From that deploy on, a platform rollback to 0.26.0 or earlier breaks the project; see the rollback notes below. + +Only Workers named `user-`, from before per-deployment Worker names (2026-03-06), can still hold the legacy token in `__AI_TOKEN`, its name before the binding rename. Only AI calls used that binding: ISR revalidation then used `VOID_REVALIDATE_TOKEN`, and email did not exist yet. A project still served by a `user-` Worker has not been deployed since, so its `projects.active_deployment_id` is null while a `live` deployment exists. A rotation of such a project fails at START (`The active deployment is not recorded.`) and retries every 15 minutes. The issued generation and the floor never rise, so the app, AI included, keeps working, and after about an hour owners and admins see the stuck-rotation warning. A deploy would give the project a per-deployment Worker and let the rotation finish, but deploys are refused for the same reason (409 `project_requires_normalization`), and `void platform project normalize` refuses a route whose `workerName` is `user-`. Each of these projects needs an operator repair before it can deploy or rotate; until then its job stays failed in `--plan`. This query lists every project whose rotation stops at START this way; `used_ai` marks those with AI in a live deployment: + +```sql +SELECT p.id, p.slug, max(d.created_at) AS last_live, + bool_or(d.manifest::jsonb->'bindings'->>'ai' = 'true') AS used_ai +FROM projects p +JOIN deployments d ON d.project_id = p.id AND d.status = 'live' +WHERE p.active_deployment_id IS NULL + AND p.status <> 'deleting' +GROUP BY p.id, p.slug +ORDER BY last_live; +``` + +Each deployment row records the generation its Worker holds (`deployments.proxy_token_gen`). The proxy, the queue relay and the cron consumer send exactly that token, once, and never retry a 401/403 with another token, because the Worker's own middleware has already run. A rotation raises the issued generation, then PATCHes one Worker per step and, only after Cloudflare confirms the PATCH, records the new generation on its row and (for the active Worker) its queue entries and cron scheduler. The remaining gap is the drain right after each PATCH: until the step commits and Cloudflare stops serving the old version, that Worker's proxy calls get 403, cron fires are dropped with `AUTH_DRIFT`, and queue batches are retried after 120 seconds. If a step fails after its PATCH, the gap lasts until the step is retried. REVOKE raises the floor only when every patched row records the new generation. + +The API also requests a rotation when a collaborator or admin loses deploy access, and again when a deploy or rollback replaces code from a user who can no longer deploy. A rotation patches the active Worker and every retained Worker except those whose author can no longer deploy (`[proxy-token-rotation] skipped`): they keep the old token, so after a rollback to one, its own calls to the proxy (AI, email, ISR revalidation) fail with 403 until the project is redeployed. Remote D1/KV/R2 keep working, because the proxy sends the token its deployment row records. The rollback itself is not refused. The API cron advances rotations every two minutes. A failed step is retried after 15 minutes and logs `[proxy-token-rotation] FAILED phase= attempts= worker=: `. A step that failed after its PATCH may have left that Worker ahead of its row, so it is retried on every cron run until it commits; a run tries each job once. Alert on `attempts >= 4` (about one hour of retries, or about 8 minutes after a failed PATCH step); project owners and admins see a CLI warning at the same point. `--plan --project ` shows the job's phase and last error. A Cloudflare rate limit is not a failed attempt: the step logs `[proxy-token-rotation] RATE-LIMITED phase= retryAfter=s` and waits. Alert when that line keeps appearing for one project for more than an hour; the same limit also fails deploys, which use the same `CF_API_TOKEN`. + +Accept floors live in the ROUTING namespace. If it is ever recreated, every floor reads as 0 and old tokens work again: run `rotate-proxy-tokens --all --yes`. + +Do not roll the platform back to 0.26.0 or earlier, the releases before token generations, once a release with rotation has run. `void platform rollback` allows it, because `drizzle/compatibility.json` lists 0.25.0 and 0.26.0 for schema reasons, but their proxy accepts and sends only the legacy token. Every project deployed since the upgrade (its Worker holds a generation 1 or later token) and every rotated project then loses AI, email, ISR revalidation, remote D1/KV/R2 and prerendering until the platform is upgraded again. Those releases also drop rotations that have not finished, and Workers deployed while they run carry the legacy token, which a rotated project rejects after the upgrade. If such a rollback happened, upgrade again, then run the recovery below. + +Before rolling the platform back, let running rotations finish (`--plan` shows nothing pending). After a rollback to the release before rotation, a project in the middle of a rotation gets 403 on remote D1/KV/R2, and so does a project deployed since the upgrade, until it is redeployed or the platform is upgraded again; unfinished rotations resume after the upgrade. A rotated project deployed while the release before rotation runs gets a deployment row with generation 0 for a Worker that holds a newer token, so it gets 403 after the upgrade until it is rotated or redeployed. After upgrading again from any rollback, run `rotate-proxy-tokens --all --yes` and check with `--plan` until nothing is pending or failed, apart from the projects above without an active deployment: every Worker gets a new token and every floor rises again. Keep the API and proxy on the same release: a proxy from before token generations cannot call the metadata routes of an API that has rotation. + ## Secrets Use separate credentials for CI deployment and platform runtime access. Runtime permissions cover tenant resource provisioning; the deploy credential also needs the permissions required to upload the configured Workers and build-container images. diff --git a/platform/packages/api/README.md b/platform/packages/api/README.md index 26b3fa0c2..56b0cb174 100644 --- a/platform/packages/api/README.md +++ b/platform/packages/api/README.md @@ -145,9 +145,9 @@ HMAC. See [Platform Development](../../../docs/guide/platform/development/runtim - `POST /internal/verify-owner/:projectId` — legacy URL that live-checks the caller's project deploy capability `resolve-resource`, `resolve-worker` and `verify-prerender-build` authenticate -the proxy with its own key (`x-void-proxy-internal`); a request without it is -checked like a project token. `revalidate` checks the project token forwarded -by the proxy against the project's accept floor (see the proxy README). +the proxy with its own key (`x-void-proxy-internal`) and do not accept project +tokens. `revalidate` checks the project token forwarded by the proxy against +the project's accept floor (see the proxy README). ### Operator API diff --git a/platform/packages/api/drizzle/0058_project_proxy_token_gen.sql b/platform/packages/api/drizzle/0058_project_proxy_token_gen.sql new file mode 100644 index 000000000..5345f1252 --- /dev/null +++ b/platform/packages/api/drizzle/0058_project_proxy_token_gen.sql @@ -0,0 +1,2 @@ +ALTER TABLE "deployments" ADD COLUMN "proxy_token_gen" integer DEFAULT 0 NOT NULL;--> statement-breakpoint +ALTER TABLE "projects" ADD COLUMN "proxy_token_gen" integer DEFAULT 0 NOT NULL; \ No newline at end of file diff --git a/platform/packages/api/drizzle/compatibility.json b/platform/packages/api/drizzle/compatibility.json index 869729275..3e5377562 100644 --- a/platform/packages/api/drizzle/compatibility.json +++ b/platform/packages/api/drizzle/compatibility.json @@ -57,6 +57,14 @@ { "platformVersion": "0.24.0", "schemaDigest": "3e3458ba1268853dc8e9a887270863557c2a4c5a5210f8615d247bf1f4bf6bcc" + }, + { + "platformVersion": "0.25.0", + "schemaDigest": "7581930465664b7c578518bea54f36a83d20fcc47d9528183419033930d7ff4a" + }, + { + "platformVersion": "0.26.0", + "schemaDigest": "7581930465664b7c578518bea54f36a83d20fcc47d9528183419033930d7ff4a" } ] } diff --git a/platform/packages/api/drizzle/meta/0058_snapshot.json b/platform/packages/api/drizzle/meta/0058_snapshot.json new file mode 100644 index 000000000..2320520ce --- /dev/null +++ b/platform/packages/api/drizzle/meta/0058_snapshot.json @@ -0,0 +1,6531 @@ +{ + "id": "7de15010-25c6-463f-9b9e-395589b18901", + "prevId": "075ff55d-533d-44d8-acba-66fa0e8d29ae", + "version": "7", + "dialect": "postgresql", + "tables": { + "public.auth_codes": { + "name": "auth_codes", + "schema": "", + "columns": { + "code": { + "name": "code", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "jwt": { + "name": "jwt", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "cli_state": { + "name": "cli_state", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "code_challenge": { + "name": "code_challenge", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "code_challenge_method": { + "name": "code_challenge_method", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.builds": { + "name": "builds", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "executor": { + "name": "executor", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'container'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'queued'" + }, + "commit_sha": { + "name": "commit_sha", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "branch": { + "name": "branch", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "deployment_id": { + "name": "deployment_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "github_run_id": { + "name": "github_run_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "github_run_attempt": { + "name": "github_run_attempt", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "logs_url": { + "name": "logs_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "check_run_id": { + "name": "check_run_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "error": { + "name": "error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "exit_code": { + "name": "exit_code", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "instance_type": { + "name": "instance_type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "started_at": { + "name": "started_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_builds_project_created": { + "name": "idx_builds_project_created", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_builds_status": { + "name": "idx_builds_status", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_builds_user_status": { + "name": "idx_builds_user_status", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_builds_project_github_run_id": { + "name": "idx_builds_project_github_run_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "github_run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "github_run_attempt", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"builds\".\"github_run_id\" IS NOT NULL AND \"builds\".\"github_run_attempt\" IS NOT NULL AND \"builds\".\"status\" IN ('starting', 'building', 'deploying')", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "builds_project_id_projects_id_fk": { + "name": "builds_project_id_projects_id_fk", + "tableFrom": "builds", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + }, + "builds_user_id_users_id_fk": { + "name": "builds_user_id_users_id_fk", + "tableFrom": "builds", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + }, + "builds_deployment_id_deployments_id_fk": { + "name": "builds_deployment_id_deployments_id_fk", + "tableFrom": "builds", + "tableTo": "deployments", + "columnsFrom": ["deployment_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.deployment_operations": { + "name": "deployment_operations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "actor_id": { + "name": "actor_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "build_id": { + "name": "build_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "original_active_id": { + "name": "original_active_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "cli_version": { + "name": "cli_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "idempotency_key": { + "name": "idempotency_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "input_digest": { + "name": "input_digest", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "input_prefix": { + "name": "input_prefix", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'queued'" + }, + "owner_id": { + "name": "owner_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "lease_expires_at": { + "name": "lease_expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "next_dispatch_at": { + "name": "next_dispatch_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "events": { + "name": "events", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'[]'::jsonb" + }, + "last_seq": { + "name": "last_seq", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "inputs_deleted": { + "name": "inputs_deleted", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_deployment_operations_key": { + "name": "idx_deployment_operations_key", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "actor_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "idempotency_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_deployment_operations_dispatch": { + "name": "idx_deployment_operations_dispatch", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "next_dispatch_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "deployment_operations_project_id_projects_id_fk": { + "name": "deployment_operations_project_id_projects_id_fk", + "tableFrom": "deployment_operations", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.deployments": { + "name": "deployments", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "manifest": { + "name": "manifest", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "commit": { + "name": "commit", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "repository_url": { + "name": "repository_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "commit_url": { + "name": "commit_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "git_ref": { + "name": "git_ref", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "cli_version": { + "name": "cli_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "admission_protocol": { + "name": "admission_protocol", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "asset_hashes": { + "name": "asset_hashes", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "deployed_at": { + "name": "deployed_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "retained": { + "name": "retained", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "applied_migrations": { + "name": "applied_migrations", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "bundle_stored": { + "name": "bundle_stored", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "billing_user_id": { + "name": "billing_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "initiated_by_user_id": { + "name": "initiated_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "proxy_token_gen": { + "name": "proxy_token_gen", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + } + }, + "indexes": { + "idx_deployments_project_id": { + "name": "idx_deployments_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_deployments_project_status": { + "name": "idx_deployments_project_status", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_deployments_created_id": { + "name": "idx_deployments_created_id", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "deployments_project_id_projects_id_fk": { + "name": "deployments_project_id_projects_id_fk", + "tableFrom": "deployments", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.domains": { + "name": "domains", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "hostname": { + "name": "hostname", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "cf_hostname_id": { + "name": "cf_hostname_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "verification_data": { + "name": "verification_data", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_domains_project_id": { + "name": "idx_domains_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "domains_project_id_projects_id_fk": { + "name": "domains_project_id_projects_id_fk", + "tableFrom": "domains", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "domains_hostname_unique": { + "name": "domains_hostname_unique", + "nullsNotDistinct": false, + "columns": ["hostname"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_destination_cleanups": { + "name": "email_destination_cleanups", + "schema": "", + "columns": { + "cf_destination_id": { + "name": "cf_destination_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "attempts": { + "name": "attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "next_attempt_at": { + "name": "next_attempt_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "lease_started_at": { + "name": "lease_started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_email_destination_cleanups_due": { + "name": "idx_email_destination_cleanups_due", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "next_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "email_destination_cleanups_status_check": { + "name": "email_destination_cleanups_status_check", + "value": "\"email_destination_cleanups\".\"status\" IN ('pending', 'deleting', 'completed')" + }, + "email_destination_cleanups_attempts_check": { + "name": "email_destination_cleanups_attempts_check", + "value": "\"email_destination_cleanups\".\"attempts\" >= 0" + } + }, + "isRLSEnabled": false + }, + "public.email_destinations": { + "name": "email_destinations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "address": { + "name": "address", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "cf_destination_id": { + "name": "cf_destination_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "verified_at": { + "name": "verified_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "verification_source": { + "name": "verification_source", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'cf'" + }, + "verified_by_user_id": { + "name": "verified_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "verified_by_email_snapshot": { + "name": "verified_by_email_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_email_destinations_project_id": { + "name": "idx_email_destinations_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_email_destinations_cf_destination_id": { + "name": "idx_email_destinations_cf_destination_id", + "columns": [ + { + "expression": "cf_destination_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_email_destinations_verified_by_user_id": { + "name": "idx_email_destinations_verified_by_user_id", + "columns": [ + { + "expression": "verified_by_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "verified_by_email_snapshot", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_destinations_project_id_projects_id_fk": { + "name": "email_destinations_project_id_projects_id_fk", + "tableFrom": "email_destinations", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "email_destinations_verified_by_user_id_users_id_fk": { + "name": "email_destinations_verified_by_user_id_users_id_fk", + "tableFrom": "email_destinations", + "tableTo": "users", + "columnsFrom": ["verified_by_user_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "email_destinations_project_address_uq": { + "name": "email_destinations_project_address_uq", + "nullsNotDistinct": false, + "columns": ["project_id", "address"] + } + }, + "policies": {}, + "checkConstraints": { + "email_destinations_status_check": { + "name": "email_destinations_status_check", + "value": "\"email_destinations\".\"status\" IN ('pending', 'verified')" + }, + "email_destinations_verification_source_check": { + "name": "email_destinations_verification_source_check", + "value": "\"email_destinations\".\"verification_source\" IN ('cf', 'platform')" + } + }, + "isRLSEnabled": false + }, + "public.email_domains": { + "name": "email_domains", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "mode": { + "name": "mode", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "credential_kind": { + "name": "credential_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "token_ciphertext": { + "name": "token_ciphertext", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "token_iv": { + "name": "token_iv", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "token_key_version": { + "name": "token_key_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "managed_by": { + "name": "managed_by", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'owner'" + }, + "relay_ciphertext": { + "name": "relay_ciphertext", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "relay_iv": { + "name": "relay_iv", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "relay_key_version": { + "name": "relay_key_version", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "relay_worker_name": { + "name": "relay_worker_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "relay_version": { + "name": "relay_version", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "token_id": { + "name": "token_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "token_expires_on": { + "name": "token_expires_on", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "attempt_id": { + "name": "attempt_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "setup_state": { + "name": "setup_state", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_email_domains_project_id": { + "name": "idx_email_domains_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_domains_zone_uq": { + "name": "email_domains_zone_uq", + "columns": [ + { + "expression": "zone_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"email_domains\".\"zone_id\" IS NOT NULL AND \"email_domains\".\"status\" IN ('active', 'pending')", + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_domains_relay_uq": { + "name": "email_domains_relay_uq", + "columns": [ + { + "expression": "account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "relay_worker_name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"email_domains\".\"relay_worker_name\" IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_domains_project_id_projects_id_fk": { + "name": "email_domains_project_id_projects_id_fk", + "tableFrom": "email_domains", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "email_domains_domain_uq": { + "name": "email_domains_domain_uq", + "nullsNotDistinct": false, + "columns": ["domain"] + } + }, + "policies": {}, + "checkConstraints": { + "email_domains_mode_check": { + "name": "email_domains_mode_check", + "value": "\"email_domains\".\"mode\" IN ('apex', 'subdomain')" + }, + "email_domains_status_check": { + "name": "email_domains_status_check", + "value": "\"email_domains\".\"status\" IN ('pending', 'active', 'token_revoked', 'token_expired', 'failed')" + }, + "email_domains_credential_kind_check": { + "name": "email_domains_credential_kind_check", + "value": "\"email_domains\".\"credential_kind\" IN ('oauth_grant', 'api_token', 'platform')" + }, + "email_domains_platform_token_check": { + "name": "email_domains_platform_token_check", + "value": "(\"email_domains\".\"credential_kind\" = 'platform' AND \"email_domains\".\"token_ciphertext\" IS NULL AND \"email_domains\".\"token_iv\" IS NULL AND \"email_domains\".\"token_key_version\" IS NULL) OR (\"email_domains\".\"credential_kind\" <> 'platform' AND \"email_domains\".\"token_ciphertext\" IS NOT NULL AND \"email_domains\".\"token_iv\" IS NOT NULL AND \"email_domains\".\"token_key_version\" IS NOT NULL)" + }, + "email_domains_managed_by_check": { + "name": "email_domains_managed_by_check", + "value": "\"email_domains\".\"managed_by\" IN ('owner', 'admin')" + } + }, + "isRLSEnabled": false + }, + "public.github_connections": { + "name": "github_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "repo_full_name": { + "name": "repo_full_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "repository_id": { + "name": "repository_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "authorized_by_user_id": { + "name": "authorized_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "branch": { + "name": "branch", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'main'" + }, + "build_executor": { + "name": "build_executor", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'container'" + }, + "workflow_path": { + "name": "workflow_path", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'.github/workflows/void-deploy.yml'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_github_connections_installation_id": { + "name": "idx_github_connections_installation_id", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_connections_repo_full_name": { + "name": "idx_github_connections_repo_full_name", + "columns": [ + { + "expression": "repo_full_name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_connections_installation_repo_id": { + "name": "idx_github_connections_installation_repo_id", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "github_connections_project_id_projects_id_fk": { + "name": "github_connections_project_id_projects_id_fk", + "tableFrom": "github_connections", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "github_connections_authorized_by_user_id_users_id_fk": { + "name": "github_connections_authorized_by_user_id_users_id_fk", + "tableFrom": "github_connections", + "tableTo": "users", + "columnsFrom": ["authorized_by_user_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "github_connections_project_id_unique": { + "name": "github_connections_project_id_unique", + "nullsNotDistinct": false, + "columns": ["project_id"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.github_installation_members": { + "name": "github_installation_members", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "github_account_id": { + "name": "github_account_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "verified_at": { + "name": "verified_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_github_installation_members_install_user": { + "name": "idx_github_installation_members_install_user", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_installation_members_user_id": { + "name": "idx_github_installation_members_user_id", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "github_installation_members_user_id_users_id_fk": { + "name": "github_installation_members_user_id_users_id_fk", + "tableFrom": "github_installation_members", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.github_installation_revocations": { + "name": "github_installation_revocations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "github_account_id": { + "name": "github_account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "revoked_at": { + "name": "revoked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_github_installation_revocations_install_account": { + "name": "idx_github_installation_revocations_install_account", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "github_account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.github_installations": { + "name": "github_installations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "account_login": { + "name": "account_login", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_type": { + "name": "account_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "repositories": { + "name": "repositories", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'[]'::jsonb" + }, + "repository_selection": { + "name": "repository_selection", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'selected'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_github_installations_user_id": { + "name": "idx_github_installations_user_id", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_installations_account_id": { + "name": "idx_github_installations_account_id", + "columns": [ + { + "expression": "account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "github_installations_user_id_users_id_fk": { + "name": "github_installations_user_id_users_id_fk", + "tableFrom": "github_installations", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "github_installations_installation_id_unique": { + "name": "github_installations_installation_id_unique", + "nullsNotDistinct": false, + "columns": ["installation_id"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.github_webhook_redeliveries": { + "name": "github_webhook_redeliveries", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "guid": { + "name": "guid", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delivery_id": { + "name": "delivery_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "event": { + "name": "event", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status_code": { + "name": "status_code", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "redelivered_at": { + "name": "redelivered_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_github_webhook_redeliveries_guid": { + "name": "idx_github_webhook_redeliveries_guid", + "columns": [ + { + "expression": "guid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.invitations": { + "name": "invitations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "email_delivery_status": { + "name": "email_delivery_status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "invited_by": { + "name": "invited_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "accepted_by": { + "name": "accepted_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "revoked_at": { + "name": "revoked_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "revoked_by": { + "name": "revoked_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "accepted_at": { + "name": "accepted_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_invitations_email": { + "name": "idx_invitations_email", + "columns": [ + { + "expression": "email", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_invitations_status": { + "name": "idx_invitations_status", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "invitations_email_delivery_status_check": { + "name": "invitations_email_delivery_status_check", + "value": "\"invitations\".\"email_delivery_status\" IS NULL OR \"invitations\".\"email_delivery_status\" IN ('sent', 'failed', 'unavailable')" + } + }, + "isRLSEnabled": false + }, + "public.migration_renames": { + "name": "migration_renames", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "from_name": { + "name": "from_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "to_name": { + "name": "to_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_migration_renames_project_id": { + "name": "idx_migration_renames_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "migration_renames_project_id_projects_id_fk": { + "name": "migration_renames_project_id_projects_id_fk", + "tableFrom": "migration_renames", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "migration_renames_project_id_from_name_to_name_pk": { + "name": "migration_renames_project_id_from_name_to_name_pk", + "columns": ["project_id", "from_name", "to_name"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.oauth_accounts": { + "name": "oauth_accounts", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_oauth_accounts_provider_id": { + "name": "idx_oauth_accounts_provider_id", + "columns": [ + { + "expression": "provider", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "provider_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_oauth_accounts_user_id": { + "name": "idx_oauth_accounts_user_id", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "oauth_accounts_user_id_users_id_fk": { + "name": "oauth_accounts_user_id_users_id_fk", + "tableFrom": "oauth_accounts", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.oauth_nonces": { + "name": "oauth_nonces", + "schema": "", + "columns": { + "nonce": { + "name": "nonce", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.operator_events": { + "name": "operator_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "login": { + "name": "login", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target": { + "name": "target", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "http_status": { + "name": "http_status", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_operator_events_created": { + "name": "idx_operator_events_created", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.operator_sessions": { + "name": "operator_sessions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "human_session_id": { + "name": "human_session_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_operator_sessions_expiry": { + "name": "idx_operator_sessions_expiry", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "operator_sessions_user_id_users_id_fk": { + "name": "operator_sessions_user_id_users_id_fk", + "tableFrom": "operator_sessions", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "operator_sessions_human_session_id_platform_human_sessions_id_fk": { + "name": "operator_sessions_human_session_id_platform_human_sessions_id_fk", + "tableFrom": "operator_sessions", + "tableTo": "platform_human_sessions", + "columnsFrom": ["human_session_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_auth_attempts": { + "name": "platform_auth_attempts", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "purpose": { + "name": "purpose", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "browser_hash": { + "name": "browser_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "context": { + "name": "context", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "platform_auth_attempts_expiry": { + "name": "platform_auth_attempts_expiry", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "platform_auth_attempts_connection_id_platform_login_connections_id_fk": { + "name": "platform_auth_attempts_connection_id_platform_login_connections_id_fk", + "tableFrom": "platform_auth_attempts", + "tableTo": "platform_login_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_auth_bootstrap": { + "name": "platform_auth_bootstrap", + "schema": "", + "columns": { + "singleton": { + "name": "singleton", + "type": "boolean", + "primaryKey": true, + "notNull": true, + "default": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "credential_hash": { + "name": "credential_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "request_hash": { + "name": "request_hash", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "claim_hash": { + "name": "claim_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "pending_identity": { + "name": "pending_identity", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "consumed_at": { + "name": "consumed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": { + "platform_auth_bootstrap_connection_id_platform_login_connections_id_fk": { + "name": "platform_auth_bootstrap_connection_id_platform_login_connections_id_fk", + "tableFrom": "platform_auth_bootstrap", + "tableTo": "platform_login_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "platform_auth_bootstrap_singleton": { + "name": "platform_auth_bootstrap_singleton", + "value": "\"platform_auth_bootstrap\".\"singleton\" = true" + } + }, + "isRLSEnabled": false + }, + "public.platform_auth_settings": { + "name": "platform_auth_settings", + "schema": "", + "columns": { + "singleton": { + "name": "singleton", + "type": "boolean", + "primaryKey": true, + "notNull": true, + "default": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "admission": { + "name": "admission", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "protection": { + "name": "protection", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "session_seconds": { + "name": "session_seconds", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 2592000 + }, + "sessions_required": { + "name": "sessions_required", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "platform_auth_settings_singleton": { + "name": "platform_auth_settings_singleton", + "value": "\"platform_auth_settings\".\"singleton\" = true" + }, + "platform_auth_settings_revision": { + "name": "platform_auth_settings_revision", + "value": "\"platform_auth_settings\".\"revision\" > 0" + }, + "platform_auth_settings_session_seconds": { + "name": "platform_auth_settings_session_seconds", + "value": "\"platform_auth_settings\".\"session_seconds\" BETWEEN 300 AND 2592000" + } + }, + "isRLSEnabled": false + }, + "public.platform_company_identities": { + "name": "platform_company_identities", + "schema": "", + "columns": { + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject": { + "name": "subject", + "type": "text", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "platform_company_identity_user": { + "name": "platform_company_identity_user", + "columns": [ + { + "expression": "issuer", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "platform_company_identities_user_id_users_id_fk": { + "name": "platform_company_identities_user_id_users_id_fk", + "tableFrom": "platform_company_identities", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "platform_company_identities_issuer_subject_pk": { + "name": "platform_company_identities_issuer_subject_pk", + "columns": ["issuer", "subject"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_events": { + "name": "platform_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "batch_id": { + "name": "batch_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "worker_name": { + "name": "worker_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "environment": { + "name": "environment", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "version_id": { + "name": "version_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "script_name": { + "name": "script_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "admin_user_id": { + "name": "admin_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "admin_login": { + "name": "admin_login", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_platform_events_batch_id": { + "name": "idx_platform_events_batch_id", + "columns": [ + { + "expression": "batch_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_platform_events_env_created": { + "name": "idx_platform_events_env_created", + "columns": [ + { + "expression": "environment", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_human_sessions": { + "name": "platform_human_sessions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "audience": { + "name": "audience", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "access_issuer": { + "name": "access_issuer", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "access_subject": { + "name": "access_subject", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "revoked_at": { + "name": "revoked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "platform_human_sessions_connection": { + "name": "platform_human_sessions_connection", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "platform_human_sessions_user": { + "name": "platform_human_sessions_user", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "platform_human_sessions_expiry": { + "name": "platform_human_sessions_expiry", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "platform_human_sessions_user_id_users_id_fk": { + "name": "platform_human_sessions_user_id_users_id_fk", + "tableFrom": "platform_human_sessions", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "platform_human_sessions_connection_id_platform_login_connections_id_fk": { + "name": "platform_human_sessions_connection_id_platform_login_connections_id_fk", + "tableFrom": "platform_human_sessions", + "tableTo": "platform_login_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "platform_human_sessions_audience": { + "name": "platform_human_sessions_audience", + "value": "\"platform_human_sessions\".\"audience\" IN ('api', 'admin')" + }, + "platform_human_sessions_access_identity": { + "name": "platform_human_sessions_access_identity", + "value": "(\"platform_human_sessions\".\"access_issuer\" IS NULL) = (\"platform_human_sessions\".\"access_subject\" IS NULL)" + } + }, + "isRLSEnabled": false + }, + "public.platform_login_connections": { + "name": "platform_login_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "enabled": { + "name": "enabled", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "active_revision": { + "name": "active_revision", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "platform_login_connections_kind": { + "name": "platform_login_connections_kind", + "value": "\"platform_login_connections\".\"kind\" IN ('github', 'google', 'oidc', 'cloudflare-access')" + }, + "platform_login_connections_revision": { + "name": "platform_login_connections_revision", + "value": "\"platform_login_connections\".\"revision\" > 0 AND \"platform_login_connections\".\"generation\" > 0" + }, + "platform_login_connections_active_revision": { + "name": "platform_login_connections_active_revision", + "value": "\"platform_login_connections\".\"active_revision\" IS NULL OR \"platform_login_connections\".\"active_revision\" BETWEEN 1 AND \"platform_login_connections\".\"revision\"" + }, + "platform_login_connections_enabled": { + "name": "platform_login_connections_enabled", + "value": "NOT \"platform_login_connections\".\"enabled\" OR \"platform_login_connections\".\"active_revision\" IS NOT NULL" + } + }, + "isRLSEnabled": false + }, + "public.platform_login_identities": { + "name": "platform_login_identities", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject": { + "name": "subject", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "last_authenticated_at": { + "name": "last_authenticated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "platform_login_identity_subject": { + "name": "platform_login_identity_subject", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issuer", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "subject", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "platform_login_identity_user": { + "name": "platform_login_identity_user", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "platform_login_identities_user_id_users_id_fk": { + "name": "platform_login_identities_user_id_users_id_fk", + "tableFrom": "platform_login_identities", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "platform_login_identities_connection_id_platform_login_connections_id_fk": { + "name": "platform_login_identities_connection_id_platform_login_connections_id_fk", + "tableFrom": "platform_login_identities", + "tableTo": "platform_login_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_login_revisions": { + "name": "platform_login_revisions", + "schema": "", + "columns": { + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "configuration": { + "name": "configuration", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "secret": { + "name": "secret", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "platform_login_revisions_connection_id_platform_login_connections_id_fk": { + "name": "platform_login_revisions_connection_id_platform_login_connections_id_fk", + "tableFrom": "platform_login_revisions", + "tableTo": "platform_login_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "platform_login_revisions_connection_id_revision_pk": { + "name": "platform_login_revisions_connection_id_revision_pk", + "columns": ["connection_id", "revision"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_login_tests": { + "name": "platform_login_tests", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject": { + "name": "subject", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "login": { + "name": "login", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "platform_login_tests_expiry": { + "name": "platform_login_tests_expiry", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "platform_login_tests_user_id_users_id_fk": { + "name": "platform_login_tests_user_id_users_id_fk", + "tableFrom": "platform_login_tests", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "platform_login_tests_connection_id_platform_login_connections_id_fk": { + "name": "platform_login_tests_connection_id_platform_login_connections_id_fk", + "tableFrom": "platform_login_tests", + "tableTo": "platform_login_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.platform_settings": { + "name": "platform_settings", + "schema": "", + "columns": { + "key": { + "name": "key", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "value": { + "name": "value", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.project_access_events": { + "name": "project_access_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "actor_user_id": { + "name": "actor_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject_user_id": { + "name": "subject_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "invitation_email_snapshot": { + "name": "invitation_email_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "previous_role": { + "name": "previous_role", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "new_role": { + "name": "new_role", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_project_access_events_project_created": { + "name": "idx_project_access_events_project_created", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_project_access_events_subject_created": { + "name": "idx_project_access_events_subject_created", + "columns": [ + { + "expression": "subject_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_access_events_project_id_projects_id_fk": { + "name": "project_access_events_project_id_projects_id_fk", + "tableFrom": "project_access_events", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_access_events_previous_role_check": { + "name": "project_access_events_previous_role_check", + "value": "\"project_access_events\".\"previous_role\" IS NULL OR \"project_access_events\".\"previous_role\" IN ('reader', 'collaborator', 'admin')" + }, + "project_access_events_new_role_check": { + "name": "project_access_events_new_role_check", + "value": "\"project_access_events\".\"new_role\" IS NULL OR \"project_access_events\".\"new_role\" IN ('reader', 'collaborator', 'admin')" + }, + "project_access_events_action_check": { + "name": "project_access_events_action_check", + "value": "\"project_access_events\".\"action\" IN ('invited', 'invitation_resent', 'member_added', 'accepted', 'declined', 'invitation_revoked', 'role_changed', 'member_removed', 'left')" + } + }, + "isRLSEnabled": false + }, + "public.project_cleanup_jobs": { + "name": "project_cleanup_jobs", + "schema": "", + "columns": { + "target": { + "name": "target", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "attempts": { + "name": "attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "last_error": { + "name": "last_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "next_attempt_at": { + "name": "next_attempt_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "project_cleanup_jobs_due": { + "name": "project_cleanup_jobs_due", + "columns": [ + { + "expression": "next_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.project_database_credentials": { + "name": "project_database_credentials", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "ciphertext": { + "name": "ciphertext", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "iv": { + "name": "iv", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "key_version": { + "name": "key_version", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "project_database_credentials_project_id_projects_id_fk": { + "name": "project_database_credentials_project_id_projects_id_fk", + "tableFrom": "project_database_credentials", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.project_deploy_credentials": { + "name": "project_deploy_credentials", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issued_by": { + "name": "issued_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "revoked_at": { + "name": "revoked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "project_deploy_credentials_project": { + "name": "project_deploy_credentials_project", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "project_deploy_credentials_expiry": { + "name": "project_deploy_credentials_expiry", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_deploy_credentials_project_id_projects_id_fk": { + "name": "project_deploy_credentials_project_id_projects_id_fk", + "tableFrom": "project_deploy_credentials", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "project_deploy_credentials_issued_by_users_id_fk": { + "name": "project_deploy_credentials_issued_by_users_id_fk", + "tableFrom": "project_deploy_credentials", + "tableTo": "users", + "columnsFrom": ["issued_by"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_deploy_credentials_name": { + "name": "project_deploy_credentials_name", + "value": "length(\"project_deploy_credentials\".\"name\") BETWEEN 1 AND 80" + } + }, + "isRLSEnabled": false + }, + "public.project_invitations": { + "name": "project_invitations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target_user_id": { + "name": "target_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target_email_snapshot": { + "name": "target_email_snapshot", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "invited_by_user_id": { + "name": "invited_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "accepted_by_user_id": { + "name": "accepted_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true + }, + "accepted_at": { + "name": "accepted_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "revoked_at": { + "name": "revoked_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "revoked_by_user_id": { + "name": "revoked_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_project_invitations_project_status": { + "name": "idx_project_invitations_project_status", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_project_invitations_target_status": { + "name": "idx_project_invitations_target_status", + "columns": [ + { + "expression": "target_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_project_invitations_pending_target": { + "name": "idx_project_invitations_pending_target", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "target_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"project_invitations\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_invitations_project_id_projects_id_fk": { + "name": "project_invitations_project_id_projects_id_fk", + "tableFrom": "project_invitations", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "project_invitations_target_user_id_users_id_fk": { + "name": "project_invitations_target_user_id_users_id_fk", + "tableFrom": "project_invitations", + "tableTo": "users", + "columnsFrom": ["target_user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_invitations_role_check": { + "name": "project_invitations_role_check", + "value": "\"project_invitations\".\"role\" IN ('reader', 'collaborator', 'admin')" + }, + "project_invitations_status_check": { + "name": "project_invitations_status_check", + "value": "\"project_invitations\".\"status\" IN ('pending', 'accepted', 'declined', 'revoked', 'expired')" + } + }, + "isRLSEnabled": false + }, + "public.project_members": { + "name": "project_members", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "added_by_user_id": { + "name": "added_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_project_members_user_project": { + "name": "idx_project_members_user_project", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_members_project_id_projects_id_fk": { + "name": "project_members_project_id_projects_id_fk", + "tableFrom": "project_members", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "project_members_user_id_users_id_fk": { + "name": "project_members_user_id_users_id_fk", + "tableFrom": "project_members", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "project_members_project_id_user_id_pk": { + "name": "project_members_project_id_user_id_pk", + "columns": ["project_id", "user_id"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_members_role_check": { + "name": "project_members_role_check", + "value": "\"project_members\".\"role\" IN ('reader', 'collaborator', 'admin')" + } + }, + "isRLSEnabled": false + }, + "public.project_migrations": { + "name": "project_migrations", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "applied_at": { + "name": "applied_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "source_deployment_id": { + "name": "source_deployment_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_project_migrations_project_id": { + "name": "idx_project_migrations_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "applied_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_migrations_project_id_projects_id_fk": { + "name": "project_migrations_project_id_projects_id_fk", + "tableFrom": "project_migrations", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "project_migrations_source_deployment_id_deployments_id_fk": { + "name": "project_migrations_source_deployment_id_deployments_id_fk", + "tableFrom": "project_migrations", + "tableTo": "deployments", + "columnsFrom": ["source_deployment_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "project_migrations_project_id_name_pk": { + "name": "project_migrations_project_id_name_pk", + "columns": ["project_id", "name"] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.project_ownership_periods": { + "name": "project_ownership_periods", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "owner_user_id": { + "name": "owner_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "starts_at": { + "name": "starts_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true + }, + "ends_at": { + "name": "ends_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "changed_by_user_id": { + "name": "changed_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_project_ownership_periods_owner_time": { + "name": "idx_project_ownership_periods_owner_time", + "columns": [ + { + "expression": "owner_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "starts_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "ends_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_project_ownership_periods_project_time": { + "name": "idx_project_ownership_periods_project_time", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "starts_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_project_ownership_periods_open": { + "name": "idx_project_ownership_periods_open", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"project_ownership_periods\".\"ends_at\" IS NULL", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_ownership_periods_project_id_projects_id_fk": { + "name": "project_ownership_periods_project_id_projects_id_fk", + "tableFrom": "project_ownership_periods", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_ownership_periods_time_check": { + "name": "project_ownership_periods_time_check", + "value": "\"project_ownership_periods\".\"ends_at\" IS NULL OR \"project_ownership_periods\".\"ends_at\" >= \"project_ownership_periods\".\"starts_at\"" + } + }, + "isRLSEnabled": false + }, + "public.project_secrets": { + "name": "project_secrets", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "ciphertext": { + "name": "ciphertext", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "iv": { + "name": "iv", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "key_version": { + "name": "key_version", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "managed_reason": { + "name": "managed_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_project_secrets_project_id": { + "name": "idx_project_secrets_project_id", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_secrets_project_id_projects_id_fk": { + "name": "project_secrets_project_id_projects_id_fk", + "tableFrom": "project_secrets", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "project_secrets_project_id_name_unique": { + "name": "project_secrets_project_id_name_unique", + "nullsNotDistinct": false, + "columns": ["project_id", "name"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.project_zero_trust": { + "name": "project_zero_trust", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "desired_protected": { + "name": "desired_protected", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "effective_protected": { + "name": "effective_protected", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'ready'" + }, + "bypass_application_id": { + "name": "bypass_application_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "bypass_application_name": { + "name": "bypass_application_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "protected_application_id": { + "name": "protected_application_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "protected_application_name": { + "name": "protected_application_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "protected_application_audience": { + "name": "protected_application_audience", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "routing_cursor": { + "name": "routing_cursor", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reconciled_operation": { + "name": "reconciled_operation", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "last_error": { + "name": "last_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_project_zero_trust_status": { + "name": "idx_project_zero_trust_status", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_zero_trust_project_id_projects_id_fk": { + "name": "project_zero_trust_project_id_projects_id_fk", + "tableFrom": "project_zero_trust", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_zero_trust_status_check": { + "name": "project_zero_trust_status_check", + "value": "\"project_zero_trust\".\"status\" IN ('ready', 'reconciling', 'error')" + }, + "project_zero_trust_ready_check": { + "name": "project_zero_trust_ready_check", + "value": "\"project_zero_trust\".\"status\" <> 'ready' OR \"project_zero_trust\".\"desired_protected\" = \"project_zero_trust\".\"effective_protected\"" + } + }, + "isRLSEnabled": false + }, + "public.projects": { + "name": "projects", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "mode": { + "name": "mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'void'" + }, + "active_deployment_id": { + "name": "active_deployment_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "websocket_version": { + "name": "websocket_version", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "proxy_token_gen": { + "name": "proxy_token_gen", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_projects_slug": { + "name": "idx_projects_slug", + "columns": [ + { + "expression": "slug", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_projects_user_id": { + "name": "idx_projects_user_id", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "projects_user_id_users_id_fk": { + "name": "projects_user_id_users_id_fk", + "tableFrom": "projects", + "tableTo": "users", + "columnsFrom": ["user_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "projects_user_id_slug_unique": { + "name": "projects_user_id_slug_unique", + "nullsNotDistinct": false, + "columns": ["user_id", "slug"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.resources": { + "name": "resources", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "cf_id": { + "name": "cf_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "binding_name": { + "name": "binding_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "resources_project_id_projects_id_fk": { + "name": "resources_project_id_projects_id_fk", + "tableFrom": "resources", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "resources_project_id_type_binding_name_unique": { + "name": "resources_project_id_type_binding_name_unique", + "nullsNotDistinct": false, + "columns": ["project_id", "type", "binding_name"] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.sandbox_drain_checkpoints": { + "name": "sandbox_drain_checkpoints", + "schema": "", + "columns": { + "singleton": { + "name": "singleton", + "type": "boolean", + "primaryKey": true, + "notNull": true + }, + "admission_protocol": { + "name": "admission_protocol", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "generation": { + "name": "generation", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "phase": { + "name": "phase", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "cursor": { + "name": "cursor", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "lease_owner": { + "name": "lease_owner", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "lease_expires_at": { + "name": "lease_expires_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + }, + "started_at": { + "name": "started_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.users": { + "name": "users", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'github'" + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "login": { + "name": "login", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "plan": { + "name": "plan", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'free'" + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'user'" + }, + "billing_cycle_start": { + "name": "billing_cycle_start", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "suspended_reason": { + "name": "suspended_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_users_provider_id": { + "name": "idx_users_provider_id", + "columns": [ + { + "expression": "provider", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "provider_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_users_email": { + "name": "idx_users_email", + "columns": [ + { + "expression": "email", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_cache_outbox": { + "name": "email_cache_outbox", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_kind": { + "name": "resource_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_id": { + "name": "resource_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "version": { + "name": "version", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "published_at": { + "name": "published_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "email_cache_outbox_version_uq": { + "name": "email_cache_outbox_version_uq", + "columns": [ + { + "expression": "resource_kind", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "resource_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "version", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_cache_outbox_pending_idx": { + "name": "email_cache_outbox_pending_idx", + "columns": [ + { + "expression": "published_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_credentials": { + "name": "email_credentials", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "managed_by": { + "name": "managed_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "manager_user_id": { + "name": "manager_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "ciphertext": { + "name": "ciphertext", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "iv": { + "name": "iv", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "key_version": { + "name": "key_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "token_id": { + "name": "token_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "expires_on": { + "name": "expires_on", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'healthy'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "email_credentials_scope_idx": { + "name": "email_credentials_scope_idx", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "zone_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "email_credentials_material_check": { + "name": "email_credentials_material_check", + "value": "(\"email_credentials\".\"kind\" = 'platform' AND \"email_credentials\".\"ciphertext\" IS NULL AND \"email_credentials\".\"iv\" IS NULL AND \"email_credentials\".\"key_version\" IS NULL) OR (\"email_credentials\".\"kind\" = 'api_token' AND \"email_credentials\".\"ciphertext\" IS NOT NULL AND \"email_credentials\".\"iv\" IS NOT NULL AND \"email_credentials\".\"key_version\" IS NOT NULL)" + } + }, + "isRLSEnabled": false + }, + "public.email_domain_routes": { + "name": "email_domain_routes", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "addressing": { + "name": "addressing", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "managed_by": { + "name": "managed_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "assigned_by_user_id": { + "name": "assigned_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "desired_state": { + "name": "desired_state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "inbound_status": { + "name": "inbound_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "inbound_reason": { + "name": "inbound_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "inbound_observed_at": { + "name": "inbound_observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "outbound_status": { + "name": "outbound_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "outbound_reason": { + "name": "outbound_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "outbound_observed_at": { + "name": "outbound_observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "desired_generation": { + "name": "desired_generation", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "email_domain_routes_domain_uq": { + "name": "email_domain_routes_domain_uq", + "columns": [ + { + "expression": "domain", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_domain_routes_connection_idx": { + "name": "email_domain_routes_connection_idx", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_domain_routes_project_idx": { + "name": "email_domain_routes_project_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_domain_routes_connection_id_email_zone_connections_id_fk": { + "name": "email_domain_routes_connection_id_email_zone_connections_id_fk", + "tableFrom": "email_domain_routes", + "tableTo": "email_zone_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "email_domain_routes_assignment_check": { + "name": "email_domain_routes_assignment_check", + "value": "(\"email_domain_routes\".\"addressing\" = 'custom' AND \"email_domain_routes\".\"project_id\" IS NOT NULL) OR (\"email_domain_routes\".\"addressing\" = 'shared' AND \"email_domain_routes\".\"project_id\" IS NULL)" + } + }, + "isRLSEnabled": false + }, + "public.email_operation_steps": { + "name": "email_operation_steps", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "operation_id": { + "name": "operation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "ordinal": { + "name": "ordinal", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_identity": { + "name": "resource_identity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "request_digest": { + "name": "request_digest", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "attempt": { + "name": "attempt", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "outcome": { + "name": "outcome", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "failure_code": { + "name": "failure_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "failure_detail": { + "name": "failure_detail", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "finished_at": { + "name": "finished_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "email_operation_steps_order_uq": { + "name": "email_operation_steps_order_uq", + "columns": [ + { + "expression": "operation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "ordinal", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_operation_steps_operation_id_email_operations_id_fk": { + "name": "email_operation_steps_operation_id_email_operations_id_fk", + "tableFrom": "email_operation_steps", + "tableTo": "email_operations", + "columnsFrom": ["operation_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_operations": { + "name": "email_operations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "route_id": { + "name": "route_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "route_generation": { + "name": "route_generation", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "actor_kind": { + "name": "actor_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "actor_user_id": { + "name": "actor_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "actor_scope": { + "name": "actor_scope", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "idempotency_key": { + "name": "idempotency_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "request_digest": { + "name": "request_digest", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "desired_generation": { + "name": "desired_generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "current_step": { + "name": "current_step", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "failure_code": { + "name": "failure_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "failure_detail": { + "name": "failure_detail", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "email_operations_key_uq": { + "name": "email_operations_key_uq", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "actor_scope", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "idempotency_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_operations_work_idx": { + "name": "email_operations_work_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_operations_connection_idx": { + "name": "email_operations_connection_idx", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_operations_connection_id_email_zone_connections_id_fk": { + "name": "email_operations_connection_id_email_zone_connections_id_fk", + "tableFrom": "email_operations", + "tableTo": "email_zone_connections", + "columnsFrom": ["connection_id"], + "columnsTo": ["id"], + "onDelete": "no action", + "onUpdate": "no action" + }, + "email_operations_route_id_email_domain_routes_id_fk": { + "name": "email_operations_route_id_email_domain_routes_id_fk", + "tableFrom": "email_operations", + "tableTo": "email_domain_routes", + "columnsFrom": ["route_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_shared_recipients": { + "name": "email_shared_recipients", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "route_id": { + "name": "route_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "recipient": { + "name": "recipient", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "rule_name": { + "name": "rule_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "rule_id": { + "name": "rule_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "create_started_at": { + "name": "create_started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "removing": { + "name": "removing", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "reason": { + "name": "reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "attempt": { + "name": "attempt", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "lease_started_at": { + "name": "lease_started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "observed_at": { + "name": "observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "email_shared_recipients_project_address_uq": { + "name": "email_shared_recipients_project_address_uq", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "recipient", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_shared_recipients_rule_name_uq": { + "name": "email_shared_recipients_rule_name_uq", + "columns": [ + { + "expression": "rule_name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_shared_recipients_reconcile_idx": { + "name": "email_shared_recipients_reconcile_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_zone_connections": { + "name": "email_zone_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "zone_id": { + "name": "zone_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "zone_name": { + "name": "zone_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "managed_by": { + "name": "managed_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "manager_user_id": { + "name": "manager_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "credential_id": { + "name": "credential_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "ingress_worker_name": { + "name": "ingress_worker_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "ingress_version": { + "name": "ingress_version", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "desired_generation": { + "name": "desired_generation", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "active_secret_generation": { + "name": "active_secret_generation", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "staged_secret_generation": { + "name": "staged_secret_generation", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "secret_ciphertext": { + "name": "secret_ciphertext", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "secret_iv": { + "name": "secret_iv", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "secret_key_version": { + "name": "secret_key_version", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "staged_secret_ciphertext": { + "name": "staged_secret_ciphertext", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "staged_secret_iv": { + "name": "staged_secret_iv", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "staged_secret_key_version": { + "name": "staged_secret_key_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "previous_secret_hash": { + "name": "previous_secret_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "previous_secret_until": { + "name": "previous_secret_until", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "inbound_status": { + "name": "inbound_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "inbound_reason": { + "name": "inbound_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "inbound_observed_at": { + "name": "inbound_observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "management_status": { + "name": "management_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'probe_failed'" + }, + "management_reason": { + "name": "management_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "management_observed_at": { + "name": "management_observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "email_zone_connections_zone_uq": { + "name": "email_zone_connections_zone_uq", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "zone_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_zone_connections_ingress_uq": { + "name": "email_zone_connections_ingress_uq", + "columns": [ + { + "expression": "account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "ingress_worker_name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "email_zone_connections_manager_idx": { + "name": "email_zone_connections_manager_idx", + "columns": [ + { + "expression": "manager_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "email_zone_connections_manager_check": { + "name": "email_zone_connections_manager_check", + "value": "(\"email_zone_connections\".\"managed_by\" = 'owner' AND \"email_zone_connections\".\"manager_user_id\" IS NOT NULL) OR (\"email_zone_connections\".\"managed_by\" = 'admin')" + } + }, + "isRLSEnabled": false + }, + "public.email_action_recipients": { + "name": "email_action_recipients", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "action_id": { + "name": "action_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "recipient": { + "name": "recipient", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "error_code": { + "name": "error_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "provider_reference": { + "name": "provider_reference", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reserved_at": { + "name": "reserved_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "attempt_started_at": { + "name": "attempt_started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_email_action_recipients_project_reserved": { + "name": "idx_email_action_recipients_project_reserved", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "reserved_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_email_action_recipients_expiring": { + "name": "idx_email_action_recipients_expiring", + "columns": [ + { + "expression": "reserved_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"email_action_recipients\".\"state\" = 'reserved'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_action_recipients_action_id_email_actions_id_fk": { + "name": "email_action_recipients_action_id_email_actions_id_fk", + "tableFrom": "email_action_recipients", + "tableTo": "email_actions", + "columnsFrom": ["action_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "email_action_recipients_action_recipient_uq": { + "name": "email_action_recipients_action_recipient_uq", + "nullsNotDistinct": false, + "columns": ["action_id", "recipient"] + } + }, + "policies": {}, + "checkConstraints": { + "email_action_recipients_state_check": { + "name": "email_action_recipients_state_check", + "value": "\"email_action_recipients\".\"state\" IN ('rejected', 'reserved', 'attempt_started', 'provider_accepted', 'failed', 'outcome_unknown', 'cancelled')" + } + }, + "isRLSEnabled": false + }, + "public.email_actions": { + "name": "email_actions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "sender_domain": { + "name": "sender_domain", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "receipt_id": { + "name": "receipt_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "idempotency_key": { + "name": "idempotency_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "payload_digest": { + "name": "payload_digest", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "transport": { + "name": "transport", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "policy_version": { + "name": "policy_version", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_email_actions_project_created": { + "name": "idx_email_actions_project_created", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "email_actions_receipt_id_email_receipts_id_fk": { + "name": "email_actions_receipt_id_email_receipts_id_fk", + "tableFrom": "email_actions", + "tableTo": "email_receipts", + "columnsFrom": ["receipt_id"], + "columnsTo": ["id"], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "email_actions_project_key_uq": { + "name": "email_actions_project_key_uq", + "nullsNotDistinct": false, + "columns": ["project_id", "idempotency_key"] + } + }, + "policies": {}, + "checkConstraints": { + "email_actions_kind_check": { + "name": "email_actions_kind_check", + "value": "\"email_actions\".\"kind\" IN ('direct', 'forward', 'reply')" + } + }, + "isRLSEnabled": false + }, + "public.email_monthly_usage": { + "name": "email_monthly_usage", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "month": { + "name": "month", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "recipient_attempts": { + "name": "recipient_attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "inbound_receipts": { + "name": "inbound_receipts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "email_monthly_usage_project_month_uq": { + "name": "email_monthly_usage_project_month_uq", + "nullsNotDistinct": false, + "columns": ["project_id", "month"] + } + }, + "policies": {}, + "checkConstraints": { + "email_monthly_usage_nonnegative_check": { + "name": "email_monthly_usage_nonnegative_check", + "value": "\"email_monthly_usage\".\"recipient_attempts\" >= 0" + }, + "email_monthly_usage_inbound_nonnegative_check": { + "name": "email_monthly_usage_inbound_nonnegative_check", + "value": "\"email_monthly_usage\".\"inbound_receipts\" >= 0" + } + }, + "isRLSEnabled": false + }, + "public.email_project_controls": { + "name": "email_project_controls", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "monthly_limit": { + "name": "monthly_limit", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 200 + }, + "burst_limit": { + "name": "burst_limit", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 10 + }, + "suspended_at": { + "name": "suspended_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "version": { + "name": "version", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + } + }, + "indexes": {}, + "foreignKeys": { + "email_project_controls_project_id_projects_id_fk": { + "name": "email_project_controls_project_id_projects_id_fk", + "tableFrom": "email_project_controls", + "tableTo": "projects", + "columnsFrom": ["project_id"], + "columnsTo": ["id"], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_receipts": { + "name": "email_receipts", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'received'" + }, + "payload_digest": { + "name": "payload_digest", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "final_reject": { + "name": "final_reject", + "type": "boolean", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_email_receipts_project_created": { + "name": "idx_email_receipts_project_created", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.email_unresolved_attempts": { + "name": "email_unresolved_attempts", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "sender_domain": { + "name": "sender_domain", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "idx_email_unresolved_attempts_project": { + "name": "idx_email_unresolved_attempts_project", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_email_unresolved_attempts_domain": { + "name": "idx_email_unresolved_attempts_domain", + "columns": [ + { + "expression": "sender_domain", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + } + }, + "enums": {}, + "schemas": {}, + "sequences": {}, + "roles": {}, + "policies": {}, + "views": {}, + "_meta": { + "columns": {}, + "schemas": {}, + "tables": {} + } +} diff --git a/platform/packages/api/drizzle/meta/_journal.json b/platform/packages/api/drizzle/meta/_journal.json index d6d34adb0..4e5988aee 100644 --- a/platform/packages/api/drizzle/meta/_journal.json +++ b/platform/packages/api/drizzle/meta/_journal.json @@ -358,6 +358,13 @@ "when": 1790912739624, "tag": "0057_github_authority_constraints", "breakpoints": true + }, + { + "idx": 58, + "version": "7", + "when": 1791311435335, + "tag": "0058_project_proxy_token_gen", + "breakpoints": true } ] } diff --git a/platform/packages/api/src/cron-fires-queue.ts b/platform/packages/api/src/cron-fires-queue.ts index 00a91b460..a1f0818d9 100644 --- a/platform/packages/api/src/cron-fires-queue.ts +++ b/platform/packages/api/src/cron-fires-queue.ts @@ -117,7 +117,10 @@ export async function processCronFiresBatch( // user worker's baked __VOID_PROXY_TOKEN. Retrying would yield // the same 401 (same token in, same status out), so we ack; the // next deploy's /sync resyncs the token. Logged with a stable - // greppable marker so ops can alert on this. + // greppable marker so ops can alert on this. Right after a token + // rotation patches the Worker, its old version can still answer + // for a moment; that fire is dropped too, never replayed with + // another token, because the Worker's own middleware already ran. console.error( `[cron-fires] AUTH_DRIFT non-delivery for ${msg.body.workerName} (cron ${msg.body.cron}): status=${status} body=${outcome.bodySnippet ?? ''}; acking — next deploy will resync token`, ); diff --git a/platform/packages/api/src/db/admin.ts b/platform/packages/api/src/db/admin.ts index 26395125b..b1f2ed2f5 100644 --- a/platform/packages/api/src/db/admin.ts +++ b/platform/packages/api/src/db/admin.ts @@ -16,6 +16,10 @@ import { import { alias } from 'drizzle-orm/pg-core'; import { lockAuthSettings } from '../auth/configuration.ts'; import { generateId } from '../id.ts'; +import { + requestProxyTokenRotation, + requestProxyTokenRotationsForMember, +} from '../proxy-token-rotation-request.ts'; import { users, projects, @@ -175,6 +179,11 @@ async function withAdminRemovalGuard( }); } +/** An admin suspension ends deploy access, like account deletion. */ +async function requestRotationsForAdminSuspension(tx: Db, id: string): Promise { + await requestProxyTokenRotationsForMember(tx, id, 'user-suspended'); +} + /** * Revoke administrator access before external deletion/suspension work begins. * This durable change prevents another removal from counting this administrator @@ -192,6 +201,11 @@ export async function prepareUserRemoval( .update(users) .set(action.kind === 'delete' ? { role: 'user' } : { suspended_reason: action.reason }) .where(eq(users.id, id)); + // The suspension is in force from here, even when the rest of the + // removal fails. + if (action.kind === 'suspend' && isAdminSuspended(action.reason)) { + await requestRotationsForAdminSuspension(tx, id); + } } return user; }); @@ -202,7 +216,10 @@ export async function suspendUser(db: Db, id: string, reason: string): Promise { + await suspend(tx); + await requestRotationsForAdminSuspension(tx, id); + }); } else { // Usage limits suspend app traffic, not administrator access. A delayed // quota write must not undo an administrator's manual access revocation. @@ -290,6 +307,21 @@ export async function deleteUser(db: Db, id: string): Promise): Promis projectId, resolvePersistedDeploymentType(cutoverManifest, protectedDeployment), ); + await requestRotationIfReplacingUntrustedCode(tx, projectId, protectedDeployment.id); await setActiveDeployment(tx, projectId, protectedDeployment.id); await markDeploymentRetained(tx, protectedDeployment.id, true); recoveredCutover = { ...protectedDeployment, status: 'live', retained: true }; diff --git a/platform/packages/api/src/deploy/recovery.ts b/platform/packages/api/src/deploy/recovery.ts index 746fac866..64bb3e94f 100644 --- a/platform/packages/api/src/deploy/recovery.ts +++ b/platform/packages/api/src/deploy/recovery.ts @@ -29,6 +29,7 @@ import { removeQueueConsumer, getDispatchWorkerProxyToken, } from '../provisioning/cloudflare-api.ts'; +import { requestRotationIfReplacingUntrustedCode } from '../proxy-token-rotation.ts'; import { buildRoutingJson, buildStaticRoutingJson, computeDispatchLimits } from '../routing.ts'; import { type DeployManifest, type Deployment, type Project } from '../types.ts'; import { @@ -102,6 +103,7 @@ export async function commitCutoverLive( // began, so no final object-store scan is needed. await updateDeploymentStatus(tx, deploymentId, 'live', undefined, { force: true }); await updateProjectType(tx, projectId, input.projectType); + await requestRotationIfReplacingUntrustedCode(tx, projectId, deploymentId); await setActiveDeployment(tx, projectId, deploymentId); await markDeploymentRetained(tx, deploymentId, true); }); @@ -329,6 +331,7 @@ export async function replayDeploymentCutover( desiredQueueResources.set(queue.name, resource); } const needsProxyToken = !isStatic && (!!manifest.schedules?.length || desiredQueues.length > 0); + // The token the replayed Worker holds, which may be older than the issued one. const proxyToken = needsProxyToken ? await resolveCutoverProxyToken(env, workerName, project.id) : null; diff --git a/platform/packages/api/src/deploy/upload-worker.ts b/platform/packages/api/src/deploy/upload-worker.ts index 084d66e91..3a65f47c3 100644 --- a/platform/packages/api/src/deploy/upload-worker.ts +++ b/platform/packages/api/src/deploy/upload-worker.ts @@ -24,7 +24,7 @@ import { provisionSandboxController, sandboxControllerWorkerName, } from '../provisioning/sandbox-controller.ts'; -import { issueProxyToken } from '../proxy-token.ts'; +import { issueProjectProxyToken } from '../proxy-token-rotation.ts'; import { RESERVED_BINDING_NAMES } from '../reserved-bindings.ts'; import { deployments } from '../schema.ts'; import { type DeployManifest, type DeployEvent, type Deployment, type Project } from '../types.ts'; @@ -108,6 +108,18 @@ export async function uploadDeploymentWorker(input: { emit({ event: 'provisioning', bindings: bindingLabels }); } + // Recorded before the upload: this deployment's Worker does not exist until + // the upload creates it holding this token, and only a rotation changes it. + const proxyToken = await issueProjectProxyToken( + context.get('db'), + context.env.JWT_SECRET, + projectId, + ); + await context + .get('db') + .update(deployments) + .set({ proxy_token_gen: proxyToken.gen }) + .where(eq(deployments.id, deployment.id)); const workerBindings = await provisionBindings( context.get('db'), context.env.CF_ACCOUNT_ID, @@ -117,7 +129,7 @@ export async function uploadDeploymentWorker(input: { context.env.VOID_PROXY_SERVICE ?? 'void-proxy', { projectSlug: project.slug, - proxyToken: await issueProxyToken(context.env.ROUTING, context.env.JWT_SECRET, projectId), + proxyToken: proxyToken.token, emailSenderDomain: context.env.EMAIL_SENDER_DOMAIN, deploymentId: deployment.id, }, diff --git a/platform/packages/api/src/index.ts b/platform/packages/api/src/index.ts index 4c07f3dc5..cbc5f54c3 100644 --- a/platform/packages/api/src/index.ts +++ b/platform/packages/api/src/index.ts @@ -63,7 +63,7 @@ import { isValidEmailDomain, MAX_DOMAIN_LENGTH } from './email/domain-name.ts'; import emailIngressResolution from './email/ingress-resolution.ts'; import { repairEmailRecipientPolicyMirror } from './email/settings.ts'; import { reconcileSharedEmailRecipients } from './email/shared-recipient-service.ts'; -import type { ApiEnv } from './env.ts'; +import { isDevBypass, type ApiEnv } from './env.ts'; import { ErrorAlertMonitor } from './error-alert-monitor.ts'; import { cancelContainerBuildDos } from './github/cancel-builds.ts'; import { authMiddleware } from './middleware/auth.ts'; @@ -94,6 +94,7 @@ import { } from './provisioning/cloudflare-api.ts'; import { reconcileEmailDestinationCleanups } from './provisioning/email-destination-cleanup.ts'; import { retrySandboxControllerCleanup } from './provisioning/sandbox-controller.ts'; +import { reconcileProxyTokenRotations } from './proxy-token-rotation.ts'; import { oidcRoutes } from './routes/auth-oidc.ts'; import auth from './routes/auth.ts'; import deleteRoute from './routes/delete.ts'; @@ -668,6 +669,14 @@ async function scheduled( } } + // Proxy token rotations, bounded per tick so deploys keep their share of the + // Cloudflare API rate limit. + if (!isDevBypass(env) && env.CF_API_TOKEN) { + await reconcileProxyTokenRotations(db, env, { maxCfCalls: 100, maxMs: 25_000 }).catch((error) => + console.error('[scheduled] proxy token rotation failed:', error), + ); + } + if (capabilities.email) { await reconcileEmailDestinationCleanups(db, { accountId: env.CF_ACCOUNT_ID, diff --git a/platform/packages/api/src/internal-proxy-auth.ts b/platform/packages/api/src/internal-proxy-auth.ts index 6b14b1acb..b272d2859 100644 --- a/platform/packages/api/src/internal-proxy-auth.ts +++ b/platform/packages/api/src/internal-proxy-auth.ts @@ -44,9 +44,7 @@ export async function checkTenantProxyToken( /** * Authenticates the proxy on an internal metadata route. The proxy signs * `x-void-proxy-internal` with its own key, bound to the route and project. - * A request without that header is checked like a tenant project token in - * `x-void-internal`, so a token below the project's accept floor is refused - * here as it is at the proxy. + * Tenant project tokens are not accepted here. * * Returns null when the call may proceed, else the error response. */ @@ -56,23 +54,13 @@ export async function requireProxyInternalAuth( projectId: string, ): Promise { const proxyAuth = context.req.header(PROXY_INTERNAL_AUTH_HEADER); - const token = context.req.header('x-void-internal'); - if (!proxyAuth && !token) { + if (!proxyAuth) { return context.json({ error: 'missing internal auth' }, 401); } - if (!isProjectId(projectId)) { - return context.json({ error: 'invalid internal auth' }, 403); - } - if (proxyAuth) { - return (await verifyProxyInternalAuth(context.env.JWT_SECRET, route, projectId, proxyAuth)) - ? null - : context.json({ error: 'invalid internal auth' }, 403); - } - const tokenCheck = await checkTenantProxyToken(context.env, projectId, token ?? ''); - if (tokenCheck === 'unavailable') { - return context.json({ error: 'project token state unavailable' }, 503); - } - if (tokenCheck === 'invalid') { + if ( + !isProjectId(projectId) || + !(await verifyProxyInternalAuth(context.env.JWT_SECRET, route, projectId, proxyAuth)) + ) { return context.json({ error: 'invalid internal auth' }, 403); } return null; diff --git a/platform/packages/api/src/operator/audit.ts b/platform/packages/api/src/operator/audit.ts index 877f00398..cd6590802 100644 --- a/platform/packages/api/src/operator/audit.ts +++ b/platform/packages/api/src/operator/audit.ts @@ -3,6 +3,7 @@ import { count, desc, eq } from 'drizzle-orm'; import { Hono, type Context, type MiddlewareHandler } from 'hono'; import { AuthenticationConfigurationError } from '../auth/configuration.ts'; import type { ApiEnv } from '../env.ts'; +import { isProjectId } from '../id.ts'; import { operatorEvents } from '../schema.ts'; import { dryRun, OperatorError, pagination, readBody } from './common.ts'; @@ -272,6 +273,9 @@ function auditTarget( if (segments[0] === 'system' && segments[1] === 'sandbox-drain') { return JSON.stringify({ applications: 'managed-sandbox-containers' }); } + if (segments[0] === 'system' && segments[1] === 'rotate-proxy-tokens') { + return JSON.stringify({ projects: isProjectId(body.project) ? body.project : 'all' }); + } if (segments[0] === 'workers') { const environment = boundedString(body.environment ?? platformEnvironment, 512) ?? 'production'; if (segments[1] === 'rollback-all') { diff --git a/platform/packages/api/src/operator/operations-routes.ts b/platform/packages/api/src/operator/operations-routes.ts index 1363eb0cd..410799ec7 100644 --- a/platform/packages/api/src/operator/operations-routes.ts +++ b/platform/packages/api/src/operator/operations-routes.ts @@ -9,6 +9,7 @@ import { listOperatorDeployments, requestDeploymentCancellation, } from './deployment-operations.ts'; +import { rotateProxyTokens } from './proxy-token-operations.ts'; import { drainManagedSandboxes } from './sandbox-operations.ts'; import { backfillPlatformQueueTokens, @@ -124,6 +125,16 @@ operationsRoutes.post('/system/sandbox-drain', async (context) => { const result = await drainManagedSandboxes(context.get('db'), context.env, preview, cursor); return context.json(result, !preview && !result.complete ? 207 : 200); }); +operationsRoutes.post('/system/rotate-proxy-tokens', async (context) => { + const body = await readBody(context); + const preview = dryRun(body); + const result = await rotateProxyTokens(context.get('db'), context.env, { + dryRun: preview, + project: body.project, + all: body.all, + }); + return context.json(result, !preview && 'complete' in result && !result.complete ? 207 : 200); +}); operationsRoutes.get('/workers', async (context) => context.json( diff --git a/platform/packages/api/src/operator/proxy-token-operations.ts b/platform/packages/api/src/operator/proxy-token-operations.ts new file mode 100644 index 000000000..bb56ca644 --- /dev/null +++ b/platform/packages/api/src/operator/proxy-token-operations.ts @@ -0,0 +1,219 @@ +import { and, asc, eq, gt, sql, type SQL } from 'drizzle-orm'; +import type { Db } from '../db.ts'; +import { type ApiEnv, isDevBypass } from '../env.ts'; +import { isProjectId } from '../id.ts'; +import { + advanceProxyTokenRotation, + listProxyTokenInventory, + PROXY_TOKEN_ROTATION_JOB_KIND, + proxyTokenRotationTarget, + reconcileProxyTokenRotations, + requestBulkProxyTokenRotations, + requestProxyTokenRotation, +} from '../proxy-token-rotation.ts'; +import { projectCleanupJobs, projects } from '../schema.ts'; +import { OperatorError, preview } from './common.ts'; + +// Inline work per request stays well under the CLI's 5 minute request timeout; +// the scheduled run finishes whatever is left. +const APPLY_MAX_CF_CALLS = 150; +const APPLY_MAX_MS = 60_000; +const LISTED_JOBS = 50; + +type RotateInput = Readonly<{ dryRun: boolean; project?: unknown; all?: unknown }>; + +const rotationJobs = (projectId: string | undefined): SQL | undefined => + and( + eq(projectCleanupJobs.kind, PROXY_TOKEN_ROTATION_JOB_KIND), + projectId ? eq(projectCleanupJobs.target, proxyTokenRotationTarget(projectId)) : undefined, + ); + +/** Rotation jobs in scope, oldest first. A job that has failed since its last good step counts as failed. */ +async function listJobs(db: Db, projectId: string | undefined, failedOnly = false) { + const rows = await db + .select({ + projectId: projectCleanupJobs.project_id, + slug: projects.slug, + payload: projectCleanupJobs.payload, + attempts: projectCleanupJobs.attempts, + lastError: projectCleanupJobs.last_error, + nextAttemptAt: projectCleanupJobs.next_attempt_at, + }) + .from(projectCleanupJobs) + .leftJoin(projects, eq(projects.id, projectCleanupJobs.project_id)) + .where( + and(rotationJobs(projectId), failedOnly ? gt(projectCleanupJobs.attempts, 0) : undefined), + ) + .orderBy(asc(projectCleanupJobs.created_at), asc(projectCleanupJobs.target)) + .limit(LISTED_JOBS); + return rows.map((row) => ({ + projectId: row.projectId, + slug: row.slug, + phase: row.payload.phase ?? null, + reason: row.payload.reason ?? null, + again: row.payload.again === '1', + attempts: row.attempts, + lastError: row.lastError, + nextAttemptAt: row.nextAttemptAt, + waitUntil: row.payload.waitUntil ?? null, + })); +} + +async function jobCounts(db: Db, projectId: string | undefined) { + const [row] = await db + .select({ + pending: sql`count(*) FILTER (WHERE ${projectCleanupJobs.attempts} = 0)::integer`, + failed: sql`count(*) FILTER (WHERE ${projectCleanupJobs.attempts} > 0)::integer`, + retryAfterSeconds: sql< + number | null + >`min(greatest(0, ceil(extract(epoch FROM ${projectCleanupJobs.next_attempt_at}::timestamptz - now()))))::integer`, + }) + .from(projectCleanupJobs) + .where(rotationJobs(projectId)); + return { + pending: row?.pending ?? 0, + failed: row?.failed ?? 0, + retryAfterSeconds: row?.retryAfterSeconds ?? null, + }; +} + +/** Projects --all would queue or flag: every project without a job or with one past START. */ +async function countWouldQueue(db: Db): Promise { + const job = sql`${projectCleanupJobs.target} = ${PROXY_TOKEN_ROTATION_JOB_KIND}::text || ':' || ${projects.id}`; + const [row] = await db + .select({ + wouldQueue: sql`count(*) FILTER (WHERE ${projectCleanupJobs.target} IS NULL + OR ${projectCleanupJobs.payload}->>'phase' <> 'requested')::integer`, + }) + .from(projects) + .leftJoin(projectCleanupJobs, job) + .where(sql`${projects.status} <> 'deleting'`); + return row?.wouldQueue ?? 0; +} + +/** Progress of every rotation job; with `all`, also what an --all apply would queue. */ +async function bulkPreview(db: Db, all: boolean) { + const counts = await jobCounts(db, undefined); + return preview( + 'system.rotate-proxy-tokens', + { projects: 'all' }, + { + summary: { + ...(all ? { wouldQueue: await countWouldQueue(db) } : {}), + pending: counts.pending, + failed: counts.failed, + }, + projects: await listJobs(db, undefined), + }, + ); +} + +/** + * `void platform system rotate-proxy-tokens`. An apply needs `project` or + * `all` (every project); a preview without either shows the progress of + * every rotation. A preview reads the database only and never returns a + * token. An apply requests the rotations and advances them within a bounded + * budget; the scheduled run finishes the rest. + */ +export async function rotateProxyTokens(db: Db, env: ApiEnv['Bindings'], input: RotateInput) { + if (input.project !== undefined && !isProjectId(input.project)) { + throw new OperatorError(400, 'project must be a project ID.', 'invalid_argument'); + } + if (input.all !== undefined && typeof input.all !== 'boolean') { + throw new OperatorError(400, 'all must be a boolean.', 'invalid_argument'); + } + if (input.project !== undefined && input.all === true) { + throw new OperatorError(400, 'Choose either project or all.', 'invalid_argument'); + } + if (!input.dryRun && input.project === undefined && input.all !== true) { + throw new OperatorError(400, 'Supply --project or --all.', 'invalid_argument'); + } + const all = input.all === true; + const projectId = input.project as string | undefined; + const [project] = projectId + ? await db + .select({ + id: projects.id, + slug: projects.slug, + active_deployment_id: projects.active_deployment_id, + proxy_token_gen: projects.proxy_token_gen, + }) + .from(projects) + .where(eq(projects.id, projectId)) + : []; + if (projectId && !project) { + throw new OperatorError(404, 'Project not found.', 'not_found'); + } + + if (input.dryRun) { + if (!project) { + return bulkPreview(db, all); + } + const [job] = await listJobs(db, project.id); + return preview( + 'system.rotate-proxy-tokens', + { projects: project.id }, + { + projectId: project.id, + slug: project.slug, + gen: project.proxy_token_gen, + job: job + ? { + phase: job.phase, + reason: job.reason, + again: job.again, + attempts: job.attempts, + lastError: job.lastError, + nextAttemptAt: job.nextAttemptAt, + waitUntil: job.waitUntil, + } + : null, + workers: (await listProxyTokenInventory(db, project)).map((row) => row.workerName), + }, + ); + } + + if (isDevBypass(env) || !env.CF_API_TOKEN) { + throw new OperatorError( + 409, + 'Proxy token rotation needs the platform Cloudflare API token.', + 'rotation_unavailable', + ); + } + + const budget = { cfCallsRemaining: APPLY_MAX_CF_CALLS, deadline: Date.now() + APPLY_MAX_MS }; + let queued: number; + if (project) { + await requestProxyTokenRotation(db, project.id, 'operator', 'operator'); + await advanceProxyTokenRotation(db, env, project.id, budget); + queued = 1; + } else { + queued = await requestBulkProxyTokenRotations(db); + await reconcileProxyTokenRotations(db, env, { + maxCfCalls: APPLY_MAX_CF_CALLS, + maxMs: APPLY_MAX_MS, + }); + } + const counts = await jobCounts(db, project?.id); + const complete = counts.pending + counts.failed === 0; + const failures = (await listJobs(db, project?.id, true)).map((job) => ({ + projectId: job.projectId, + slug: job.slug, + phase: job.phase, + error: job.lastError, + })); + return { + ok: counts.failed === 0, + complete, + queued, + pending: counts.pending, + failed: counts.failed, + failures, + retryAfterSeconds: complete ? null : counts.retryAfterSeconds, + message: complete + ? 'Every requested rotation is finished.' + : counts.failed > 0 + ? 'Some rotations keep failing and are retried in the background. Check progress with --plan.' + : 'Rotations continue in the background. Check progress with --plan.', + }; +} diff --git a/platform/packages/api/src/project-access.ts b/platform/packages/api/src/project-access.ts index e7b2d00cb..5bad26b0b 100644 --- a/platform/packages/api/src/project-access.ts +++ b/platform/packages/api/src/project-access.ts @@ -43,10 +43,20 @@ export const PROJECT_PERMISSION_MESSAGES = { string >; -function hasProjectCapability(role: ProjectRole, capability: ProjectCapability): boolean { +export function hasProjectCapability(role: ProjectRole, capability: ProjectCapability): boolean { return (PROJECT_ROLE_CAPABILITIES[role] as ReadonlyArray).includes(capability); } +/** Whether a member change from `previous` to `next` (null = removed) takes away deploy. */ +export function roleLosesDeployCapability( + previous: ProjectRole, + next: ProjectRole | null, +): boolean { + return ( + hasProjectCapability(previous, 'deploy') && !(next && hasProjectCapability(next, 'deploy')) + ); +} + const projectOwner = alias(users, 'project_owner'); async function selectProjectAccess( @@ -79,6 +89,15 @@ async function selectProjectAccess( }; } +/** The user's current role on the project, null without one. Owner suspension is not checked. */ +export async function projectRoleForUser( + db: Db, + projectId: string, + userId: string, +): Promise { + return (await selectProjectAccess(db, projectId, userId))?.memberRole ?? null; +} + export class ProjectNotFoundError extends Error { readonly status = 404 as const; readonly userMessage = PROJECT_PERMISSION_MESSAGES['not-found']; diff --git a/platform/packages/api/src/provisioning/cleanup.ts b/platform/packages/api/src/provisioning/cleanup.ts index 09efa809f..d63b2b973 100644 --- a/platform/packages/api/src/provisioning/cleanup.ts +++ b/platform/packages/api/src/provisioning/cleanup.ts @@ -41,6 +41,7 @@ import { cancelContainerBuildDos } from '../github/cancel-builds.ts'; import { generateId } from '../id.ts'; import { getLegacyProjectWorkerName, getProjectDeploymentWorkerName } from '../naming.ts'; import { requireProjectCapability } from '../project-access.ts'; +import { CLOSED_PROXY_TOKEN_STATE, proxyTokenStateKey } from '../proxy-token.ts'; import { deployments as deploymentsTable, projectCleanupJobs, @@ -130,6 +131,7 @@ const RETRYABLE_CLEANUP_JOB_KINDS = [ 'worker-bundle', 'project-assets', 'routing-key', + 'routing-put', 'cron', 'custom-hostname', ] as const satisfies ReadonlyArray; @@ -173,6 +175,9 @@ async function executeCleanupJob( case 'routing-key': await env.ROUTING.delete(jobValue(job, 'key')); return; + case 'routing-put': + await env.ROUTING.put(jobValue(job, 'key'), jobValue(job, 'value')); + return; case 'cron': await clearCronScheduler(env, jobValue(job, 'projectId')); return; @@ -457,6 +462,14 @@ async function cleanupBestEffortResources( cleanup.push( cleanupJob(projectId, 'routing-key', `project:${projectId}`, { key: `project:${projectId}` }), ); + // Proxy token accept floor: closed, not deleted. A missing floor accepts + // the legacy token, so deleting it would revive every token ever issued. + cleanup.push( + cleanupJob(projectId, 'routing-put', proxyTokenStateKey(projectId), { + key: proxyTokenStateKey(projectId), + value: CLOSED_PROXY_TOKEN_STATE, + }), + ); // Queue routing KV entries for (const q of resources.filter((r) => r.type === 'queue')) { diff --git a/platform/packages/api/src/proxy-token-rotation-cloudflare.ts b/platform/packages/api/src/proxy-token-rotation-cloudflare.ts new file mode 100644 index 000000000..ec3708eec --- /dev/null +++ b/platform/packages/api/src/proxy-token-rotation-cloudflare.ts @@ -0,0 +1,168 @@ +import { CF_API_BASE, CloudflareApiError } from './provisioning/cloudflare-api.ts'; +import type { RotationBudget } from './proxy-token-rotation.ts'; + +/** + * Cloudflare calls a proxy token rotation makes against tenant Workers: read a + * script's bindings, and replace its `__VOID_PROXY_TOKEN` through a settings + * PATCH that inherits every other binding (no code upload). + * + * Every call is counted against the run's budget and never retried or slept + * on here; the rotation decides what to do with a rate limit or an error. + */ + +const PROXY_TOKEN_BINDING = '__VOID_PROXY_TOKEN'; +const REQUEST_TIMEOUT_MS = 60_000; +/** + * A call needs at least this much of the run's time left. Starting one closer + * to the deadline would cut a settings PATCH off with its outcome unknown. + */ +const MIN_REQUEST_MS = 5_000; +const DEFAULT_RETRY_AFTER_SECONDS = 60; +/** Cloudflare's "script not found" error code. */ +const SCRIPT_NOT_FOUND_CODE = 10007; + +export class CloudflareRateLimitError extends Error { + readonly retryAfterSeconds: number; + constructor(retryAfterSeconds: number) { + super(`Cloudflare API rate limited; retry after ${retryAfterSeconds} seconds`); + this.name = 'CloudflareRateLimitError'; + this.retryAfterSeconds = retryAfterSeconds; + } +} + +/** Thrown before a call when the run has no Cloudflare calls or time left. */ +export class RotationBudgetExhaustedError extends Error { + constructor() { + super('proxy token rotation budget exhausted'); + this.name = 'RotationBudgetExhaustedError'; + } +} + +export type ScriptBinding = Readonly<{ name: string; type: string; text?: string }>; + +/** + * Settings PATCH bindings that keep every binding as it is (`inherit`, in the + * original order) and set the proxy token. A binding left out of the list + * would be removed, so every name is listed. + */ +export function buildProxyTokenBindingsPatch( + existing: ReadonlyArray, + token: string, +): ReadonlyArray> { + if (!existing.some((binding) => binding.name === PROXY_TOKEN_BINDING)) { + throw new Error(`script has no ${PROXY_TOKEN_BINDING} binding`); + } + return existing.map((binding): Record => + binding.name === PROXY_TOKEN_BINDING + ? { type: 'plain_text', name: PROXY_TOKEN_BINDING, text: token } + : { type: 'inherit', name: binding.name }, + ); +} + +export type RotationCloudflareClient = Readonly<{ + /** The script's bindings, or null when the script does not exist. */ + getScriptBindings(scriptName: string): Promise | null>; + /** The bindings in the PATCH result, or null when the result has none. */ + patchScriptBindings( + scriptName: string, + bindings: ReadonlyArray>, + ): Promise | null>; +}>; + +type CfEnvelope = Readonly<{ + success: boolean; + errors?: ReadonlyArray<{ code: number; message: string }>; + result: T; +}>; + +function parseRetryAfter(value: string | null): number { + const seconds = Number(value?.trim() || Number.NaN); + return Number.isFinite(seconds) && seconds >= 0 + ? Math.ceil(seconds) + : DEFAULT_RETRY_AFTER_SECONDS; +} + +export function createRotationCloudflareClient( + input: Readonly<{ + accountId: string; + apiToken: string; + dispatchNamespace: string; + budget: RotationBudget; + fetch?: typeof fetch; + }>, +): RotationCloudflareClient { + const send = input.fetch ?? ((url, init) => fetch(url, init)); + const scriptsUrl = `${CF_API_BASE}/${input.accountId}/workers/dispatch/namespaces/${input.dispatchNamespace}/scripts`; + + async function request(method: string, path: string, body?: BodyInit): Promise { + const { budget } = input; + const remainingMs = budget.deadline - Date.now(); + if (budget.cfCallsRemaining <= 0 || remainingMs < MIN_REQUEST_MS) { + throw new RotationBudgetExhaustedError(); + } + budget.cfCallsRemaining -= 1; + const headers: Record = { Authorization: `Bearer ${input.apiToken}` }; + if (body !== undefined && !(body instanceof FormData)) { + headers['Content-Type'] = 'application/json'; + } + const response = await send(`${scriptsUrl}${path}`, { + method, + headers, + body, + signal: AbortSignal.timeout(Math.min(REQUEST_TIMEOUT_MS, remainingMs)), + }); + if (response.status === 429) { + throw new CloudflareRateLimitError(parseRetryAfter(response.headers.get('Retry-After'))); + } + let json: CfEnvelope; + try { + json = (await response.json()) as CfEnvelope; + } catch { + throw new CloudflareApiError( + `Cloudflare API error: HTTP ${response.status} ${method} ${path}`, + response.status, + ); + } + if (!json.success) { + const errors = json.errors ?? []; + throw new CloudflareApiError( + `Cloudflare API error: ${errors.map((error) => error.message).join(', ')}`, + response.status, + errors.map((error) => error.code), + ); + } + return json.result; + } + + return { + async getScriptBindings(scriptName) { + try { + return await request>( + 'GET', + `/${encodeURIComponent(scriptName)}/bindings`, + ); + } catch (error) { + if ( + error instanceof CloudflareApiError && + (error.status === 404 || error.codes.includes(SCRIPT_NOT_FOUND_CODE)) + ) { + return null; + } + throw error; + } + }, + async patchScriptBindings(scriptName, bindings) { + const form = new FormData(); + form.append( + 'settings', + new Blob([JSON.stringify({ bindings })], { type: 'application/json' }), + ); + const result = await request<{ bindings?: ReadonlyArray } | null>( + 'PATCH', + `/${encodeURIComponent(scriptName)}/settings`, + form, + ); + return Array.isArray(result?.bindings) ? result.bindings : null; + }, + }; +} diff --git a/platform/packages/api/src/proxy-token-rotation-request.ts b/platform/packages/api/src/proxy-token-rotation-request.ts new file mode 100644 index 000000000..58871d34c --- /dev/null +++ b/platform/packages/api/src/proxy-token-rotation-request.ts @@ -0,0 +1,98 @@ +import { and, eq, inArray, sql } from 'drizzle-orm'; +import type { DbOrTx } from './db/client.ts'; +import { projectCleanupJobs, projectMembers } from './schema.ts'; + +// Rotation requests, apart from the runner in `proxy-token-rotation.ts` so the +// `db/` modules can write one inside their transactions without importing +// `db.ts` back. + +export const PROXY_TOKEN_ROTATION_JOB_KIND = 'proxy-token-rotation'; + +/** Primary key of a project's rotation row in `project_cleanup_jobs`. */ +export function proxyTokenRotationTarget(projectId: string): string { + return `${PROXY_TOKEN_ROTATION_JOB_KIND}:${projectId}`; +} + +export type ProxyTokenRotationReason = + | 'member-removed' + | 'member-role-changed' + | 'user-deleted' + | 'user-suspended' + | 'untrusted-deploy-replaced' + | 'operator' + | 'operator-bulk'; + +export function rotationLog(event: string, projectId: string, detail = ''): void { + console.log(`[proxy-token-rotation] ${event} ${projectId}${detail ? ` ${detail}` : ''}`); +} + +// What a request does to an existing job. Both writers below share these, so +// one statement for many projects leaves the rows one request per project would. +const jobStarted = sql`${projectCleanupJobs.payload}->>'phase' <> 'requested'`; +const flagAgain = sql`CASE WHEN ${projectCleanupJobs.payload}->>'phase' <> 'requested' + THEN ${projectCleanupJobs.payload} || '{"again":"1"}'::jsonb + ELSE ${projectCleanupJobs.payload} END`; + +/** + * Asks for a new token generation. A job that has not started yet already + * covers the request; one past START may have handed out tok(T), so it + * finishes and then runs once more (`again`). Operator requests also make the + * job due now. Call inside the caller's transaction. + */ +export async function requestProxyTokenRotation( + tx: DbOrTx, + projectId: string, + reason: ProxyTokenRotationReason, + mode: 'member' | 'operator', +): Promise { + const rows = await tx + .insert(projectCleanupJobs) + .values({ + target: proxyTokenRotationTarget(projectId), + project_id: projectId, + kind: PROXY_TOKEN_ROTATION_JOB_KIND, + payload: { phase: 'requested', reason }, + next_attempt_at: sql`now()`, + }) + .onConflictDoUpdate({ + target: projectCleanupJobs.target, + set: + mode === 'operator' + ? { payload: flagAgain, next_attempt_at: sql`now()` } + : { payload: flagAgain }, + ...(mode === 'member' ? { setWhere: jobStarted } : {}), + }) + .returning({ phase: sql`${projectCleanupJobs.payload}->>'phase'` }); + if (rows.length > 0) { + rotationLog('requested', projectId, `reason=${reason}`); + } +} + +/** + * `requestProxyTokenRotation` in `member` mode for every project where the + * user is a collaborator or admin, in one statement whatever the number of + * projects. Call inside the caller's transaction. + */ +export async function requestProxyTokenRotationsForMember( + tx: DbOrTx, + userId: string, + reason: ProxyTokenRotationReason, +): Promise { + const deployMember = and( + eq(projectMembers.user_id, userId), + inArray(projectMembers.role, ['collaborator', 'admin']), + ); + const result = await tx.execute<{ project_id: string }>(sql` + INSERT INTO ${projectCleanupJobs} (target, project_id, kind, payload, next_attempt_at) + SELECT ${PROXY_TOKEN_ROTATION_JOB_KIND}::text || ':' || ${projectMembers.project_id}, + ${projectMembers.project_id}, ${PROXY_TOKEN_ROTATION_JOB_KIND}::text, + jsonb_build_object('phase', 'requested', 'reason', ${reason}::text), now() + FROM ${projectMembers} + WHERE ${deployMember} + ORDER BY ${projectMembers.project_id} + ON CONFLICT (target) DO UPDATE SET payload = ${flagAgain} WHERE ${jobStarted} + RETURNING project_id`); + for (const { project_id: projectId } of result.rows) { + rotationLog('requested', projectId, `reason=${reason}`); + } +} diff --git a/platform/packages/api/src/proxy-token-rotation.ts b/platform/packages/api/src/proxy-token-rotation.ts new file mode 100644 index 000000000..a947b7397 --- /dev/null +++ b/platform/packages/api/src/proxy-token-rotation.ts @@ -0,0 +1,1004 @@ +import { and, asc, eq, or, sql, type SQL } from 'drizzle-orm'; +import { DEFAULT_DISPATCH_NAMESPACE } from './constants.ts'; +import { snapshotCronScheduler, syncCronScheduler } from './cron-scheduler-client.ts'; +import { getActiveDeployment, type Db, type DbOrTx } from './db.ts'; +import type { ApiEnv } from './env.ts'; +import { getProjectDeploymentWorkerName, getProjectQueueName } from './naming.ts'; +import { hasProjectCapability, projectRoleForUser } from './project-access.ts'; +import { requireNormalizedProject } from './project-state.ts'; +import { + buildProxyTokenBindingsPatch, + CloudflareRateLimitError, + createRotationCloudflareClient, + RotationBudgetExhaustedError, + type RotationCloudflareClient, + type ScriptBinding, +} from './proxy-token-rotation-cloudflare.ts'; +import { + PROXY_TOKEN_ROTATION_JOB_KIND, + proxyTokenRotationTarget, + requestProxyTokenRotation, + rotationLog, + type ProxyTokenRotationReason, +} from './proxy-token-rotation-request.ts'; +import { deriveProxyToken, proxyTokenStateKey, serializeProxyTokenState } from './proxy-token.ts'; +import { deployments, projectCleanupJobs, projects, users } from './schema.ts'; +import { isAdminSuspended } from './suspension.ts'; +import type { DeployManifest, Project, ProjectRole } from './types.ts'; + +export { + PROXY_TOKEN_ROTATION_JOB_KIND, + proxyTokenRotationTarget, + requestProxyTokenRotation, + type ProxyTokenRotationReason, +}; + +/** + * Proxy token generations per project. + * + * `projects.proxy_token_gen` is the highest generation the platform has issued + * for the project (0 = the legacy token). Deploys bake the token for that + * generation into the Worker. A Worker deploy moves 0 to 1 + * (`issueProjectProxyToken`); after that only a rotation raises it. The KV + * accept floor (`proxy-token:`) never exceeds it. + * + * `deployments.proxy_token_gen` is the generation that deployment's Worker + * holds. Senders (the proxy through resolve-worker, the queue entries and the + * cron scheduler) present exactly that token, once: the Worker runs its own + * middleware before the generated handlers check the token, so a rejected + * call is never replayed with another token. A rotation writes the row only + * after Cloudflare confirms the Worker holds the new token. + */ + +/** How long a rotation waits for KV copies and old Worker versions to drain. */ +export const PROXY_TOKEN_ROTATION_WAIT_MS = 120_000; + +/** Failed steps in a row after which project owners and admins see a warning. */ +export const PROXY_TOKEN_ROTATION_STUCK_ATTEMPTS = 4; + +/** Cloudflare API calls and wall-clock time one rotation run may still spend. */ +export type RotationBudget = { cfCallsRemaining: number; readonly deadline: number }; + +/** + * The token a deploy bakes into its Worker: the one for the project's issued + * generation. A project still on generation 0 moves to generation 1 here, so + * new code never receives the legacy token. The floor stays where it is: + * older Workers of the project keep the legacy token, which stays accepted + * until a rotation raises the floor. The caller records `gen` on its + * deployment row. + */ +export async function issueProjectProxyToken( + db: Db, + jwtSecret: string, + projectId: string, +): Promise> { + const [initialized] = await db + .update(projects) + .set({ proxy_token_gen: 1 }) + .where(and(eq(projects.id, projectId), eq(projects.proxy_token_gen, 0))) + .returning({ gen: projects.proxy_token_gen }); + if (initialized) { + console.info('[proxy-token] initialized', { projectId }); + return { + gen: initialized.gen, + token: await deriveProxyToken(jwtSecret, projectId, initialized.gen), + }; + } + const [row] = await db + .select({ gen: projects.proxy_token_gen }) + .from(projects) + .where(eq(projects.id, projectId)) + .limit(1); + if (!row) { + throw new Error(`project ${projectId} not found`); + } + return { gen: row.gen, token: await deriveProxyToken(jwtSecret, projectId, row.gen) }; +} + +// --- Rotation --- +// +// A rotation is one `project_cleanup_jobs` row (kind `proxy-token-rotation`) +// that moves through these phases, one short transaction per step: +// +// requested START: raise projects.proxy_token_gen to T. Postgres only, so +// T is committed before anything outside sees tok(T). +// patch one Worker per step: GET its bindings, PATCH tok(T) in, then +// record T on its deployment row and, for the active Worker, +// point the queue entries and the cron scheduler at tok(T). +// revoke every Worker holds tok(T): write the accept floor T. +// settle wait until every floor reader sees T, then finish. +// +// Every step locks the project row, then the job row, and refuses while a +// deployment is in flight (the same fence as deploys, rollback and secret +// writes). A wait is stored in the payload as `waitUntil`, taken from +// `clock_timestamp()` after the step's last outside write, so a kick of +// `next_attempt_at` never cuts it short. Until the floor rises both tokens are +// accepted, so a failed or unfinished rotation never breaks a tenant. Each +// sender presents only the token its deployment row records, so right after a +// PATCH, until the step commits and Cloudflare stops serving the Worker's old +// version, a call can get a 403 that is not retried. + +export type ProxyTokenRotationPhase = 'requested' | 'patch' | 'revoke' | 'settle'; + +export type ProxyTokenRotationEnv = Pick< + ApiEnv['Bindings'], + | 'ROUTING' + | 'CRON_SCHEDULER' + | 'JWT_SECRET' + | 'CF_ACCOUNT_ID' + | 'CF_API_TOKEN' + | 'DISPATCH_NAMESPACE' +>; + +export type ProjectRotationOutcome = + | { status: 'done'; gen: number } + | { + status: 'pending'; + phase: ProxyTokenRotationPhase; + reason: 'waiting' | 'busy' | 'budget' | 'rate-limited'; + retryAfterSeconds: number; + } + | { status: 'failed'; phase: ProxyTokenRotationPhase; error: string; workerName?: string }; + +export type RotationRunSummary = { + done: number; + pending: number; + failed: number; + retryAfterSeconds: number | null; +}; + +const PROXY_TOKEN_BINDING = '__VOID_PROXY_TOKEN'; +const CLAIM_LEASE = sql`interval '15 minutes'`; +const BUSY_RETRY_SECONDS = 120; + +function isStaticProjectType(type: Project['type']): boolean { + return type === 'static' || type === 'spa'; +} + +/** `expr` (a timestamptz) as an ISO-8601 UTC string with microseconds. */ +function isoUtc(expr: SQL): SQL { + return sql`to_char((${expr}) AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`; +} + +function waitInterval(waitMs: number): SQL { + return sql`(${Math.max(0, Math.trunc(waitMs))}::integer * interval '1 millisecond')`; +} + +const jobTarget = (projectId: string) => + eq(projectCleanupJobs.target, proxyTokenRotationTarget(projectId)); + +// --- Requests --- + +/** + * Queues every project not being deleted, with the member rule on an + * existing job. Returns the number of jobs inserted or flagged. + */ +export async function requestBulkProxyTokenRotations(db: DbOrTx): Promise { + const reason: ProxyTokenRotationReason = 'operator-bulk'; + const result = await db.execute(sql` + INSERT INTO ${projectCleanupJobs} (target, project_id, kind, payload, next_attempt_at) + SELECT ${PROXY_TOKEN_ROTATION_JOB_KIND}::text || ':' || ${projects.id}, ${projects.id}, + ${PROXY_TOKEN_ROTATION_JOB_KIND}::text, jsonb_build_object('phase', 'requested', 'reason', ${reason}::text), now() + FROM ${projects} + WHERE ${projects.status} <> 'deleting' + ON CONFLICT (target) DO UPDATE + SET payload = project_cleanup_jobs.payload || '{"again":"1"}'::jsonb + WHERE project_cleanup_jobs.payload->>'phase' <> 'requested' + RETURNING target`); + const count = result.rows.length; + if (count > 0) { + console.log(`[proxy-token-rotation] requested ${count} project(s)`); + } + return count; +} + +/** + * The user who deployed a deployment's code. Deployments from before project + * teams record no author; only the owner could deploy then, and their billing + * user (backfilled with the owner when teams shipped) is that owner. + */ +const deploymentAuthor = sql< + string | null +>`coalesce(${deployments.initiated_by_user_id}, ${deployments.billing_user_id})`; + +type CodeAuthor = Readonly<{ userId: string | null; suspendedReason: string | null }>; + +/** + * Whether the author of some code can still deploy it to the project. Owner + * suspension does not count against it; a collaborator or admin suspended by + * an admin does. Code with no known author counts as trusted. + */ +async function authorCanDeploy( + tx: DbOrTx, + projectId: string, + author: CodeAuthor, +): Promise { + if (!author.userId) { + return true; + } + const role = await projectRoleForUser(tx, projectId, author.userId); + return ( + role === 'owner' || + (!!role && hasProjectCapability(role, 'deploy') && !isAdminSuspended(author.suspendedReason)) + ); +} + +/** + * While code from a user who can no longer deploy is active, it receives every + * new token, so a rotation protects only once that code stops running. Call in + * the cutover transaction (project lock held) right before the active + * deployment changes: when the code being replaced was deployed by such a user, + * this requests one more generation. + */ +export async function requestRotationIfReplacingUntrustedCode( + tx: DbOrTx, + projectId: string, + nextDeploymentId: string, +): Promise { + const [current] = await tx + .select({ + deploymentId: projects.active_deployment_id, + userId: deploymentAuthor, + suspendedReason: users.suspended_reason, + }) + .from(projects) + .innerJoin(deployments, eq(deployments.id, projects.active_deployment_id)) + .leftJoin(users, eq(users.id, deploymentAuthor)) + .where(eq(projects.id, projectId)) + .limit(1); + if ( + !current || + current.deploymentId === nextDeploymentId || + (await authorCanDeploy(tx, projectId, current)) + ) { + return false; + } + await requestProxyTokenRotation(tx, projectId, 'untrusted-deploy-replaced', 'member'); + return true; +} + +// --- Inventory --- + +type InventoryRow = Readonly<{ + deploymentId: string; + workerName: string; + createdAt: string; + active: boolean; + /** Generation recorded for the token this Worker holds. */ + proxyTokenGen: number; + /** A retained Worker whose author can no longer deploy: it keeps its old token. */ + skipped: boolean; +}>; + +function inventoryWhere( + project: Pick, + after?: Readonly<{ createdAt: string; id: string }>, +): SQL | undefined { + return and( + eq(deployments.project_id, project.id), + project.active_deployment_id + ? or(eq(deployments.retained, true), eq(deployments.id, project.active_deployment_id)) + : eq(deployments.retained, true), + sql`${deployments.source} IS DISTINCT FROM 'rollback-lease'`, + after + ? sql`(${deployments.created_at}, ${deployments.id}) > (${after.createdAt}::timestamp, ${after.id})` + : undefined, + ); +} + +async function selectInventory( + db: DbOrTx, + project: Pick, + options: Readonly<{ after?: { createdAt: string; id: string }; limit?: number }> = {}, +): Promise> { + const query = db + .select({ + id: deployments.id, + createdAt: deployments.created_at, + proxyTokenGen: deployments.proxy_token_gen, + userId: deploymentAuthor, + suspendedReason: users.suspended_reason, + }) + .from(deployments) + .leftJoin(users, eq(users.id, deploymentAuthor)) + .where(inventoryWhere(project, options.after)) + .orderBy(asc(deployments.created_at), asc(deployments.id)); + const rows = options.limit === undefined ? await query : await query.limit(options.limit); + const inventory: Array = []; + for (const row of rows) { + const active = row.id === project.active_deployment_id; + inventory.push({ + deploymentId: row.id, + workerName: getProjectDeploymentWorkerName(project.slug, row.id), + createdAt: row.createdAt, + active, + proxyTokenGen: row.proxyTokenGen, + skipped: !active && !(await authorCanDeploy(db, project.id, row)), + }); + } + return inventory; +} + +/** + * Every Worker that gets the project's next token: the active deployment and + * every retained one (rollback leases excluded), oldest first. A retained + * Worker whose author can no longer deploy is left out: it keeps its old + * token, which the raised floor rejects, so rolling back to it needs a new + * deploy before platform services work again. A Worker the running rotation + * found without a proxy token binding is left out the same way; one the walk + * has not reached yet looks like any other until its GET. Rows of a static + * deployment stay in; their GET finds no script. + */ +export async function listProxyTokenInventory( + db: DbOrTx, + project: Pick, +): Promise> { + const [job] = await db + .select({ payload: projectCleanupJobs.payload }) + .from(projectCleanupJobs) + .where(and(jobTarget(project.id), eq(projectCleanupJobs.kind, PROXY_TOKEN_ROTATION_JOB_KIND))); + const unbound = unboundDeployments(job?.payload); + return (await selectInventory(db, project)).filter( + (row) => !row.skipped && !unbound.has(row.deploymentId), + ); +} + +/** + * Deployments whose Worker this rotation found without a proxy token binding + * (`unbound` in the job payload, comma-separated). Their rows keep the + * generation they hold. + */ +function unboundDeployments(payload: JobPayload | undefined): ReadonlySet { + return new Set(payload?.unbound ? payload.unbound.split(',') : []); +} + +function parseCursor(cursor: string | undefined): { createdAt: string; id: string } | undefined { + const split = cursor ? cursor.indexOf('|') : -1; + return split > 0 + ? { createdAt: cursor!.slice(0, split), id: cursor!.slice(split + 1) } + : undefined; +} + +// --- Steps --- + +type JobPayload = Readonly>; + +type StepContext = { + readonly projectId: string; + readonly env: ProxyTokenRotationEnv; + readonly cf: RotationCloudflareClient; + readonly wait: SQL; + phase: ProxyTokenRotationPhase; + workerName?: string; + /** A Worker may hold a token that its row (and, if active, its copies) do not name yet. */ + workerAheadOfRow?: boolean; +}; + +type StepResult = + | { kind: 'continue' } + | { kind: 'done'; gen: number } + | { kind: 'waiting'; retryAfterSeconds: number } + | { kind: 'busy' }; + +type LockedStep = { + project: Project; + payload: JobPayload; + /** Failed attempts of this step since the last one that committed. */ + attempts: number; +}; + +async function issuedGeneration(db: DbOrTx, projectId: string): Promise { + const [row] = await db + .select({ gen: projects.proxy_token_gen }) + .from(projects) + .where(eq(projects.id, projectId)) + .limit(1); + return row?.gen ?? 0; +} + +/** Locks the job row and checks its wait. Null when the job is gone. */ +async function lockJob( + tx: DbOrTx, + projectId: string, +): Promise<{ payload: JobPayload; attempts: number; waitSeconds: number | null } | null> { + const [job] = await tx + .select({ + payload: projectCleanupJobs.payload, + attempts: projectCleanupJobs.attempts, + waitSeconds: sql`CASE + WHEN (${projectCleanupJobs.payload}->>'waitUntil')::timestamptz > now() + THEN ceil(extract(epoch FROM (${projectCleanupJobs.payload}->>'waitUntil')::timestamptz - now()))::integer + END`, + }) + .from(projectCleanupJobs) + .where(and(jobTarget(projectId), eq(projectCleanupJobs.kind, PROXY_TOKEN_ROTATION_JOB_KIND))) + .for('update'); + return job + ? { payload: job.payload, attempts: job.attempts, waitSeconds: job.waitSeconds } + : null; +} + +/** Project lock, then job lock, then the wait, then no deployment in flight. */ +async function lockStep(tx: DbOrTx, ctx: StepContext): Promise { + const [project] = await tx + .select() + .from(projects) + .where(eq(projects.id, ctx.projectId)) + .for('update'); + if (!project || project.status === 'deleting') { + await tx.delete(projectCleanupJobs).where(jobTarget(ctx.projectId)); + rotationLog('dropped', ctx.projectId, 'project deleted'); + return { kind: 'done', gen: project?.proxy_token_gen ?? 0 }; + } + const job = await lockJob(tx, ctx.projectId); + if (!job) { + return { kind: 'done', gen: project.proxy_token_gen }; + } + ctx.phase = job.payload.phase as ProxyTokenRotationPhase; + if (job.waitSeconds !== null) { + return { kind: 'waiting', retryAfterSeconds: job.waitSeconds }; + } + if (await getActiveDeployment(tx, ctx.projectId)) { + return { kind: 'busy' }; + } + return { project, payload: job.payload, attempts: job.attempts }; +} + +/** Final statement of a successful step: merge `fields` into the payload. */ +async function finishStep(tx: DbOrTx, projectId: string, fields: SQL): Promise { + await tx + .update(projectCleanupJobs) + .set({ + payload: sql`${projectCleanupJobs.payload} || ${fields}`, + attempts: 0, + last_error: null, + }) + .where(jobTarget(projectId)); +} + +type ActiveWorker = Readonly<{ workerName: string; manifest: DeployManifest | null }>; + +async function activeWorker(tx: DbOrTx, project: Project): Promise { + if (!project.active_deployment_id) { + return null; + } + const [row] = await tx + .select({ manifest: deployments.manifest }) + .from(deployments) + .where(eq(deployments.id, project.active_deployment_id)) + .limit(1); + let manifest: DeployManifest | null = null; + try { + manifest = row?.manifest ? (JSON.parse(row.manifest) as DeployManifest) : null; + } catch { + manifest = null; + } + return { + workerName: getProjectDeploymentWorkerName(project.slug, project.active_deployment_id), + manifest, + }; +} + +/** Whether the active Worker receives queue or cron traffic through a copy of its token. */ +function hasTokenCopies(project: Project, active: ActiveWorker | null): active is ActiveWorker { + return ( + !!active && + !isStaticProjectType(project.type) && + (!!active.manifest?.schedules?.length || !!active.manifest?.queues?.length) + ); +} + +/** + * Points the queue routing entries and the cron scheduler of the active Worker + * at `proxyToken`, the one token that Worker holds. Built from the active + * manifest, never from what KV holds. + */ +async function writeCopies( + env: ProxyTokenRotationEnv, + projectId: string, + active: ActiveWorker, + proxyToken: string, +): Promise { + if (active.manifest?.schedules?.length) { + const state = await snapshotCronScheduler(env, projectId); + if (state && state.workerName === active.workerName) { + await syncCronScheduler(env, projectId, { ...state, proxyToken }); + } + } + for (const queue of active.manifest?.queues ?? []) { + await env.ROUTING.put( + `queue:${getProjectQueueName(projectId, queue.name)}`, + JSON.stringify({ + workerName: active.workerName, + projectId, + queueName: queue.name, + proxyToken, + }), + ); + } +} + +/** The proxy token binding, `undefined` when the Worker has none. */ +function tokenBinding(bindings: ReadonlyArray): ScriptBinding | undefined { + return bindings.find((binding) => binding.name === PROXY_TOKEN_BINDING); +} + +async function start(tx: DbOrTx, ctx: StepContext, locked: LockedStep): Promise { + await requireNormalizedProject(tx, locked.project); + const gen = locked.project.proxy_token_gen + 1; + await tx.update(projects).set({ proxy_token_gen: gen }).where(eq(projects.id, ctx.projectId)); + await tx + .update(projectCleanupJobs) + .set({ + payload: sql`jsonb_build_object('phase', 'patch', 'reason', ${projectCleanupJobs.payload}->>'reason', 'gen', ${String(gen)}::text, 'cursor', '')`, + attempts: 0, + last_error: null, + }) + .where(jobTarget(ctx.projectId)); + rotationLog('started', ctx.projectId, `gen=${gen}`); + return { kind: 'continue' }; +} + +async function patchNext(tx: DbOrTx, ctx: StepContext, locked: LockedStep): Promise { + const { project, payload } = locked; + const [row] = await selectInventory(tx, project, { + after: parseCursor(payload.cursor), + limit: 1, + }); + if (!row) { + // Old versions of a patched Worker drain for one wait after the last PATCH. + await finishStep( + tx, + ctx.projectId, + sql`jsonb_build_object('phase', 'revoke', 'waitUntil', CASE + WHEN ${projectCleanupJobs.payload}->>'lastPatchAt' IS NOT NULL + THEN ${isoUtc(sql`(${projectCleanupJobs.payload}->>'lastPatchAt')::timestamptz + ${ctx.wait}`)} + ELSE ${isoUtc(sql`clock_timestamp()`)} END)`, + ); + return { kind: 'continue' }; + } + const cursor = `${row.createdAt}|${row.deploymentId}`; + if (row.skipped) { + rotationLog('skipped', ctx.projectId, `worker=${row.workerName} author cannot deploy`); + await finishStep(tx, ctx.projectId, sql`jsonb_build_object('cursor', ${cursor}::text)`); + return { kind: 'continue' }; + } + ctx.workerName = row.workerName; + const gen = project.proxy_token_gen; + const token = await deriveProxyToken(ctx.env.JWT_SECRET, ctx.projectId, gen); + const bindings = await ctx.cf.getScriptBindings(row.workerName); + const current = bindings ? tokenBinding(bindings) : undefined; + if (bindings && !current) { + // No token binding to PATCH. Every per-deployment Worker the platform + // creates has one, so it was removed outside the platform (Workers from + // before the binding rename are `user-` scripts that never reach + // this step). Its row and copies keep the generation it holds, so senders + // still reach it; its own proxy calls fail once the floor rises, until a + // redeploy. + rotationLog( + 'skipped', + ctx.projectId, + `worker=${row.workerName} has no ${PROXY_TOKEN_BINDING} binding; redeploy the project to give it one`, + ); + const unbound = [...new Set(unboundDeployments(payload)).add(row.deploymentId)].join(','); + await finishStep( + tx, + ctx.projectId, + sql`jsonb_build_object('cursor', ${cursor}::text, 'unbound', ${unbound}::text)`, + ); + return { kind: 'continue' }; + } + let patched = false; + if (bindings === null) { + if (row.active && !isStaticProjectType(project.type)) { + throw new Error(`active Worker ${row.workerName} not found`); + } + } else if (current) { + if (current.type !== 'plain_text') { + throw new Error(`Worker ${row.workerName} has a ${current.type} proxy token binding`); + } + if (current.text !== token) { + // Cloudflare can apply the PATCH even when its response is lost. + ctx.workerAheadOfRow = true; + const result = + (await ctx.cf.patchScriptBindings( + row.workerName, + buildProxyTokenBindingsPatch(bindings, token), + )) ?? (await ctx.cf.getScriptBindings(row.workerName)); + if (tokenBinding(result ?? [])?.text !== token) { + throw new Error(`Worker ${row.workerName} did not take the new proxy token`); + } + patched = true; + rotationLog('patched', ctx.projectId, `worker=${row.workerName} gen=${gen}`); + } else if (locked.attempts > 0 || row.proxyTokenGen !== gen) { + // An earlier attempt patched this Worker and then lost the response, + // failed or rolled back before its step committed. Its old version can + // still serve, so it drains as if patched now. + patched = true; + ctx.workerAheadOfRow = row.proxyTokenGen !== gen; + } + } + // The Worker now holds tok(gen), or has no script. Only now do the row and + // the copies name it, so senders never run ahead of the Worker. + if (patched && row.active) { + const active = await activeWorker(tx, project); + if (hasTokenCopies(project, active)) { + await writeCopies(ctx.env, ctx.projectId, active, token); + } + } + if (row.proxyTokenGen !== gen) { + await tx + .update(deployments) + .set({ proxy_token_gen: gen }) + .where(eq(deployments.id, row.deploymentId)); + } + await finishStep( + tx, + ctx.projectId, + patched + ? sql`jsonb_build_object('cursor', ${cursor}::text, 'lastPatchAt', ${isoUtc(sql`clock_timestamp()`)})` + : sql`jsonb_build_object('cursor', ${cursor}::text)`, + ); + return { kind: 'continue' }; +} + +async function revoke(tx: DbOrTx, ctx: StepContext, locked: LockedStep): Promise { + const { project, payload } = locked; + const [later] = await selectInventory(tx, project, { + after: parseCursor(payload.cursor), + limit: 1, + }); + if (later) { + // A deploy finished after the walk passed; patch (or skip) it first. + await finishStep(tx, ctx.projectId, sql`'{"phase":"patch"}'::jsonb`); + return { kind: 'continue' }; + } + const gen = project.proxy_token_gen; + const unbound = unboundDeployments(payload); + const stale = (await selectInventory(tx, project)).find( + (row) => !row.skipped && !unbound.has(row.deploymentId) && row.proxyTokenGen !== gen, + ); + if (stale) { + // The floor would reject the token a sender still presents to this Worker. + rotationLog('rewalk', ctx.projectId, `worker=${stale.workerName} gen=${stale.proxyTokenGen}`); + await finishStep(tx, ctx.projectId, sql`'{"phase":"patch","cursor":""}'::jsonb`); + return { kind: 'continue' }; + } + const token = await deriveProxyToken(ctx.env.JWT_SECRET, ctx.projectId, gen); + const active = await activeWorker(tx, project); + if (active) { + ctx.workerName = active.workerName; + const bindings = await ctx.cf.getScriptBindings(active.workerName); + if (bindings === null && !isStaticProjectType(project.type)) { + throw new Error(`active Worker ${active.workerName} not found`); + } + const current = bindings ? tokenBinding(bindings) : undefined; + if (current && current.text !== token) { + await finishStep(tx, ctx.projectId, sql`'{"phase":"patch","cursor":""}'::jsonb`); + return { kind: 'continue' }; + } + // An active Worker without the binding keeps the copies of the token it holds. + if (current && hasTokenCopies(project, active)) { + await writeCopies(ctx.env, ctx.projectId, active, token); + } + } + // Written before COMMIT: KV may only run ahead, and every patched Worker holds T. + await ctx.env.ROUTING.put(proxyTokenStateKey(ctx.projectId), serializeProxyTokenState({ gen })); + await finishStep( + tx, + ctx.projectId, + sql`jsonb_build_object('phase', 'settle', 'waitUntil', ${isoUtc(sql`clock_timestamp() + ${ctx.wait}`)})`, + ); + rotationLog('revoked', ctx.projectId, `gen=${gen}`); + return { kind: 'continue' }; +} + +/** Holds only the job row lock: it touches nothing outside Postgres. */ +async function settle(tx: DbOrTx, ctx: StepContext): Promise { + const job = await lockJob(tx, ctx.projectId); + if (!job) { + return { kind: 'done', gen: await issuedGeneration(tx, ctx.projectId) }; + } + ctx.phase = job.payload.phase as ProxyTokenRotationPhase; + if (ctx.phase !== 'settle') { + return { kind: 'continue' }; + } + if (job.waitSeconds !== null) { + return { kind: 'waiting', retryAfterSeconds: job.waitSeconds }; + } + if (job.payload.again === '1') { + await tx + .update(projectCleanupJobs) + .set({ + payload: sql`jsonb_build_object('phase', 'requested', 'reason', ${projectCleanupJobs.payload}->>'reason')`, + attempts: 0, + last_error: null, + }) + .where(jobTarget(ctx.projectId)); + rotationLog('rotated', ctx.projectId, `gen=${job.payload.gen} again`); + return { kind: 'continue' }; + } + await tx.delete(projectCleanupJobs).where(jobTarget(ctx.projectId)); + rotationLog('rotated', ctx.projectId, `gen=${job.payload.gen}`); + return { kind: 'done', gen: Number(job.payload.gen) }; +} + +function runStep(db: Db, ctx: StepContext): Promise { + ctx.workerAheadOfRow = false; + return db.transaction(async (tx) => { + const [peek] = await tx + .select({ phase: sql`${projectCleanupJobs.payload}->>'phase'` }) + .from(projectCleanupJobs) + .where(jobTarget(ctx.projectId)); + if (!peek) { + return { kind: 'done', gen: await issuedGeneration(tx, ctx.projectId) }; + } + ctx.phase = peek.phase as ProxyTokenRotationPhase; + ctx.workerName = undefined; + if (peek.phase === 'settle') { + return settle(tx, ctx); + } + const locked = await lockStep(tx, ctx); + if (!('project' in locked)) { + return locked; + } + switch (locked.payload.phase as ProxyTokenRotationPhase) { + case 'requested': + return start(tx, ctx, locked); + case 'patch': + return patchNext(tx, ctx, locked); + case 'revoke': + return revoke(tx, ctx, locked); + default: + throw new Error(`unknown rotation phase ${String(locked.payload.phase)}`); + } + }); +} + +// --- Runs --- + +async function setNextAttempt(db: Db, projectId: string, at: SQL): Promise { + await db.update(projectCleanupJobs).set({ next_attempt_at: at }).where(jobTarget(projectId)); +} + +/** Runs the due steps of a job this run has claimed. */ +async function runClaimed( + db: Db, + env: ProxyTokenRotationEnv, + projectId: string, + budget: RotationBudget, + options: Readonly<{ waitMs?: number; fetch?: typeof fetch }>, +): Promise { + const ctx: StepContext = { + projectId, + env, + cf: createRotationCloudflareClient({ + accountId: env.CF_ACCOUNT_ID, + apiToken: env.CF_API_TOKEN, + dispatchNamespace: env.DISPATCH_NAMESPACE ?? DEFAULT_DISPATCH_NAMESPACE, + budget, + fetch: options.fetch, + }), + wait: waitInterval(options.waitMs ?? PROXY_TOKEN_ROTATION_WAIT_MS), + phase: 'requested', + }; + for (;;) { + let result: StepResult; + try { + if (Date.now() >= budget.deadline) { + throw new RotationBudgetExhaustedError(); + } + result = await runStep(db, ctx); + } catch (error) { + if (error instanceof CloudflareRateLimitError) { + // Not a failed attempt: owners cannot fix a platform-wide rate limit. + // The log line lets operators alert on a rotation that stays limited. + rotationLog( + 'RATE-LIMITED', + projectId, + `phase=${ctx.phase} retryAfter=${error.retryAfterSeconds}s`, + ); + await setNextAttempt( + db, + projectId, + sql`now() + ${error.retryAfterSeconds}::integer * interval '1 second'`, + ); + return { + status: 'pending', + phase: ctx.phase, + reason: 'rate-limited', + retryAfterSeconds: error.retryAfterSeconds, + }; + } + if (error instanceof RotationBudgetExhaustedError) { + await setNextAttempt(db, projectId, sql`now()`); + return { status: 'pending', phase: ctx.phase, reason: 'budget', retryAfterSeconds: 0 }; + } + const message = (error instanceof Error ? error.message : String(error)) + .replaceAll(env.CF_API_TOKEN || '\u0000', '[redacted]') + .slice(0, 500); + const [row] = await db + .update(projectCleanupJobs) + .set({ + attempts: sql`${projectCleanupJobs.attempts} + 1`, + last_error: message, + // A Worker ahead of its row rejects the token that the proxy, queue + // relays and cron fires send it, now if it is active or after a + // rollback to it. Retry on the next run; a run tries each job once. + next_attempt_at: ctx.workerAheadOfRow ? sql`now()` : sql`now() + ${CLAIM_LEASE}`, + }) + .where(jobTarget(projectId)) + .returning({ attempts: projectCleanupJobs.attempts }); + console.error( + `[proxy-token-rotation] FAILED ${projectId} phase=${ctx.phase} attempts=${row?.attempts ?? 0} worker=${ctx.workerName ?? '-'}: ${message}`, + ); + return { + status: 'failed', + phase: ctx.phase, + error: message, + ...(ctx.workerName ? { workerName: ctx.workerName } : {}), + }; + } + switch (result.kind) { + case 'continue': + continue; + case 'done': + return { status: 'done', gen: result.gen }; + case 'waiting': + await setNextAttempt( + db, + projectId, + sql`(${projectCleanupJobs.payload}->>'waitUntil')::timestamptz`, + ); + return { + status: 'pending', + phase: ctx.phase, + reason: 'waiting', + retryAfterSeconds: result.retryAfterSeconds, + }; + case 'busy': + await setNextAttempt( + db, + projectId, + sql`now() + ${BUSY_RETRY_SECONDS}::integer * interval '1 second'`, + ); + return { + status: 'pending', + phase: ctx.phase, + reason: 'busy', + retryAfterSeconds: BUSY_RETRY_SECONDS, + }; + } + } +} + +/** Takes the lease on one due job. A busy or waiting job is not a failure, so `attempts` stays. */ +async function claimDueJob( + db: Db, + projectId?: string, + skip: ReadonlySet = new Set(), +): Promise { + const skipped = [...skip].map((id) => sql`${proxyTokenRotationTarget(id)}`); + const result = await db.execute<{ project_id: string }>(sql` + UPDATE ${projectCleanupJobs} SET next_attempt_at = now() + ${CLAIM_LEASE} + WHERE target = ( + SELECT target FROM ${projectCleanupJobs} + WHERE kind = ${PROXY_TOKEN_ROTATION_JOB_KIND} AND next_attempt_at <= now() + ${projectId ? sql`AND target = ${proxyTokenRotationTarget(projectId)}` : sql``} + ${skipped.length > 0 ? sql`AND target NOT IN (${sql.join(skipped, sql`, `)})` : sql``} + ORDER BY next_attempt_at, created_at + LIMIT 1 + FOR UPDATE SKIP LOCKED + ) + RETURNING project_id`); + return result.rows[0]?.project_id ?? null; +} + +/** Runs the due steps of one project's job. Claims it first. */ +export async function advanceProxyTokenRotation( + db: Db, + env: ProxyTokenRotationEnv, + projectId: string, + budget: RotationBudget, + options: Readonly<{ waitMs?: number; fetch?: typeof fetch }> = {}, +): Promise { + if (await claimDueJob(db, projectId)) { + return runClaimed(db, env, projectId, budget, options); + } + const [job] = await db + .select({ + phase: sql`${projectCleanupJobs.payload}->>'phase'`, + retryAfterSeconds: sql`greatest(0, ceil(extract(epoch FROM ${projectCleanupJobs.next_attempt_at}::timestamptz - now())))::integer`, + }) + .from(projectCleanupJobs) + .where(jobTarget(projectId)); + if (!job) { + return { status: 'done', gen: await issuedGeneration(db, projectId) }; + } + return { + status: 'pending', + phase: job.phase, + reason: 'waiting', + retryAfterSeconds: job.retryAfterSeconds, + }; +} + +/** Advances due rotation jobs, one at a time, within a Cloudflare call and time budget. */ +export async function reconcileProxyTokenRotations( + db: Db, + env: ProxyTokenRotationEnv, + limits: Readonly<{ + maxCfCalls?: number; + maxMs?: number; + waitMs?: number; + fetch?: typeof fetch; + }> = {}, +): Promise { + const budget: RotationBudget = { + cfCallsRemaining: limits.maxCfCalls ?? 100, + deadline: Date.now() + (limits.maxMs ?? 25_000), + }; + const summary: RotationRunSummary = { done: 0, pending: 0, failed: 0, retryAfterSeconds: null }; + // One attempt per job per run: a failed step can be due again at once. + const attempted = new Set(); + while (budget.cfCallsRemaining > 0 && Date.now() < budget.deadline) { + const projectId = await claimDueJob(db, undefined, attempted); + if (!projectId) { + break; + } + attempted.add(projectId); + const outcome = await runClaimed(db, env, projectId, budget, limits); + if (outcome.status === 'done') { + summary.done += 1; + continue; + } + if (outcome.status === 'failed') { + summary.failed += 1; + continue; + } + summary.pending += 1; + summary.retryAfterSeconds = + summary.retryAfterSeconds === null + ? outcome.retryAfterSeconds + : Math.min(summary.retryAfterSeconds, outcome.retryAfterSeconds); + if (outcome.reason === 'rate-limited' || outcome.reason === 'budget') { + break; + } + } + return summary; +} + +// --- Owner warning --- + +export function proxyTokenRotationStuckWarning(projectId: string): string { + return ( + "Replacing this project's platform credentials keeps failing. Until it finishes, older " + + 'credentials still work, including any a removed member may still hold. The platform keeps ' + + 'retrying. If this warning stays, ask your platform administrator to run ' + + `\`void platform system rotate-proxy-tokens --plan --project ${projectId}\`.` + ); +} + +/** + * Warnings for project owners and admins when the project's rotation keeps + * failing. Other roles and deploy tokens always get none, so they cannot tell + * whether a rotation is stuck. Busy steps, waits and budget cut-offs do not + * raise `attempts`, so a long queue of rotations does not warn. + */ +export async function proxyTokenRotationWarnings( + db: DbOrTx, + projectId: string, + role: ProjectRole, + deployScoped: boolean, +): Promise> { + if (deployScoped || !hasProjectCapability(role, 'members')) { + return []; + } + const [job] = await db + .select({ attempts: projectCleanupJobs.attempts }) + .from(projectCleanupJobs) + .where(and(jobTarget(projectId), eq(projectCleanupJobs.kind, PROXY_TOKEN_ROTATION_JOB_KIND))); + return job && job.attempts >= PROXY_TOKEN_ROTATION_STUCK_ATTEMPTS + ? [proxyTokenRotationStuckWarning(projectId)] + : []; +} diff --git a/platform/packages/api/src/proxy-token.ts b/platform/packages/api/src/proxy-token.ts index dd31d528c..541ad5fa6 100644 --- a/platform/packages/api/src/proxy-token.ts +++ b/platform/packages/api/src/proxy-token.ts @@ -81,6 +81,15 @@ export function serializeProxyTokenState(state: ProxyTokenState): string { return JSON.stringify({ gen: state.gen }); } +/** + * The floor a deleted project keeps. Issued generations are a Postgres + * `integer`, far below it, so no token is accepted. Deleting the key instead + * would bring back floor 0 and the legacy token. + */ +export const CLOSED_PROXY_TOKEN_STATE = serializeProxyTokenState({ + gen: Number.MAX_SAFE_INTEGER - 1, +}); + export function isAcceptedGeneration(state: ProxyTokenState, gen: number): boolean { return gen >= state.gen; } @@ -273,19 +282,6 @@ export async function verifyProxyToken( return true; } -/** - * Returns the token for the current floor (floor 0 is the legacy token). - * Stopgap until the issued generation is stored with the project. - */ -export async function issueProxyToken( - routing: Pick, - jwtSecret: string, - projectId: string, -): Promise { - const state = await readProxyTokenState(routing, projectId); - return deriveProxyToken(jwtSecret, projectId, state.gen); -} - // --- Isolate cache --- export type ProxyTokenStateCache = Readonly<{ diff --git a/platform/packages/api/src/routes/internal-resolve-worker.ts b/platform/packages/api/src/routes/internal-resolve-worker.ts index 5e00cbaa5..9dced7514 100644 --- a/platform/packages/api/src/routes/internal-resolve-worker.ts +++ b/platform/packages/api/src/routes/internal-resolve-worker.ts @@ -12,7 +12,8 @@ const app = new Hono(); * Authenticated by the proxy's own credential (see requireProxyInternalAuth). * * POST /internal/resolve-worker/:projectId - * Returns: { "workerName": "user--", "deploymentId": "..." } + * Returns: { "workerName": "user--", "deploymentId": "...", + * "proxyTokenGen": } * or 409 when the active deployment is static-only. */ app.post('/:projectId', async (context) => { @@ -47,6 +48,7 @@ app.post('/:projectId', async (context) => { return context.json({ workerName: getProjectDeploymentWorkerName(project.slug, deployment.id), deploymentId: deployment.id, + proxyTokenGen: deployment.proxy_token_gen, }); }); diff --git a/platform/packages/api/src/routes/project-team.ts b/platform/packages/api/src/routes/project-team.ts index 245a50472..d66e8c3a4 100644 --- a/platform/packages/api/src/routes/project-team.ts +++ b/platform/packages/api/src/routes/project-team.ts @@ -18,6 +18,7 @@ import { sendProjectInvitationEmail } from '../email.ts'; import type { ApiEnv } from '../env.ts'; import { cancelContainerBuildDos } from '../github/cancel-builds.ts'; import { PROJECT_PERMISSION_MESSAGES } from '../project-access.ts'; +import { proxyTokenRotationWarnings } from '../proxy-token-rotation.ts'; import type { ProjectInvitation, ProjectInvitationStatus, @@ -215,6 +216,12 @@ export const projectTeamRoutes = new Hono() return access; } const members = await getProjectMembers(context.get('db'), projectId); + const warnings = await proxyTokenRotationWarnings( + context.get('db'), + projectId, + access.role, + !!context.get('deployScope'), + ); return context.json({ members: [ { @@ -224,6 +231,7 @@ export const projectTeamRoutes = new Hono() }, ...members.map(({ member, user }) => memberJson(member, user)), ], + ...(warnings.length > 0 ? { warnings } : {}), }); }) diff --git a/platform/packages/api/src/routes/projects.ts b/platform/packages/api/src/routes/projects.ts index 193172644..59ca82040 100644 --- a/platform/packages/api/src/routes/projects.ts +++ b/platform/packages/api/src/routes/projects.ts @@ -87,6 +87,10 @@ import { isAlreadyExistsError as isDestinationAlreadyExistsError, } from '../provisioning/email-routing.ts'; import { isManagedSandbox, verifySandboxController } from '../provisioning/sandbox-controller.ts'; +import { + proxyTokenRotationWarnings, + requestRotationIfReplacingUntrustedCode, +} from '../proxy-token-rotation.ts'; import { EMAIL_RESERVED_SLUGS } from '../reserved-slugs.ts'; import { buildRoutingJson, @@ -977,9 +981,16 @@ const app = new Hono() manifest: sanitizeDeploymentManifest(deployment.manifest), })) : deployments; + const warnings = await proxyTokenRotationWarnings( + context.get('db'), + project.id, + access.role, + !!context.get('deployScope'), + ); return context.json({ deployments: sanitized, activeDeploymentId: project.active_deployment_id, + ...(warnings.length > 0 ? { warnings } : {}), }); }) @@ -1455,6 +1466,7 @@ const app = new Hono() // PG flip FIRST. If this throws, no external state is mutated // yet — the tx rolls back cleanly and the caller can retry. await updateProjectType(tx, projectId, targetType); + await requestRotationIfReplacingUntrustedCode(tx, projectId, target.id); await setActiveDeployment(tx, projectId, target.id); const snapshottedRoutingKeys = new Set(); @@ -1516,6 +1528,8 @@ const app = new Hono() // /__void/scheduled and /__void/queue handlers both validate // against the SAME __VOID_PROXY_TOKEN binding, so one lookup // is authoritative for both. + // The target may hold an older generation than the issued one; + // its copies get the token it holds. const needsProxyToken = !!manifest?.schedules?.length || !!manifest?.queues?.length; const targetProxyToken = needsProxyToken ? await resolveRollbackProxyToken(context.env, workerName, projectId) diff --git a/platform/packages/api/src/schema.ts b/platform/packages/api/src/schema.ts index 1741a4eb4..c00393460 100644 --- a/platform/packages/api/src/schema.ts +++ b/platform/packages/api/src/schema.ts @@ -306,6 +306,8 @@ export const projects = pgTable( type: text('mode').notNull().default('void').$type<'void' | 'spa' | 'static' | 'framework'>(), active_deployment_id: text('active_deployment_id'), websocket_version: integer('websocket_version').notNull().default(0), + // Highest proxy token generation issued for this project. 0 = legacy token. + proxy_token_gen: integer('proxy_token_gen').notNull().default(0), status: text('status').notNull().default('active').$type<'active' | 'deleting'>(), created_at: timestamp('created_at', { mode: 'string' }).notNull().defaultNow(), }, @@ -566,6 +568,9 @@ export const deployments = pgTable( // attribution survives account deletion. billing_user_id: text('billing_user_id'), initiated_by_user_id: text('initiated_by_user_id'), + // Proxy token generation this deployment's Worker holds. 0 = legacy token, + // which every Worker deployed before generations were stored holds. + proxy_token_gen: integer('proxy_token_gen').notNull().default(0), }, (table) => [ index('idx_deployments_project_id').on(table.project_id, table.created_at), @@ -634,8 +639,10 @@ export const projectCleanupJobs = pgTable( | 'worker-bundle' | 'project-assets' | 'routing-key' + | 'routing-put' | 'cron' | 'custom-hostname' + | 'proxy-token-rotation' >(), payload: jsonb('payload').notNull().$type>(), attempts: integer('attempts').notNull().default(0), diff --git a/platform/packages/api/test/integration/admin-dashboard.test.ts b/platform/packages/api/test/integration/admin-dashboard.test.ts index 6489fdb03..9c555df41 100644 --- a/platform/packages/api/test/integration/admin-dashboard.test.ts +++ b/platform/packages/api/test/integration/admin-dashboard.test.ts @@ -22,6 +22,7 @@ import { platformLoginRevisions, operatorEvents, projectAccessEvents, + projectCleanupJobs, projectInvitations, projectMembers, projectOwnershipPeriods, @@ -1871,6 +1872,7 @@ describe('GET /admin/projects/:id', () => { ); expect(roleChange.status).toBe(302); expect((await env.db.select().from(projectMembers))[0]?.role).toBe('collaborator'); + expect(await env.db.select().from(projectCleanupJobs)).toEqual([]); const ownerRemoval = await req( `/admin/projects/${project.id}/team/${owner.id}/remove`, @@ -1895,6 +1897,14 @@ describe('GET /admin/projects/:id', () => { ); expect(removal.status).toBe(302); expect(await env.db.select().from(projectMembers)).toEqual([]); + // The removed collaborator could deploy: the project's proxy token rotates. + expect(await env.db.select().from(projectCleanupJobs)).toMatchObject([ + { + target: `proxy-token-rotation:${project.id}`, + kind: 'proxy-token-rotation', + payload: { phase: 'requested', reason: 'member-removed' }, + }, + ]); expect( (await env.db.select().from(projects).where(eq(projects.id, project.id)))[0]?.user_id, ).toBe(owner.id); diff --git a/platform/packages/api/test/integration/api-delete.test.ts b/platform/packages/api/test/integration/api-delete.test.ts index 0f70d410d..6ebc6ceb7 100644 --- a/platform/packages/api/test/integration/api-delete.test.ts +++ b/platform/packages/api/test/integration/api-delete.test.ts @@ -13,6 +13,12 @@ import * as db from '../../src/db.ts'; import app, { FAST_CRON, scheduled } from '../../src/index.ts'; import * as projectAccess from '../../src/project-access.ts'; import { reconcileProjectCleanupJobs } from '../../src/provisioning/cleanup.ts'; +import { + CLOSED_PROXY_TOKEN_STATE, + deriveProxyToken, + readProxyTokenState, + verifyProxyToken, +} from '../../src/proxy-token.ts'; import { resources, projects, @@ -610,6 +616,85 @@ describe('DELETE /projects/:id', () => { expect(await env.ROUTING.get(project.slug)).toBe(JSON.stringify({ projectId: replacement.id })); }); + it('closes the proxy token floor and retries a failed write from the cleanup queue', async () => { + const user = await createTestUser(env.db); + const project = await createTestProject(env.db, user.id, 'proxy-token-floor'); + const floorKey = `proxy-token:${project.id}`; + await env.ROUTING.put(floorKey, JSON.stringify({ gen: 2 })); + mockCfApi(); + const putRoute = env.ROUTING.put.bind(env.ROUTING); + const routePut = vi.spyOn(env.ROUTING, 'put').mockImplementation(async (key, value) => { + if (key === floorKey) { + throw new Error('temporary KV failure'); + } + return putRoute(key, value); + }); + + const res = await app.fetch( + new Request(`http://localhost/projects/${project.id}`, { + method: 'DELETE', + headers: { Authorization: `Bearer ${user.jwt}` }, + }), + env, + ); + expect(res.status).toBe(200); + expect(await db.getProject(env.db, project.id)).toBeNull(); + expect(await env.ROUTING.get(floorKey, 'json')).toEqual({ gen: 2 }); + const pending = await env.db.execute<{ kind: string; payload: unknown }>(sql` + SELECT kind, payload FROM project_cleanup_jobs + WHERE target = ${`routing-put:${project.id}:${floorKey}`} + `); + expect(pending.rows).toEqual([ + { kind: 'routing-put', payload: { key: floorKey, value: CLOSED_PROXY_TOKEN_STATE } }, + ]); + + routePut.mockRestore(); + await scheduled( + { cron: FAST_CRON, scheduledTime: Date.now() } as ScheduledController, + env as unknown as Parameters[1], + createMockExecutionContext(), + ); + expect(await env.ROUTING.get(floorKey)).toBe(CLOSED_PROXY_TOKEN_STATE); + const finished = await env.db.execute(sql` + SELECT target FROM project_cleanup_jobs + WHERE target = ${`routing-put:${project.id}:${floorKey}`} + `); + expect(finished.rows).toEqual([]); + }); + + it.each([ + ['a never-rotated project', null, [0, 1]], + ['a rotated project', 2, [1, 2, 3]], + ])('rejects every proxy token of %s once it is deleted', async (_label, floor, gens) => { + const user = await createTestUser(env.db); + const project = await createTestProject(env.db, user.id, `token-after-delete-${floor ?? 0}`); + if (floor !== null) { + await env.ROUTING.put(`proxy-token:${project.id}`, JSON.stringify({ gen: floor })); + } + mockCfApi(); + const accepted = async (gen: number) => + verifyProxyToken({ + jwtSecret: env.JWT_SECRET, + projectId: project.id, + token: await deriveProxyToken(env.JWT_SECRET, project.id, gen), + readState: () => readProxyTokenState(env.ROUTING, project.id), + }); + expect(await accepted(gens.at(-1)!)).toBe(true); + + const res = await app.fetch( + new Request(`http://localhost/projects/${project.id}`, { + method: 'DELETE', + headers: { Authorization: `Bearer ${user.jwt}` }, + }), + env, + ); + expect(res.status).toBe(200); + vi.spyOn(console, 'warn').mockImplementation(() => {}); + for (const gen of gens) { + expect(await accepted(gen)).toBe(false); + } + }); + it('deletes Zero Trust applications only after retryable cleanup succeeds', async () => { const user = await createTestUser(env.db); const project = await createTestProject(env.db, user.id, 'zero-trust-teardown'); diff --git a/platform/packages/api/test/integration/api-deploy.test.ts b/platform/packages/api/test/integration/api-deploy.test.ts index 8b8b0966c..4eb48a668 100644 --- a/platform/packages/api/test/integration/api-deploy.test.ts +++ b/platform/packages/api/test/integration/api-deploy.test.ts @@ -12,7 +12,6 @@ import { vi, } from 'vite-plus/test'; import { computeAssetHash } from '../../src/assets.ts'; -import { hmacSignState } from '../../src/auth/hmac.ts'; import { beginDeploymentCutover, cancelDeployment, @@ -28,7 +27,7 @@ import app from '../../src/index.ts'; import { writePlanCatalog } from '../../src/plans.ts'; import * as projectAccess from '../../src/project-access.ts'; import * as projectIngress from '../../src/project-ingress.ts'; -import { deriveProxyToken, proxyTokenStateKey } from '../../src/proxy-token.ts'; +import { deriveProxyToken } from '../../src/proxy-token.ts'; import { validateManifestShape } from '../../src/routes/deploy.ts'; import { builds, @@ -41,6 +40,7 @@ import { projectZeroTrust, projectSecrets, platformSettings, + projectCleanupJobs, users, } from '../../src/schema.ts'; import { @@ -2002,15 +2002,15 @@ describe('POST /projects/:id/deploy', () => { workerName: expect.stringMatching(/^user-queue-kv-app-dpl_/), projectId: project.id, queueName: 'emails', - proxyToken: await hmacSignState(project.id, JWT_SECRET), + proxyToken: await deriveProxyToken(JWT_SECRET, project.id, 1), }); }); - it('issues the proxy token for the project accept floor', async () => { + it('issues generation 1 on the first deploy of a new project', async () => { const user = await createTestUser(env.db); const project = await createTestProject(env.db, user.id, 'queue-kv-floor-app'); - await env.ROUTING.put(proxyTokenStateKey(project.id), JSON.stringify({ gen: 1 })); const fetchMock = mockCfApi(); + const kvPut = vi.spyOn(env.ROUTING, 'put'); const res = await app.fetch( await buildDeployRequest(env, project.id, user.jwt, { @@ -2046,6 +2046,56 @@ describe('POST /projects/:id/deploy', () => { proxyToken: string; } | null; expect(queueRouting?.proxyToken).toBe(expected); + + const [row] = await env.db + .select({ gen: projects.proxy_token_gen }) + .from(projects) + .where(eq(projects.id, project.id)); + expect(row!.gen).toBe(1); + // The deployment records the generation its Worker was uploaded with. + expect( + await env.db + .select({ gen: deployments.proxy_token_gen }) + .from(deployments) + .where(eq(deployments.project_id, project.id)), + ).toEqual([{ gen: 1 }]); + expect(kvPut.mock.calls.map(([key]) => key)).not.toContainEqual( + expect.stringMatching(/^proxy-token:/), + ); + expect( + await env.db + .select() + .from(projectCleanupJobs) + .where(eq(projectCleanupJobs.kind, 'proxy-token-rotation')), + ).toEqual([]); + }); + + it('moves a project on the legacy token to generation 1 without a rotation', async () => { + const user = await createTestUser(env.db); + const project = await createTestProject(env.db, user.id, 'legacy-redeploy-app'); + mockCfApi(); + const deploy = async () => { + const res = await app.fetch(await buildDeployRequest(env, project.id, user.jwt), env); + expect(res.status).toBe(200); + await res.text(); + }; + await deploy(); + // A project whose Workers predate token generations. + await env.db.update(projects).set({ proxy_token_gen: 0 }).where(eq(projects.id, project.id)); + + await deploy(); + + const [row] = await env.db + .select({ gen: projects.proxy_token_gen }) + .from(projects) + .where(eq(projects.id, project.id)); + expect(row!.gen).toBe(1); + expect( + await env.db + .select() + .from(projectCleanupJobs) + .where(eq(projectCleanupJobs.kind, 'proxy-token-rotation')), + ).toEqual([]); }); it('removes queue routing entries omitted by the next deployment', async () => { diff --git a/platform/packages/api/test/integration/api-internal-project-token.test.ts b/platform/packages/api/test/integration/api-internal-project-token.test.ts index 4b0160fc5..b5763fb8d 100644 --- a/platform/packages/api/test/integration/api-internal-project-token.test.ts +++ b/platform/packages/api/test/integration/api-internal-project-token.test.ts @@ -1,3 +1,4 @@ +import { eq } from 'drizzle-orm'; import { afterAll, afterEach, @@ -17,6 +18,7 @@ import { deriveProxyToken, type ProxyInternalRoute, } from '../../src/proxy-token.ts'; +import { deployments, projects } from '../../src/schema.ts'; import { clearTables, createTestDeployment, @@ -131,17 +133,30 @@ describe('per-project token checks', () => { expect(run).toHaveBeenCalledOnce(); }); - it.each(['revalidate', 'resolve-resource', 'resolve-worker', 'verify-prerender-build'])( - 'rejects a login JWT presented as a project id and token at /internal/%s', + it('rejects a login JWT presented as a project id and token at /internal/revalidate', async () => { + const forged = await loginJwtAsProjectToken(); + const response = await internalRequest('revalidate', forged.projectId, forged.token); + expect(response.status).toBe(403); + expect(await response.json()).toEqual({ error: 'invalid internal auth' }); + }); + + it.each(['resolve-resource', 'resolve-worker', 'verify-prerender-build'])( + 'rejects a login JWT presented as a project id and credential at /internal/%s', async (route) => { const forged = await loginJwtAsProjectToken(); - const response = await internalRequest(route, forged.projectId, forged.token); - expect(response.status).toBe(403); - expect(await response.json()).toEqual({ error: 'invalid internal auth' }); + const asProxy = await internalRequest(route, forged.projectId, { + 'x-void-proxy-internal': forged.token, + }); + expect(asProxy.status).toBe(403); + expect(await asProxy.json()).toEqual({ error: 'invalid internal auth' }); + + const asProject = await internalRequest(route, forged.projectId, forged.token); + expect(asProject.status).toBe(401); + expect(await asProject.json()).toEqual({ error: 'missing internal auth' }); }, ); - it('accepts a generated project id with its token at every internal route', async () => { + it('accepts a generated project id at every internal route', async () => { const user = await createTestUser(env.db); const project = await createTestProject(env.db, user.id); const deployment = await createTestDeployment(env.db, project.id, 'live'); @@ -149,29 +164,42 @@ describe('per-project token checks', () => { await env.ROUTING.put(project.slug, JSON.stringify({ version: deployment.id })); mockCfApi(); const token = await hmacSignState(project.id, JWT_SECRET); + const proxyAuth = async (route: ProxyInternalRoute) => ({ + 'x-void-proxy-internal': await deriveProxyInternalAuth(JWT_SECRET, route, project.id), + }); const revalidated = await internalRequest('revalidate', project.id, token, { paths: ['/about'], }); expect(revalidated.status).toBe(200); - const resource = await internalRequest('resolve-resource', project.id, token, { - type: 'kv', - binding: 'CACHE', - }); + const resource = await internalRequest( + 'resolve-resource', + project.id, + await proxyAuth('resolve-resource'), + { type: 'kv', binding: 'CACHE' }, + ); expect(resource.status).toBe(200); expect(await resource.json()).toEqual({ cf_id: 'cf-kv-id' }); - const worker = await internalRequest('resolve-worker', project.id, token); + const worker = await internalRequest( + 'resolve-worker', + project.id, + await proxyAuth('resolve-worker'), + ); expect(worker.status).toBe(200); expect(await worker.json()).toEqual({ workerName: `user-${project.slug}-${deployment.id}`, deploymentId: deployment.id, + proxyTokenGen: 0, }); - const prerender = await internalRequest('verify-prerender-build', project.id, token, { - userId: user.id, - }); + const prerender = await internalRequest( + 'verify-prerender-build', + project.id, + await proxyAuth('verify-prerender-build'), + { userId: user.id }, + ); expect(prerender.status).toBe(200); expect(await prerender.json()).toEqual({ ok: true }); }); @@ -286,11 +314,12 @@ describe('proxy credential on the metadata routes', () => { 'resolve-worker': { workerName: `user-${project.slug}-${deployment.id}`, deploymentId: deployment.id, + proxyTokenGen: 0, }, 'resolve-resource': { cf_id: 'cf-kv-id' }, 'verify-prerender-build': { ok: true }, }; - return { project, bodies, expected }; + return { project, deployment, bodies, expected }; } function floorReads(get: { mock: { calls: Array> } }) { @@ -319,6 +348,65 @@ describe('proxy credential on the metadata routes', () => { }, ); + async function resolvedGeneration(projectId: string): Promise { + const response = await internalRequest( + 'resolve-worker', + projectId, + { + 'x-void-proxy-internal': await deriveProxyInternalAuth( + JWT_SECRET, + 'resolve-worker', + projectId, + ), + }, + {}, + ); + expect(response.status).toBe(200); + return ((await response.json()) as { proxyTokenGen?: unknown }).proxyTokenGen; + } + + async function setGenerations( + projectId: string, + issued: number, + held: Readonly>, + ): Promise { + await env.db + .update(projects) + .set({ proxy_token_gen: issued }) + .where(eq(projects.id, projectId)); + for (const [deploymentId, gen] of Object.entries(held)) { + await env.db + .update(deployments) + .set({ proxy_token_gen: gen }) + .where(eq(deployments.id, deploymentId)); + } + } + + it('reports the generation the active Worker holds from /internal/resolve-worker', async () => { + const { project, deployment } = await liveProject(); + await setGenerations(project.id, 3, { [deployment.id]: 3 }); + expect(await resolvedGeneration(project.id)).toBe(3); + }); + + it('reports the held generation, not the issued one, when they differ', async () => { + const { project, deployment } = await liveProject(); + await setGenerations(project.id, 2, { [deployment.id]: 0 }); + expect(await resolvedGeneration(project.id)).toBe(0); + }); + + it('reports generation 0 after a rollback to a Worker that still holds the legacy token', async () => { + const { project, deployment: older } = await liveProject(); + const newer = await createTestDeployment(env.db, project.id, 'live'); + await setGenerations(project.id, 1, { [older.id]: 0, [newer.id]: 1 }); + expect(await resolvedGeneration(project.id)).toBe(1); + + await env.db + .update(projects) + .set({ active_deployment_id: older.id }) + .where(eq(projects.id, project.id)); + expect(await resolvedGeneration(project.id)).toBe(0); + }); + it.each(routes)( 'rejects a proxy credential made for another route at /internal/%s', async (route) => { @@ -357,40 +445,21 @@ describe('proxy credential on the metadata routes', () => { }, ); - it.each(routes)('applies the floor to a project token at /internal/%s', async (route) => { - const { project, bodies, expected } = await liveProject(); - const legacy = await hmacSignState(project.id, JWT_SECRET); - - const accepted = await internalRequest(route, project.id, legacy, bodies[route]); - expect(accepted.status).toBe(200); - expect(await accepted.json()).toEqual(expected[route]); - - await env.ROUTING.put(`proxy-token:${project.id}`, '{"gen":1}'); - const rejected = await internalRequest(route, project.id, legacy, bodies[route]); - expect(rejected.status).toBe(403); - expect(await rejected.json()).toEqual({ error: 'invalid internal auth' }); - - await env.ROUTING.put(`proxy-token:${project.id}`, 'not json'); - const unavailable = await internalRequest(route, project.id, legacy, bodies[route]); - expect(unavailable.status).toBe(503); - expect(await unavailable.json()).toEqual({ error: 'project token state unavailable' }); - }); - - it.each(routes)('requires internal auth at /internal/%s', async (route) => { + it.each(routes)('requires the proxy credential at /internal/%s', async (route) => { const { project, bodies } = await liveProject(); const missing = await internalRequest(route, project.id, {}, bodies[route]); expect(missing.status).toBe(401); expect(await missing.json()).toEqual({ error: 'missing internal auth' }); const get = vi.spyOn(env.ROUTING, 'get'); - const forged = await internalRequest( - route, - project.id, - await hmacSignState('proj_000000000999', JWT_SECRET), - bodies[route], - ); - expect(forged.status).toBe(403); - expect(await forged.json()).toEqual({ error: 'invalid internal auth' }); + for (const token of [ + await hmacSignState(project.id, JWT_SECRET), + await deriveProxyToken(JWT_SECRET, project.id, 1), + ]) { + const projectToken = await internalRequest(route, project.id, token, bodies[route]); + expect(projectToken.status).toBe(401); + expect(await projectToken.json()).toEqual({ error: 'missing internal auth' }); + } expect(floorReads(get)).toEqual([]); }); }); diff --git a/platform/packages/api/test/integration/api-project-team.test.ts b/platform/packages/api/test/integration/api-project-team.test.ts index 89106c410..51a61fac5 100644 --- a/platform/packages/api/test/integration/api-project-team.test.ts +++ b/platform/packages/api/test/integration/api-project-team.test.ts @@ -12,14 +12,20 @@ import { import { createGitHubConnection, getBuild, upsertGitHubInstallation } from '../../src/db.ts'; import app from '../../src/index.ts'; import * as projectAccess from '../../src/project-access.ts'; +import { + proxyTokenRotationStuckWarning, + proxyTokenRotationTarget, +} from '../../src/proxy-token-rotation.ts'; import { projectAccessEvents, + projectCleanupJobs, projectInvitations, projectMembers, projects, users, } from '../../src/schema.ts'; import { + captureQueries, createTestBuild, clearTables, createTestProject, @@ -1176,3 +1182,167 @@ describe('project team API', () => { expect(await env.db.select().from(projectMembers)).toEqual([]); }); }); + +describe('proxy token rotation on member changes', () => { + async function rotationJob(projectId: string) { + const [job] = await env.db + .select() + .from(projectCleanupJobs) + .where(eq(projectCleanupJobs.target, proxyTokenRotationTarget(projectId))); + return job; + } + + async function projectWithMember(role: 'reader' | 'collaborator' | 'admin') { + const owner = await createTestUser(env.db, 'rotation-owner'); + const member = await createTestUser(env.db, `rotation-${role}`); + const project = await createTestProject(env.db, owner.id); + await env.db.insert(projectMembers).values({ + project_id: project.id, + user_id: member.id, + role, + added_by_user_id: owner.id, + }); + return { owner, member, project }; + } + + function change( + projectId: string, + jwt: string, + userId: string, + next: 'reader' | 'collaborator' | 'admin' | null, + ): Promise { + return Promise.resolve( + app.fetch( + request( + `/projects/${projectId}/members/${userId}`, + jwt, + next ? { method: 'PATCH', body: { role: next } } : { method: 'DELETE' }, + ), + env, + ), + ); + } + + it.each([ + { role: 'collaborator', next: null, reason: 'member-removed' }, + { role: 'admin', next: null, reason: 'member-removed' }, + { role: 'collaborator', next: 'reader', reason: 'member-role-changed' }, + { role: 'admin', next: 'reader', reason: 'member-role-changed' }, + ] as const)('requests a rotation when $role -> $next', async ({ role, next, reason }) => { + const { owner, member, project } = await projectWithMember(role); + const response = await change(project.id, owner.jwt, member.id, next); + expect(response.ok).toBe(true); + expect(await rotationJob(project.id)).toMatchObject({ + kind: 'proxy-token-rotation', + project_id: project.id, + payload: { phase: 'requested', reason }, + }); + }); + + it.each([ + { role: 'reader', next: null }, + { role: 'reader', next: 'collaborator' }, + { role: 'collaborator', next: 'admin' }, + ] as const)('requests nothing when $role -> $next', async ({ role, next }) => { + const { owner, member, project } = await projectWithMember(role); + const response = await change(project.id, owner.jwt, member.id, next); + expect(response.ok).toBe(true); + expect(await rotationJob(project.id)).toBeUndefined(); + }); + + it.each([ + { role: 'collaborator', rotates: true }, + { role: 'reader', rotates: false }, + ] as const)('requests a rotation when a $role leaves: $rotates', async ({ role, rotates }) => { + const { member, project } = await projectWithMember(role); + const response = await app.fetch( + request(`/projects/${project.id}/leave`, member.jwt, { method: 'POST' }), + env, + ); + expect(response.status).toBe(204); + const job = await rotationJob(project.id); + if (rotates) { + expect(job?.payload).toEqual({ phase: 'requested', reason: 'member-removed' }); + } else { + expect(job).toBeUndefined(); + } + }); + + it('requests nothing when the member change fails', async () => { + const { owner, member, project } = await projectWithMember('admin'); + const conflict = await change(project.id, member.jwt, owner.id, null); + expect(conflict.status).toBe(409); + expect(await rotationJob(project.id)).toBeUndefined(); + }); + + it('writes the request under the project lock', async () => { + const { owner, member, project } = await projectWithMember('collaborator'); + const { result, queries } = await captureQueries(env.db, () => + change(project.id, owner.jwt, member.id, null), + ); + expect(result.status).toBe(204); + const lockAt = queries.findIndex((query) => /from "projects".* for update/i.test(query)); + const insertAt = queries.findIndex((query) => + /insert into "project_cleanup_jobs"/i.test(query), + ); + expect(lockAt).toBeGreaterThanOrEqual(0); + expect(insertAt).toBeGreaterThan(lockAt); + }); +}); + +describe('stuck rotation warning on the member list', () => { + it('warns only owners and admins', async () => { + const owner = await createTestUser(env.db, 'stuck-owner'); + const project = await createTestProject(env.db, owner.id); + const jwts: Record<'reader' | 'collaborator' | 'admin', string> = { + reader: '', + collaborator: '', + admin: '', + }; + for (const role of ['reader', 'collaborator', 'admin'] as const) { + const user = await createTestUser(env.db, `stuck-${role}`); + await env.db.insert(projectMembers).values({ + project_id: project.id, + user_id: user.id, + role, + added_by_user_id: owner.id, + }); + jwts[role] = user.jwt; + } + + const listAs = async (jwt: string) => { + const response = await app.fetch(request(`/projects/${project.id}/members`, jwt), env); + expect(response.status).toBe(200); + const text = await response.text(); + expect(text).not.toContain('cf boom'); + expect(text).not.toContain('user-x-dep'); + expect(text).not.toContain('mock-proxy-token'); + return JSON.parse(text) as { members: Array; warnings?: Array }; + }; + + expect(await listAs(owner.jwt)).not.toHaveProperty('warnings'); + + await env.db.insert(projectCleanupJobs).values({ + target: proxyTokenRotationTarget(project.id), + project_id: project.id, + kind: 'proxy-token-rotation', + payload: { phase: 'patch', reason: 'member-removed' }, + attempts: 3, + last_error: 'cf boom user-x-dep', + }); + expect(await listAs(owner.jwt)).not.toHaveProperty('warnings'); + + await env.db + .update(projectCleanupJobs) + .set({ attempts: 4 }) + .where(eq(projectCleanupJobs.target, proxyTokenRotationTarget(project.id))); + for (const jwt of [owner.jwt, jwts.admin]) { + const body = await listAs(jwt); + expect(body.warnings).toEqual([proxyTokenRotationStuckWarning(project.id)]); + expect(body.members).toHaveLength(4); + } + for (const jwt of [jwts.reader, jwts.collaborator]) { + expect(await listAs(jwt)).not.toHaveProperty('warnings'); + } + }); +}); diff --git a/platform/packages/api/test/integration/api-projects.test.ts b/platform/packages/api/test/integration/api-projects.test.ts index 0f34016b5..a2b8b81e9 100644 --- a/platform/packages/api/test/integration/api-projects.test.ts +++ b/platform/packages/api/test/integration/api-projects.test.ts @@ -18,7 +18,11 @@ import { import * as db from '../../src/db.ts'; import app from '../../src/index.ts'; import * as projectAccess from '../../src/project-access.ts'; -import { deriveProxyInternalAuth } from '../../src/proxy-token.ts'; +import { + proxyTokenRotationStuckWarning, + proxyTokenRotationTarget, +} from '../../src/proxy-token-rotation.ts'; +import { deriveProxyInternalAuth, deriveProxyToken } from '../../src/proxy-token.ts'; import { users } from '../../src/schema.ts'; import { deployments, @@ -31,6 +35,7 @@ import { projectDeployCredentials, platformLoginConnections, platformSettings, + projectCleanupJobs, } from '../../src/schema.ts'; import { createTestBuild, @@ -2271,6 +2276,76 @@ describe('GET /projects/:id/deployments', () => { ssr: true, }); }); + + it('warns only owners and admins about a stuck credential rotation', async () => { + const owner = await createTestUser(env.db, 'stuck-owner'); + const project = await createTestProject(env.db, owner.id, 'stuck-project'); + const members: Record<'reader' | 'collaborator' | 'admin', string> = { + reader: '', + collaborator: '', + admin: '', + }; + for (const role of ['reader', 'collaborator', 'admin'] as const) { + const user = await createTestUser(env.db, `stuck-${role}`); + await env.db.insert(projectMembers).values({ + project_id: project.id, + user_id: user.id, + role, + added_by_user_id: owner.id, + }); + members[role] = user.jwt; + } + const build = await createTestBuild(env.db, { + projectId: project.id, + userId: owner.id, + status: 'building', + }); + const deployToken = await signDeployToken( + { projectId: project.id, userId: owner.id, login: owner.login, buildId: build.id }, + env.JWT_SECRET, + ); + await createDeployment(env.db, project.id, JSON.stringify({ version: 3, bindings: {} })); + + const listAs = async (jwt: string) => { + const response = await app.fetch( + new Request(`http://localhost/projects/${project.id}/deployments`, { + headers: { Authorization: `Bearer ${jwt}` }, + }), + env, + ); + expect(response.status).toBe(200); + const text = await response.text(); + expect(text).not.toContain('cf boom'); + expect(text).not.toContain('user-x-dep'); + expect(text).not.toContain('mock-proxy-token'); + return JSON.parse(text) as { deployments: Array; warnings?: Array }; + }; + + expect(await listAs(owner.jwt)).not.toHaveProperty('warnings'); + + await env.db.insert(projectCleanupJobs).values({ + target: proxyTokenRotationTarget(project.id), + project_id: project.id, + kind: 'proxy-token-rotation', + payload: { phase: 'patch', reason: 'member-removed' }, + attempts: 3, + last_error: 'cf boom user-x-dep', + }); + expect(await listAs(owner.jwt)).not.toHaveProperty('warnings'); + + await env.db + .update(projectCleanupJobs) + .set({ attempts: 4 }) + .where(eq(projectCleanupJobs.target, proxyTokenRotationTarget(project.id))); + for (const jwt of [owner.jwt, members.admin]) { + const body = await listAs(jwt); + expect(body.warnings).toEqual([proxyTokenRotationStuckWarning(project.id)]); + expect(body.deployments).toHaveLength(1); + } + for (const jwt of [members.reader, members.collaborator, deployToken]) { + expect(await listAs(jwt)).not.toHaveProperty('warnings'); + } + }); }); describe('POST /projects/:id/rollback', () => { @@ -3138,6 +3213,87 @@ describe('POST /projects/:id/rollback', () => { proxyToken: 'token-baked-into-d1', }); }); + + it.each([ + { issued: 1, held: 0 }, + { issued: 1, held: 1 }, + { issued: 0, held: 0 }, + ])( + 'rollback to a Worker on generation $held while generation $issued is issued writes only its token', + async ({ issued, held }) => { + const user = await createTestUser(env.db); + const project = await createTestProject(env.db, user.id, 'roll-token-gen'); + await env.db + .update(projects) + .set({ proxy_token_gen: issued }) + .where(eq(projects.id, project.id)); + const queueRoutingKey = `queue:${project.id.replaceAll('_', '-')}-emails`; + await env.db.insert(resources).values({ + id: 'res_roll_token_gen', + project_id: project.id, + type: 'queue', + cf_id: '25832cb0ce3d4f60b1e538911a4629f4', + binding_name: 'emails', + }); + const manifest = JSON.stringify({ + version: 3, + bindings: {}, + schedules: [{ cron: '*/5 * * * *', filePath: 'crons/tick.ts' }], + queues: [{ name: 'emails', bindingName: 'QUEUE_EMAILS' }], + }); + const d1 = await createDeployment(env.db, project.id, manifest); + await updateDeploymentStatus(env.db, d1.id, 'live'); + await markDeploymentRetained(env.db, d1.id, true); + const d2 = await createDeployment(env.db, project.id, manifest); + await updateDeploymentStatus(env.db, d2.id, 'live'); + await markDeploymentRetained(env.db, d2.id, true); + await setActiveDeployment(env.db, project.id, d2.id); + + const heldToken = await deriveProxyToken(env.JWT_SECRET, project.id, held); + mockCfApi({ + listWorkerBindings: () => + cfSuccess([{ type: 'plain_text', name: '__VOID_PROXY_TOKEN', text: heldToken }]), + }); + const syncBodies: Array = []; + const originalGet = env.CRON_SCHEDULER.get.bind(env.CRON_SCHEDULER); + vi.spyOn(env.CRON_SCHEDULER, 'get').mockImplementation((id) => { + const stub = originalGet(id); + return { + ...stub, + fetch: async (req: Request) => { + if (new URL(req.url).pathname === '/sync') { + syncBodies.push(await req.text()); + } + return new Response('{}', { status: 200 }); + }, + } as unknown as DurableObjectStub; + }); + + const res = await app.fetch( + new Request(`http://localhost/projects/${project.id}/rollback`, { + method: 'POST', + headers: { Authorization: `Bearer ${user.jwt}`, 'Content-Type': 'application/json' }, + body: JSON.stringify({ deploymentId: d1.id }), + }), + env, + ); + expect(res.status).toBe(200); + + expect(await env.ROUTING.get(queueRoutingKey, 'json')).toEqual({ + workerName: `user-roll-token-gen-${d1.id}`, + projectId: project.id, + queueName: 'emails', + proxyToken: heldToken, + }); + expect(syncBodies).toHaveLength(1); + const sync = JSON.parse(syncBodies[0]!) as Record; + // Exactly one token: the one the Worker holds. + expect(Object.keys(sync).filter((key) => key.startsWith('proxyToken'))).toEqual([ + 'proxyToken', + ]); + expect(sync.proxyToken).toBe(heldToken); + }, + ); }); describe('GET /projects/:id/logs', () => { diff --git a/platform/packages/api/test/integration/api-resolve-worker.test.ts b/platform/packages/api/test/integration/api-resolve-worker.test.ts index 004a5caa2..1d234babf 100644 --- a/platform/packages/api/test/integration/api-resolve-worker.test.ts +++ b/platform/packages/api/test/integration/api-resolve-worker.test.ts @@ -51,6 +51,7 @@ describe('POST /internal/resolve-worker/:projectId', () => { expect(await current.json()).toEqual({ workerName: `user-${project.slug}-${newer.id}`, deploymentId: newer.id, + proxyTokenGen: 0, }); await setActiveDeployment(env.db, project.id, rollbackTarget.id); @@ -59,6 +60,7 @@ describe('POST /internal/resolve-worker/:projectId', () => { expect(await rolledBack.json()).toEqual({ workerName: `user-${project.slug}-${rollbackTarget.id}`, deploymentId: rollbackTarget.id, + proxyTokenGen: 0, }); }); diff --git a/platform/packages/api/test/integration/deployment-recovery.test.ts b/platform/packages/api/test/integration/deployment-recovery.test.ts index d2dad46f1..044f67dac 100644 --- a/platform/packages/api/test/integration/deployment-recovery.test.ts +++ b/platform/packages/api/test/integration/deployment-recovery.test.ts @@ -1,14 +1,18 @@ import { eq } from 'drizzle-orm'; import { afterAll, afterEach, beforeAll, beforeEach, expect, test, vi } from 'vite-plus/test'; import { getDeployment, getProject, type Db } from '../../src/db.ts'; +import { replayDeploymentCutover } from '../../src/deploy/recovery.ts'; import { settleExpiredDeployment } from '../../src/deployment-recovery.ts'; import type { ApiEnv } from '../../src/env.ts'; +import { deriveProxyToken } from '../../src/proxy-token.ts'; import { deployments, domains, projectMigrations, projects, resources } from '../../src/schema.ts'; import { clearTables, createTestDeployment, createTestProject, createTestUser, + cfSuccess, + mockCfApi, setupEnv, teardownEnv, type TestEnv, @@ -289,3 +293,80 @@ test('preserves retained historical deployment artifacts while terminalizing its expect(fetch).not.toHaveBeenCalled(); expect(await env.ROUTING.get(`manifest:${candidate.id}`)).toBe('{}'); }); + +test.each([ + { issued: 1, held: 0 }, + { issued: 1, held: 1 }, + { issued: 0, held: 0 }, +])( + 'cutover replay to a Worker on generation $held while generation $issued is issued writes only its token', + async ({ issued, held }) => { + await env.db + .update(projects) + .set({ proxy_token_gen: issued }) + .where(eq(projects.id, project.id)); + const deployment = await createTestDeployment(env.db, project.id); + await env.db + .update(deployments) + .set({ + manifest: JSON.stringify({ + version: 3, + type: 'void', + bindings: {}, + schedules: [{ cron: '*/5 * * * *', filePath: 'crons/tick.ts' }], + queues: [{ name: 'emails', bindingName: 'QUEUE_EMAILS' }], + }), + }) + .where(eq(deployments.id, deployment.id)); + await env.db.insert(resources).values({ + id: 'resource-replay-queue', + project_id: project.id, + type: 'queue', + cf_id: '25832cb0ce3d4f60b1e538911a4629f4', + binding_name: 'emails', + }); + const heldToken = await deriveProxyToken(env.JWT_SECRET, project.id, held); + mockCfApi({ + listWorkerBindings: () => + cfSuccess([{ type: 'plain_text', name: '__VOID_PROXY_TOKEN', text: heldToken }]), + }); + const syncBodies: Array = []; + const originalGet = env.CRON_SCHEDULER.get.bind(env.CRON_SCHEDULER); + vi.spyOn(env.CRON_SCHEDULER, 'get').mockImplementation((id) => { + const stub = originalGet(id); + return { + ...stub, + fetch: async (request: Request) => { + if (new URL(request.url).pathname === '/sync') { + syncBodies.push(await request.text()); + } + return new Response('{}', { status: 200 }); + }, + } as unknown as DurableObjectStub; + }); + + const lockedProject = (await getProject(env.db, project.id))!; + await replayDeploymentCutover( + env as unknown as ApiEnv['Bindings'], + env.db, + lockedProject, + (await getDeployment(env.db, deployment.id))!, + lockedProject.type, + { restoreCommitted: true }, + ); + + expect( + await env.ROUTING.get(`queue:${project.id.replaceAll('_', '-')}-emails`, 'json'), + ).toEqual({ + workerName: `user-${project.slug}-${deployment.id}`, + projectId: project.id, + queueName: 'emails', + proxyToken: heldToken, + }); + expect(syncBodies).toHaveLength(1); + const sync = JSON.parse(syncBodies[0]!) as Record; + // Exactly one token: the one the Worker holds. + expect(Object.keys(sync).filter((key) => key.startsWith('proxyToken'))).toEqual(['proxyToken']); + expect(sync.proxyToken).toBe(heldToken); + }, +); diff --git a/platform/packages/api/test/integration/helpers.ts b/platform/packages/api/test/integration/helpers.ts index bef0ab931..1a9f63dc3 100644 --- a/platform/packages/api/test/integration/helpers.ts +++ b/platform/packages/api/test/integration/helpers.ts @@ -5,6 +5,7 @@ import { drizzle } from 'drizzle-orm/pglite'; import pg from 'pg'; import { vi } from 'vite-plus/test'; import { signJwt } from '../../src/auth/jwt.ts'; +import type { CronSyncState } from '../../src/cron-scheduler-client.ts'; import type { Db } from '../../src/db.ts'; import { generateId } from '../../src/id.ts'; import { encryptProjectSecret } from '../../src/project-secrets.ts'; @@ -436,6 +437,47 @@ export async function captureQueries( } } +/** Queries captured so far by the active `captureQueries` on `db`. */ +export function capturedQueryCount(db: Db): number { + const capture = queryCaptures.get(db); + if (!capture?.active) { + throw new Error('no query capture is active'); + } + return capture.queries.length; +} + +/** + * Stateful CronScheduler namespace: `/sync` stores the posted state per + * project, `/snapshot` returns it, `/clear` removes it. Records every call. + */ +export function createCronSchedulerMock(): { + namespace: DurableObjectNamespace; + states: Map; + calls: Array<{ name: string; path: string }>; +} { + const states = new Map(); + const calls: Array<{ name: string; path: string }> = []; + const namespace = { + idFromName: (name: string) => ({ toString: () => name }), + get: (id: { toString(): string }) => ({ + fetch: async (request: Request) => { + const name = id.toString(); + const path = new URL(request.url).pathname; + calls.push({ name, path }); + if (path === '/sync') { + states.set(name, (await request.json()) as CronSyncState); + } else if (path === '/clear') { + states.delete(name); + } else if (path === '/snapshot') { + return Response.json({ state: states.get(name) ?? null }); + } + return Response.json({ ok: true }); + }, + }), + } as unknown as DurableObjectNamespace; + return { namespace, states, calls }; +} + export async function setupEnv(): Promise { const kv = new MemoryKVNamespace(); const isrCache = new MemoryKVNamespace(); @@ -1056,6 +1098,7 @@ export type CfApiMocks = { // Dispatch script / container-app endpoints. State-aware defaults are wired // up when overrides are not supplied. Tests can still override individually. getDispatchScript?: (url: string, init?: RequestInit) => Response | Promise; + patchScriptSettings?: (url: string, init?: RequestInit) => Response | Promise; createContainerApp?: (url: string, init?: RequestInit) => Response | Promise; patchContainerApp?: (url: string, init?: RequestInit) => Response | Promise; deleteContainerApp?: (url: string, init?: RequestInit) => Response | Promise; @@ -1367,6 +1410,42 @@ export function mockCfApi(overrides: Partial = {}): MockCfApiHandle }); }; + // PATCH dispatch script settings — the binding list replaces the stored one: + // `inherit` keeps the stored binding of that name (unknown name → 400) and a + // name left out is removed. Echoes the new bindings like Cloudflare does. + const defaultPatchScriptSettings = async (url: string, init?: RequestInit): Promise => { + const scriptName = parseScriptNameFromUrl(url); + const script = scriptName ? state.hostScripts.get(scriptName) : undefined; + if (!script) { + return cfError('workers.api.error.script_not_found', 404, 10007); + } + const part = init?.body instanceof FormData ? init.body.get('settings') : null; + const settings = ( + part instanceof Blob + ? JSON.parse(await part.text()) + : typeof part === 'string' + ? JSON.parse(part) + : null + ) as { bindings?: Array<{ type: string; name: string; [key: string]: unknown }> } | null; + if (!Array.isArray(settings?.bindings)) { + return cfError('settings.bindings is required', 400); + } + const bindings: SandboxMockHostScript['bindings'] = []; + for (const binding of settings.bindings) { + if (binding.type !== 'inherit') { + bindings.push(binding); + continue; + } + const stored = script.bindings.find((entry) => entry.name === binding.name); + if (!stored) { + return cfError(`cannot inherit unknown binding ${binding.name}`, 400); + } + bindings.push(stored); + } + script.bindings = bindings; + return cfSuccess({ bindings }); + }; + // Container app handlers. const defaultListContainerApps = (url: string): Response => { const name = parseAppNameQuery(url); @@ -1513,6 +1592,7 @@ export function mockCfApi(overrides: Partial = {}): MockCfApiHandle listWorkerBindings: defaultListWorkerBindings, deleteWorker: defaultDeleteWorker, getDispatchScript: defaultGetDispatchScript, + patchScriptSettings: defaultPatchScriptSettings, createContainerApp: defaultCreateContainerApp, patchContainerApp: defaultPatchContainerApp, deleteContainerApp: defaultDeleteContainerApp, @@ -1665,6 +1745,9 @@ export function mockCfApi(overrides: Partial = {}): MockCfApiHandle if (method === 'GET' && url.includes('/workers/dispatch/') && url.includes('/bindings')) { return mocks.listWorkerBindings(url, init); } + if (method === 'PATCH' && url.includes('/workers/dispatch/') && url.endsWith('/settings')) { + return mocks.patchScriptSettings!(url, init); + } // GET on a dispatch script (no /bindings, no /secrets) — read the // script's metadata + migration_tag. Must come AFTER /bindings and // /secrets matches because those URLs share the prefix. diff --git a/platform/packages/api/test/integration/operator-audit.test.ts b/platform/packages/api/test/integration/operator-audit.test.ts index 62ffa663a..dea2bd8bf 100644 --- a/platform/packages/api/test/integration/operator-audit.test.ts +++ b/platform/packages/api/test/integration/operator-audit.test.ts @@ -412,3 +412,25 @@ test('records only the project and registered user IDs for ownership transfer', 'must-not-be-audited', ); }); + +test('records the project scope of a proxy token rotation and nothing for its preview', async () => { + const owner = await createTestUser(env.db, 'audit-rotation-owner'); + const project = await createTestProject(env.db, owner.id, 'audit-rotation-project'); + + expect( + (await request('/system/rotate-proxy-tokens', { dryRun: true, project: project.id })).status, + ).toBe(200); + expect(await env.db.select().from(operatorEvents)).toHaveLength(0); + + expect((await request('/system/rotate-proxy-tokens', { project: project.id })).status).toBe(207); + expect((await request('/system/rotate-proxy-tokens', { all: true })).status).toBe(207); + + const events = await env.db.select().from(operatorEvents); + expect(events.map((event) => [event.action, JSON.parse(event.target), event.status])).toEqual( + expect.arrayContaining([ + ['system/rotate-proxy-tokens', { projects: project.id }, 'success'], + ['system/rotate-proxy-tokens', { projects: 'all' }, 'success'], + ]), + ); + expect(events).toHaveLength(2); +}); diff --git a/platform/packages/api/test/integration/operator-operations.test.ts b/platform/packages/api/test/integration/operator-operations.test.ts index eb778ce93..558f94618 100644 --- a/platform/packages/api/test/integration/operator-operations.test.ts +++ b/platform/packages/api/test/integration/operator-operations.test.ts @@ -12,27 +12,41 @@ import { } from 'vite-plus/test'; import { getDeployment } from '../../src/db.ts'; import { app } from '../../src/index.ts'; +import { getProjectDeploymentWorkerName } from '../../src/naming.ts'; import { loadSandboxDeploymentPageProjection } from '../../src/operator/sandbox-operations.ts'; +import { + advanceProxyTokenRotation, + reconcileProxyTokenRotations, + requestProxyTokenRotation, + type ProxyTokenRotationEnv, +} from '../../src/proxy-token-rotation.ts'; +import { deriveProxyToken } from '../../src/proxy-token.ts'; import { databaseTimestampMs } from '../../src/routes/deploy.ts'; import { readSandboxDrainCheckpoint } from '../../src/sandbox-drain.ts'; import { deployments, operatorEvents, platformEvents, + projectCleanupJobs, + projects, resources, sandboxDrainCheckpoints, } from '../../src/schema.ts'; import { BUILD_INSTANCE_LARGE } from '../../src/tiers.ts'; import { + cfError, + cfSuccess, createTestBuild, clearTables, createOperatorToken, createTestDeployment, createTestProject, createTestUser, + JWT_SECRET, mockCfApi, setupEnv, teardownEnv, + type MockCfApiHandle, type TestEnv, } from './helpers.ts'; @@ -1450,3 +1464,368 @@ describe('operator installation-aware system and worker operations', () => { expect(cf).not.toHaveBeenCalled(); }); }); + +describe('proxy token rotation', () => { + const rotationTarget = (projectId: string) => `proxy-token-rotation:${projectId}`; + const tok = (projectId: string, gen: number) => deriveProxyToken(JWT_SECRET, projectId, gen); + + /** Cloudflare mock whose script bindings come from seeded scripts; a missing script is a 404. */ + function cloudflare() { + const cf: MockCfApiHandle = mockCfApi({ + listWorkerBindings: (url) => { + const name = decodeURIComponent(/\/scripts\/([^/?#]+)/.exec(url)?.[1] ?? ''); + const script = cf.state.hostScripts.get(name); + return script + ? cfSuccess(script.bindings) + : cfError('workers.api.error.script_not_found', 404, 10007); + }, + }); + return cf; + } + const patchCount = (cf: MockCfApiHandle) => + cf.mock.calls.filter(([, init]) => (init as RequestInit | undefined)?.method === 'PATCH') + .length; + + /** A project, optionally with an active Worker (plus a retained one) holding tok(gen). */ + async function rotationProject( + cf: MockCfApiHandle, + slug: string, + options: Readonly<{ gen?: number; workers?: boolean }> = {}, + ) { + const project = await createTestProject(env.db, ownerId, slug); + const gen = options.gen ?? 0; + await env.db.update(projects).set({ proxy_token_gen: gen }).where(eq(projects.id, project.id)); + const workers: Array = []; + if (options.workers) { + const retained = await createTestDeployment(env.db, project.id, 'live'); + const active = await createTestDeployment(env.db, project.id, 'live'); + await env.db + .update(deployments) + .set({ retained: true }) + .where(eq(deployments.id, retained.id)); + for (const id of [retained.id, active.id]) { + const scriptName = getProjectDeploymentWorkerName(slug, id); + cf.seedHostScript({ + scriptName, + bindings: [ + { type: 'plain_text', name: '__VOID_PROXY_TOKEN', text: await tok(project.id, gen) }, + { type: 'secret_text', name: 'API_KEY' }, + ], + }); + workers.push(scriptName); + } + } + return { ...project, workers }; + } + + async function rotationJob(projectId: string) { + const [row] = await env.db + .select() + .from(projectCleanupJobs) + .where(eq(projectCleanupJobs.target, rotationTarget(projectId))); + return row; + } + + async function issued(projectId: string) { + const [row] = await env.db + .select({ gen: projects.proxy_token_gen }) + .from(projects) + .where(eq(projects.id, projectId)); + return row!.gen; + } + + const scriptToken = (cf: MockCfApiHandle, script: string) => + cf.state.hostScripts.get(script)?.bindings.find((b) => b.name === '__VOID_PROXY_TOKEN')?.text; + + /** Moves every rotation wait an hour back and makes the jobs due. */ + async function endWaits() { + const shifted = (field: string) => sql`CASE WHEN payload->>${field} IS NOT NULL + THEN jsonb_build_object(${field}::text, to_char(((payload->>${field})::timestamptz + - interval '1 hour') AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')) + ELSE '{}'::jsonb END`; + await env.db.execute(sql`UPDATE project_cleanup_jobs + SET next_attempt_at = now(), + payload = payload || ${shifted('waitUntil')} || ${shifted('lastPatchAt')} + WHERE kind = 'proxy-token-rotation'`); + } + + /** One scheduled rotation run past the current waits. */ + async function runRotations() { + await endWaits(); + await reconcileProxyTokenRotations(env.db, env as unknown as ProxyTokenRotationEnv); + } + + /** Scheduled rotation runs until no job is left. */ + async function finishRotations() { + for (let run = 0; run < 6; run++) { + await runRotations(); + const plan = await req('/system/rotate-proxy-tokens', { dryRun: true }); + const body = (await plan.json()) as { summary: { pending: number; failed: number } }; + if (body.summary.pending + body.summary.failed === 0) { + return; + } + } + throw new Error('Rotation did not complete within six runs.'); + } + + test('previews every project without writing anything or returning a token', async () => { + const cf = cloudflare(); + const withWorkers = await rotationProject(cf, 'rotate-plan-workers', { workers: true }); + const empty = await rotationProject(cf, 'rotate-plan-empty'); + const rotated = await rotationProject(cf, 'rotate-plan-rotated', { gen: 1 }); + + const progress = await req('/system/rotate-proxy-tokens', { dryRun: true }); + expect(progress.status).toBe(200); + const text = await progress.text(); + expect(JSON.parse(text)).toEqual({ + summary: { pending: 0, failed: 0 }, + projects: [], + dryRun: true, + action: 'system.rotate-proxy-tokens', + target: { projects: 'all' }, + }); + const all = await req('/system/rotate-proxy-tokens', { dryRun: true, all: true }); + expect(await all.json()).toMatchObject({ + summary: { wouldQueue: 3, pending: 0, failed: 0 }, + target: { projects: 'all' }, + }); + + const project = await req('/system/rotate-proxy-tokens', { + dryRun: true, + project: withWorkers.id, + }); + expect(project.status).toBe(200); + const projectText = await project.text(); + expect(JSON.parse(projectText)).toEqual({ + projectId: withWorkers.id, + slug: 'rotate-plan-workers', + gen: 0, + job: null, + workers: withWorkers.workers, + dryRun: true, + action: 'system.rotate-proxy-tokens', + target: { projects: withWorkers.id }, + }); + + for (const body of [text, projectText]) { + expect(body).not.toContain('mock-proxy-token'); + for (const id of [withWorkers.id, empty.id, rotated.id]) { + expect(body).not.toContain(await tok(id, 0)); + expect(body).not.toContain(await tok(id, 1)); + } + } + expect(await env.db.select().from(projectCleanupJobs)).toEqual([]); + expect((await env.ROUTING.list({ prefix: 'proxy-token:' })).keys).toEqual([]); + expect(patchCount(cf)).toBe(0); + expect(await env.db.select().from(operatorEvents)).toEqual([]); + }); + + test('rotates every project with --all in bounded runs', async () => { + const cf = cloudflare(); + const withWorkers = await rotationProject(cf, 'rotate-bulk-workers', { workers: true }); + const empty = await rotationProject(cf, 'rotate-bulk-empty'); + const rotated = await rotationProject(cf, 'rotate-bulk-rotated', { gen: 1 }); + + const first = await req('/system/rotate-proxy-tokens', { all: true }); + expect(first.status).toBe(207); + expect(await first.json()).toMatchObject({ + ok: true, + complete: false, + queued: 3, + pending: 3, + failed: 0, + failures: [], + }); + + await finishRotations(); + for (const project of [withWorkers, empty]) { + expect(await issued(project.id)).toBe(1); + expect(await env.ROUTING.get(`proxy-token:${project.id}`, 'json')).toEqual({ gen: 1 }); + } + for (const worker of withWorkers.workers) { + expect(scriptToken(cf, worker)).toBe(await tok(withWorkers.id, 1)); + } + expect(await issued(rotated.id)).toBe(2); + expect(await env.ROUTING.get(`proxy-token:${rotated.id}`, 'json')).toEqual({ gen: 2 }); + expect(await env.db.select().from(projectCleanupJobs)).toEqual([]); + }); + + test('shows a project rotation in progress', async () => { + const cf = cloudflare(); + const project = await rotationProject(cf, 'rotate-progress', { workers: true }); + + const apply = await req('/system/rotate-proxy-tokens', { project: project.id }); + expect(apply.status).toBe(207); + expect(await apply.json()).toMatchObject({ + ok: true, + complete: false, + queued: 1, + pending: 1, + failed: 0, + }); + + const plan = await req('/system/rotate-proxy-tokens', { dryRun: true, project: project.id }); + const body = (await plan.json()) as { job: { waitUntil: string } }; + expect(body).toMatchObject({ + projectId: project.id, + gen: 1, + job: { + phase: 'revoke', + reason: 'operator', + again: false, + attempts: 0, + lastError: null, + }, + workers: project.workers, + dryRun: true, + action: 'system.rotate-proxy-tokens', + target: { projects: project.id }, + }); + // Every Worker is patched; the floor waits for their old versions to drain. + expect(Date.parse(body.job.waitUntil)).toBeGreaterThan(Date.now()); + expect(patchCount(cf)).toBe(2); + expect(await env.ROUTING.get(`proxy-token:${project.id}`)).toBeNull(); + + const progress = await req('/system/rotate-proxy-tokens', { dryRun: true }); + expect(await progress.json()).toMatchObject({ + summary: { pending: 1, failed: 0 }, + projects: [{ projectId: project.id, slug: 'rotate-progress', phase: 'revoke' }], + }); + }); + + test.each(['patch', 'revoke', 'settle'])( + 'rotates once more when --project lands on a job in %s', + async (phase) => { + const cf = cloudflare(); + const project = await rotationProject(cf, `rotate-again-${phase}`, { workers: true }); + if (phase === 'patch') { + // Stop right after START, before the first Worker is patched. + await requestProxyTokenRotation(env.db, project.id, 'operator', 'operator'); + await advanceProxyTokenRotation( + env.db, + env as unknown as ProxyTokenRotationEnv, + project.id, + { cfCallsRemaining: 0, deadline: Date.now() + 60_000 }, + ); + } else { + await req('/system/rotate-proxy-tokens', { project: project.id }); + } + while ((await rotationJob(project.id))!.payload.phase !== phase) { + await runRotations(); + } + + await req('/system/rotate-proxy-tokens', { project: project.id }); + expect((await rotationJob(project.id))!.payload.again).toBe('1'); + + await finishRotations(); + expect(await issued(project.id)).toBe(2); + for (const worker of project.workers) { + expect(scriptToken(cf, worker)).toBe(await tok(project.id, 2)); + } + }, + ); + + test('does not add a generation for a job that has not started', async () => { + const cf = cloudflare(); + const project = await rotationProject(cf, 'rotate-requested', { workers: true }); + await env.db.insert(projectCleanupJobs).values({ + target: rotationTarget(project.id), + project_id: project.id, + kind: 'proxy-token-rotation', + payload: { phase: 'requested', reason: 'member-removed' }, + next_attempt_at: sql`now() + interval '1 hour'`, + }); + + await req('/system/rotate-proxy-tokens', { project: project.id }); + const job = await rotationJob(project.id); + expect(job!.payload).toMatchObject({ phase: 'revoke', reason: 'member-removed' }); + expect(job!.payload.again).toBeUndefined(); + + await finishRotations(); + expect(await issued(project.id)).toBe(1); + }); + + test('flags a running rotation with --all', async () => { + const cf = cloudflare(); + const project = await rotationProject(cf, 'rotate-flag', { workers: true }); + await req('/system/rotate-proxy-tokens', { project: project.id }); + expect((await rotationJob(project.id))!.payload.again).toBeUndefined(); + + const all = await req('/system/rotate-proxy-tokens', { all: true }); + expect(await all.json()).toMatchObject({ queued: 1 }); + expect((await rotationJob(project.id))!.payload.again).toBe('1'); + }); + + test('reports a rotation that keeps failing', async () => { + const cf = cloudflare(); + const project = await rotationProject(cf, 'rotate-failing', { workers: true }); + cf.state.hostScripts.delete(project.workers[1]!); + vi.spyOn(console, 'error').mockImplementation(() => {}); + + const failed = await req('/system/rotate-proxy-tokens', { project: project.id }); + expect(failed.status).toBe(207); + expect(await failed.json()).toMatchObject({ + ok: false, + complete: false, + pending: 0, + failed: 1, + failures: [ + { + projectId: project.id, + slug: 'rotate-failing', + phase: 'patch', + error: expect.stringContaining('not found'), + }, + ], + }); + const plan = await req('/system/rotate-proxy-tokens', { dryRun: true, project: project.id }); + expect(await plan.json()).toMatchObject({ + job: { phase: 'patch', attempts: 1, lastError: expect.stringContaining('not found') }, + }); + expect(await env.ROUTING.get(`proxy-token:${project.id}`)).toBeNull(); + }); + + test('rejects invalid scopes and platforms without Cloudflare access', async () => { + const cf = cloudflare(); + for (const body of [ + { project: 'not-a-project' }, + { project: 42 }, + { all: 'yes' }, + { project: 'proj_000000000000', all: true }, + ]) { + for (const dryRun of [true, false]) { + const response = await req('/system/rotate-proxy-tokens', { ...body, dryRun }); + expect(response.status).toBe(400); + expect(await response.json()).toMatchObject({ error: 'invalid_argument' }); + } + } + for (const body of [{}, { all: false }]) { + const response = await req('/system/rotate-proxy-tokens', body); + expect(response.status).toBe(400); + expect(await response.json()).toMatchObject({ + error: 'invalid_argument', + message: 'Supply --project or --all.', + }); + } + const unknown = await req('/system/rotate-proxy-tokens', { + project: 'proj_zzzzzzzzzzzz', + dryRun: true, + }); + expect(unknown.status).toBe(404); + + await rotationProject(cf, 'rotate-unavailable'); + for (const extraEnv of [{ DEV_AUTH_BYPASS: 'true' }, { CF_API_TOKEN: '' }]) { + const preview = await req( + '/system/rotate-proxy-tokens', + { all: true, dryRun: true }, + extraEnv, + ); + expect(preview.status).toBe(200); + const response = await req('/system/rotate-proxy-tokens', { all: true }, extraEnv); + expect(response.status).toBe(409); + expect(await response.json()).toMatchObject({ error: 'rotation_unavailable' }); + } + expect(await env.db.select().from(projectCleanupJobs)).toEqual([]); + expect(cf).not.toHaveBeenCalled(); + }); +}); diff --git a/platform/packages/api/test/integration/operator-tenants.test.ts b/platform/packages/api/test/integration/operator-tenants.test.ts index e1f9e10ca..a1771fe5d 100644 --- a/platform/packages/api/test/integration/operator-tenants.test.ts +++ b/platform/packages/api/test/integration/operator-tenants.test.ts @@ -29,6 +29,7 @@ import { platformLoginConnections, platformLoginIdentities, platformLoginRevisions, + projectCleanupJobs, projectDeployCredentials, projectInvitations, projectMembers, @@ -997,6 +998,57 @@ describe('operator tenant mutations', () => { expect(JSON.parse((await env.ROUTING.get(project.slug))!).suspended).toBeUndefined(); }); + test('queues proxy token rotations for a suspended administrator when routing fails', async () => { + vi.spyOn(console, 'error').mockImplementation(() => {}); + const owner = await createTestUser(env.db, 'rotation-owner'); + const target = await createTestUser( + env.db, + 'rotation-admin', + 'github', + undefined, + 'free', + 'admin', + ); + const owned = await createTestProject(env.db, target.id, 'rotation-admin-owned'); + const shared = await createTestProject(env.db, owner.id, 'rotation-shared'); + await env.db + .insert(projectMembers) + .values({ project_id: shared.id, user_id: target.id, role: 'collaborator' }); + await env.ROUTING.put( + owned.slug, + JSON.stringify({ workerName: 'worker', projectId: owned.id }), + ); + const failingRouting = { + get: (key: string) => env.ROUTING.get(key), + put: async () => { + throw new Error('KV unavailable'); + }, + delete: (key: string) => env.ROUTING.delete(key), + } as unknown as KVNamespace; + + const response = await req( + `/users/${target.id}/suspend`, + { reason: 'admin_suspended' }, + { ROUTING: failingRouting }, + ); + + expect(response.status).toBe(500); + expect(await getUser(env.db, target.id)).toMatchObject({ suspended_reason: 'admin_suspended' }); + expect( + await env.db + .select({ projectId: projectCleanupJobs.project_id, payload: projectCleanupJobs.payload }) + .from(projectCleanupJobs), + ).toEqual([ + { projectId: shared.id, payload: { phase: 'requested', reason: 'user-suspended' } }, + ]); + + // A retry runs the whole suspension again; the queued request covers it. + expect((await req(`/users/${target.id}/suspend`, { reason: 'admin_suspended' })).status).toBe( + 200, + ); + expect(await env.db.select().from(projectCleanupJobs)).toHaveLength(1); + }); + test('reports persisted plan changes and preserves suspension on failed restore cleanup', async () => { const user = await createTestUser(env.db, 'cleanup-user'); await env.db diff --git a/platform/packages/api/test/integration/project-team-lifecycle.test.ts b/platform/packages/api/test/integration/project-team-lifecycle.test.ts index 085f54daf..cbc1efb3b 100644 --- a/platform/packages/api/test/integration/project-team-lifecycle.test.ts +++ b/platform/packages/api/test/integration/project-team-lifecycle.test.ts @@ -1,4 +1,4 @@ -import { eq } from 'drizzle-orm'; +import { eq, sql } from 'drizzle-orm'; import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vite-plus/test'; import { createGitHubConnection, @@ -12,14 +12,18 @@ import { getGitHubConnection, getProject, respondToProjectInvitation, + suspendUser, } from '../../src/db.ts'; +import { proxyTokenRotationTarget } from '../../src/proxy-token-rotation-request.ts'; import { projectAccessEvents, + projectCleanupJobs, projectInvitations, projectMembers, projectOwnershipPeriods, users, } from '../../src/schema.ts'; +import { REASON_REQUESTS_EXCEEDED } from '../../src/suspension.ts'; import { createTestBuild, captureQueries, @@ -137,6 +141,131 @@ describe('project team lifecycle', () => { expect(events.every((event) => event.subject_user_id === member.id)).toBe(true); }); + it('rotates the proxy token of every project the deleted user could deploy to', async () => { + const owner = await createTestUser(env.db, 'rotation-owner'); + const member = await createTestUser(env.db, 'rotation-member'); + const collaborated = await createProject(env.db, owner.id, 'rotation-collaborator'); + const read = await createProject(env.db, owner.id, 'rotation-reader'); + const administered = await createProject(env.db, owner.id, 'rotation-admin'); + await env.db.insert(projectMembers).values([ + { project_id: collaborated.id, user_id: member.id, role: 'collaborator' }, + { project_id: read.id, user_id: member.id, role: 'reader' }, + { project_id: administered.id, user_id: member.id, role: 'admin' }, + ]); + + await deleteUser(env.db, member.id); + + const jobs = await env.db.select().from(projectCleanupJobs); + expect(jobs.map((job) => job.project_id).sort()).toEqual( + [collaborated.id, administered.id].sort(), + ); + for (const job of jobs) { + expect(job).toMatchObject({ + kind: 'proxy-token-rotation', + payload: { phase: 'requested', reason: 'user-deleted' }, + }); + } + }); + + it('rotates the proxy token of every project an admin-suspended user could deploy to', async () => { + const owner = await createTestUser(env.db, 'suspension-owner'); + const member = await createTestUser(env.db, 'suspension-member'); + const collaborated = await createProject(env.db, owner.id, 'suspension-collaborator'); + const read = await createProject(env.db, owner.id, 'suspension-reader'); + const administered = await createProject(env.db, owner.id, 'suspension-admin'); + await env.db.insert(projectMembers).values([ + { project_id: collaborated.id, user_id: member.id, role: 'collaborator' }, + { project_id: read.id, user_id: member.id, role: 'reader' }, + { project_id: administered.id, user_id: member.id, role: 'admin' }, + ]); + + await suspendUser(env.db, member.id, REASON_REQUESTS_EXCEEDED); + await suspendUser(env.db, owner.id, 'admin_suspended'); + expect(await env.db.select().from(projectCleanupJobs)).toEqual([]); + + await suspendUser(env.db, member.id, 'admin_suspended'); + + const jobs = await env.db.select().from(projectCleanupJobs); + expect(jobs.map((job) => job.project_id).sort()).toEqual( + [collaborated.id, administered.id].sort(), + ); + for (const job of jobs) { + expect(job).toMatchObject({ + kind: 'proxy-token-rotation', + payload: { phase: 'requested', reason: 'user-suspended' }, + }); + } + }); + + it('queues the rotations of an admin suspension in one statement, keeping queued jobs', async () => { + const owner = await createTestUser(env.db, 'many-owner'); + const member = await createTestUser(env.db, 'many-member'); + const collaborated = await createProject(env.db, owner.id, 'many-collaborator'); + const administered = await createProject(env.db, owner.id, 'many-admin'); + const read = await createProject(env.db, owner.id, 'many-reader'); + const queued = await createProject(env.db, owner.id, 'many-queued'); + const started = await createProject(env.db, owner.id, 'many-started'); + await env.db.insert(projectMembers).values([ + { project_id: collaborated.id, user_id: member.id, role: 'collaborator' }, + { project_id: administered.id, user_id: member.id, role: 'admin' }, + { project_id: read.id, user_id: member.id, role: 'reader' }, + { project_id: queued.id, user_id: member.id, role: 'collaborator' }, + { project_id: started.id, user_id: member.id, role: 'admin' }, + ]); + const later = sql`now() + interval '1 hour'`; + await env.db.insert(projectCleanupJobs).values([ + { + target: proxyTokenRotationTarget(queued.id), + project_id: queued.id, + kind: 'proxy-token-rotation', + payload: { phase: 'requested', reason: 'member-role-changed' }, + next_attempt_at: later, + }, + { + target: proxyTokenRotationTarget(started.id), + project_id: started.id, + kind: 'proxy-token-rotation', + payload: { phase: 'patch', reason: 'operator', gen: '1', cursor: '' }, + attempts: 2, + next_attempt_at: later, + }, + ]); + const before = new Map( + (await env.db.select().from(projectCleanupJobs)).map((job) => [job.project_id, job]), + ); + + const { queries } = await captureQueries(env.db, () => + suspendUser(env.db, member.id, 'admin_suspended'), + ); + + const jobs = new Map( + (await env.db.select().from(projectCleanupJobs)).map((job) => [job.project_id, job]), + ); + expect([...jobs.keys()].sort()).toEqual( + [collaborated.id, administered.id, queued.id, started.id].sort(), + ); + for (const projectId of [collaborated.id, administered.id]) { + expect(jobs.get(projectId)).toMatchObject({ + target: proxyTokenRotationTarget(projectId), + kind: 'proxy-token-rotation', + payload: { phase: 'requested', reason: 'user-suspended' }, + attempts: 0, + last_error: null, + }); + } + // A job that has not started covers the request as it is. + expect(jobs.get(queued.id)).toEqual(before.get(queued.id)); + // A started job finishes and runs once more, on its own schedule. + const startedBefore = before.get(started.id)!; + expect(jobs.get(started.id)).toEqual({ + ...startedBefore, + payload: { ...startedBefore.payload, again: '1' }, + }); + expect( + queries.filter((query) => /^insert into "project_cleanup_jobs"/i.test(query.trim())), + ).toHaveLength(1); + }); + it('atomically discovers and locks pending-invitation projects before user deletion', async () => { const owner = await createTestUser(env.db, 'owner'); const invitee = await createTestUser(env.db, 'invitee'); diff --git a/platform/packages/api/test/integration/proxy-token-rotation.test.ts b/platform/packages/api/test/integration/proxy-token-rotation.test.ts new file mode 100644 index 000000000..61af99421 --- /dev/null +++ b/platform/packages/api/test/integration/proxy-token-rotation.test.ts @@ -0,0 +1,1943 @@ +import { eq, sql } from 'drizzle-orm'; +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + describe, + expect, + it, + vi, +} from 'vite-plus/test'; +import { hmacSignState } from '../../src/auth/hmac.ts'; +import { + createDeployment, + deleteUser, + markDeploymentRetained, + setActiveDeployment, + updateDeploymentStatus, +} from '../../src/db.ts'; +import app from '../../src/index.ts'; +import { getProjectDeploymentWorkerName, getProjectQueueName } from '../../src/naming.ts'; +import { reconcileProjectCleanupJobs } from '../../src/provisioning/cleanup.ts'; +import { + advanceProxyTokenRotation, + issueProjectProxyToken, + listProxyTokenInventory, + PROXY_TOKEN_ROTATION_STUCK_ATTEMPTS, + PROXY_TOKEN_ROTATION_WAIT_MS, + proxyTokenRotationStuckWarning, + proxyTokenRotationTarget, + proxyTokenRotationWarnings, + reconcileProxyTokenRotations, + requestBulkProxyTokenRotations, + requestProxyTokenRotation, + requestRotationIfReplacingUntrustedCode, + type ProjectRotationOutcome, + type ProxyTokenRotationEnv, + type ProxyTokenRotationReason, +} from '../../src/proxy-token-rotation.ts'; +import { + deriveProxyInternalAuth, + deriveProxyToken, + PROXY_INTERNAL_AUTH_HEADER, + proxyTokenStateKey, + type ProxyInternalRoute, +} from '../../src/proxy-token.ts'; +import { + deployments, + projectCleanupJobs, + projectMembers, + projects, + resources, + users, +} from '../../src/schema.ts'; +import { + captureQueries, + capturedQueryCount, + cfError, + cfSuccess, + clearTables, + createCronSchedulerMock, + createTestDeployment, + createTestProject, + createTestUser, + JWT_SECRET, + MemoryKVNamespace, + mockCfApi, + readAllEvents, + setupEnv, + teardownEnv, + type CfApiMocks, + type MockCfApiHandle, + type TestEnv, +} from './helpers.ts'; + +let env: TestEnv; + +beforeAll(async () => { + env = await setupEnv(); +}); + +afterAll(async () => { + await teardownEnv(); +}); + +const MANIFEST = { + version: 3, + bindings: {}, + schedules: [{ cron: '*/5 * * * *', filePath: 'crons/tick.ts' }], + queues: [{ name: 'emails', bindingName: 'QUEUE_EMAILS' }], +}; + +/** Ordered log of Cloudflare calls (`GET