Repository navigation
Record the project and hostname decisions (D2, D3) and the runbook for the hostname (C2) and the move (C5) - #21
Merged
Conversation
docs/project-and-domain-decision.md sets out decisions D2 (a dedicated Google Cloud project) and D3 (a custom hostname): the options, their costs and risks, what changes for the emitter, the calibration dashboard and microcosm-runs, and the choices to make. Neither decision is made. docs/project-move-runbook.md is plan item C2: creating the project, the Artifact Registry store, the Cloud Run service and migration job, Cloud SQL with an export and import inside a maintenance window, the Secret Manager secrets (including S1's Hugging Face read token), Workload Identity Federation for this repo, the hostname and DNS, the GitHub environment and dashboard settings, verification and rollback. It applies only after the deploy switch-over (C1) is verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From staging's window on, staging builds the image in the new project and production deploys that same image, still in the old project, so the old deployment account and the old project's Cloud Run service agent need Artifact Registry Reader on the new project. Without it, production's deployment fails in staging's window and on every merge until production moves. The runbook now grants the role before staging's window and removes it when the old resources are retired. Production's window had no merge to deploy it: the C2 code PR merges in staging's window. The runbook now names a second PR for production's window. The stated downtime left out the release pipeline, which sets most of a window's length. The memo and the runbook now give the measured range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production's old instance lists a third user, telemetry_app, besides the migration and runtime users. Google's import guide says every user that owns objects or holds privileges in the dumped database must exist on the target, so the runbook now creates every user the record lists. Re-creating the database after the timing rehearsal drops the database-level privileges granted in step 5.2; the runbook now grants them again, and the copy check compares the database listing too. The command that shows the database URL's form now hides everything up to the last "@", so a password containing one isn't printed in part. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runbook listed only part of what #12, #16 and #19 need in the new project, so following it after any of them merges would stop a deployment or lose writes: - #12's worker job and recovery job write to the database while the service is in maintenance mode, so the export could miss their writes. The runbook now pauses the queue and the recovery schedule before the export, resumes them on rollback, and lists the recovery job, its schedule and the Cloud Scheduler API. - #16's signing key is a fifth secret that the deployment checks and mounts. The runbook now creates it, grants it like the other four and lists it with the settings. - #19's daily check wasn't mentioned. The runbook now lists its job, schedule, alert policies, permissions and three production variables, one of which names the project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The dashboard's identity-provider setting is one Vercel entry for Production and Preview (since 2026-10-07), not Production only, and Preview deployments can read staging's collector. - Only Cloud Run domain mapping needs Search Console verification; Certificate Manager's DNS authorization needs only a DNS record. - Events are not always kept through a window: a machine discarded when its build ends, such as a Modal container, loses the events not sent by then. - The emitter's Orrery uploads do get their credential from the collector's token exchange; they already wait today, so the window changes nothing for them. - #23 adds the emitter's Hugging Face module; #27 makes it send submissions to the collector. - A new emitter release reaches builds once it is installed on the machine or in the build image. - The backup check needs a format that shows the description. - Whether a re-run of the dashboard's deployment reads a changed GitHub variable is unchecked, so the documents now point to a merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 6.2 now pauses #12's queue and recovery schedule together and lists unfinished executions of both the worker and the recovery job. It gives the possible wait (three attempts of up to 1 hour each), how to cancel a worker execution and what recovery then does with its job, and why the queue must never be paused while recovery runs. The intake bucket stays in the old project. Each accepted submission records its files' Cloud Storage generation numbers, and the worker reads every file at that generation; a copy into a new bucket gets new generation numbers, so the copy step is removed. Uploads still in flight need nothing, because the bucket doesn't change. Step 6.5 and section 12 now point the queue, worker job, signing account and invoker account settings at the new project. A new step 6.7 creates the recovery schedule only after the deployment. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new part of step 6.7 creates #19's schedule in the new project once production's deployment has created the check job, then runs the job once by hand: the README says the absence policy can report only after a run that follows its creation. It then disables the old project's two alert policies and pauses its schedule, in that order, before section 11 removes the old check job or revokes S1's old token, so the old project sends no false alerts. Rollback reverses this, and section 9 checks it. Step 5.3's #19 bullet now names what the deployment needs: S1's secret, the worker account, and the deployment account's viewer and service-account-user roles. The channel and the policies don't affect it. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 3.2 now reads the access policy of every secret the environment uses, including #16's signing key and S1's token if merged, and records #12's queue, worker and recovery jobs, schedule, bucket roles and invoker account, and #19's check job, schedule, alert policies and notification channel. Section 11 now retires those resources: every recorded secret, the old worker account, #12's queue, jobs, schedule and accounts (but not the intake bucket, which stays in use), and #19's job, schedule, account, policies and channel. S1's old token is revoked only after step 6.7 has paused the old check that uses it. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production redeploys in staging's window and on every merge until its own window, possibly with migrations, so the revision step 3.2 records can be older than the database schema by the time of a rollback. Step 6.1 now records the revision the old service serves just before step 6.2, and rollback routes traffic to that one. Rollback also says which GitHub values it restores: only those step 6.5 changed in that window. In production's window that is production's values and TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION in staging's environment; staging stays on the new project. The rollback after the emitter release notes when the recorded revision is too old for the imported schema. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cloud SQL's "About PostgreSQL users and roles" and "Create and manage users" pages say a user created with gcloud sql users create is a member of cloudsqlsuperuser with CREATEROLE, CREATEDB and LOGIN. An old user created in SQL may have less, and an export carries no roles. Step 3.2 now records every role's attributes and memberships with one SQL query, which works in any client (psql 16 and later no longer show memberships in \du). Step 5.2 compares each new user with that record and removes what it doesn't show: the membership with gcloud sql users assign-roles --revoke-existing-roles, the attributes with ALTER ROLE as postgres. Step 6.4 compares the query's rows on both instances. The extra users are now a list set from the record, so pasting the block for staging no longer creates telemetry_app. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#16 adds the signing-key secret to both environments' deployments, so without it that environment's deployment stops, not only production's. The quiet-day figure counted merge to tag only (25 minutes on 2026-10-15). The two deploy jobs add about 10 minutes, and the export and import come on top, so the runbook's section 6 and the memo's D2 B now say about 40 minutes. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo's D2 B sentence put the quiet-day figure on the redeploy alone; the 40 minutes includes the export and import, so it now says "in all". Two paragraphs added in this round are rewrapped, and one sentence in step 5.2 reads in order. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Section 1 asked only for read access and Cloud SQL admin in the old project, but the steps also read secret values, change the intake bucket's IAM policy, update the old Cloud Run service, pause and resume Cloud Tasks and Cloud Scheduler, change alert policies and delete resources there. A table now names the role for each step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If #12's intake is switched on before the move, its bucket stays in policyengine-api, so its storage cost, data-access logs and administration stay there, and shutting that project down or removing its billing would make every submission's files unreadable. The memo's D2 B row, "What a dedicated project gives" and "Order and timing" now say so, recommend switching intake on only after the move, and record that intake is off today (read-only check: the Cloud Tasks and Cloud Scheduler APIs aren't enabled, and no intake bucket exists). In the runbook, step 3.2 also records the old accounts' project-level roles, because #12's unwritten deployment wiring may grant storage access on the project. Step 5.3 names the permissions the code needs on the bucket (create and list for the runtime account, get for the worker's) and grants them on the bucket. Moving the bucket later is a one-off data change that rewrites the recorded generation numbers, not necessarily a code change. Section 9 checks the bucket's policy and one test submission, and section 11 says the old project must stay active while the collector reads the files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 5.2 still set other_database_users=(telemetry_app) whatever the environment, so pasting the block for staging created a user staging doesn't have, and step 6.4's roles comparison would then stop the window. Step 3.2 now records each environment's users as listed on 2026-10-20 (staging: postgres, telemetry_migrator, telemetry_runtime; production: the same plus telemetry_app) and sets OTHER_DATABASE_USERS per environment, and step 5.2 creates that list, which is empty for staging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The roles comparison in step 6.4 would differ on the postgres row if the old users were made in SQL by postgres: PostgreSQL 16 grants each role a non-superuser creates back to its creator with ADMIN TRUE, SET FALSE, INHERIT FALSE, and users made by gcloud sql users create don't get that grant. Step 3.2's roles query now shows each membership's admin, inherit and set options. Step 5.2 creates a user whose row lists cloudsqlsuperuser with gcloud sql users create, and any other user in SQL as the role that holds it as "(admin)", normally postgres, with exactly the recorded attributes, so both rows match without removing anything. Step 5.2 compares the roles query before the window, and step 6.4 accepts only the differences step 5.2 names. Whether postgres may ALTER ROLE a user Cloud SQL made is not documented: PostgreSQL 16 requires ADMIN OPTION on the target, and Cloud SQL's pages don't say whether postgres holds it. Step 5.2 now checks it once on staging's new instance, with a throwaway user, before production relies on it, and gives the fallback: create that user in SQL as postgres, which then holds ADMIN OPTION, and accept the two resulting differences. The local check of this behaviour on PostgreSQL 16.4, with a non-superuser postgres, refused ALTER ROLE on a role it hadn't created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On PostgreSQL 15 and later only the database's owner may create objects in schema public unless granted CREATE there. If the old migration user isn't a cloudsqlsuperuser member, it got that by owning the database or public, or by a grant, and none of these exists on the new database before the import. The dump's own grants on public come after its tables and run as the importing user, so the import's first CREATE TABLE would fail. Step 5.2 now compares the new database's \l and \dn+ public with the record before any import (step 5.5's rehearsal and step 6.3) and applies what differs: privileges first, then owners. A Cloud SQL-made owner makes itself the owner; for a role postgres made, postgres grants itself SET on it for the change and revokes that afterwards, which PostgreSQL 16 allows and which leaves the automatic grant in place (checked on a local PostgreSQL 16.4 emulation). Step 5.5 repeats these steps after re-creating the database, and step 6.4 also compares \dn+ public. The example loop's comment now says a placeholder makes each call fail rather than stopping the loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first rollback row re-enabled the old project's alert policies but didn't run the old check job, so after a rollback the old "no success for 23 hours" policy couldn't report a stopped schedule: #19's README says a metric-absence condition needs a successful run after the policy is created or changed. The row now runs the old check job once and waits for it, and pauses the new queue along with the new schedules. The "after the emitter release" row paused only the new project's queue and recovery schedule. It now also pauses the new check schedule and disables the new alert policies, and in the old project resumes the queue, the recovery schedule and the check schedule, enables the old alert policies and runs the old check job once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 6.2 said every reprocess run is a recovery-job execution, so the loop covers it. #12's README also allows running reprocess locally through the Cloud SQL Auth Proxy, which writes to the database without a Cloud Run execution. The step now says so and asks that nobody runs it that way during the window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Section 12's closing paragraph now lists the open questions plainly: whether postgres may ALTER ROLE a user Cloud SQL made (step 5.2's staging check settles it); who owns a database gcloud sql databases create makes, and the old databases' owners and grants; whether either project is inside a VPC Service Controls perimeter; whether policyengine-api has a project lien; and #12's unwritten deployment wiring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo said the bucket "can't move" and that the files depend on policyengine-api "for good", while the same text says a one-off change could move them. It now says the files depend on that project until such a change, not yet written, moves them. Section 12 no longer says nothing has been run: some read-only commands have been. One sentence in step 5.5 is rewrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo offers a hostname-first order, but the runbook had no step that points the hostname at today's service, releases the emitter with it, and waits for builds to leave the default address before the move; section 7 assumed the hostname was "already mapped", and section 8.2 placed the emitter release after the move. Step 3.4 now does that phase against policyengine-api, says why emitter 0.2.1 decides the wait, and section 11 retires the mapping or load balancer it created there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runbook created the new project's Certificate Manager DNS authorization with the default fixed record name, `_acme-challenge.HOSTNAME`. In the hostname-first order the old project's authorization already holds that name, and DNS allows one CNAME per name, so the new certificate couldn't be issued before the A record moved, as the memo says it is. The command now passes `--type=PER_PROJECT_RECORD`, whose record name carries a suffix for the project, and the text says why, quoting Certificate Manager's page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For the hostname-first order with domain mapping, the runbook moved the hostname by creating the new project's mapping with --force-override and said this "moves the hostname at once". gcloud's help describes the flag as mapping a domain "even if it is already mapped to another service", and nothing says it reaches a mapping in another project. The runbook now deletes the old project's mapping and then creates the new one, and the rollback row does the reverse; section 11 no longer lists that mapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A read-only check of policyengine-api on 2026-10-22 found a fifth production collector secret, `microcosm-telemetry-database-url`, made four days before the runtime database URL secret. No GitHub variable names it and no service or job mounts it, but production's runtime account may read it. Step 3.2 recorded secrets only from the deployment's settings, so section 11 would have left it in policyengine-api. Step 3.2 now lists the old project's secrets and notes unused ones, section 11 deletes them, and the memo says what the 2026-10-22 check found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 5.2 said to create each instance with the recorded settings and to "keep" deletion protection on, but on 2026-10-22 production's instance had deletion protection off, and its command always passed --backup-start-time, which turns on automated backups that staging's instance doesn't have. It now says deletion protection is on for both new instances, and to use --no-backup where the record shows backups off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new project inherits its parent's organization policies. A policy such as domain-restricted sharing would refuse the allUsers invoker binding that --allow-unauthenticated adds, and the deployment's maintenance check would then fail. Whether one applies couldn't be read: the Organization Policy API isn't enabled in policyengine-api. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second review pass over this round's fixes: the memo's hostname-first paragraph had overlong lines and still said the D3 B mapping is "re-created" rather than deleted and created again; the organization policy note implied the new project shares policyengine-api's organization, which choice 2 decides, and now names the checked invoker binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
added a commit
that referenced
this pull request
Oct 27, 2026
The README's "Database access" said each environment sets the migration user's default privileges before 0002_artifact_intake is applied. A read-only check through each environment's migration job on 2026-10-27 found that staging and production both have them, so before a merge there is only a \ddp check, and nothing to run. It now also says when the statement applies again: when the database users are recreated, as the project move in #21 does. A default-privileges entry belongs to the role, not its name, so the new database needs a \ddp check before a migration job runs there. The intake checklist now lists all four privileges the default privileges grant, not three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memo now records Anthony's decisions: a dedicated project in the same organization and billing account (D2 B), a global external load balancer with one hostname per environment (D3 C), DNS records added by hand outside Google Cloud, hostname first with a cross-project backend service until each move, no scheduled windows, staging copied as the rehearsal, the dashboard's identity kept in the current project, and microcosm-runs left until O7 and O8. Plan codes follow the 2026-10-27 renumbering (C2 to C8). Concrete values, such as project IDs, domains and database user names, are replaced by named placeholders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C2 builds each environment's load balancer in the new project, sending requests to today's service through a backend service left in the current project, and lists the four DNS records to add. C5 copies each database with the service in maintenance mode, sets the migration user's default privileges before any migration runs, deploys into the new project and changes the URL map. Rollback is the reverse URL map change. Intake, S1, S2, C6 and C7 are created once in the new project after the move. The third database user, domain mapping and the "both together" order are removed, and no concrete values remain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
changed the title
…vision The memo says where the hostnames and the domain are kept, and states that the copy recreates only the migration and runtime users. The runbook sets the new project's number right after creating the project (step 4.2's binding needs it), reads the operator's account from gcloud's configuration, and in step 8.1 checks that the serving revision is the latest created one, since step 8.2's update builds the maintenance revision from the latest created revision's settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #17
Summary
Docs only. The memo now records decisions D2 and D3 and the choices that follow from them, made by Anthony on 2026-10-27, and the runbook is rewritten for the chosen order: the hostname first (C2), then the move behind it (C5). Nothing in Google Cloud, DNS, GitHub settings or Vercel was created or changed.
No concrete values. Neither document names a hostname, domain, IP address, project ID or number, organization or billing account ID, Cloud SQL instance, database or user name, connection name, service-account email or secret value. Each uses a named placeholder, such as
<production hostname>or<new project ID>, and says where the value is kept (a named GitHub variable or secret, a value read from the current project withgcloud, or "recorded by the operator").The decisions (2026-10-27)
Arecord for the load balancer's static IP, and the Certificate Manager DNS-authorizationCNAME.Plan codes follow the 2026-10-27 renumbering: C2 the hostname, C3 the emitter's default, C4 builds running that release, C5 the move, C6 read API v2, C7 ES256 signing, C8 the dashboard's Vercel OIDC token.
docs/project-and-domain-decision.mdMICROCOSM_TELEMETRY_COLLECTOR_URLonce, in C2;MICROCOSM_TELEMETRY_READ_SECRET_VERSION,TELEMETRY_GCP_PROJECT_IDandTELEMETRY_READ_SECRET_NAMEafter each copy; the emitter's constants in C3.telemetry-service.ymlis never run after the move, Accept artifact submissions into private storage and run worker jobs #12 merged with intake off, and the default privileges are set again on each new instance.docs/project-move-runbook.md: what changedroles/compute.loadBalancerServiceUsergranted on that backend service to the operator; then the static IP, per-project DNS authorization, certificate, certificate map, URL map (default service: the current project's backend service), HTTPS proxy and forwarding rule in the new project. Step 5.3 lists the four DNS records Anthony adds (per environment, the authorizationCNAMEand theArecord) and how to print their values. Then the hostname checks, and the dashboard's switch to the hostname.ALTER DEFAULT PRIVILEGES FOR ROLE <migration user> IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO <runtime user>), with the admin user granting itselfINHERITon the migration user first and revoking it after, before any migration runs, and checks\ddp; the GitHub environment switch; the deployment; then step 8.8 changes the URL map to the new backend service. Intake pausing and the intake-bucket handling are gone, since intake is off.telemetry-service.ymlafter the move.Values check
Before pushing, I searched all text the PR adds, in its diff against
main, for hostnames, domains, IP addresses, long numbers, emails, project IDs, instance, connection and user names. None of PolicyEngine's remain. What matches are file names, IAM role and permission names, and Google's and GitHub's public endpoint names that the commands need (*.googleapis.com, the GitHub OIDC issuer, the Artifact Registry and service-account address formats built from placeholders, and Google's public sample image).What a merge does
Merging releases 0.6.0 (an
addedfragment). With deployment enabled, that release redeploys staging and production with unchanged runtime code, including the usual short maintenance phase.Testing
Docs only, so there is no failing-test-first step.
tests/test_deployment_workflow.py(its vocabulary test scans every Markdown file) andtests/test_packaging.py: 18 passed. The full suite: 361 passed, 51 skipped.tools/check-quality.sh(Ruff check and format, script syntax) passes, andtools/bump-version.py --checkgives 0.5.0 to 0.6.0.mainwas merged in with a merge commit.Earlier review rounds (2026-10-20 to 2026-10-22) are in the commit history; their fixes are kept where they still apply to the chosen design.
🤖 Generated with Claude Code