From db3eaa8df75d7dab97ccfe8c40660693e501ad7d Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 17:06:41 +0400 Subject: [PATCH 01/31] Document the project and hostname decision and the move runbook docs/project-and-domain-decision.md sets out decisions D2 (a dedicated Google Cloud project) and D3 (a custom hostname): the options, their costs and risks, what changes for the emitter, the calibration dashboard and microcosm-runs, and the choices to make. Neither decision is made. docs/project-move-runbook.md is plan item C2: creating the project, the Artifact Registry store, the Cloud Run service and migration job, Cloud SQL with an export and import inside a maintenance window, the Secret Manager secrets (including S1's Hugging Face read token), Workload Identity Federation for this repo, the hostname and DNS, the GitHub environment and dashboard settings, verification and rollback. It applies only after the deploy switch-over (C1) is verified. Co-Authored-By: Claude Opus 5.5 --- .../project-and-domain-decision.added.md | 1 + docs/project-and-domain-decision.md | 245 ++++++ docs/project-move-runbook.md | 743 ++++++++++++++++++ 3 files changed, 989 insertions(+) create mode 100644 changelog.d/project-and-domain-decision.added.md create mode 100644 docs/project-and-domain-decision.md create mode 100644 docs/project-move-runbook.md diff --git a/changelog.d/project-and-domain-decision.added.md b/changelog.d/project-and-domain-decision.added.md new file mode 100644 index 0000000..38fe1c4 --- /dev/null +++ b/changelog.d/project-and-domain-decision.added.md @@ -0,0 +1 @@ +Document the choices for moving the collector to its own Google Cloud project and a custom hostname (plan decisions D2 and D3) in `docs/project-and-domain-decision.md`, and the steps for the move, including the Postgres dump and restore and rollback, in `docs/project-move-runbook.md`. diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md new file mode 100644 index 0000000..347801f --- /dev/null +++ b/docs/project-and-domain-decision.md @@ -0,0 +1,245 @@ +# The collector's Google Cloud project and address: decision memo + +Status: **for decision.** Decisions D2 and D3 of the collector plan are not +made, and nothing described here has been created. Written 2026-10-20. The +steps that carry out the decisions are in +[`project-move-runbook.md`](project-move-runbook.md). + +- **D2:** does the collector stay in the Google Cloud project it shares with + the Public API, or move to a project of its own? +- **D3:** do builds and consumers keep using Cloud Run's default address for + the collector, or a hostname of our own? + +The plan recommends both a dedicated project and a custom hostname, and pairs +them: moving the project changes the default address, so without D3 every +consumer changes twice. The [choices](#the-choices-to-make) are listed at the +end. + +## Today + +Taken from this repo's deployment code, the GitHub environment settings (names +compared, values not printed), the Vercel project settings (names only), and +calibration-diagnostics#233's read-only checks of the live resources. The live +Google Cloud resources were not re-read for this memo. + +| What | Where | +|---|---| +| Google Cloud project | `policyengine-api`, which also hosts the Public API. Both GitHub environments, `staging` and `production`, name it. | +| Per environment | a Cloud Run service, a Cloud Run migration job, a Cloud SQL for PostgreSQL instance, a runtime service account, a migration service account, and four Secret Manager secrets: the runtime database URL, the migration database URL, the session-token signing secret and the dashboard's read token | +| Shared by both environments | one Artifact Registry Docker store and image name; the deployment service account, which calibration-diagnostics also uses; the Workload Identity providers for calibration-diagnostics and for this repo | +| The dashboard's Google identity | a Workload Identity provider for Vercel's OIDC tokens, in the same project, named by its project number | +| Address | Cloud Run's default address for the production service (`https://-..run.app`). The project number in it is `policyengine-api`'s. | + +Where the address appears: + +- **The emitter** fixes it as `PRODUCTION_COLLECTOR_URL` in + `src/microcosm_emitter/core/constants.py`, in 0.2.1 (which Microcosm pins), + 0.3.0 and `main`. PolicyEngine/microcosm-emitter#29 adds a fixed set of + trusted origins and a `collector_origin` module setting, without changing + the default. +- **microcosm-runs** fixes it as `PRODUCTION_COLLECTOR_ORIGIN` in + `lib/auth.ts`, and repeats it in `.env.example`. +- **The calibration dashboard** reads it from the setting + `MICROCOSM_TELEMETRY_COLLECTOR_URL`. +- **Microcosm** doesn't name it. `docs/orrery-publication.md` says the origin + is the emitter's tracked constant, with no override. + +## D2: which Google Cloud project + +| Option | Cost | Risks and limits | +|---|---|---| +| **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | +| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed. Estimate 30 to 60 minutes for production; the staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | +| **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | + +What a dedicated project gives: its own permissions (a deployment identity +that can reach only the collector), its own quotas and billing, audit logs +that contain only the collector, and room for the intake and worker resources +without adding them to the Public API's project. + +## D3: which address + +| Option | Cost | Risks and limits | +|---|---|---| +| **A. Keep Cloud Run's default address** | Nothing. | The address contains the project number, so D2 B or C changes it. Every change needs an emitter release, and builds on older emitters lose delivery until they move. | +| **B. Cloud Run domain mapping** | No charge. | Google lists it as Preview: "not production-ready", and "not recommended for production services" because of latency. TLS 1.0 and 1.1 can't be switched off, and only Google's certificate can be used. The certificate is issued after the mapping exists and DNS points at it: "usually about 15 minutes but can take up to 24 hours." A hostname has one active mapping, so moving the project means re-creating the mapping there and waiting for a new certificate. `us-central1` supports mappings. | +| **C. Global external Application Load Balancer** with a serverless network endpoint group and a Certificate Manager certificate | About $0.025 an hour for the forwarding rule (about $18 a month) plus about $0.008 per GB processed; Certificate Manager's first 100 certificates are free. Check Google's current price lists. | Generally available, and Google's recommended option. With DNS authorization the certificate is issued before any traffic moves, so a later project move is a DNS change with no certificate gap. More resources to keep: a static IP, a network endpoint group, a backend service, a URL map, an HTTPS proxy, a forwarding rule and a certificate map. Cloud Armor can be added later. | +| **D. Route through Vercel or Cloudflare** | Varies. | Not recommended: another provider would end TLS for requests that carry builders' Hugging Face tokens. | + +Which DNS zone (from public name-server records, 2026-10-20): + +- **`microcosm.institute`** is served by Cloudflare's name servers; its apex + points at Vercel, which serves the dashboard. A collector record there must + be "DNS only", not proxied, so that Cloudflare doesn't end TLS and doesn't + interfere with Google's certificate checks. +- **`policyengine.org`** is served by Google Cloud DNS name servers. Which + project holds that zone wasn't checked. +- Neither zone has CAA records today. If one is added, it must allow + `pki.goog` and `letsencrypt.org`. +- Domain mapping (B) and the load balancer's certificate (C) both need the + base domain verified in Google Search Console by the account that creates + them. + +Whatever is chosen, three things stay as they are: + +- **Cloud Run's default addresses stay on** (`--default-url` in + `scripts/deploy.sh`). The deployment checks the maintenance and candidate + revisions through their tagged default addresses, and builds on older + emitters use the production default address. +- **Ingress stays open to the internet.** Build machines and Modal containers + reach the collector directly. Allowing only load-balancer traffic would + break the tagged-address checks and older emitters. +- **The old address can't redirect to the new one.** The emitter doesn't + follow redirects on requests that carry tokens (`_NoRedirectHandler` in + `core/auth.py`). After a move, the old service either keeps serving or stops. + +## What changes for each consumer + +### The emitter + +- **Where:** `PRODUCTION_COLLECTOR_URL`, and after + PolicyEngine/microcosm-emitter#29 also `TRUSTED_COLLECTOR_ORIGINS`, both in + `core/constants.py`. Every service module resolves the origin through + `core/collector_origin.py`. The emitter stack's #27 adds a fourth module + (`huggingface/service/module.py`), which should use the same resolver. +- **The change:** one emitter PR adds the new origin to + `TRUSTED_COLLECTOR_ORIGINS` and points `PRODUCTION_COLLECTOR_URL` at it. + Merging it publishes a release to PyPI. +- **Who picks it up:** + - Builds that start the separately installed emitter (emitter #21 and #22, + and Microcosm's H2a) use the newest installed emitter at their next + start. They need no Microcosm change. A running build keeps the emitter it + started with. + - Builds pinned to `microcosm-emitter==0.2.1` (Microcosm's + `packages/microcosm-build/pyproject.toml` today) keep the old address + until Microcosm moves off that pin. +- **If the old address stops answering,** a build that still uses it keeps + its telemetry on its own machine: the emitter retries with a backoff of up + to 60 seconds and keeps undelivered events for seven days (up to 100 MiB). + A build never fails because of telemetry. + +### The calibration dashboard (calibration-diagnostics) + +| Setting | Provider | Read by | Changes under D2 B | Changes under D3 B or C | +|---|---|---|---|---| +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | GitHub `Production` environment variable; also set in the Vercel project for Production and Preview | `deploy.yml` passes the GitHub value to `.github/scripts/deploy-dashboard.sh`, which requires it and sets it on the production deployment with `vercel deploy --env`. Preview deployments use the Vercel value. The server reads it in `frontend/lib/server/microcosm/telemetry-collector-client.ts`. | the new default address (if D3 is A) | the custom hostname, in all three places (GitHub Production, Vercel Production, and Vercel Preview with the staging address) | +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel project setting, Production and Preview | `telemetry-collector-client.ts`: the full Secret Manager version resource, which names the project | the new project's read secrets | no | +| `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` | Vercel project setting, Production only | `frontend/lib/server/secret-manager.ts`, on Vercel: the Workload Identity provider resource, named by project number | only if the provider moves (see choice 10) | no | +| `TELEMETRY_GCP_PROJECT_ID`, `TELEMETRY_READ_SECRET_NAME` | GitHub `Production` environment variables | `deploy.yml` step "Check collector read secret exists" | the new project | no | +| `TELEMETRY_GCP_WIF_PROVIDER`, `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | GitHub `Production` environment secrets | the same step's Google authentication | no, if the dashboard's own identity in `policyengine-api` is granted metadata read on the new read secret | no | + +Two things found while checking: + +- **Preview has no `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`.** On a + Preview deployment the server can't read its read token ("Configure + MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER."), so the build monitor shows + only the Hugging Face runs and lists `collector` under "Runs that could not + be read". Whether that's intended wasn't checked. It is independent of D2 + and D3. +- **After C6,** the dashboard authenticates to the collector with its own + Vercel OIDC token instead of a shared read token, so it no longer reads + Secret Manager for the collector. If C6 lands before the move, only + `MICROCOSM_TELEMETRY_COLLECTOR_URL` changes. + +Settings take effect on the next deployment: a merge to calibration-diagnostics +`main`, or a re-run of its latest "Deploy production" run. + +### microcosm-runs + +- **Where:** `PRODUCTION_COLLECTOR_ORIGIN` in `lib/auth.ts`, overridable with + the server-only setting `ORRERY_COLLECTOR_ORIGIN`; `.env.example` repeats the + address, and the README names the setting. The Vercel project sets no + `ORRERY_COLLECTOR_ORIGIN` (checked with `vercel env ls`), so production uses + the tracked default. +- **What uses it:** only the upload API's authorization call, + `POST /v1/auth/graph-publication/authorize`. The collector doesn't implement + that endpoint (calibration-diagnostics#208 was closed), and O7 removes the + upload API. +- **So:** the move breaks nothing there. Either update the constant and + `.env.example` in a small PR when the emitter's default changes, or leave + them for O7 to delete (choice 11). If O6 later reads collector outputs + through its API, it should use the custom hostname from the start. + +### Microcosm + +No change. It doesn't name the address, and builds don't set +`collector_origin`. Its emitter pin decides how long builds keep the old +address. + +### The collector itself + +- **GitHub `staging` and `production` environments:** every project-specific + value changes. The runbook lists them. +- **Session tokens:** their issuer and audience are names + (`TELEMETRY_JWT_ISSUER` and `TELEMETRY_JWT_AUDIENCE` default to + `policyengine-telemetry` and `microcosm-telemetry-collector`), not + addresses, so they don't change. C5 publishes a JWKS; whoever fetches it + will use the collector's address, which is one more reason to fix the + address before consumers depend on it. +- **The README** says the production address isn't repeated in + documentation. A custom hostname is stable, so it could be documented. + +## Order and timing + +The plan's constraints: + +- The move happens only after the collector deploy switch-over (C1) is + verified. From then on, every merge to this repo's `main` deploys + production, so nothing else merges during a window. +- calibration-diagnostics' `telemetry-service.yml` can still redeploy the + collector for rollback until calibration-diagnostics#234 merges. Its + settings name the old project, so merge #234 before the move, or never + dispatch that workflow afterwards. +- The new project has to include whatever has merged by then: #12's intake + resources, S1's secret and its mount, C5's signing key. + +If both D2 and D3 are adopted, there are two orders: + +1. **Hostname first.** Point the custom hostname at today's service in + `policyengine-api`, release the emitter with the hostname as its default, + and wait until builds use it (after H2a, and after Microcosm leaves the + 0.2.1 pin). Then move the project behind the hostname: only the hostname's + target changes, and the old default address can be retired, because no + build uses it. With the load balancer (D3 C), the certificate is issued in + the new project before the DNS change. With domain mapping (D3 B), the + mapping is re-created in the new project, and requests fail TLS checks + until the new certificate is issued (the emitter retries). +2. **Together.** Create the new project and the hostname, move each database + in one window, then release the emitter. Builds keep the old address until + they start a newer emitter, and builds pinned to 0.2.1 keep it until + Microcosm moves. Meanwhile the old service either keeps serving, connected + to the new database across projects, or stays in maintenance mode, and + those builds' telemetry stays on their machines (choice 7). + +## The choices to make + +1. **D2:** A (stay), B (a dedicated project, one window per environment) or C + (a dedicated project, database later). The plan recommends B. +2. **If B or C:** the project ID and display name; its parent organization or + folder; the billing account; who holds the owner and IAM-admin roles. +3. **D3:** A (default address), B (Cloud Run domain mapping) or C (load + balancer). The plan recommends a custom domain without naming a mechanism. +4. **Hostnames:** the production hostname, for example + `collector.microcosm.institute`; and whether staging gets one, for example + `collector-staging.microcosm.institute`, or keeps its default address. +5. **DNS:** which zone (`microcosm.institute` on Cloudflare, or + `policyengine.org` on Cloud DNS); who adds the records; which Google account + verifies the base domain in Search Console. +6. **Order:** hostname first, or both together. +7. **The old address after the move:** keep the old service serving, + connected to the new database, until no build uses emitter 0.2.1; or leave + it in maintenance mode, so builds still on it keep their telemetry locally; + or delete it. A redirect isn't possible. +8. **The window:** when, and how long is acceptable. During it, running builds + keep their events locally and deliver them afterwards, and the build monitor + shows only Hugging Face runs. +9. **Staging data:** dump and restore it like production (which also rehearses + the production move), or start staging empty. +10. **The dashboard's Google identity:** create a new Vercel Workload Identity + provider in the new project (changes + `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`), or keep it in + `policyengine-api` and grant its identities access to the new secrets + across projects (no Vercel change, but a lasting dependency on the shared + project), or finish C6 first so the dashboard needs neither. +11. **microcosm-runs:** update its default address with the emitter release, + or leave it for O7. diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md new file mode 100644 index 0000000..ed9da15 --- /dev/null +++ b/docs/project-move-runbook.md @@ -0,0 +1,743 @@ +# Moving the collector to its own project and hostname (runbook) + +This is plan item C2. Use it only when both of these hold: + +- **The collector deploy switch-over (C1) is verified:** this repo deploys + staging and production, and every check in the "Verify the deployment from + microcosm-collector" section of calibration-diagnostics#233 has passed. +- **Decisions D2 and D3 are made.** See + [`project-and-domain-decision.md`](project-and-domain-decision.md); the + numbered choices there are referred to below as "choice N". + +The runbook assumes D2 option B (a dedicated project, one maintenance window +per environment) and a custom hostname (D3 B, Cloud Run domain mapping, or D3 +C, a load balancer). Steps that depend on a choice say so. It hasn't been +rehearsed. Moving staging first is the rehearsal, so check each command against +the decided names there before running it for production. + +No command here prints a secret value. Values are generated in the shell and +piped into Secret Manager, or piped from one Secret Manager secret into +another. + +## 1. People and access + +- **Google Cloud:** permission to create a project under the chosen parent and + link the billing account; then the owner role on the new project. Read access + to `policyengine-api`, and Cloud SQL admin on the old instances for the + export. +- **DNS:** edit access to the chosen zone (choice 5), and a Google account that + is a verified owner of the base domain in Search Console. +- **GitHub:** an admin of PolicyEngine/microcosm-collector (environment + settings), and someone who can change the calibration-diagnostics + `Production` environment and merge in both repos. +- **Vercel:** access to the calibration-diagnostics project's settings. +- **The emitter:** someone who can merge in PolicyEngine/microcosm-emitter + (each merge publishes a release). + +## 2. Names used below + +Set these once per environment, in a `bash` shell. Every name a choice decides +is marked. + +```bash +OLD_PROJECT_ID=policyengine-api +NEW_PROJECT_ID=... # choice 2 +ENVIRONMENT_NAME=staging # then production +COLLECTOR_HOSTNAME=... # choice 4, for this environment + +collector_variable() { + gh api "repos/PolicyEngine/microcosm-collector/environments/$ENVIRONMENT_NAME/variables/$1" --jq .value +} +REGION=$(collector_variable TELEMETRY_GCP_REGION) +SERVICE=$(collector_variable TELEMETRY_SERVICE) +MIGRATION_JOB=$(collector_variable TELEMETRY_MIGRATION_JOB) +ARTIFACT_REGISTRY_NAME=$(collector_variable TELEMETRY_ARTIFACT_REGISTRY_NAME) +OLD_CLOUD_SQL_CONNECTION=$(collector_variable TELEMETRY_CLOUD_SQL_CONNECTION) +OLD_INSTANCE=${OLD_CLOUD_SQL_CONNECTION##*:} +DATABASE_SECRET_NAME=$(collector_variable TELEMETRY_DATABASE_SECRET_NAME) +MIGRATION_DATABASE_SECRET_NAME=$(collector_variable TELEMETRY_MIGRATION_DATABASE_SECRET_NAME) +JWT_SECRET_NAME=$(collector_variable TELEMETRY_JWT_SECRET_NAME) +READ_SECRET_NAME=$(collector_variable TELEMETRY_READ_SECRET_NAME) + +# Names are scoped to a project, so the new resources can reuse the old ones. +NEW_INSTANCE=$OLD_INSTANCE +NEW_CLOUD_SQL_CONNECTION="$NEW_PROJECT_ID:$REGION:$NEW_INSTANCE" +RUNTIME_SERVICE_ACCOUNT="collector-runtime-$ENVIRONMENT_NAME@$NEW_PROJECT_ID.iam.gserviceaccount.com" +MIGRATION_SERVICE_ACCOUNT="collector-migration-$ENVIRONMENT_NAME@$NEW_PROJECT_ID.iam.gserviceaccount.com" +DEPLOY_SERVICE_ACCOUNT="collector-deploy@$NEW_PROJECT_ID.iam.gserviceaccount.com" +EXPORT_BUCKET="gs://$NEW_PROJECT_ID-collector-move" +``` + +`DATABASE_NAME`, `RUNTIME_DATABASE_USER` and `MIGRATION_DATABASE_USER` come +from the old instance in step 3.2. + +## 3. Before the windows + +### 3.1 Check the prerequisites + +- `COLLECTOR_DEPLOY_ENABLED` is `true`, and C1 is verified. +- calibration-diagnostics#234 has merged, or everyone has agreed never to + dispatch calibration-diagnostics' `telemetry-service.yml` again. Its settings + name the old project. +- PolicyEngine/microcosm-emitter#29 has merged, so changing the emitter's + default is a constant change (section 8.2). +- List what has merged here that needs its own resources in the new project, + and add those steps where this runbook marks them: #12's artifact intake (a + bucket, a Cloud Tasks queue, a worker job and their permissions, as in the + README's "Artifact intake" section), S1's Hugging Face read token (#13), and + C5's session signing key. + +### 3.2 Record the current state + +For each environment (read-only): + +- The service's URL, serving revision, image, Cloud SQL connection and secret + names: the `gcloud run services describe` command in step 0 of + calibration-diagnostics#233. The serving revision is the rollback target. +- The instance's settings: + `gcloud sql instances describe "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --format=yaml`. + Keep the database version, edition, tier, disk size, availability type, + backup and point-in-time-recovery settings, maintenance window, database + flags, and IP and SSL configuration. +- The database and its users: + `gcloud sql databases list` and `gcloud sql users list` with + `--instance "$OLD_INSTANCE" --project "$OLD_PROJECT_ID"`. Set + `DATABASE_NAME`, `RUNTIME_DATABASE_USER` and `MIGRATION_DATABASE_USER` from + them. +- **The database's roles and privileges.** Connect as the migration user + through Cloud SQL Studio, or with `psql` through the Cloud SQL Auth Proxy, + and save the output of `\du`, `\l`, `\dn+ public`, `\dp public.*` and + `\ddp`. Note which role owns the tables and what the runtime user is + granted. These were provisioned by scripts kept outside this repo, so this + record is the only specification of them. +- Who may read each secret: + `gcloud secrets get-iam-policy SECRET --project "$OLD_PROJECT_ID" --format=yaml` + for each of the four secret names. The read secret's policy names the + dashboard's Vercel identity. +- The deployment account's roles. Its email is the + `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` secret, which GitHub can't show. It is + the account whose IAM policy grants `roles/iam.workloadIdentityUser` to this + repo's principal set (calibration-diagnostics#233 checked that binding); + `gcloud iam service-accounts get-iam-policy ACCOUNT --project "$OLD_PROJECT_ID"` + shows it. Then list its roles on the project with + `gcloud projects get-iam-policy "$OLD_PROJECT_ID" --flatten=bindings --filter="bindings.members:serviceAccount:DEPLOY_ACCOUNT" --format="value(bindings.role)"`, + and its roles on individual resources (the runtime and migration accounts, + the secrets and the Artifact Registry store) from their IAM policies. +- The dashboard's Vercel identity provider: the resource named by the Vercel + setting `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`, read with + `gcloud iam workload-identity-pools providers describe` (issuer, allowed + audiences, attribute mapping and condition). + +### 3.3 Agree the windows + +One for staging, one for production (choice 8). From the start of a window +until its verification passes, nothing merges to this repo's `main`: every +merge deploys. + +## 4. Create the project (once, no downtime) + +```bash +gcloud projects create "$NEW_PROJECT_ID" --name "DISPLAY_NAME" --organization ORGANIZATION_ID # or --folder FOLDER_ID +gcloud billing projects link "$NEW_PROJECT_ID" --billing-account BILLING_ACCOUNT_ID +NEW_PROJECT_NUMBER=$(gcloud projects describe "$NEW_PROJECT_ID" --format='value(projectNumber)') +gcloud services enable run.googleapis.com sqladmin.googleapis.com \ + secretmanager.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \ + iamcredentials.googleapis.com sts.googleapis.com storage.googleapis.com \ + --project "$NEW_PROJECT_ID" +``` + +Also enable `compute.googleapis.com` and `certificatemanager.googleapis.com` +for a load balancer (D3 C), and `cloudtasks.googleapis.com` if #12's intake is +switched on. + +### 4.1 Artifact Registry + +Create a Docker-format Artifact Registry store named `$ARTIFACT_REGISTRY_NAME` +in the region `$REGION`, in standard mode (in the console: Artifact Registry, +then Create, format Docker, location type Region). Google's +`gcloud artifacts` command for this, in its quickstart "Store Docker container +images in Artifact Registry", names its subcommand and flag after Google's word +for the store, which this repo's vocabulary test keeps out of documentation. +`scripts/deploy.sh` pushes to +`${REGION}-docker.pkg.dev/${PROJECT_ID}/${ARTIFACT_REGISTRY_NAME}/${IMAGE_NAME}`, +so the store must be in the new project. + +### 4.2 The deployment account and Workload Identity Federation for this repo + +The new project gets a deployment account used only by this repo. The dashboard +keeps its own deployment identity in `policyengine-api`. + +```bash +gcloud iam service-accounts create collector-deploy --project "$NEW_PROJECT_ID" \ + --display-name "microcosm-collector deployments" +gcloud iam workload-identity-pools create github --project "$NEW_PROJECT_ID" \ + --location global --display-name "GitHub Actions" +gcloud iam workload-identity-pools providers create-oidc microcosm-collector \ + --project "$NEW_PROJECT_ID" --location global --workload-identity-pool github \ + --issuer-uri https://token.actions.githubusercontent.com \ + --attribute-mapping "google.subject=assertion.sub,attribute.actor=assertion.actor,attribute.ref=assertion.ref,attribute.repository=assertion.repository" \ + --attribute-condition "assertion.repository=='PolicyEngine/microcosm-collector' && assertion.ref=='refs/heads/main'" +gcloud iam service-accounts add-iam-policy-binding "$DEPLOY_SERVICE_ACCOUNT" \ + --project "$NEW_PROJECT_ID" --role roles/iam.workloadIdentityUser \ + --member "principalSet://iam.googleapis.com/projects/$NEW_PROJECT_NUMBER/locations/global/workloadIdentityPools/github/attribute.repository/PolicyEngine/microcosm-collector" +``` + +The mapping and condition are the ones in the README's "Google Cloud access" +section. The provider's resource name, for the `TELEMETRY_GCP_WIF_PROVIDER` +secret, is +`projects/$NEW_PROJECT_NUMBER/locations/global/workloadIdentityPools/github/providers/microcosm-collector`. + +What `scripts/deploy.sh` needs the deployment account to do, and the role that +allows it. Compare with the record from step 3.2 before granting: + +| `deploy.sh` command | Needs | Grant | +|---|---|---| +| `gcloud run deploy`, `run jobs deploy` and `execute`, `services describe`, `update-traffic`, `--allow-unauthenticated` | Cloud Run administration | `roles/run.admin` on the project | +| `--service-account` for the service and the migration job | acting as those accounts | `roles/iam.serviceAccountUser` on each runtime and migration account (step 5.1) | +| `build-and-push-image` | pushing images | `roles/artifactregistry.writer` on the store | +| `check-secrets-exist` (`gcloud secrets describe`) | reading secret metadata | `roles/secretmanager.viewer` on each secret (step 5.3) | +| `create-database-backup` (production) | `cloudsql.backupRuns.create` | `roles/cloudsql.editor` on the project, or a custom role with `cloudsql.backupRuns.create` and `cloudsql.instances.get` | + +### 4.3 The export bucket + +It will hold complete copies of the telemetry databases, so it is private and +is deleted after the move (section 11). + +```bash +gcloud storage buckets create "$EXPORT_BUCKET" --project "$NEW_PROJECT_ID" \ + --location "$REGION" --uniform-bucket-level-access --public-access-prevention +``` + +## 5. Create each environment's resources (no downtime) + +Staging first. Production's can be created days before its window. + +### 5.1 Service accounts + +```bash +gcloud iam service-accounts create "collector-runtime-$ENVIRONMENT_NAME" --project "$NEW_PROJECT_ID" +gcloud iam service-accounts create "collector-migration-$ENVIRONMENT_NAME" --project "$NEW_PROJECT_ID" +for service_account in "$RUNTIME_SERVICE_ACCOUNT" "$MIGRATION_SERVICE_ACCOUNT"; do + gcloud projects add-iam-policy-binding "$NEW_PROJECT_ID" \ + --member "serviceAccount:$service_account" --role roles/cloudsql.client + gcloud iam service-accounts add-iam-policy-binding "$service_account" \ + --project "$NEW_PROJECT_ID" --role roles/iam.serviceAccountUser \ + --member "serviceAccount:$DEPLOY_SERVICE_ACCOUNT" +done +``` + +If the record from step 3.2 shows these accounts limited further today (for +example, an IAM condition on the instance name), repeat that here. + +### 5.2 Cloud SQL + +Create the instance with the settings recorded in step 3.2. Keep deletion +protection on. + +```bash +gcloud sql instances create "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --region "$REGION" \ + --database-version DATABASE_VERSION --edition EDITION --tier TIER \ + --storage-size DISK_SIZE --availability-type AVAILABILITY_TYPE \ + --backup-start-time BACKUP_START_TIME --deletion-protection +gcloud sql databases create "$DATABASE_NAME" --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" +``` + +Match the record's IP configuration (Cloud Run connects through the Cloud SQL +connector, so a public IP needs no authorized networks), SSL mode, database +flags, maintenance window and point-in-time recovery. + +Create the two users with the **same names** as on the old instance, so the +ownership and `GRANT` statements in the export apply, and new random passwords: + +```bash +migration_password="$(openssl rand -hex 32)" +runtime_password="$(openssl rand -hex 32)" +gcloud sql users create "$MIGRATION_DATABASE_USER" --instance "$NEW_INSTANCE" \ + --project "$NEW_PROJECT_ID" --password "$migration_password" +gcloud sql users create "$RUNTIME_DATABASE_USER" --instance "$NEW_INSTANCE" \ + --project "$NEW_PROJECT_ID" --password "$runtime_password" +``` + +Keep the two password variables in this shell for step 5.3. Then recreate what +an export doesn't carry: the database-level privileges (`GRANT CONNECT` and +`CREATE ON DATABASE`) and role memberships from the `\l` and `\du` record. +Table privileges, schema privileges and default privileges (`\dp`, `\dn+`, +`\ddp`) come with the export, provided the roles exist. + +Let both instances' service agents use the bucket. Google's export and import +pages name `roles/storage.objectAdmin` for both: + +```bash +for sql_service_agent in \ + "$(gcloud sql instances describe "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --format='value(serviceAccountEmailAddress)')" \ + "$(gcloud sql instances describe "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --format='value(serviceAccountEmailAddress)')"; do + gcloud storage buckets add-iam-policy-binding "$EXPORT_BUCKET" \ + --member "serviceAccount:$sql_service_agent" --role roles/storage.objectAdmin +done +``` + +### 5.3 Secret Manager + +```bash +for secret_name in "$DATABASE_SECRET_NAME" "$MIGRATION_DATABASE_SECRET_NAME" "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do + gcloud secrets create "$secret_name" --project "$NEW_PROJECT_ID" --replication-policy automatic +done +``` + +**The two database URLs are new:** the new instance and the new passwords. +First see the form of the old value with its credentials hidden: + +```bash +gcloud secrets versions access latest --secret "$DATABASE_SECRET_NAME" --project "$OLD_PROJECT_ID" \ + | sed -E 's#//[^@]*@#//USER:PASSWORD@#' +``` + +Build the new values in the same form. For the usual Cloud SQL socket form: + +```bash +printf 'postgresql://%s:%s@/%s?host=/cloudsql/%s' "$RUNTIME_DATABASE_USER" "$runtime_password" \ + "$DATABASE_NAME" "$NEW_CLOUD_SQL_CONNECTION" \ + | gcloud secrets versions add "$DATABASE_SECRET_NAME" --project "$NEW_PROJECT_ID" --data-file=- +printf 'postgresql://%s:%s@/%s?host=/cloudsql/%s' "$MIGRATION_DATABASE_USER" "$migration_password" \ + "$DATABASE_NAME" "$NEW_CLOUD_SQL_CONNECTION" \ + | gcloud secrets versions add "$MIGRATION_DATABASE_SECRET_NAME" --project "$NEW_PROJECT_ID" --data-file=- +unset runtime_password migration_password +``` + +**The signing secret and the read token are copied,** so session tokens issued +before the move stay valid afterwards (which also lets the old service keep +serving, choice 7), and the dashboard's credential doesn't change: + +```bash +for secret_name in "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do + gcloud secrets versions access latest --secret "$secret_name" --project "$OLD_PROJECT_ID" \ + | gcloud secrets versions add "$secret_name" --project "$NEW_PROJECT_ID" --data-file=- +done +``` + +Who may read them: + +```bash +for secret_name in "$DATABASE_SECRET_NAME" "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do + gcloud secrets add-iam-policy-binding "$secret_name" --project "$NEW_PROJECT_ID" \ + --member "serviceAccount:$RUNTIME_SERVICE_ACCOUNT" --role roles/secretmanager.secretAccessor +done +gcloud secrets add-iam-policy-binding "$MIGRATION_DATABASE_SECRET_NAME" --project "$NEW_PROJECT_ID" \ + --member "serviceAccount:$MIGRATION_SERVICE_ACCOUNT" --role roles/secretmanager.secretAccessor +for secret_name in "$DATABASE_SECRET_NAME" "$MIGRATION_DATABASE_SECRET_NAME" "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do + gcloud secrets add-iam-policy-binding "$secret_name" --project "$NEW_PROJECT_ID" \ + --member "serviceAccount:$DEPLOY_SERVICE_ACCOUNT" --role roles/secretmanager.viewer +done +``` + +The dashboard's access to the read secret is set in section 8.1. + +**If merged by now:** + +- **S1's Hugging Face read token (#13):** create the secret under the name S1 + records, and copy its value with the same `versions access | versions add` + pipe, or rotate it: create a new fine-grained token with the same scope in + Hugging Face, store that, and revoke the old token in section 11. Grant + `roles/secretmanager.secretAccessor` only to the account the worker jobs run + as, and mount it into the worker jobs as `COLLECTOR_HF_READ_TOKEN`. +- **C5's session signing key:** copy it the same way, so tokens signed before + the move verify afterwards and the key ID published in the JWKS doesn't + change. +- **#12's intake:** the bucket, queue, worker job, invoker account and + permissions from the README's "Artifact intake" section, in the new project. + Copy the intake objects in the window (step 6.3), and check whether stored + submission rows name the bucket. + +### 5.4 Create the Cloud Run service once + +`deploy-maintenance` deploys with `--no-traffic`, which Cloud Run refuses when +it creates a service. So create the service once with Google's sample image. +Nothing uses its address yet, and the first deployment replaces the revision. + +```bash +gcloud run deploy "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" \ + --image us-docker.pkg.dev/cloudrun/container/hello \ + --service-account "$RUNTIME_SERVICE_ACCOUNT" --no-allow-unauthenticated --quiet +``` + +The migration job needs no step: `configure-migration-job` runs +`gcloud run jobs deploy`, which creates it. + +### 5.5 Production only: time the database copy + +Before production's window, export the production database with `--offload` +(a serverless export, billed separately, which doesn't load the serving +instance), import it into the new instance as in step 6.3, and note both +durations. Then delete and re-create the new database +(`gcloud sql databases delete`, then `create`), so the window's import starts +empty. Staging's window gives the same timings for staging's size. + +## 6. The window + +Do staging end to end, verify it (section 9), then production. + +While the window lasts: + +- **Builds:** the emitter gets HTTP 503 from the collector and retries, with + a backoff of up to 60 seconds. Events wait on the build machine (kept for + seven days, up to 100 MiB). A build never fails because of telemetry. +- **The dashboard:** the build monitor lists `collector` under "Runs that + could not be read" and still shows the Hugging Face runs. +- **Orrery uploads** don't use the collector today. + +### 6.1 Check that no deployment is running + +```bash +gh run list -R PolicyEngine/microcosm-collector --workflow push.yaml --status in_progress +gh run list -R PolicyEngine/microcosm-collector --workflow push.yaml --status queued +``` + +Both must list nothing. + +### 6.2 Put the old service in maintenance mode + +`services update` makes a new revision from the serving one with only +`TELEMETRY_MAINTENANCE_MODE` changed, so it keeps the same image, secrets and +database: + +```bash +gcloud run services update "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" \ + --update-env-vars TELEMETRY_MAINTENANCE_MODE=1 --no-traffic --tag project-move --quiet +gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" \ + --to-tags project-move=100 --quiet +old_service_url=$(gcloud run services describe "$SERVICE" --project "$OLD_PROJECT_ID" \ + --region "$REGION" --format='value(status.url)') +curl --silent --output /dev/null --write-out '%{http_code}\n' \ + --request POST "$old_service_url/v1/auth/huggingface/exchange" +sleep 35 +``` + +The `curl` must print `503`, as `resolve-and-verify-maintenance` checks. The +pause outlasts the 30-second request timeout, so no write is still running. +From here the old database doesn't change. + +### 6.3 Export and import + +```bash +export_object="$EXPORT_BUCKET/$ENVIRONMENT_NAME-$(date -u +%Y%m%dT%H%M%SZ).sql.gz" +gcloud sql export sql "$OLD_INSTANCE" "$export_object" \ + --database "$DATABASE_NAME" --project "$OLD_PROJECT_ID" +gcloud sql import sql "$NEW_INSTANCE" "$export_object" \ + --database "$DATABASE_NAME" --user "$MIGRATION_DATABASE_USER" --project "$NEW_PROJECT_ID" --quiet +``` + +Import as the role that owns the tables in the step 3.2 record (normally the +migration user), so the restored objects keep their owner. If #12's intake is +switched on, copy the intake objects too: +`gcloud storage cp --recursive gs://OLD_INTAKE_BUCKET/* gs://NEW_INTAKE_BUCKET/`. + +### 6.4 Check the copy + +On both instances, as the migration user: + +- `SELECT version_num FROM alembic_version;` gives the same revision. +- `SELECT count(*)` on every table in the `public` schema gives the same + counts. At revision `0001` these are `telemetry_runs`, `telemetry_producers` + and `telemetry_events`; later migrations add more. +- `\dp public.*` and `\ddp` match the step 3.2 record. + +If anything differs, stop and roll back (section 10, first row). + +### 6.5 Point the GitHub environment at the new project + +```bash +collector_settings=(--repo PolicyEngine/microcosm-collector --env "$ENVIRONMENT_NAME") +gh variable set TELEMETRY_GCP_PROJECT_ID "${collector_settings[@]}" --body "$NEW_PROJECT_ID" +gh variable set TELEMETRY_CLOUD_SQL_CONNECTION "${collector_settings[@]}" --body "$NEW_CLOUD_SQL_CONNECTION" +gh variable set TELEMETRY_RUNTIME_SERVICE_ACCOUNT "${collector_settings[@]}" --body "$RUNTIME_SERVICE_ACCOUNT" +gh variable set TELEMETRY_MIGRATION_SERVICE_ACCOUNT "${collector_settings[@]}" --body "$MIGRATION_SERVICE_ACCOUNT" +gh secret set TELEMETRY_GCP_WIF_PROVIDER "${collector_settings[@]}" \ + --body "projects/$NEW_PROJECT_NUMBER/locations/global/workloadIdentityPools/github/providers/microcosm-collector" +gh secret set TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT "${collector_settings[@]}" --body "$DEPLOY_SERVICE_ACCOUNT" +``` + +In production's window, also set `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` +to production's new connection in **both** environments, since staging's +`validate-environment-target` compares against it. The other variables keep +their values if the new resources reused the old names (section 12). + +### 6.6 Deploy + +Merge the C2 code PR (section 13). Its release runs the Push workflow, which +deploys staging and then production with the environments' current settings. +In staging's window, production redeploys in the old project with unchanged +code: a normal deployment, with its usual short maintenance phase. + +Re-running the deployment jobs of the latest Push run might do the same +without a merge, but whether a re-run reads the environments' changed values +hasn't been checked. In staging's window, confirm it from the run log (the +"Require deployment configuration" step shows `PROJECT_ID`) before relying on +it for production. In production, a re-run that used the old values would take +the old service out of maintenance mode, on the old database. + +The first deployment replaces the sample revision with a maintenance revision, +runs the migration job (the restored database is already at the current +revision, and the job checks the schema against the SQLAlchemy metadata), +deploys and checks a candidate revision, and moves traffic to it. In +production it also takes a backup of the new instance first. + +## 7. The hostname and DNS + +Records at Cloudflare must be "DNS only". If the zone ever gets CAA records, +they must allow `pki.goog` and `letsencrypt.org`. + +### D3 B: Cloud Run domain mapping + +```bash +gcloud domains verify BASE_DOMAIN # once, by the account that creates mappings +gcloud beta run domain-mappings create --service "$SERVICE" --domain "$COLLECTOR_HOSTNAME" \ + --region "$REGION" --project "$NEW_PROJECT_ID" +gcloud beta run domain-mappings describe --domain "$COLLECTOR_HOSTNAME" --region "$REGION" \ + --project "$NEW_PROJECT_ID" --format='yaml(status.resourceRecords,status.conditions)' +``` + +Add the records `describe` lists at the DNS provider, then wait for the +`CertificateProvisioned` condition to be `True`: "usually about 15 minutes but +can take up to 24 hours." + +- **Both together (choice 6):** nothing uses the hostname yet, so map it any + time after the first deployment, and wait for the certificate before the + emitter release (section 8.2). +- **Hostname first:** the hostname is already mapped to the old project's + service. Re-create the mapping in the new project with `--force-override` + inside production's window, after the deployment. That moves the hostname + at once, and requests fail TLS checks until the new certificate is issued; + the emitter retries. + +### D3 C: load balancer + +```bash +gcloud compute addresses create collector-address --global --project "$NEW_PROJECT_ID" +gcloud certificate-manager dns-authorizations create collector-hostname \ + --domain "$COLLECTOR_HOSTNAME" --project "$NEW_PROJECT_ID" +gcloud certificate-manager dns-authorizations describe collector-hostname \ + --project "$NEW_PROJECT_ID" --format='yaml(dnsResourceRecord)' +``` + +Add that CNAME at the DNS provider. It authorizes the certificate and moves no +traffic. Then: + +```bash +gcloud certificate-manager certificates create collector-certificate \ + --domains "$COLLECTOR_HOSTNAME" --dns-authorizations collector-hostname --project "$NEW_PROJECT_ID" +gcloud certificate-manager maps create collector-certificates --project "$NEW_PROJECT_ID" +gcloud certificate-manager maps entries create collector-hostname --map collector-certificates \ + --certificates collector-certificate --hostname "$COLLECTOR_HOSTNAME" --project "$NEW_PROJECT_ID" +gcloud compute network-endpoint-groups create collector-endpoints --region "$REGION" \ + --network-endpoint-type serverless --cloud-run-service "$SERVICE" --project "$NEW_PROJECT_ID" +gcloud compute backend-services create collector-backend --global \ + --load-balancing-scheme EXTERNAL_MANAGED --project "$NEW_PROJECT_ID" +gcloud compute backend-services add-backend collector-backend --global \ + --network-endpoint-group collector-endpoints --network-endpoint-group-region "$REGION" \ + --project "$NEW_PROJECT_ID" +gcloud compute url-maps create collector-routes --default-service collector-backend \ + --global --project "$NEW_PROJECT_ID" +gcloud compute target-https-proxies create collector-proxy --url-map collector-routes \ + --certificate-map collector-certificates --global --project "$NEW_PROJECT_ID" +gcloud compute forwarding-rules create collector-https --global \ + --load-balancing-scheme EXTERNAL_MANAGED --address collector-address \ + --target-https-proxy collector-proxy --ports 443 --project "$NEW_PROJECT_ID" +``` + +Wait until +`gcloud certificate-manager certificates describe collector-certificate --project "$NEW_PROJECT_ID" --format='value(managed.state)'` +prints `ACTIVE`. Then point the hostname's `A` record at the address +(`gcloud compute addresses describe collector-address --global --project "$NEW_PROJECT_ID" --format='value(address)'`). +Lower the record's TTL to 300 seconds a day before, so a rollback propagates +quickly. In the hostname-first order, the old project's certificate +authorization record may still exist at the same name; Certificate Manager's +per-project DNS authorizations (`--type=PER_PROJECT_RECORD`) let both exist at +once. Check its documentation before relying on that. + +### Check the hostname + +```bash +curl -fsS "https://$COLLECTOR_HOSTNAME/ready" +curl --silent --output /dev/null --write-out '%{http_code}\n' \ + --request POST "https://$COLLECTOR_HOSTNAME/v1/auth/huggingface/exchange" +``` + +The first must succeed. The second must print `401` (no credential sent), which +shows the request reached the application rather than a maintenance revision. + +## 8. Point the consumers at the new collector + +### 8.1 The calibration dashboard + +**Give the dashboard's identity access to the new read secrets first** (choice +10): + +- **Keep its provider in `policyengine-api`:** grant the principal (or + principals) that the step 3.2 record lists on the old read secret the same + role on the new one: + `gcloud secrets add-iam-policy-binding "$READ_SECRET_NAME" --project "$NEW_PROJECT_ID" --member "PRINCIPAL" --role roles/secretmanager.secretAccessor`. + Production's identity gets only production's read secret, and Preview's only + staging's. +- **A new provider in the new project:** create a pool and provider with the + issuer, allowed audiences, attribute mapping and condition recorded in step + 3.2, grant its principals as above, and set the Vercel setting + `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` to the new provider's + resource. +- **C6 done:** nothing; the dashboard no longer reads Secret Manager for the + collector. + +The dashboard's deployment checks that its read secret exists ("Check +collector read secret exists" in `deploy.yml`). Grant its own deployment +identity, the account in calibration-diagnostics' `Production` secret +`TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT`, `roles/secretmanager.viewer` on +production's new read secret. + +**Then change the settings:** + +| Setting | Where | New value | +|---|---|---| +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | calibration-diagnostics GitHub environment `Production` | `https://$COLLECTOR_HOSTNAME` for production | +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | Vercel project calibration-diagnostics, Production and Preview | production's and staging's addresses | +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Production and Preview | `projects/$NEW_PROJECT_ID/secrets/READ_SECRET/versions/VERSION`, for production's and staging's read secrets, in the form used today | +| `TELEMETRY_GCP_PROJECT_ID` | calibration-diagnostics GitHub environment `Production` | `$NEW_PROJECT_ID` | +| `TELEMETRY_READ_SECRET_NAME` | calibration-diagnostics GitHub environment `Production` | unchanged if the name was reused | + +Settings take effect on the next deployment. Re-run calibration-diagnostics' +latest "Deploy production" run, or merge its next PR, and open a new Preview +deployment. The build monitor should then list collector runs from before the +move. + +### 8.2 The emitter + +In PolicyEngine/microcosm-emitter, one PR: add `https://COLLECTOR_HOSTNAME` +(production's) to `TRUSTED_COLLECTOR_ORIGINS` and point +`PRODUCTION_COLLECTOR_URL` at it in `src/microcosm_emitter/core/constants.py`. +Keep the old origin in the set while the old address serves. Merging it +publishes the release. + +- Builds that start the separately installed emitter pick it up at their next + start. A running build keeps the emitter it started with. +- Builds pinned to `microcosm-emitter==0.2.1` keep the old address until + Microcosm moves off that pin. + +### 8.3 microcosm-runs (choice 11) + +Either change `PRODUCTION_COLLECTOR_ORIGIN` in `lib/auth.ts` and the value in +`.env.example`, or leave both for O7, which removes the upload API that uses +them. Its Vercel project sets no `ORRERY_COLLECTOR_ORIGIN`. + +### 8.4 The old service (choice 7) + +- **Keep serving, connected to the new database:** grant the old runtime + account `roles/cloudsql.client` in the new project and + `roles/secretmanager.secretAccessor` on the new runtime database secret. + Then update the old service, which keeps the image it ran before the move: + + ```bash + gcloud run services update "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" \ + --set-cloudsql-instances "$NEW_CLOUD_SQL_CONNECTION" \ + --update-secrets "DATABASE_URL=projects/$NEW_PROJECT_NUMBER/secrets/$DATABASE_SECRET_NAME:latest" \ + --update-env-vars TELEMETRY_MAINTENANCE_MODE=0 --no-traffic --tag new-database --quiet + ``` + + Check `/ready` on the `new-database` tag's address, then route traffic to it + with `update-traffic --to-tags new-database=100`. Retire it before any + migration that its older code can't work with, and as soon as no build uses + emitter 0.2.1. +- **Stay in maintenance mode:** nothing to do. Builds still on the old address + keep their telemetry locally. +- **Delete it:** see section 11. + +## 9. Verify + +For each environment: + +- `gcloud run services describe "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" --format=json` + shows an image in the new store, tagged with this repo's release commit (whose + message is "Update microcosm-collector"), the new Cloud SQL connection, and + the four secret names. +- `curl -fsS "$(gcloud run services describe "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" --format='value(status.url)')/ready"` + succeeds, and so do the hostname checks in section 7. +- `gcloud run jobs executions list --job "$MIGRATION_JOB" --project "$NEW_PROJECT_ID" --region "$REGION" --limit 1` + shows a successful execution. +- Production: `gcloud sql backups list --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --limit 1` + shows `telemetry-release-RUN_ID-ATTEMPT` for the Push run. +- The row counts from step 6.4 still match. +- After section 8.1: the build monitor lists runs from before the move and + doesn't list `collector` under "Runs that could not be read". +- After section 8.2: the next build that starts the new emitter shows its + events in the build monitor. +- The old service answers as choice 7 says: `503` in maintenance mode, or + `/ready` when kept serving. + +## 10. Rollback + +Keep every old resource until section 11. The old database is unchanged from +the moment the old service entered maintenance mode. + +| Situation | What to do | +|---|---| +| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. Nothing is lost: builds kept their events and deliver them to the old service. | +| A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | +| After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | +| After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to its pre-move revision, set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | +| The hostname | D3 B: delete the mapping, or map the hostname back to the old project's service with `--force-override`. D3 C: point the `A` record back to what it named before, if anything. | + +## 11. Retire the old resources + +Only when no build uses the old address (Microcosm has left the 0.2.1 pin, and +the old service's request count in Cloud Monitoring has stayed at zero), and +after an agreed holding period: + +- The old Cloud SQL instances: export a last copy into the bucket if wanted, + then `gcloud sql instances patch "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --no-deletion-protection` + and `gcloud sql instances delete`. +- The old Cloud Run services and migration jobs, the four secrets per + environment, and the old runtime and migration service accounts. +- The collector's images in the old Artifact Registry store (only the + collector's image name, unless the store holds nothing else). +- This repo's Workload Identity provider in `policyengine-api` + (`github/microcosm-collector`) and the old deployment account's + `roles/iam.workloadIdentityUser` binding for this repo's principal set. + Keep calibration-diagnostics' provider and the shared deployment account: + the dashboard's deployment still uses them. +- The export bucket and every dump in it. +- calibration-diagnostics' leftover collector settings, as listed in + calibration-diagnostics#233's "Manual steps". +- S1's old Hugging Face token, if it was rotated. + +## 12. Settings this runbook changes + +| Setting | Provider | Read by | After the move | +|---|---|---|---| +| `TELEMETRY_GCP_PROJECT_ID` | this repo's GitHub environments `staging` and `production` (variable) | `push.yaml` job `env:` as `PROJECT_ID`, then every `scripts/deploy.sh` command | the new project ID | +| `TELEMETRY_CLOUD_SQL_CONNECTION` | same (variable) | `CLOUD_SQL_CONNECTION`: `--add-cloudsql-instances`, `--set-cloudsql-instances`, `create-database-backup` | `NEW_PROJECT_ID:REGION:INSTANCE` | +| `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` | same, in both environments (variable) | `validate-environment-target` | production's new connection | +| `TELEMETRY_RUNTIME_SERVICE_ACCOUNT`, `TELEMETRY_MIGRATION_SERVICE_ACCOUNT` | same (variables) | `--service-account` for the service and the migration job | the new accounts | +| `TELEMETRY_GCP_REGION`, `TELEMETRY_ARTIFACT_REGISTRY_NAME`, `TELEMETRY_IMAGE_NAME`, `TELEMETRY_SERVICE`, `TELEMETRY_PRODUCTION_SERVICE`, `TELEMETRY_MIGRATION_JOB`, and the four `TELEMETRY_*_SECRET_NAME` variables | same (variables) | image path, Cloud Run names, `--set-secrets`, `check-secrets-exist` | unchanged when the new resources reuse the names | +| `TELEMETRY_GCP_WIF_PROVIDER` | same (secret) | `google-github-actions/auth`, `workload_identity_provider` | the new provider's resource | +| `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | same (secret) | `google-github-actions/auth`, `service_account` | the new deployment account | +| `COLLECTOR_DEPLOY_ENABLED` | this repo's variable | the deployment jobs' `if:` | unchanged (`true`) | +| `DATABASE_URL`, `TELEMETRY_JWT_SECRET`, `TELEMETRY_READ_TOKEN` | Google Secret Manager in the new project | the service, through `--set-secrets` | new database URL; copied signing secret and read token | +| `DATABASE_URL` (migration job) | Secret Manager in the new project | the migration job, through `--set-secrets` | new migration database URL | +| `COLLECTOR_HF_READ_TOKEN` (if S1 merged) | Secret Manager in the new project | the worker jobs, through a secret mount | copied or rotated | +| `TELEMETRY_MAINTENANCE_MODE` | `scripts/deploy.sh` (`--set-env-vars`), and step 6.2 on the old service | the service | unchanged | +| Dashboard settings | calibration-diagnostics GitHub `Production` environment and the Vercel project | see section 8.1 | see section 8.1 | +| `PRODUCTION_COLLECTOR_URL`, `TRUSTED_COLLECTOR_ORIGINS` | emitter source (`core/constants.py`) | every emitter service module | the hostname, by an emitter release | + +**Unverified:** nothing in this runbook has been run. The current deployment +account's roles, the database roles and privileges, and the dashboard +identity's bindings were not read for it (they are recorded in step 3.2). +Whether a workflow re-run reads changed environment values is unchecked (step +6.6). Every setting above needs checking end to end in the staging pass. + +## 13. Code changes that go with this runbook (not yet written) + +- **This repo:** a step after "Verify the stable service address" that checks + `https://$PUBLIC_ORIGIN/ready`, from a new environment variable + `TELEMETRY_PUBLIC_ORIGIN`, so every deployment proves the hostname reaches + the new revision. Its merge is the release that deploys into the new project + (step 6.6). The README's "Cloud Run deployment" and "Google Cloud access" + sections then describe the new project and the hostname. +- **The emitter:** the release in section 8.2. +- **microcosm-runs:** section 8.3, if chosen. From 54d49a2f3346ad5371f597df98ddad099f432665 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 22:47:06 +0400 Subject: [PATCH 02/31] Let production deploy between the two windows, and in its own From staging's window on, staging builds the image in the new project and production deploys that same image, still in the old project, so the old deployment account and the old project's Cloud Run service agent need Artifact Registry Reader on the new project. Without it, production's deployment fails in staging's window and on every merge until production moves. The runbook now grants the role before staging's window and removes it when the old resources are retired. Production's window had no merge to deploy it: the C2 code PR merges in staging's window. The runbook now names a second PR for production's window. The stated downtime left out the release pipeline, which sets most of a window's length. The memo and the runbook now give the measured range. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 2 +- docs/project-move-runbook.md | 68 ++++++++++++++++++++++++++--- 2 files changed, 62 insertions(+), 8 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 347801f..1044b90 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -49,7 +49,7 @@ Where the address appears: | Option | Cost | Risks and limits | |---|---|---| | **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | -| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed. Estimate 30 to 60 minutes for production; the staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | +| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners: about 30 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | | **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | What a dedicated project gives: its own permissions (a deployment identity diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index ed9da15..485a288 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -41,6 +41,8 @@ is marked. ```bash OLD_PROJECT_ID=policyengine-api +OLD_PROJECT_NUMBER=$(gcloud projects describe "$OLD_PROJECT_ID" --format='value(projectNumber)') +OLD_DEPLOY_SERVICE_ACCOUNT=... # the current deployment account, from step 3.2 NEW_PROJECT_ID=... # choice 2 ENVIRONMENT_NAME=staging # then production COLLECTOR_HOSTNAME=... # choice 4, for this environment @@ -119,7 +121,8 @@ For each environment (read-only): the account whose IAM policy grants `roles/iam.workloadIdentityUser` to this repo's principal set (calibration-diagnostics#233 checked that binding); `gcloud iam service-accounts get-iam-policy ACCOUNT --project "$OLD_PROJECT_ID"` - shows it. Then list its roles on the project with + shows it. Set `OLD_DEPLOY_SERVICE_ACCOUNT` to it. Then list its roles on the + project with `gcloud projects get-iam-policy "$OLD_PROJECT_ID" --flatten=bindings --filter="bindings.members:serviceAccount:DEPLOY_ACCOUNT" --format="value(bindings.role)"`, and its roles on individual resources (the runtime and migration accounts, the secrets and the Artifact Registry store) from their IAM policies. @@ -162,6 +165,27 @@ for the store, which this repo's vocabulary test keeps out of documentation. `${REGION}-docker.pkg.dev/${PROJECT_ID}/${ARTIFACT_REGISTRY_NAME}/${IMAGE_NAME}`, so the store must be in the new project. +**Let the old project's production deployments read it.** `deploy-staging` +builds the image with staging's `PROJECT_ID`, and `deploy-production` deploys +that same image. So from staging's window until production's window, production +deploys an image from the new project's store into the old project. Cloud Run +then needs both the deploying account and the deploying project's Cloud Run +service agent to hold Artifact Registry Reader on that store. Without it, +production's "Deploy an isolated maintenance revision" step fails (no traffic +moves) in staging's window and on every merge until production's window. The +new project holds only the collector's store, so grant the role on the project +before staging's window: + +```bash +for old_production_reader in "serviceAccount:$OLD_DEPLOY_SERVICE_ACCOUNT" \ + "serviceAccount:service-$OLD_PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com"; do + gcloud projects add-iam-policy-binding "$NEW_PROJECT_ID" \ + --member "$old_production_reader" --role roles/artifactregistry.reader +done +``` + +Remove both bindings in section 11. + ### 4.2 The deployment account and Workload Identity Federation for this repo The new project gets a deployment account used only by this repo. The dashboard @@ -376,6 +400,20 @@ empty. Staging's window gives the same timings for staging's size. Do staging end to end, verify it (section 9), then production. +**How long it lasts:** from step 6.2 until the deployment in step 6.6 +finishes. Exporting and importing take minutes at today's size (10 GB disks). +Most of the time is the release that step 6.6 starts: the merge's Push run +(versioning), then the release commit's Push run (checks, tag, staging's +deployment, then production's). From merge to tag this took 25 minutes on +2026-10-15 and 88 minutes on 2026-10-19 and 2026-10-20, when the +organization's runners were queued; on 2026-10-20, calibration-diagnostics' +last collector deploy run took 2 hours 25 minutes in all, of which its staging +and production deploy jobs ran for 4 and 6 minutes. So plan for about 30 +minutes on a quiet day and 2 to 3 hours when runners are queued, and measure +it in staging's window. The dashboard's collector reads stay down until +section 8.1's dashboard deployment finishes, unless the old service keeps +serving (section 8.4). + While the window lasts: - **Builds:** the emitter gets HTTP 503 from the collector and retries, with @@ -463,10 +501,17 @@ their values if the new resources reused the old names (section 12). ### 6.6 Deploy -Merge the C2 code PR (section 13). Its release runs the Push workflow, which +Each window needs its own merge. Its release runs the Push workflow, which deploys staging and then production with the environments' current settings. -In staging's window, production redeploys in the old project with unchanged -code: a normal deployment, with its usual short maintenance phase. + +- **Staging's window:** merge the C2 code PR (section 13). Production then + redeploys in the old project with unchanged runtime code, from the image + staging built in the new project's store. That works only with the + Artifact Registry Reader grant in step 4.1; it is otherwise a normal + deployment, with its usual short maintenance phase. +- **Production's window:** merge a second PR prepared for it, for example the + README update in section 13. This repo requires branches to be up to date + with `main`, so update it and let its checks pass before the window starts. Re-running the deployment jobs of the latest Push run might do the same without a merge, but whether a re-run reads the environments' changed values @@ -696,6 +741,9 @@ after an agreed holding period: environment, and the old runtime and migration service accounts. - The collector's images in the old Artifact Registry store (only the collector's image name, unless the store holds nothing else). +- The two `roles/artifactregistry.reader` bindings on the new project from + step 4.1 (the old deployment account and the old project's Cloud Run service + agent). - This repo's Workload Identity provider in `policyengine-api` (`github/microcosm-collector`) and the old deployment account's `roles/iam.workloadIdentityUser` binding for this repo's principal set. @@ -718,6 +766,7 @@ after an agreed holding period: | `TELEMETRY_GCP_WIF_PROVIDER` | same (secret) | `google-github-actions/auth`, `workload_identity_provider` | the new provider's resource | | `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | same (secret) | `google-github-actions/auth`, `service_account` | the new deployment account | | `COLLECTOR_DEPLOY_ENABLED` | this repo's variable | the deployment jobs' `if:` | unchanged (`true`) | +| `roles/artifactregistry.reader` on the new project for the old deployment account and the old project's Cloud Run service agent | Google Cloud IAM (step 4.1) | Cloud Run, when `deploy-production` deploys staging's image into the old project | removed in section 11 | | `DATABASE_URL`, `TELEMETRY_JWT_SECRET`, `TELEMETRY_READ_TOKEN` | Google Secret Manager in the new project | the service, through `--set-secrets` | new database URL; copied signing secret and read token | | `DATABASE_URL` (migration job) | Secret Manager in the new project | the migration job, through `--set-secrets` | new migration database URL | | `COLLECTOR_HF_READ_TOKEN` (if S1 merged) | Secret Manager in the new project | the worker jobs, through a secret mount | copied or rotated | @@ -736,8 +785,13 @@ Whether a workflow re-run reads changed environment values is unchecked (step - **This repo:** a step after "Verify the stable service address" that checks `https://$PUBLIC_ORIGIN/ready`, from a new environment variable `TELEMETRY_PUBLIC_ORIGIN`, so every deployment proves the hostname reaches - the new revision. Its merge is the release that deploys into the new project - (step 6.6). The README's "Cloud Run deployment" and "Google Cloud access" - sections then describe the new project and the hostname. + the new revision. The step skips when the variable is unset: in the "both + together" order the hostname reaches the new service only after section 7, + so set the variable once section 7's checks pass. Its merge is the release + that deploys staging into the new project (step 6.6). +- **This repo, for production's window:** the README's "Cloud Run deployment" + and "Google Cloud access" sections updated to describe the new project and + the hostname. Its merge is the release that deploys production into the new + project (step 6.6). - **The emitter:** the release in section 8.2. - **microcosm-runs:** section 8.3, if chosen. From e1c6ccfc4878d197b143cd80755db839ea06a2d9 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 22:47:34 +0400 Subject: [PATCH 03/31] Recreate every database user and keep database-level privileges Production's old instance lists a third user, telemetry_app, besides the migration and runtime users. Google's import guide says every user that owns objects or holds privileges in the dumped database must exist on the target, so the runbook now creates every user the record lists. Re-creating the database after the timing rehearsal drops the database-level privileges granted in step 5.2; the runbook now grants them again, and the copy check compares the database listing too. The command that shows the database URL's form now hides everything up to the last "@", so a password containing one isn't printed in part. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 485a288..04c4f30 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -270,8 +270,13 @@ Match the record's IP configuration (Cloud Run connects through the Cloud SQL connector, so a public IP needs no authorized networks), SSL mode, database flags, maintenance window and point-in-time recovery. -Create the two users with the **same names** as on the old instance, so the -ownership and `GRANT` statements in the export apply, and new random passwords: +Create every user the step 3.2 record lists, except `postgres`, with the +**same names** as on the old instance and new random passwords. Google's import +guide says every user that owns objects or was granted permissions on objects +in the dumped database must exist on the target instance, or the import +fails. Besides the migration and runtime users, production's instance lists +`telemetry_app` today, which the collector doesn't use; give it a random +password that isn't kept. ```bash migration_password="$(openssl rand -hex 32)" @@ -280,6 +285,9 @@ gcloud sql users create "$MIGRATION_DATABASE_USER" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$migration_password" gcloud sql users create "$RUNTIME_DATABASE_USER" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$runtime_password" +# Production only, and any other user the record lists: +gcloud sql users create telemetry_app --instance "$NEW_INSTANCE" \ + --project "$NEW_PROJECT_ID" --password "$(openssl rand -hex 32)" ``` Keep the two password variables in this shell for step 5.3. Then recreate what @@ -313,7 +321,7 @@ First see the form of the old value with its credentials hidden: ```bash gcloud secrets versions access latest --secret "$DATABASE_SECRET_NAME" --project "$OLD_PROJECT_ID" \ - | sed -E 's#//[^@]*@#//USER:PASSWORD@#' + | sed -E 's#//.*@#//USER:PASSWORD@#' ``` Build the new values in the same form. For the usual Cloud SQL socket form: @@ -394,7 +402,9 @@ Before production's window, export the production database with `--offload` instance), import it into the new instance as in step 6.3, and note both durations. Then delete and re-create the new database (`gcloud sql databases delete`, then `create`), so the window's import starts -empty. Staging's window gives the same timings for staging's size. +empty, and grant the database-level privileges from step 5.2 again: deleting +the database drops them. Staging's window gives the same timings for staging's +size. ## 6. The window @@ -477,7 +487,8 @@ On both instances, as the migration user: - `SELECT count(*)` on every table in the `public` schema gives the same counts. At revision `0001` these are `telemetry_runs`, `telemetry_producers` and `telemetry_events`; later migrations add more. -- `\dp public.*` and `\ddp` match the step 3.2 record. +- `\l`, `\dp public.*` and `\ddp` match the step 3.2 record, so the + database-level, table and default privileges all came across. If anything differs, stop and roll back (section 10, first row). @@ -775,7 +786,8 @@ after an agreed holding period: | `PRODUCTION_COLLECTOR_URL`, `TRUSTED_COLLECTOR_ORIGINS` | emitter source (`core/constants.py`) | every emitter service module | the hostname, by an emitter release | **Unverified:** nothing in this runbook has been run. The current deployment -account's roles, the database roles and privileges, and the dashboard +account's roles, the database roles and privileges (including whether +production's `telemetry_app` owns or holds anything), and the dashboard identity's bindings were not read for it (they are recorded in step 3.2). Whether a workflow re-run reads changed environment values is unchecked (step 6.6). Every setting above needs checking end to end in the staging pass. From 7c7c43c996ec19b20950995c7f978db894210406 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 22:48:36 +0400 Subject: [PATCH 04/31] Carry over what the open collector PRs add The runbook listed only part of what #12, #16 and #19 need in the new project, so following it after any of them merges would stop a deployment or lose writes: - #12's worker job and recovery job write to the database while the service is in maintenance mode, so the export could miss their writes. The runbook now pauses the queue and the recovery schedule before the export, resumes them on rollback, and lists the recovery job, its schedule and the Cloud Scheduler API. - #16's signing key is a fifth secret that the deployment checks and mounts. The runbook now creates it, grants it like the other four and lists it with the settings. - #19's daily check wasn't mentioned. The runbook now lists its job, schedule, alert policies, permissions and three production variables, one of which names the project. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 4 +- docs/project-move-runbook.md | 76 ++++++++++++++++++++++------- 2 files changed, 61 insertions(+), 19 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 1044b90..cce958c 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -191,7 +191,9 @@ The plan's constraints: settings name the old project, so merge #234 before the move, or never dispatch that workflow afterwards. - The new project has to include whatever has merged by then: #12's intake - resources, S1's secret and its mount, C5's signing key. + resources (including its recovery job and schedule), S1's secret and its + mount, C5's signing key, and S2's daily check job, schedule and alert + policies (#19). If both D2 and D3 are adopted, there are two orders: diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 04c4f30..75a865a 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -84,10 +84,16 @@ from the old instance in step 3.2. - PolicyEngine/microcosm-emitter#29 has merged, so changing the emitter's default is a constant change (section 8.2). - List what has merged here that needs its own resources in the new project, - and add those steps where this runbook marks them: #12's artifact intake (a - bucket, a Cloud Tasks queue, a worker job and their permissions, as in the - README's "Artifact intake" section), S1's Hugging Face read token (#13), and - C5's session signing key. + and add those steps where this runbook marks them: + - #12's artifact intake: a bucket, a Cloud Tasks queue, the worker job, the + recovery job and its Cloud Scheduler job, the invoker account and their + permissions, as in the README's "Artifact intake" section; + - S1's Hugging Face read token (#13, PR #18); + - C5's session signing key (#16), a fifth Secret Manager secret per + environment; + - S2's daily check (#19): production's check job, its Cloud Scheduler job + and the account that job runs as, a notification channel and two Cloud + Monitoring alert policies, and three production variables. ### 3.2 Record the current state @@ -150,8 +156,10 @@ gcloud services enable run.googleapis.com sqladmin.googleapis.com \ ``` Also enable `compute.googleapis.com` and `certificatemanager.googleapis.com` -for a load balancer (D3 C), and `cloudtasks.googleapis.com` if #12's intake is -switched on. +for a load balancer (D3 C); `cloudtasks.googleapis.com` and +`cloudscheduler.googleapis.com` if #12's intake is switched on; and +`cloudscheduler.googleapis.com` and `monitoring.googleapis.com` if #19's daily +check has merged. ### 4.1 Artifact Registry @@ -372,13 +380,27 @@ The dashboard's access to the read secret is set in section 8.1. Hugging Face, store that, and revoke the old token in section 11. Grant `roles/secretmanager.secretAccessor` only to the account the worker jobs run as, and mount it into the worker jobs as `COLLECTOR_HF_READ_TOKEN`. -- **C5's session signing key:** copy it the same way, so tokens signed before - the move verify afterwards and the key ID published in the JWKS doesn't - change. -- **#12's intake:** the bucket, queue, worker job, invoker account and - permissions from the README's "Artifact intake" section, in the new project. - Copy the intake objects in the window (step 6.3), and check whether stored - submission rows name the bucket. +- **C5's session signing key (#16):** create the secret named by + `TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME` in the new project and copy its + value the same way, so tokens signed before the move verify afterwards and + the key ID published in the JWKS doesn't change. Add it to both grant loops + above: `roles/secretmanager.secretAccessor` for the runtime account and + `roles/secretmanager.viewer` for the deployment account. The deployment + checks that it exists and mounts it as `TELEMETRY_JWT_SIGNING_KEY`, so + production's deployment stops at "Check required Secret Manager entries + exist" without it. +- **#12's intake:** the bucket, queue, worker job, recovery job and its Cloud + Scheduler job, invoker account and permissions from the README's "Artifact + intake" section, in the new project. Copy the intake objects in the window + (step 6.3), and check whether stored submission rows name the bucket. +- **S2's daily check (#19), production only:** the worker service account + (the one that may read S1's token), granted to the deployment account with + `roles/iam.serviceAccountUser`; `roles/secretmanager.viewer` on S1's secret + for the deployment account; the Cloud Scheduler job that starts the check at + 06:00 and 18:00 UTC and the account it uses; the notification channel and + both alert policies, as in the README. The deployment configures the check + job itself, before it changes the service, and fails there until these + exist. ### 5.4 Create the Cloud Run service once @@ -461,7 +483,19 @@ sleep 35 ``` The `curl` must print `503`, as `resolve-and-verify-maintenance` checks. The -pause outlasts the 30-second request timeout, so no write is still running. +pause outlasts the 30-second request timeout, so no request to the service is +still writing. + +**If #12's intake is switched on,** its worker job and its recovery job also +write to the database, and maintenance mode doesn't stop them. Pause what +starts them, in the old project, and wait until no execution is running: + +```bash +gcloud tasks queues pause WORKER_QUEUE --location "$REGION" --project "$OLD_PROJECT_ID" +gcloud scheduler jobs pause RECOVERY_SCHEDULE --location "$REGION" --project "$OLD_PROJECT_ID" +gcloud run jobs executions list --job WORKER_JOB --region "$REGION" --project "$OLD_PROJECT_ID" --limit 5 +``` + From here the old database doesn't change. ### 6.3 Export and import @@ -507,8 +541,10 @@ gh secret set TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT "${collector_settings[@]}" -- In production's window, also set `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` to production's new connection in **both** environments, since staging's -`validate-environment-target` compares against it. The other variables keep -their values if the new resources reused the old names (section 12). +`validate-environment-target` compares against it. If #19 has merged, also +set production's `TELEMETRY_WORKER_SERVICE_ACCOUNT` to the new worker account. +The other variables keep their values if the new resources reused the old +names (section 12). ### 6.6 Deploy @@ -711,7 +747,7 @@ For each environment: - `gcloud run services describe "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" --format=json` shows an image in the new store, tagged with this repo's release commit (whose message is "Update microcosm-collector"), the new Cloud SQL connection, and - the four secret names. + the four secret names (five once #16 has merged). - `curl -fsS "$(gcloud run services describe "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" --format='value(status.url)')/ready"` succeeds, and so do the hostname checks in section 7. - `gcloud run jobs executions list --job "$MIGRATION_JOB" --project "$NEW_PROJECT_ID" --region "$REGION" --limit 1` @@ -733,7 +769,7 @@ the moment the old service entered maintenance mode. | Situation | What to do | |---|---| -| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. Nothing is lost: builds kept their events and deliver them to the old service. | +| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). Nothing is lost: builds kept their events and deliver them to the old service. | | A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | | After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | | After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to its pre-move revision, set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | @@ -778,7 +814,11 @@ after an agreed holding period: | `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | same (secret) | `google-github-actions/auth`, `service_account` | the new deployment account | | `COLLECTOR_DEPLOY_ENABLED` | this repo's variable | the deployment jobs' `if:` | unchanged (`true`) | | `roles/artifactregistry.reader` on the new project for the old deployment account and the old project's Cloud Run service agent | Google Cloud IAM (step 4.1) | Cloud Run, when `deploy-production` deploys staging's image into the old project | removed in section 11 | +| `TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME` (if #16 merged) | same, in both environments (variable) | `check-secrets-exist`, `--set-secrets` | unchanged when the new secret reuses the name | +| `TELEMETRY_WORKER_SERVICE_ACCOUNT` (if #19 merged) | `production` (variable) | `configure-huggingface-check-job`, `--service-account` | the new worker account | +| `TELEMETRY_HF_CHECK_JOB`, `TELEMETRY_HF_READ_SECRET_NAME` (if #19 merged) | `production` (variables) | `configure-huggingface-check-job` | unchanged when the new resources reuse the names | | `DATABASE_URL`, `TELEMETRY_JWT_SECRET`, `TELEMETRY_READ_TOKEN` | Google Secret Manager in the new project | the service, through `--set-secrets` | new database URL; copied signing secret and read token | +| `TELEMETRY_JWT_SIGNING_KEY` (if #16 merged) | Secret Manager in the new project | the service, through `--set-secrets` | copied key | | `DATABASE_URL` (migration job) | Secret Manager in the new project | the migration job, through `--set-secrets` | new migration database URL | | `COLLECTOR_HF_READ_TOKEN` (if S1 merged) | Secret Manager in the new project | the worker jobs, through a secret mount | copied or rotated | | `TELEMETRY_MAINTENANCE_MODE` | `scripts/deploy.sh` (`--set-env-vars`), and step 6.2 on the old service | the service | unchanged | From 4c33bb650ab047c6e1c9a14c035deac39f17317e Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 22:49:45 +0400 Subject: [PATCH 05/31] Correct statements about Preview, Search Console, lost events and more - The dashboard's identity-provider setting is one Vercel entry for Production and Preview (since 2026-10-07), not Production only, and Preview deployments can read staging's collector. - Only Cloud Run domain mapping needs Search Console verification; Certificate Manager's DNS authorization needs only a DNS record. - Events are not always kept through a window: a machine discarded when its build ends, such as a Modal container, loses the events not sent by then. - The emitter's Orrery uploads do get their credential from the collector's token exchange; they already wait today, so the window changes nothing for them. - #23 adds the emitter's Hugging Face module; #27 makes it send submissions to the collector. - A new emitter release reaches builds once it is installed on the machine or in the build image. - The backup check needs a format that shows the description. - Whether a re-run of the dashboard's deployment reads a changed GitHub variable is unchecked, so the documents now point to a merge. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 52 +++++++++++++++++------------ docs/project-move-runbook.md | 37 ++++++++++++-------- 2 files changed, 53 insertions(+), 36 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index cce958c..bba324a 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -76,9 +76,10 @@ Which DNS zone (from public name-server records, 2026-10-20): project holds that zone wasn't checked. - Neither zone has CAA records today. If one is added, it must allow `pki.goog` and `letsencrypt.org`. -- Domain mapping (B) and the load balancer's certificate (C) both need the - base domain verified in Google Search Console by the account that creates - them. +- Domain mapping (B) needs the base domain verified in Google Search Console + by the account that creates the mapping. The load balancer's certificate (C) + doesn't: Certificate Manager's DNS authorization needs only a `CNAME` record + in the zone. Whatever is chosen, three things stay as they are: @@ -100,23 +101,27 @@ Whatever is chosen, three things stay as they are: - **Where:** `PRODUCTION_COLLECTOR_URL`, and after PolicyEngine/microcosm-emitter#29 also `TRUSTED_COLLECTOR_ORIGINS`, both in `core/constants.py`. Every service module resolves the origin through - `core/collector_origin.py`. The emitter stack's #27 adds a fourth module - (`huggingface/service/module.py`), which should use the same resolver. + `core/collector_origin.py`. In the emitter stack, #27 makes a third service + module, `huggingface/service/module.py` (added by #23), send submissions to + the collector; it chooses the origin with its own copy of the old expression + and should use the same resolver. - **The change:** one emitter PR adds the new origin to `TRUSTED_COLLECTOR_ORIGINS` and points `PRODUCTION_COLLECTOR_URL` at it. Merging it publishes a release to PyPI. - **Who picks it up:** - Builds that start the separately installed emitter (emitter #21 and #22, - and Microcosm's H2a) use the newest installed emitter at their next - start. They need no Microcosm change. A running build keeps the emitter it - started with. + and Microcosm's H2a) use the new release from their next start once it is + installed on that machine or in that build image. They need no Microcosm + change. A running build keeps the emitter it started with. - Builds pinned to `microcosm-emitter==0.2.1` (Microcosm's `packages/microcosm-build/pyproject.toml` today) keep the old address until Microcosm moves off that pin. - **If the old address stops answering,** a build that still uses it keeps its telemetry on its own machine: the emitter retries with a backoff of up - to 60 seconds and keeps undelivered events for seven days (up to 100 MiB). - A build never fails because of telemetry. + to 60 seconds and keeps undelivered events for seven days (up to 100 MiB), + and a later emitter on the same machine sends them. A machine discarded + when its build ends, such as a Modal container, loses the events not sent + by then. A build never fails because of telemetry. ### The calibration dashboard (calibration-diagnostics) @@ -124,25 +129,27 @@ Whatever is chosen, three things stay as they are: |---|---|---|---|---| | `MICROCOSM_TELEMETRY_COLLECTOR_URL` | GitHub `Production` environment variable; also set in the Vercel project for Production and Preview | `deploy.yml` passes the GitHub value to `.github/scripts/deploy-dashboard.sh`, which requires it and sets it on the production deployment with `vercel deploy --env`. Preview deployments use the Vercel value. The server reads it in `frontend/lib/server/microcosm/telemetry-collector-client.ts`. | the new default address (if D3 is A) | the custom hostname, in all three places (GitHub Production, Vercel Production, and Vercel Preview with the staging address) | | `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel project setting, Production and Preview | `telemetry-collector-client.ts`: the full Secret Manager version resource, which names the project | the new project's read secrets | no | -| `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` | Vercel project setting, Production only | `frontend/lib/server/secret-manager.ts`, on Vercel: the Workload Identity provider resource, named by project number | only if the provider moves (see choice 10) | no | +| `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` | Vercel project setting, one entry for Production and Preview | `frontend/lib/server/secret-manager.ts`, on Vercel: the Workload Identity provider resource, named by project number | only if the provider moves (see choice 10) | no | | `TELEMETRY_GCP_PROJECT_ID`, `TELEMETRY_READ_SECRET_NAME` | GitHub `Production` environment variables | `deploy.yml` step "Check collector read secret exists" | the new project | no | | `TELEMETRY_GCP_WIF_PROVIDER`, `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | GitHub `Production` environment secrets | the same step's Google authentication | no, if the dashboard's own identity in `policyengine-api` is granted metadata read on the new read secret | no | Two things found while checking: -- **Preview has no `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`.** On a - Preview deployment the server can't read its read token ("Configure - MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER."), so the build monitor shows - only the Hugging Face runs and lists `collector` under "Runs that could not - be read". Whether that's intended wasn't checked. It is independent of D2 - and D3. +- **Preview reads staging's collector.** The identity-provider setting covers + Production and Preview, the provider accepts both environments' Vercel + identities, and staging's read secret grants the Preview identity access. So + under choice 10 both identities need access to the new project's read + secrets, each to its own environment's. - **After C6,** the dashboard authenticates to the collector with its own Vercel OIDC token instead of a shared read token, so it no longer reads Secret Manager for the collector. If C6 lands before the move, only `MICROCOSM_TELEMETRY_COLLECTOR_URL` changes. Settings take effect on the next deployment: a merge to calibration-diagnostics -`main`, or a re-run of its latest "Deploy production" run. +`main`, or a re-run of its latest "Deploy production" run. Whether a re-run +reads a changed GitHub variable such as `MICROCOSM_TELEMETRY_COLLECTOR_URL` +hasn't been checked (the runbook's step 6.6 has the same question), so a merge +is the reliable way. ### microcosm-runs @@ -225,16 +232,17 @@ If both D2 and D3 are adopted, there are two orders: `collector.microcosm.institute`; and whether staging gets one, for example `collector-staging.microcosm.institute`, or keeps its default address. 5. **DNS:** which zone (`microcosm.institute` on Cloudflare, or - `policyengine.org` on Cloud DNS); who adds the records; which Google account - verifies the base domain in Search Console. + `policyengine.org` on Cloud DNS); who adds the records; for D3 B, which + Google account verifies the base domain in Search Console. 6. **Order:** hostname first, or both together. 7. **The old address after the move:** keep the old service serving, connected to the new database, until no build uses emitter 0.2.1; or leave it in maintenance mode, so builds still on it keep their telemetry locally; or delete it. A redirect isn't possible. 8. **The window:** when, and how long is acceptable. During it, running builds - keep their events locally and deliver them afterwards, and the build monitor - shows only Hugging Face runs. + keep their events locally and deliver them afterwards, except on a machine + discarded when its build ends, such as a Modal container, which loses the + events not sent by then. The build monitor shows only Hugging Face runs. 9. **Staging data:** dump and restore it like production (which also rehearses the production move), or start staging empty. 10. **The dashboard's Google identity:** create a new Vercel Workload Identity diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 75a865a..54f7d30 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -25,8 +25,9 @@ another. link the billing account; then the owner role on the new project. Read access to `policyengine-api`, and Cloud SQL admin on the old instances for the export. -- **DNS:** edit access to the chosen zone (choice 5), and a Google account that - is a verified owner of the base domain in Search Console. +- **DNS:** edit access to the chosen zone (choice 5). For D3 B only, a Google + account that is a verified owner of the base domain in Search Console; D3 + C's certificate needs only a DNS record. - **GitHub:** an admin of PolicyEngine/microcosm-collector (environment settings), and someone who can change the calibration-diagnostics `Production` environment and merge in both repos. @@ -394,8 +395,8 @@ The dashboard's access to the read secret is set in section 8.1. intake" section, in the new project. Copy the intake objects in the window (step 6.3), and check whether stored submission rows name the bucket. - **S2's daily check (#19), production only:** the worker service account - (the one that may read S1's token), granted to the deployment account with - `roles/iam.serviceAccountUser`; `roles/secretmanager.viewer` on S1's secret + (the one that may read S1's token), with `roles/iam.serviceAccountUser` on + it for the deployment account; `roles/secretmanager.viewer` on S1's secret for the deployment account; the Cloud Scheduler job that starts the check at 06:00 and 18:00 UTC and the account it uses; the notification channel and both alert policies, as in the README. The deployment configures the check @@ -450,10 +451,15 @@ While the window lasts: - **Builds:** the emitter gets HTTP 503 from the collector and retries, with a backoff of up to 60 seconds. Events wait on the build machine (kept for - seven days, up to 100 MiB). A build never fails because of telemetry. + seven days, up to 100 MiB). A machine discarded when its build ends, such as + a Modal container, loses the events not sent by then. A build never fails + because of telemetry. - **The dashboard:** the build monitor lists `collector` under "Runs that could not be read" and still shows the Hugging Face runs. -- **Orrery uploads** don't use the collector today. +- **Orrery uploads:** the emitter gets their credential from the collector's + token exchange, so they wait during the window. They already wait today, + because microcosm-runs' authorization call needs a collector endpoint that + doesn't exist, so nothing changes for them. ### 6.1 Check that no deployment is running @@ -694,10 +700,11 @@ production's new read secret. | `TELEMETRY_GCP_PROJECT_ID` | calibration-diagnostics GitHub environment `Production` | `$NEW_PROJECT_ID` | | `TELEMETRY_READ_SECRET_NAME` | calibration-diagnostics GitHub environment `Production` | unchanged if the name was reused | -Settings take effect on the next deployment. Re-run calibration-diagnostics' -latest "Deploy production" run, or merge its next PR, and open a new Preview -deployment. The build monitor should then list collector runs from before the -move. +Settings take effect on the next deployment. Merge calibration-diagnostics' +next PR, and open a new Preview deployment. Re-running its latest "Deploy +production" run may also work, but whether a re-run reads the changed GitHub +variable is the unchecked question from step 6.6. The build monitor should +then list collector runs from before the move. ### 8.2 The emitter @@ -708,7 +715,8 @@ Keep the old origin in the set while the old address serves. Merging it publishes the release. - Builds that start the separately installed emitter pick it up at their next - start. A running build keeps the emitter it started with. + start, once it is installed on that machine or in that build image. A + running build keeps the emitter it started with. - Builds pinned to `microcosm-emitter==0.2.1` keep the old address until Microcosm moves off that pin. @@ -752,8 +760,9 @@ For each environment: succeeds, and so do the hostname checks in section 7. - `gcloud run jobs executions list --job "$MIGRATION_JOB" --project "$NEW_PROJECT_ID" --region "$REGION" --limit 1` shows a successful execution. -- Production: `gcloud sql backups list --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --limit 1` - shows `telemetry-release-RUN_ID-ATTEMPT` for the Push run. +- Production: `gcloud sql backups list --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --limit 1 --format='table(id,windowStartTime,status,description)'` + shows `telemetry-release-RUN_ID-ATTEMPT` for the Push run. The default + columns leave out the description. - The row counts from step 6.4 still match. - After section 8.1: the build monitor lists runs from before the move and doesn't list `collector` under "Runs that could not be read". @@ -769,7 +778,7 @@ the moment the old service entered maintenance mode. | Situation | What to do | |---|---| -| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). Nothing is lost: builds kept their events and deliver them to the old service. | +| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | | A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | | After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | | After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to its pre-move revision, set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | From 4381a93d7bf1c6d009b3021fec87190bdd4b5e77 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:20:55 +0400 Subject: [PATCH 06/31] Stop both intake jobs before the export, and keep the intake bucket Step 6.2 now pauses #12's queue and recovery schedule together and lists unfinished executions of both the worker and the recovery job. It gives the possible wait (three attempts of up to 1 hour each), how to cancel a worker execution and what recovery then does with its job, and why the queue must never be paused while recovery runs. The intake bucket stays in the old project. Each accepted submission records its files' Cloud Storage generation numbers, and the worker reads every file at that generation; a copy into a new bucket gets new generation numbers, so the copy step is removed. Uploads still in flight need nothing, because the bucket doesn't change. Step 6.5 and section 12 now point the queue, worker job, signing account and invoker account settings at the new project. A new step 6.7 creates the recovery schedule only after the deployment. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 5 ++ docs/project-move-runbook.md | 115 ++++++++++++++++++++++++---- 2 files changed, 107 insertions(+), 13 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index bba324a..3baf93b 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -201,6 +201,11 @@ The plan's constraints: resources (including its recovery job and schedule), S1's secret and its mount, C5's signing key, and S2's daily check job, schedule and alert policies (#19). +- If #12's intake is switched on before the move, its bucket stays in + `policyengine-api`: each accepted submission's row records its files' + Cloud Storage generation numbers, which a copy into a new bucket doesn't + keep (the runbook's step 5.3). Switching intake on after the move creates + the bucket in the new project from the start. If both D2 and D3 are adopted, there are two orders: diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 54f7d30..c339417 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -86,9 +86,10 @@ from the old instance in step 3.2. default is a constant change (section 8.2). - List what has merged here that needs its own resources in the new project, and add those steps where this runbook marks them: - - #12's artifact intake: a bucket, a Cloud Tasks queue, the worker job, the - recovery job and its Cloud Scheduler job, the invoker account and their - permissions, as in the README's "Artifact intake" section; + - #12's artifact intake: a Cloud Tasks queue, the worker job, the recovery + job and its Cloud Scheduler job, the invoker account and their + permissions, as in the README's "Artifact intake" section. Its bucket + stays in the old project (step 5.3); - S1's Hugging Face read token (#13, PR #18); - C5's session signing key (#16), a fifth Secret Manager secret per environment; @@ -390,10 +391,25 @@ The dashboard's access to the read secret is set in section 8.1. checks that it exists and mounts it as `TELEMETRY_JWT_SIGNING_KEY`, so production's deployment stops at "Check required Secret Manager entries exist" without it. -- **#12's intake:** the bucket, queue, worker job, recovery job and its Cloud - Scheduler job, invoker account and permissions from the README's "Artifact - intake" section, in the new project. Copy the intake objects in the window - (step 6.3), and check whether stored submission rows name the bucket. +- **#12's intake:** in the new project, the Cloud Tasks queue, the worker + job, the recovery job (its three `COLLECTOR_WORKER_*` settings naming the + new queue, worker job and invoker account), the invoker account, the + account Cloud Scheduler starts the recovery job as, and the permissions + from the README's "Artifact intake" section. Create the + recovery job's Cloud Scheduler job only in step 6.7: until the window's + deployment, the new database is empty or holds step 5.5's rehearsal copy, + and a recovery run would dispatch jobs from it. + + **Keep the intake bucket in the old project.** When a submission is + completed, the collector records each stored object's generation number in + the submission's row, and the worker reads every file at that generation. + Cloud Storage assigns generation numbers, and they can't be set, so a copy + in another bucket has different ones and the worker couldn't read any + accepted submission's files. Grant the new runtime account and the new + worker job's account the roles on the old bucket that step 3.2 recorded for + the old accounts. `COLLECTOR_INTAKE_BUCKET` keeps its value. Moving the + bucket later needs a code change that records the copies' generations; + none is written. - **S2's daily check (#19), production only:** the worker service account (the one that may read S1's token), with `roles/iam.serviceAccountUser` on it for the deployment account; `roles/secretmanager.viewer` on S1's secret @@ -443,7 +459,9 @@ organization's runners were queued; on 2026-10-20, calibration-diagnostics' last collector deploy run took 2 hours 25 minutes in all, of which its staging and production deploy jobs ran for 4 and 6 minutes. So plan for about 30 minutes on a quiet day and 2 to 3 hours when runners are queued, and measure -it in staging's window. The dashboard's collector reads stay down until +it in staging's window. If #12's intake is switched on, add the wait for its +worker executions in step 6.2, up to about 3 hours, unless they are cancelled +or the queue is paused early. The dashboard's collector reads stay down until section 8.1's dashboard deployment finishes, unless the old service keeps serving (section 8.4). @@ -493,15 +511,49 @@ pause outlasts the 30-second request timeout, so no request to the service is still writing. **If #12's intake is switched on,** its worker job and its recovery job also -write to the database, and maintenance mode doesn't stop them. Pause what -starts them, in the old project, and wait until no execution is running: +write to the database, and maintenance mode doesn't stop them. In the old +project, pause the queue and the recovery schedule together, then make sure +no execution of either job is still running: ```bash gcloud tasks queues pause WORKER_QUEUE --location "$REGION" --project "$OLD_PROJECT_ID" gcloud scheduler jobs pause RECOVERY_SCHEDULE --location "$REGION" --project "$OLD_PROJECT_ID" -gcloud run jobs executions list --job WORKER_JOB --region "$REGION" --project "$OLD_PROJECT_ID" --limit 5 +for intake_job in WORKER_JOB RECOVERY_JOB; do + gcloud run jobs executions list --job "$intake_job" --region "$REGION" \ + --project "$OLD_PROJECT_ID" --format=json \ + | jq -r '.[] | select(.status.completionTime == null) | .metadata.name' +done ``` +The loop prints the executions that haven't finished; repeat it until it +prints nothing. An operator's `reprocess` run is an execution of the recovery +job, so the loop covers it too. + +- **A recovery execution** only updates job rows and creates Cloud Tasks. + Wait for it. +- **A worker execution** can run for about 3 hours: three attempts + (`--max-retries 2`) of up to 1 hour each (`--task-timeout 1h`). Either wait, + which lengthens the window, or cancel it with + `gcloud run jobs executions cancel EXECUTION --region "$REGION" --project "$OLD_PROJECT_ID" --quiet`. + A cancelled attempt records no result. Its job stays `running` in the + copied database until its lease expires, up to 65 minutes after the attempt + started; then the new project's recovery job dispatches it again, or fails + it if it has used its three attempts or three dispatches. +- **To keep the window short,** pause the queue and the schedule up to 3 + hours before it starts, so worker executions finish first (resume both if + the window is called off). Submissions completed in the meantime wait in + the paused queue, and the new project's recovery job dispatches them after + the move (step 6.7). +- **Never pause the queue alone.** Tasks can still be added to a paused + queue, and recovery dispatches a job again each time it has waited 30 + minutes without a claim, and fails it after its third dispatch. With the + queue paused and recovery running, queued jobs fail about 90 minutes after + their first dispatch. + +Uploads in flight need nothing. A signed upload URL (valid for 30 minutes) +names the bucket, which stays where it is (step 5.3), so a build's `complete` +request to the new service finds the files there. + From here the old database doesn't change. ### 6.3 Export and import @@ -516,8 +568,7 @@ gcloud sql import sql "$NEW_INSTANCE" "$export_object" \ Import as the role that owns the tables in the step 3.2 record (normally the migration user), so the restored objects keep their owner. If #12's intake is -switched on, copy the intake objects too: -`gcloud storage cp --recursive gs://OLD_INTAKE_BUCKET/* gs://NEW_INTAKE_BUCKET/`. +switched on, its bucket isn't copied: it stays in the old project (step 5.3). ### 6.4 Check the copy @@ -552,6 +603,19 @@ set production's `TELEMETRY_WORKER_SERVICE_ACCOUNT` to the new worker account. The other variables keep their values if the new resources reused the old names (section 12). +If #12's intake is switched on, point its settings at the new project, through +whichever environment variables #12's deployment wiring reads (not yet +written): + +- `COLLECTOR_WORKER_QUEUE` and `COLLECTOR_WORKER_JOB`: + `projects/$NEW_PROJECT_ID/locations/$REGION/queues/...` and `.../jobs/...`; +- `COLLECTOR_INTAKE_SIGNING_SERVICE_ACCOUNT`: the new runtime account; +- `COLLECTOR_WORKER_INVOKER_SERVICE_ACCOUNT`: the new invoker account. + +`COLLECTOR_INTAKE_BUCKET` keeps its value (step 5.3). With the old values, the +new service would try to sign uploads as the old runtime account and enqueue +into the old project's paused queue, so intake would stop. + ### 6.6 Deploy Each window needs its own merge. Its release runs the Push workflow, which @@ -579,6 +643,28 @@ revision, and the job checks the schema against the SQLAlchemy metadata), deploys and checks a candidate revision, and moves traffic to it. In production it also takes a backup of the new instance first. +### 6.7 Start the scheduled jobs in the new project + +After the deployment in step 6.6 has finished. + +**#12's intake, if switched on:** create the recovery job's schedule in the +new project, every 10 minutes as the README says, as the account created for +it in step 5.3: + +```bash +gcloud scheduler jobs create http RECOVERY_SCHEDULE --location "$REGION" --project "$NEW_PROJECT_ID" \ + --schedule '*/10 * * * *' \ + --uri "https://run.googleapis.com/v2/projects/$NEW_PROJECT_ID/locations/$REGION/jobs/RECOVERY_JOB:run" \ + --http-method post --oauth-service-account-email RECOVERY_SCHEDULER_ACCOUNT +``` + +Compare its settings with the old schedule's record from step 3.2. Jobs that +the old service queued before step 6.2 are still in the old, paused queue. +The new recovery job dispatches each one again into the new queue once 30 +minutes have passed since its last dispatch, or fails it if it has already +been dispatched three times. The old queue and schedule stay paused until +section 11. + ## 7. The hostname and DNS Records at Cloudflare must be "DNS only". If the zone ever gets CAA records, @@ -826,6 +912,9 @@ after an agreed holding period: | `TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME` (if #16 merged) | same, in both environments (variable) | `check-secrets-exist`, `--set-secrets` | unchanged when the new secret reuses the name | | `TELEMETRY_WORKER_SERVICE_ACCOUNT` (if #19 merged) | `production` (variable) | `configure-huggingface-check-job`, `--service-account` | the new worker account | | `TELEMETRY_HF_CHECK_JOB`, `TELEMETRY_HF_READ_SECRET_NAME` (if #19 merged) | `production` (variables) | `configure-huggingface-check-job` | unchanged when the new resources reuse the names | +| `COLLECTOR_WORKER_QUEUE`, `COLLECTOR_WORKER_JOB` (if #12's intake is switched on) | the environment variables #12's deployment wiring will read (not yet written); set by hand on the recovery job | the service and the recovery job | the new project's queue and worker job | +| `COLLECTOR_INTAKE_SIGNING_SERVICE_ACCOUNT`, `COLLECTOR_WORKER_INVOKER_SERVICE_ACCOUNT` (same) | same | the service signs upload URLs as the first; the service and the recovery job dispatch as the second | the new runtime account and the new invoker account | +| `COLLECTOR_INTAKE_BUCKET` (same) | same | the service and the worker job | unchanged: the bucket stays in the old project (step 5.3) | | `DATABASE_URL`, `TELEMETRY_JWT_SECRET`, `TELEMETRY_READ_TOKEN` | Google Secret Manager in the new project | the service, through `--set-secrets` | new database URL; copied signing secret and read token | | `TELEMETRY_JWT_SIGNING_KEY` (if #16 merged) | Secret Manager in the new project | the service, through `--set-secrets` | copied key | | `DATABASE_URL` (migration job) | Secret Manager in the new project | the migration job, through `--set-secrets` | new migration database URL | From d3718f16827c3980de995e63d16049d9cab0c188 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:21:30 +0400 Subject: [PATCH 07/31] Move the daily check's schedule and alerts in the right order A new part of step 6.7 creates #19's schedule in the new project once production's deployment has created the check job, then runs the job once by hand: the README says the absence policy can report only after a run that follows its creation. It then disables the old project's two alert policies and pauses its schedule, in that order, before section 11 removes the old check job or revokes S1's old token, so the old project sends no false alerts. Rollback reverses this, and section 9 checks it. Step 5.3's #19 bullet now names what the deployment needs: S1's secret, the worker account, and the deployment account's viewer and service-account-user roles. The channel and the policies don't affect it. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 50 +++++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index c339417..3b60286 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -61,6 +61,7 @@ DATABASE_SECRET_NAME=$(collector_variable TELEMETRY_DATABASE_SECRET_NAME) MIGRATION_DATABASE_SECRET_NAME=$(collector_variable TELEMETRY_MIGRATION_DATABASE_SECRET_NAME) JWT_SECRET_NAME=$(collector_variable TELEMETRY_JWT_SECRET_NAME) READ_SECRET_NAME=$(collector_variable TELEMETRY_READ_SECRET_NAME) +HF_CHECK_JOB=$(collector_variable TELEMETRY_HF_CHECK_JOB) # production, once #19 has merged # Names are scoped to a project, so the new resources can reuse the old ones. NEW_INSTANCE=$OLD_INSTANCE @@ -413,11 +414,13 @@ The dashboard's access to the read secret is set in section 8.1. - **S2's daily check (#19), production only:** the worker service account (the one that may read S1's token), with `roles/iam.serviceAccountUser` on it for the deployment account; `roles/secretmanager.viewer` on S1's secret - for the deployment account; the Cloud Scheduler job that starts the check at - 06:00 and 18:00 UTC and the account it uses; the notification channel and - both alert policies, as in the README. The deployment configures the check - job itself, before it changes the service, and fails there until these - exist. + for the deployment account; the account Cloud Scheduler starts the check + as; the notification channel and both alert policies, as in the README. + Production's deployment configures the check job itself, before it changes + the service, and stops at that step without S1's secret, the worker + account, or either of the deployment account's two roles. The channel and + the policies don't affect the deployment. Create the schedule in step 6.7, + once the deployment has created the job. ### 5.4 Create the Cloud Run service once @@ -665,6 +668,38 @@ minutes have passed since its last dispatch, or fails it if it has already been dispatched three times. The old queue and schedule stay paused until section 11. +**S2's daily check (#19), in production's window:** production's deployment +has created the check job in the new project. Create its schedule, then run +the job once by hand and wait for it: + +```bash +gcloud scheduler jobs create http HF_CHECK_SCHEDULE --location "$REGION" --project "$NEW_PROJECT_ID" \ + --schedule '0 6,18 * * *' --time-zone Etc/UTC \ + --uri "https://run.googleapis.com/v2/projects/$NEW_PROJECT_ID/locations/$REGION/jobs/$HF_CHECK_JOB:run" \ + --http-method post --oauth-service-account-email HF_CHECK_SCHEDULER_ACCOUNT +gcloud run jobs execute "$HF_CHECK_JOB" --region "$REGION" --project "$NEW_PROJECT_ID" --wait +``` + +The execution must succeed. The README says a metric-absence condition "is +met only after the job writes at least one data point after the policy is +created", so until this run the new "no successful execution for 23 hours" +policy can't report a stopped schedule. + +Then stop the old project's check, alerts first: + +```bash +for old_alert_policy in OLD_FAILURE_POLICY OLD_ABSENCE_POLICY; do + gcloud monitoring policies update "$old_alert_policy" --no-enabled --project "$OLD_PROJECT_ID" +done +gcloud scheduler jobs pause HF_CHECK_SCHEDULE --location "$REGION" --project "$OLD_PROJECT_ID" +``` + +The two policy names are in the step 3.2 record. Disable the policies before +pausing the schedule, and do both before section 11 removes the old check job +or revokes S1's old token. Otherwise the old project emails false alerts: +the failure policy at the next failed run, and the absence policy 23 hours +after the last successful one. + ## 7. The hostname and DNS Records at Cloudflare must be "DNS only". If the zone ever gets CAA records, @@ -856,6 +891,9 @@ For each environment: events in the build monitor. - The old service answers as choice 7 says: `503` in maintenance mode, or `/ready` when kept serving. +- If #19 has merged: step 6.7's execution of the check job succeeded, the + old project's two alert policies are disabled, and its check schedule is + paused. ## 10. Rollback @@ -864,7 +902,7 @@ the moment the old service entered maintenance mode. | Situation | What to do | |---|---| -| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | +| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). If step 6.7 ran, pause the schedules it created in the new project and disable the new alert policies; for #19, resume the old check schedule and enable the old policies again (`--enabled`). Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | | A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | | After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | | After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to its pre-move revision, set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | From 607f10366925632c3ee849d52a030216dc0e6974 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:21:58 +0400 Subject: [PATCH 08/31] Record and retire what the open collector PRs add Step 3.2 now reads the access policy of every secret the environment uses, including #16's signing key and S1's token if merged, and records #12's queue, worker and recovery jobs, schedule, bucket roles and invoker account, and #19's check job, schedule, alert policies and notification channel. Section 11 now retires those resources: every recorded secret, the old worker account, #12's queue, jobs, schedule and accounts (but not the intake bucket, which stays in use), and #19's job, schedule, account, policies and channel. S1's old token is revoked only after step 6.7 has paused the old check that uses it. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 3b60286..5286f06 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -123,8 +123,21 @@ For each environment (read-only): record is the only specification of them. - Who may read each secret: `gcloud secrets get-iam-policy SECRET --project "$OLD_PROJECT_ID" --format=yaml` - for each of the four secret names. The read secret's policy names the - dashboard's Vercel identity. + for each secret the environment uses: the four named above, and, if + merged, #16's signing-key secret and S1's Hugging Face token secret. The + read secret's policy names the dashboard's Vercel identity. +- What the open collector PRs added, if merged (step 3.1), each command with + `--project "$OLD_PROJECT_ID"`: + - #12: `gcloud tasks queues describe WORKER_QUEUE --location "$REGION"`; + `gcloud run jobs describe` with `--region "$REGION"` for the worker job + and the recovery job; `gcloud scheduler jobs describe RECOVERY_SCHEDULE --location "$REGION"`; + the bucket's roles with `gcloud storage buckets get-iam-policy gs://INTAKE_BUCKET`; + the invoker account's with `gcloud iam service-accounts get-iam-policy`. + - #19: `gcloud run jobs describe "$HF_CHECK_JOB" --region "$REGION"`; + `gcloud scheduler jobs describe HF_CHECK_SCHEDULE --location "$REGION"`, + which shows the account it starts the job as; the two alert policies' + names with `gcloud monitoring policies list --format='value(name,displayName,enabled)'`; + the notification channel with `gcloud beta monitoring channels list`. - The deployment account's roles. Its email is the `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` secret, which GitHub can't show. It is the account whose IAM policy grants `roles/iam.workloadIdentityUser` to this @@ -917,8 +930,16 @@ after an agreed holding period: - The old Cloud SQL instances: export a last copy into the bucket if wanted, then `gcloud sql instances patch "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --no-deletion-protection` and `gcloud sql instances delete`. -- The old Cloud Run services and migration jobs, the four secrets per - environment, and the old runtime and migration service accounts. +- The old Cloud Run services and migration jobs, every secret step 3.2 + recorded (the four per environment, and #16's signing key and S1's token + if merged), and the old runtime, migration and worker service accounts. +- If #12 merged: the queue (with any tasks still in it), the worker and + recovery jobs, the recovery schedule and the account it used, the invoker + account, and the old accounts' roles on the intake bucket. Keep the bucket: + it holds every submission's files and is still in use (step 5.3). +- If #19 merged: the old check job, its schedule and the account the + schedule used, both alert policies and the notification channel. Step 6.7 + already disabled the policies and paused the schedule. - The collector's images in the old Artifact Registry store (only the collector's image name, unless the store holds nothing else). - The two `roles/artifactregistry.reader` bindings on the new project from @@ -932,7 +953,8 @@ after an agreed holding period: - The export bucket and every dump in it. - calibration-diagnostics' leftover collector settings, as listed in calibration-diagnostics#233's "Manual steps". -- S1's old Hugging Face token, if it was rotated. +- S1's old Hugging Face token, if it was rotated. If #19 merged, revoke it + only after step 6.7 has paused the old check, which uses it. ## 12. Settings this runbook changes From 202600f6a493240594e8fbfe06b64ecca7c11257 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:22:32 +0400 Subject: [PATCH 09/31] Record the rollback revision just before each window's changes Production redeploys in staging's window and on every merge until its own window, possibly with migrations, so the revision step 3.2 records can be older than the database schema by the time of a rollback. Step 6.1 now records the revision the old service serves just before step 6.2, and rollback routes traffic to that one. Rollback also says which GitHub values it restores: only those step 6.5 changed in that window. In production's window that is production's values and TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION in staging's environment; staging stays on the new project. The rollback after the emitter release notes when the recorded revision is too old for the imported schema. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 5286f06..00f0db8 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -104,7 +104,9 @@ For each environment (read-only): - The service's URL, serving revision, image, Cloud SQL connection and secret names: the `gcloud run services describe` command in step 0 of - calibration-diagnostics#233. The serving revision is the rollback target. + calibration-diagnostics#233. This serving revision isn't the rollback + target: production redeploys between the windows, so step 6.1 records the + target again just before each window's changes. - The instance's settings: `gcloud sql instances describe "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --format=yaml`. Keep the database version, edition, tier, disk size, availability type, @@ -495,14 +497,26 @@ While the window lasts: because microcosm-runs' authorization call needs a collector endpoint that doesn't exist, so nothing changes for them. -### 6.1 Check that no deployment is running +### 6.1 Check that no deployment is running, and record the rollback target ```bash gh run list -R PolicyEngine/microcosm-collector --workflow push.yaml --status in_progress gh run list -R PolicyEngine/microcosm-collector --workflow push.yaml --status queued ``` -Both must list nothing. +Both must list nothing. Then record the revision the old service serves now. +It is the rollback target (section 10). Every merge since step 3.2 has +redeployed production, possibly with migrations, so an earlier revision may +not match the database schema. + +```bash +rollback_revision=$(gcloud run services describe "$SERVICE" --project "$OLD_PROJECT_ID" \ + --region "$REGION" --format=json \ + | jq -r '.status.traffic[] | select((.percent // 0) > 0) | .revisionName') +echo "$rollback_revision" +``` + +It must print one revision name. Write it in the window's notes too. ### 6.2 Put the old service in maintenance mode @@ -915,10 +929,10 @@ the moment the old service entered maintenance mode. | Situation | What to do | |---|---| -| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "PREVIOUS_REVISION=100"`, with the revision recorded in step 3.2. Set the GitHub environment values back to the step 3.2 record. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). If step 6.7 ran, pause the schedules it created in the new project and disable the new alert policies; for #19, resume the old check schedule and enable the old policies again (`--enabled`). Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | +| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "$rollback_revision=100"`, with the revision step 6.1 recorded in this window. Set back, to the step 3.2 record, only the GitHub values step 6.5 changed in this window: in staging's window, staging's; in production's window, production's and `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` in staging's environment. Staging's other values stay pointed at the new project, where its database now is. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). If step 6.7 ran, pause the schedules it created in the new project and disable the new alert policies; for #19, resume the old check schedule and enable the old policies again (`--enabled`). Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | | A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | | After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | -| After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to its pre-move revision, set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | +| After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to the revision step 6.1 recorded in production's window (if a release with a migration has deployed in the new project since, that revision's code is older than the imported schema: set the GitHub values back first, and let the next merge deploy the current release to the old project instead), set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | | The hostname | D3 B: delete the mapping, or map the hostname back to the old project's service with `--force-override`. D3 C: point the `A` record back to what it named before, if anything. | ## 11. Retire the old resources From 8b39d019b5b29026510ae82fc0faeb294635a842 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:23:34 +0400 Subject: [PATCH 10/31] Give each new database user no more privilege than the old one Cloud SQL's "About PostgreSQL users and roles" and "Create and manage users" pages say a user created with gcloud sql users create is a member of cloudsqlsuperuser with CREATEROLE, CREATEDB and LOGIN. An old user created in SQL may have less, and an export carries no roles. Step 3.2 now records every role's attributes and memberships with one SQL query, which works in any client (psql 16 and later no longer show memberships in \du). Step 5.2 compares each new user with that record and removes what it doesn't show: the membership with gcloud sql users assign-roles --revoke-existing-roles, the attributes with ALTER ROLE as postgres. Step 6.4 compares the query's rows on both instances. The extra users are now a list set from the record, so pasting the block for staging no longer creates telemetry_app. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 67 ++++++++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 14 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 00f0db8..775c9fc 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -119,10 +119,21 @@ For each environment (read-only): them. - **The database's roles and privileges.** Connect as the migration user through Cloud SQL Studio, or with `psql` through the Cloud SQL Auth Proxy, - and save the output of `\du`, `\l`, `\dn+ public`, `\dp public.*` and - `\ddp`. Note which role owns the tables and what the runtime user is - granted. These were provisioned by scripts kept outside this repo, so this - record is the only specification of them. + and save the output of `\l`, `\dn+ public`, `\dp public.*` and `\ddp`, and + of the roles query below. Note which role owns the tables and what the + runtime user is granted. These were provisioned by scripts kept outside + this repo, so this record is the only specification of them. The roles + query lists every role's attributes and memberships in one result; psql 16 + and later no longer show memberships in `\du`: + + ```sql + SELECT r.rolname, r.rolcanlogin, r.rolcreaterole, r.rolcreatedb, r.rolinherit, + ARRAY(SELECT g.rolname FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid + WHERE m.member = r.oid ORDER BY 1) AS member_of + FROM pg_roles r + WHERE r.rolname !~ '^(pg_|cloudsql)' + ORDER BY 1; + ``` - Who may read each secret: `gcloud secrets get-iam-policy SECRET --project "$OLD_PROJECT_ID" --format=yaml` for each secret the environment uses: the four named above, and, if @@ -311,16 +322,39 @@ gcloud sql users create "$MIGRATION_DATABASE_USER" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$migration_password" gcloud sql users create "$RUNTIME_DATABASE_USER" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$runtime_password" -# Production only, and any other user the record lists: -gcloud sql users create telemetry_app --instance "$NEW_INSTANCE" \ - --project "$NEW_PROJECT_ID" --password "$(openssl rand -hex 32)" +# Every other user the record lists: (telemetry_app) for production today, () for staging. +other_database_users=(telemetry_app) +for other_database_user in "${other_database_users[@]}"; do + gcloud sql users create "$other_database_user" --instance "$NEW_INSTANCE" \ + --project "$NEW_PROJECT_ID" --password "$(openssl rand -hex 32)" +done ``` -Keep the two password variables in this shell for step 5.3. Then recreate what -an export doesn't carry: the database-level privileges (`GRANT CONNECT` and -`CREATE ON DATABASE`) and role memberships from the `\l` and `\du` record. -Table privileges, schema privileges and default privileges (`\dp`, `\dn+`, -`\ddp`) come with the export, provided the roles exist. +Keep the two password variables in this shell for step 5.3. + +**Give each user no more than its record shows.** Cloud SQL's pages "About +PostgreSQL users and roles" and "Create and manage users" say a user created +this way is a member of `cloudsqlsuperuser`, with the `CREATEROLE`, +`CREATEDB` and `LOGIN` attributes. An old user created in SQL may have less, +and an export carries no roles, so nothing corrects it later. For each new +user, compare with its row from the roles query in step 3.2 and remove what +the row doesn't show: + +- **Membership of `cloudsqlsuperuser`:** + `gcloud sql users assign-roles USER --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --type BUILT_IN --revoke-existing-roles`, + with `--database-roles` for any membership the row does show, once that + role exists. Google's page gives this as the way to take + `cloudsqlsuperuser` away from a user. +- **Attributes:** in SQL, connected as `postgres`, for example + `ALTER ROLE telemetry_runtime NOCREATEROLE NOCREATEDB;`, and + `ALTER ROLE telemetry_app NOLOGIN;` if its row shows no `LOGIN`. Google's + page says `ALTER ROLE` changes any user's attributes. + +Step 6.4 compares the result. Then recreate what an export doesn't carry: the +database-level privileges (`GRANT CONNECT` and `CREATE ON DATABASE`) from the +`\l` record, and role memberships from the roles query. Table privileges, +schema privileges and default privileges (`\dp`, `\dn+`, `\ddp`) come with the +export, provided the roles exist. Let both instances' service agents use the bucket. Google's export and import pages name `roles/storage.objectAdmin` for both: @@ -610,8 +644,12 @@ On both instances, as the migration user: and `telemetry_events`; later migrations add more. - `\l`, `\dp public.*` and `\ddp` match the step 3.2 record, so the database-level, table and default privileges all came across. +- The roles query from step 3.2 gives the same rows, so every user has the + same attributes and memberships. If a new user has more, such as + `cloudsqlsuperuser`, `CREATEROLE` or `CREATEDB` left from + `gcloud sql users create`, remove it as in step 5.2 and compare again. -If anything differs, stop and roll back (section 10, first row). +If anything else differs, stop and roll back (section 10, first row). ### 6.5 Point the GitHub environment at the new project @@ -999,7 +1037,8 @@ after an agreed holding period: **Unverified:** nothing in this runbook has been run. The current deployment account's roles, the database roles and privileges (including whether -production's `telemetry_app` owns or holds anything), and the dashboard +production's `telemetry_app` owns or holds anything, and whether the old +users are members of `cloudsqlsuperuser`), and the dashboard identity's bindings were not read for it (they are recorded in step 3.2). Whether a workflow re-run reads changed environment values is unchecked (step 6.6). Every setting above needs checking end to end in the staging pass. From 3206a389624434b26c88280ef3f825a9986ff48a Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:24:00 +0400 Subject: [PATCH 11/31] Correct the signing-key note and the quiet-day window length #16 adds the signing-key secret to both environments' deployments, so without it that environment's deployment stops, not only production's. The quiet-day figure counted merge to tag only (25 minutes on 2026-10-15). The two deploy jobs add about 10 minutes, and the export and import come on top, so the runbook's section 6 and the memo's D2 B now say about 40 minutes. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 2 +- docs/project-move-runbook.md | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 3baf93b..0f12313 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -49,7 +49,7 @@ Where the address appears: | Option | Cost | Risks and limits | |---|---|---| | **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | -| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners: about 30 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | +| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners: about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | | **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | What a dedicated project gives: its own permissions (a deployment identity diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 775c9fc..d91753c 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -438,9 +438,9 @@ The dashboard's access to the read secret is set in section 8.1. the key ID published in the JWKS doesn't change. Add it to both grant loops above: `roles/secretmanager.secretAccessor` for the runtime account and `roles/secretmanager.viewer` for the deployment account. The deployment - checks that it exists and mounts it as `TELEMETRY_JWT_SIGNING_KEY`, so - production's deployment stops at "Check required Secret Manager entries - exist" without it. + checks that it exists and mounts it as `TELEMETRY_JWT_SIGNING_KEY` in both + environments, so without it that environment's deployment stops at "Check + required Secret Manager entries exist". - **#12's intake:** in the new project, the Cloud Tasks queue, the worker job, the recovery job (its three `COLLECTOR_WORKER_*` settings naming the new queue, worker job and invoker account), the invoker account, the @@ -509,8 +509,10 @@ deployment, then production's). From merge to tag this took 25 minutes on 2026-10-15 and 88 minutes on 2026-10-19 and 2026-10-20, when the organization's runners were queued; on 2026-10-20, calibration-diagnostics' last collector deploy run took 2 hours 25 minutes in all, of which its staging -and production deploy jobs ran for 4 and 6 minutes. So plan for about 30 -minutes on a quiet day and 2 to 3 hours when runners are queued, and measure +and production deploy jobs ran for 4 and 6 minutes. So plan for about 40 +minutes on a quiet day (25 minutes to the tag, about 10 for the two deploy +jobs, and the export and import) and 2 to 3 hours when runners are queued, +and measure it in staging's window. If #12's intake is switched on, add the wait for its worker executions in step 6.2, up to about 3 hours, unless they are cancelled or the queue is paused early. The dashboard's collector reads stay down until From fc0eb3afa4d9e240e7ee145c93d640885256f06c Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:24:47 +0400 Subject: [PATCH 12/31] Say the 40 minutes covers the whole window, and tidy new text The memo's D2 B sentence put the quiet-day figure on the redeploy alone; the 40 minutes includes the export and import, so it now says "in all". Two paragraphs added in this round are rewrapped, and one sentence in step 5.2 reads in order. Documentation only, so there is no failing test to add first. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 2 +- docs/project-move-runbook.md | 21 ++++++++++----------- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 0f12313..195d701 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -49,7 +49,7 @@ Where the address appears: | Option | Cost | Risks and limits | |---|---|---| | **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | -| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners: about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | +| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | | **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | What a dedicated project gives: its own permissions (a deployment identity diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index d91753c..bc32167 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -350,7 +350,7 @@ the row doesn't show: `ALTER ROLE telemetry_app NOLOGIN;` if its row shows no `LOGIN`. Google's page says `ALTER ROLE` changes any user's attributes. -Step 6.4 compares the result. Then recreate what an export doesn't carry: the +Step 6.4 compares the result. Also recreate what an export doesn't carry: the database-level privileges (`GRANT CONNECT` and `CREATE ON DATABASE`) from the `\l` record, and role memberships from the roles query. Table privileges, schema privileges and default privileges (`\dp`, `\dn+`, `\ddp`) come with the @@ -445,10 +445,10 @@ The dashboard's access to the read secret is set in section 8.1. job, the recovery job (its three `COLLECTOR_WORKER_*` settings naming the new queue, worker job and invoker account), the invoker account, the account Cloud Scheduler starts the recovery job as, and the permissions - from the README's "Artifact intake" section. Create the - recovery job's Cloud Scheduler job only in step 6.7: until the window's - deployment, the new database is empty or holds step 5.5's rehearsal copy, - and a recovery run would dispatch jobs from it. + from the README's "Artifact intake" section. Create the recovery job's + Cloud Scheduler job only in step 6.7: until the window's deployment, the + new database is empty or holds step 5.5's rehearsal copy, and a recovery + run would dispatch jobs from it. **Keep the intake bucket in the old project.** When a submission is completed, the collector records each stored object's generation number in @@ -512,12 +512,11 @@ last collector deploy run took 2 hours 25 minutes in all, of which its staging and production deploy jobs ran for 4 and 6 minutes. So plan for about 40 minutes on a quiet day (25 minutes to the tag, about 10 for the two deploy jobs, and the export and import) and 2 to 3 hours when runners are queued, -and measure -it in staging's window. If #12's intake is switched on, add the wait for its -worker executions in step 6.2, up to about 3 hours, unless they are cancelled -or the queue is paused early. The dashboard's collector reads stay down until -section 8.1's dashboard deployment finishes, unless the old service keeps -serving (section 8.4). +and measure it in staging's window. If #12's intake is switched on, add the +wait for its worker executions in step 6.2, up to about 3 hours, unless they +are cancelled or the queue is paused early. The dashboard's collector reads +stay down until section 8.1's dashboard deployment finishes, unless the old +service keeps serving (section 8.4). While the window lasts: From 27411f125cda5a66025886b5d957d845f6fc764a Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:57:29 +0400 Subject: [PATCH 13/31] List the access each step needs in policyengine-api Section 1 asked only for read access and Cloud SQL admin in the old project, but the steps also read secret values, change the intake bucket's IAM policy, update the old Cloud Run service, pause and resume Cloud Tasks and Cloud Scheduler, change alert policies and delete resources there. A table now names the role for each step. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index bc32167..0d6b839 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -21,10 +21,24 @@ another. ## 1. People and access -- **Google Cloud:** permission to create a project under the chosen parent and - link the billing account; then the owner role on the new project. Read access - to `policyengine-api`, and Cloud SQL admin on the old instances for the - export. +- **Google Cloud, the new project:** permission to create a project under the + chosen parent and link the billing account; then the owner role on the new + project. +- **Google Cloud, `policyengine-api`:** the steps change resources there, not + only read them. The project owner role covers all of these, if that is + simpler for the windows: + + | Steps | What they do in `policyengine-api` | Role | + |---|---|---| + | 2, 3.2, 6.1, 9 | read resources and their IAM policies | `roles/viewer` and `roles/iam.securityReviewer` | + | 3.2, 5.3 | read the collector's secret values (step 3.2's database connection uses the migration user's password) | `roles/secretmanager.secretAccessor` on the collector's secrets | + | 3.2, 5.2, 5.5, 6.3, 11 | connect to and export the old databases; delete the old instances | `roles/cloudsql.admin` | + | 5.3, 11 | change the intake bucket's IAM policy (#12) | `roles/storage.admin` on that bucket | + | 6.2, 8.4, 10, 11 | update the old service and its traffic; run and cancel job executions; delete the old services and jobs | `roles/run.admin`, and `roles/iam.serviceAccountUser` on the old runtime account, because a service update deploys a revision that runs as it | + | 6.2, 10, 11 | pause, resume and delete #12's queue | `roles/cloudtasks.queueAdmin` | + | 6.2, 6.7, 10, 11 | pause, resume and delete #12's and #19's schedules | `roles/cloudscheduler.admin` | + | 6.7, 10, 11 | disable, enable and delete #19's alert policies; delete its notification channel | `roles/monitoring.alertPolicyEditor` and `roles/monitoring.notificationChannelEditor` | + | 11 | delete secrets, service accounts, this repo's Workload Identity provider and the collector's images | `roles/secretmanager.admin`, `roles/iam.serviceAccountAdmin`, `roles/iam.workloadIdentityPoolAdmin` and `roles/artifactregistry.repoAdmin` | - **DNS:** edit access to the chosen zone (choice 5). For D3 B only, a Google account that is a verified owner of the base domain in Search Console; D3 C's certificate needs only a DNS record. From c9c04d9c754da958388ac6301663fab0d6238336 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:58:35 +0400 Subject: [PATCH 14/31] Say what keeping the intake bucket ties to policyengine-api If #12's intake is switched on before the move, its bucket stays in policyengine-api, so its storage cost, data-access logs and administration stay there, and shutting that project down or removing its billing would make every submission's files unreadable. The memo's D2 B row, "What a dedicated project gives" and "Order and timing" now say so, recommend switching intake on only after the move, and record that intake is off today (read-only check: the Cloud Tasks and Cloud Scheduler APIs aren't enabled, and no intake bucket exists). In the runbook, step 3.2 also records the old accounts' project-level roles, because #12's unwritten deployment wiring may grant storage access on the project. Step 5.3 names the permissions the code needs on the bucket (create and list for the runtime account, get for the worker's) and grants them on the bucket. Moving the bucket later is a one-off data change that rewrites the recorded generation numbers, not necessarily a code change. Section 9 checks the bucket's policy and one test submission, and section 11 says the old project must stay active while the collector reads the files. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 19 +++++++--- docs/project-move-runbook.md | 57 ++++++++++++++++++++++++----- 2 files changed, 61 insertions(+), 15 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 195d701..d9a3bfc 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -20,7 +20,8 @@ end. Taken from this repo's deployment code, the GitHub environment settings (names compared, values not printed), the Vercel project settings (names only), and calibration-diagnostics#233's read-only checks of the live resources. The live -Google Cloud resources were not re-read for this memo. +Google Cloud resources were not re-read for this memo, except for whether +#12's intake is switched on (see "Order and timing"). | What | Where | |---|---| @@ -49,13 +50,15 @@ Where the address appears: | Option | Cost | Risks and limits | |---|---|---| | **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | -| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project. One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. | +| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project (except #12's intake bucket, if intake is switched on before the move). One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. **If #12's intake is switched on before the move,** its bucket can't move (see "Order and timing"), so its storage cost, its data-access audit logs and its administration stay in `policyengine-api`, and anyone who administers storage there can change or delete the collector's files. Shutting that project down, or removing its billing, would then make every submission's files unreadable: Google's "Delete and restore projects" page says "Shutting down a project stops all billing and resource usage", and that Cloud Storage "might delete resources much sooner" than the 30-day recovery period. | | **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | What a dedicated project gives: its own permissions (a deployment identity that can reach only the collector), its own quotas and billing, audit logs that contain only the collector, and room for the intake and worker resources -without adding them to the Public API's project. +without adding them to the Public API's project. For intake's bucket this holds +only if intake is switched on after the move; otherwise the bucket and the +collector's stored files stay in the Public API's project. ## D3: which address @@ -204,8 +207,14 @@ The plan's constraints: - If #12's intake is switched on before the move, its bucket stays in `policyengine-api`: each accepted submission's row records its files' Cloud Storage generation numbers, which a copy into a new bucket doesn't - keep (the runbook's step 5.3). Switching intake on after the move creates - the bucket in the new project from the start. + keep (the runbook's step 5.3). The collector's files then depend on that + project for good, as D2 B's row says. Moving them later needs a one-off + change, not yet written, that copies the files, checks their SHA-256 values + again and rewrites each submission's recorded generation numbers. +- Recommended, if D2 is B or C: switch intake on only after the move, which + creates the bucket in the new project from the start. Intake is off today: a read-only check of + `policyengine-api` on 2026-10-20 found the Cloud Tasks and Cloud Scheduler + APIs not enabled, and none of its six buckets is an intake bucket. If both D2 and D3 are adopted, there are two orders: diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 0d6b839..71a880a 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -104,7 +104,10 @@ from the old instance in step 3.2. - #12's artifact intake: a Cloud Tasks queue, the worker job, the recovery job and its Cloud Scheduler job, the invoker account and their permissions, as in the README's "Artifact intake" section. Its bucket - stays in the old project (step 5.3); + stays in the old project (step 5.3), which ties the collector's files to + `policyengine-api` (the memo's D2 B). Intake was off on 2026-10-20: the + Cloud Tasks and Cloud Scheduler APIs weren't enabled there, and no intake + bucket existed. Switching it on only after the move avoids this; - S1's Hugging Face read token (#13, PR #18); - C5's session signing key (#16), a fifth Secret Manager secret per environment; @@ -159,7 +162,12 @@ For each environment (read-only): `gcloud run jobs describe` with `--region "$REGION"` for the worker job and the recovery job; `gcloud scheduler jobs describe RECOVERY_SCHEDULE --location "$REGION"`; the bucket's roles with `gcloud storage buckets get-iam-policy gs://INTAKE_BUCKET`; - the invoker account's with `gcloud iam service-accounts get-iam-policy`. + the invoker account's with `gcloud iam service-accounts get-iam-policy`; + and the project-level roles of the old runtime, worker and recovery + accounts, with the `gcloud projects get-iam-policy` command shown below for + the deployment account. #12's deployment wiring isn't written, so its + storage grants may be on the project rather than on the bucket, and the + bucket's policy wouldn't show them. - #19: `gcloud run jobs describe "$HF_CHECK_JOB" --region "$REGION"`; `gcloud scheduler jobs describe HF_CHECK_SCHEDULE --location "$REGION"`, which shows the account it starts the job as; the two alert policies' @@ -468,12 +476,29 @@ The dashboard's access to the read secret is set in section 8.1. completed, the collector records each stored object's generation number in the submission's row, and the worker reads every file at that generation. Cloud Storage assigns generation numbers, and they can't be set, so a copy - in another bucket has different ones and the worker couldn't read any - accepted submission's files. Grant the new runtime account and the new - worker job's account the roles on the old bucket that step 3.2 recorded for - the old accounts. `COLLECTOR_INTAKE_BUCKET` keeps its value. Moving the - bucket later needs a code change that records the copies' generations; - none is written. + in another bucket has different ones, and the worker would reject every + accepted submission as `object_missing`. On the old bucket, grant the new + runtime account `storage.objects.create` (its signed upload URLs create + the objects) and `storage.objects.list` (completion lists them), and the + new worker job's account `storage.objects.get` (it reads each file); the + recovery job doesn't use the bucket. Use the roles step 3.2 recorded for the + old accounts, whether they were granted on the bucket or on the project, + and grant them on the bucket, for example `roles/storage.objectCreator` and + `roles/storage.objectViewer` for the runtime account and + `roles/storage.objectViewer` for the worker's: + + ```bash + gcloud storage buckets add-iam-policy-binding gs://INTAKE_BUCKET --project "$OLD_PROJECT_ID" \ + --member "serviceAccount:$RUNTIME_SERVICE_ACCOUNT" --role ROLE + ``` + + `COLLECTOR_INTAKE_BUCKET` keeps its value. This ties the collector's files + to `policyengine-api`: their storage cost, data-access audit logs and + administration stay there, and shutting that project down, or removing its + billing, makes them unreadable (the memo's D2 B). Moving the bucket later + needs a one-off change that copies the files, checks their SHA-256 values + again and rewrites each submission's recorded generation numbers; none is + written. - **S2's daily check (#19), production only:** the worker service account (the one that may read S1's token), with `roles/iam.serviceAccountUser` on it for the deployment account; `roles/secretmanager.viewer` on S1's secret @@ -974,6 +999,16 @@ For each environment: - If #19 has merged: step 6.7's execution of the check job succeeded, the old project's two alert policies are disabled, and its check schedule is paused. +- If #12's intake is switched on: + `gcloud storage buckets get-iam-policy gs://INTAKE_BUCKET --project "$OLD_PROJECT_ID"` + lists the new runtime and worker accounts with the roles from step 5.3, and + one small test submission, made through the flow in #12's + `docs/artifact-submissions.md` (token exchange for `artifacts:submit`, + create, upload, complete), reaches the status `succeeded` at + `GET /v1/submissions/{submission_id}`. That shows the new runtime account + can sign uploads and list the bucket, the new queue starts the new worker + job, and that job can read the files. The test files stay in the bucket, + which has no lifecycle deletion. ## 10. Rollback @@ -1002,8 +1037,10 @@ after an agreed holding period: if merged), and the old runtime, migration and worker service accounts. - If #12 merged: the queue (with any tasks still in it), the worker and recovery jobs, the recovery schedule and the account it used, the invoker - account, and the old accounts' roles on the intake bucket. Keep the bucket: - it holds every submission's files and is still in use (step 5.3). + account, and the old accounts' roles on the intake bucket, or on the + project if step 3.2 found them there. Keep the bucket: it holds every + submission's files and is still in use (step 5.3). So `policyengine-api` + has to stay active, and billed, for as long as the collector reads them. - If #19 merged: the old check job, its schedule and the account the schedule used, both alert policies and the notification channel. Step 6.7 already disabled the policies and paused the schedule. From f0d6d4f5b084dd19710d5402f6f9416b6fcdc013 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 20 Oct 2026 23:59:10 +0400 Subject: [PATCH 15/31] Take the extra database users from each environment's record Step 5.2 still set other_database_users=(telemetry_app) whatever the environment, so pasting the block for staging created a user staging doesn't have, and step 6.4's roles comparison would then stop the window. Step 3.2 now records each environment's users as listed on 2026-10-20 (staging: postgres, telemetry_migrator, telemetry_runtime; production: the same plus telemetry_app) and sets OTHER_DATABASE_USERS per environment, and step 5.2 creates that list, which is empty for staging. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 39 ++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 71a880a..b87fbda 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -86,8 +86,8 @@ DEPLOY_SERVICE_ACCOUNT="collector-deploy@$NEW_PROJECT_ID.iam.gserviceaccount.com EXPORT_BUCKET="gs://$NEW_PROJECT_ID-collector-move" ``` -`DATABASE_NAME`, `RUNTIME_DATABASE_USER` and `MIGRATION_DATABASE_USER` come -from the old instance in step 3.2. +`DATABASE_NAME`, `RUNTIME_DATABASE_USER`, `MIGRATION_DATABASE_USER` and +`OTHER_DATABASE_USERS` come from the old instance in step 3.2. ## 3. Before the windows @@ -131,9 +131,27 @@ For each environment (read-only): flags, and IP and SSL configuration. - The database and its users: `gcloud sql databases list` and `gcloud sql users list` with - `--instance "$OLD_INSTANCE" --project "$OLD_PROJECT_ID"`. Set - `DATABASE_NAME`, `RUNTIME_DATABASE_USER` and `MIGRATION_DATABASE_USER` from - them. + `--instance "$OLD_INSTANCE" --project "$OLD_PROJECT_ID"`. On 2026-10-20 + both instances held the databases `postgres` and `telemetry`, and listed + these users: + + | Environment | Users | + |---|---| + | staging | `postgres`, `telemetry_migrator`, `telemetry_runtime` | + | production | `postgres`, `telemetry_app`, `telemetry_migrator`, `telemetry_runtime` | + + Set `DATABASE_NAME`, `RUNTIME_DATABASE_USER` and `MIGRATION_DATABASE_USER` + from them. Set `OTHER_DATABASE_USERS` to every other role the roles query + below lists, apart from `postgres`; as on 2026-10-20: + + ```bash + case "$ENVIRONMENT_NAME" in + production) OTHER_DATABASE_USERS=(telemetry_app) ;; + *) OTHER_DATABASE_USERS=() ;; + esac + ``` + + Change it if the record shows otherwise. - **The database's roles and privileges.** Connect as the migration user through Cloud SQL Studio, or with `psql` through the Cloud SQL Auth Proxy, and save the output of `\l`, `\dn+ public`, `\dp public.*` and `\ddp`, and @@ -333,9 +351,10 @@ Create every user the step 3.2 record lists, except `postgres`, with the **same names** as on the old instance and new random passwords. Google's import guide says every user that owns objects or was granted permissions on objects in the dumped database must exist on the target instance, or the import -fails. Besides the migration and runtime users, production's instance lists -`telemetry_app` today, which the collector doesn't use; give it a random -password that isn't kept. +fails. Besides the migration and runtime users, create each role in +`OTHER_DATABASE_USERS` (step 3.2: `telemetry_app` for production today, which +the collector doesn't use, and none for staging), with a random password that +isn't kept. ```bash migration_password="$(openssl rand -hex 32)" @@ -344,9 +363,7 @@ gcloud sql users create "$MIGRATION_DATABASE_USER" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$migration_password" gcloud sql users create "$RUNTIME_DATABASE_USER" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$runtime_password" -# Every other user the record lists: (telemetry_app) for production today, () for staging. -other_database_users=(telemetry_app) -for other_database_user in "${other_database_users[@]}"; do +for other_database_user in "${OTHER_DATABASE_USERS[@]}"; do # none for staging gcloud sql users create "$other_database_user" --instance "$NEW_INSTANCE" \ --project "$NEW_PROJECT_ID" --password "$(openssl rand -hex 32)" done From c2bd713e1ce9a637347001899dbf55c91ad7ce31 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Wed, 21 Oct 2026 00:02:44 +0400 Subject: [PATCH 16/31] Make each database user the way its old one was made The roles comparison in step 6.4 would differ on the postgres row if the old users were made in SQL by postgres: PostgreSQL 16 grants each role a non-superuser creates back to its creator with ADMIN TRUE, SET FALSE, INHERIT FALSE, and users made by gcloud sql users create don't get that grant. Step 3.2's roles query now shows each membership's admin, inherit and set options. Step 5.2 creates a user whose row lists cloudsqlsuperuser with gcloud sql users create, and any other user in SQL as the role that holds it as "(admin)", normally postgres, with exactly the recorded attributes, so both rows match without removing anything. Step 5.2 compares the roles query before the window, and step 6.4 accepts only the differences step 5.2 names. Whether postgres may ALTER ROLE a user Cloud SQL made is not documented: PostgreSQL 16 requires ADMIN OPTION on the target, and Cloud SQL's pages don't say whether postgres holds it. Step 5.2 now checks it once on staging's new instance, with a throwaway user, before production relies on it, and gives the fallback: create that user in SQL as postgres, which then holds ADMIN OPTION, and accept the two resulting differences. The local check of this behaviour on PostgreSQL 16.4, with a non-superuser postgres, refused ALTER ROLE on a role it hadn't created. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 141 +++++++++++++++++++++++++++-------- 1 file changed, 108 insertions(+), 33 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index b87fbda..eb9af37 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -158,12 +158,18 @@ For each environment (read-only): of the roles query below. Note which role owns the tables and what the runtime user is granted. These were provisioned by scripts kept outside this repo, so this record is the only specification of them. The roles - query lists every role's attributes and memberships in one result; psql 16 - and later no longer show memberships in `\du`: + query lists every role's attributes and memberships in one result, since + psql 16 and later no longer show memberships in `\du`. Each membership + carries its PostgreSQL 16 options (`admin`, `inherit`, `set`, from + `pg_auth_members`), which show how each role was made (step 5.2): ```sql SELECT r.rolname, r.rolcanlogin, r.rolcreaterole, r.rolcreatedb, r.rolinherit, - ARRAY(SELECT g.rolname FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid + ARRAY(SELECT g.rolname || ' (' || concat_ws(', ', + CASE WHEN m.admin_option THEN 'admin' END, + CASE WHEN m.inherit_option THEN 'inherit' END, + CASE WHEN m.set_option THEN 'set' END) || ')' + FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE m.member = r.oid ORDER BY 1) AS member_of FROM pg_roles r WHERE r.rolname !~ '^(pg_|cloudsql)' @@ -347,7 +353,7 @@ Match the record's IP configuration (Cloud Run connects through the Cloud SQL connector, so a public IP needs no authorized networks), SSL mode, database flags, maintenance window and point-in-time recovery. -Create every user the step 3.2 record lists, except `postgres`, with the +Create every role the step 3.2 roles query lists, except `postgres`, with the **same names** as on the old instance and new random passwords. Google's import guide says every user that owns objects or was granted permissions on objects in the dumped database must exist on the target instance, or the import @@ -356,38 +362,106 @@ fails. Besides the migration and runtime users, create each role in the collector doesn't use, and none for staging), with a random password that isn't kept. +**Make each role the way its old one was made.** An export carries no roles, +so nothing corrects a role's attributes or memberships later. Made the same +way, each role's row and its creator's row match the record, so step 6.4's +comparison of the roles query shows real differences only. The record shows +how each was made: + +- **Its row lists `cloudsqlsuperuser`:** Cloud SQL made it. Cloud SQL's pages + "About PostgreSQL users and roles" and "Create and manage users" say a user + created with `gcloud sql users create` is a member of `cloudsqlsuperuser`, + with the `CREATEROLE`, `CREATEDB` and `LOGIN` attributes. Create it that way. + If its row lacks one of those attributes, see the staging check below. +- **Its row doesn't list `cloudsqlsuperuser`:** it was made in SQL, or its + membership was taken away later. On PostgreSQL 16, a role made by a role + that isn't a superuser, such as Cloud SQL's `postgres`, "is automatically + granted back to the creating user, just as if the bootstrap superuser had + executed the command `GRANT created_user TO creating_user WITH ADMIN TRUE, + SET FALSE, INHERIT FALSE`" (PostgreSQL 16, "Role Attributes"). The roles + query shows that grant as `ROLE (admin)` in the creator's `member_of`, + normally `postgres`'s. Create the role in SQL, connected as that creator, + with exactly the attributes its row shows. PostgreSQL then makes the same + grant, so both rows match the record, and nothing has to be removed + afterwards. If no role lists it as `(admin)`, create it as `postgres`; + step 6.4 then accepts that one extra entry in `postgres`'s row. + +For the SQL, run `psql` through the Cloud SQL Auth Proxy +(`cloud-sql-proxy "$NEW_CLOUD_SQL_CONNECTION"` in another shell, listening on +127.0.0.1:5432). First set `postgres`'s password on the new instance, typed +rather than printed: +`gcloud sql users set-password postgres --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --prompt-for-password`. + ```bash migration_password="$(openssl rand -hex 32)" runtime_password="$(openssl rand -hex 32)" -gcloud sql users create "$MIGRATION_DATABASE_USER" --instance "$NEW_INSTANCE" \ - --project "$NEW_PROJECT_ID" --password "$migration_password" -gcloud sql users create "$RUNTIME_DATABASE_USER" --instance "$NEW_INSTANCE" \ - --project "$NEW_PROJECT_ID" --password "$runtime_password" +create_user_with_cloud_sql() { # USER PASSWORD + gcloud sql users create "$1" --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --password "$2" +} +create_user_in_sql() { # CREATOR USER PASSWORD 'ATTRIBUTES' + psql "host=127.0.0.1 port=5432 dbname=postgres user=$1" --set ON_ERROR_STOP=1 \ + --set user_name="$2" --set user_password="$3" --set attributes="$4" <<'SQL' +CREATE ROLE :"user_name" WITH :attributes PASSWORD :'user_password'; +SQL +} +# The calls below are examples: for each role, make the one call its step 3.2 row +# calls for, with the attributes its row shows. The placeholder stops the loop. +create_user_with_cloud_sql "$MIGRATION_DATABASE_USER" "$migration_password" +create_user_in_sql postgres "$RUNTIME_DATABASE_USER" "$runtime_password" 'LOGIN NOCREATEROLE NOCREATEDB INHERIT' for other_database_user in "${OTHER_DATABASE_USERS[@]}"; do # none for staging - gcloud sql users create "$other_database_user" --instance "$NEW_INSTANCE" \ - --project "$NEW_PROJECT_ID" --password "$(openssl rand -hex 32)" + create_user_in_sql postgres "$other_database_user" "$(openssl rand -hex 32)" 'ATTRIBUTES FROM ITS ROW' done ``` -Keep the two password variables in this shell for step 5.3. - -**Give each user no more than its record shows.** Cloud SQL's pages "About -PostgreSQL users and roles" and "Create and manage users" say a user created -this way is a member of `cloudsqlsuperuser`, with the `CREATEROLE`, -`CREATEDB` and `LOGIN` attributes. An old user created in SQL may have less, -and an export carries no roles, so nothing corrects it later. For each new -user, compare with its row from the roles query in step 3.2 and remove what -the row doesn't show: - -- **Membership of `cloudsqlsuperuser`:** - `gcloud sql users assign-roles USER --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --type BUILT_IN --revoke-existing-roles`, - with `--database-roles` for any membership the row does show, once that - role exists. Google's page gives this as the way to take - `cloudsqlsuperuser` away from a user. -- **Attributes:** in SQL, connected as `postgres`, for example - `ALTER ROLE telemetry_runtime NOCREATEROLE NOCREATEDB;`, and - `ALTER ROLE telemetry_app NOLOGIN;` if its row shows no `LOGIN`. Google's - page says `ALTER ROLE` changes any user's attributes. +psql puts the variables' values into the statement, quoted as a name +(`:"user_name"`), as a string (`:'user_password'`) or as written +(`:attributes`), so neither the shell's history nor psql's holds a password. +PostgreSQL's `CREATE ROLE` page warns that a password in the statement "might +also be logged in ... the server log": the server logs it if the statement +fails, or if the `log_statement` flag recorded in step 3.2 is `ddl` or `all`. +Keep the two password variables in this shell for step 5.3. Then grant the +other memberships the rows show, and run the roles query on the new instance: +it must already match the record, apart from the differences this step names. + +**Check once, on staging's new instance, whether `postgres` can change a user +Cloud SQL made.** PostgreSQL 16's `ALTER ROLE` page says a role with +`CREATEROLE` that isn't a superuser, as Cloud SQL's `postgres` is, can change +another role's attributes "only for non-superuser and non-replication roles +for which they have been granted `ADMIN OPTION`". Cloud SQL's pages say "You +can change the attributes of any user using the ALTER ROLE command", but not +whether `postgres` holds `ADMIN OPTION` on users Cloud SQL makes. So don't +rely on it until this check has passed, in staging's pass and before +production's step 5.2: + +```bash +gcloud sql users create move_check --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" \ + --password "$(openssl rand -hex 32)" +psql "host=127.0.0.1 port=5432 dbname=postgres user=postgres" <<'SQL' +SELECT g.rolname AS holder, m.admin_option, m.inherit_option, m.set_option +FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.member +WHERE m.roleid = 'move_check'::regrole; +ALTER ROLE move_check NOCREATEROLE NOCREATEDB NOLOGIN; +SQL +gcloud sql users delete move_check --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --quiet +``` + +Write the result in the window notes. It matters only for a row that lists +`cloudsqlsuperuser` but lacks `CREATEROLE`, `CREATEDB` or `LOGIN`: + +- **`ALTER ROLE` succeeds:** create that user with `gcloud sql users create`, + then remove what its row lacks with `ALTER ROLE` as `postgres`, for example + `ALTER ROLE telemetry_app NOLOGIN;`. +- **It fails** with "permission denied to alter role": create that user in + SQL as `postgres` instead, with the attributes its row shows, so that + `postgres` holds `ADMIN OPTION` on it. Its row then lacks + `cloudsqlsuperuser`, and `postgres`'s row lists it as `(admin)`; step 6.4 + accepts exactly these two differences for that user. It then has less + privilege than its record. The README asks no more of the two users the + collector uses: the migration credential needs "schema-modification + access", and the runtime credential "only the data access required by the + application". If that user is the importing user, the `public` schema step + below must give it `CREATE` on `public`, and step 6.4 accepts that + difference too. Step 6.4 compares the result. Also recreate what an export doesn't carry: the database-level privileges (`GRANT CONNECT` and `CREATE ON DATABASE`) from the @@ -701,10 +775,11 @@ On both instances, as the migration user: and `telemetry_events`; later migrations add more. - `\l`, `\dp public.*` and `\ddp` match the step 3.2 record, so the database-level, table and default privileges all came across. -- The roles query from step 3.2 gives the same rows, so every user has the - same attributes and memberships. If a new user has more, such as - `cloudsqlsuperuser`, `CREATEROLE` or `CREATEDB` left from - `gcloud sql users create`, remove it as in step 5.2 and compare again. +- The roles query from step 3.2 gives the same rows, `postgres`'s included, + so every role has the same attributes and the same memberships with the + same options. Step 5.2 already compared them before the window; the only + accepted differences are the ones it names for a role it had to make + differently, as written in the window notes. If anything else differs, stop and roll back (section 10, first row). From d86897ec8ec2310150dcad894a6de60064d56b85 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Wed, 21 Oct 2026 00:21:04 +0400 Subject: [PATCH 17/31] Set the new database's owner and public grants before importing On PostgreSQL 15 and later only the database's owner may create objects in schema public unless granted CREATE there. If the old migration user isn't a cloudsqlsuperuser member, it got that by owning the database or public, or by a grant, and none of these exists on the new database before the import. The dump's own grants on public come after its tables and run as the importing user, so the import's first CREATE TABLE would fail. Step 5.2 now compares the new database's \l and \dn+ public with the record before any import (step 5.5's rehearsal and step 6.3) and applies what differs: privileges first, then owners. A Cloud SQL-made owner makes itself the owner; for a role postgres made, postgres grants itself SET on it for the change and revokes that afterwards, which PostgreSQL 16 allows and which leaves the automatic grant in place (checked on a local PostgreSQL 16.4 emulation). Step 5.5 repeats these steps after re-creating the database, and step 6.4 also compares \dn+ public. The example loop's comment now says a placeholder makes each call fail rather than stopping the loop. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 79 +++++++++++++++++++++++++++++------- 1 file changed, 64 insertions(+), 15 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index eb9af37..8aa01df 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -405,7 +405,8 @@ CREATE ROLE :"user_name" WITH :attributes PASSWORD :'user_password'; SQL } # The calls below are examples: for each role, make the one call its step 3.2 row -# calls for, with the attributes its row shows. The placeholder stops the loop. +# calls for, with the attributes its row shows. Until the placeholder in the loop +# is replaced, PostgreSQL refuses each call there and creates nothing. create_user_with_cloud_sql "$MIGRATION_DATABASE_USER" "$migration_password" create_user_in_sql postgres "$RUNTIME_DATABASE_USER" "$runtime_password" 'LOGIN NOCREATEROLE NOCREATEDB INHERIT' for other_database_user in "${OTHER_DATABASE_USERS[@]}"; do # none for staging @@ -459,15 +460,61 @@ Write the result in the window notes. It matters only for a row that lists privilege than its record. The README asks no more of the two users the collector uses: the migration credential needs "schema-modification access", and the runtime credential "only the data access required by the - application". If that user is the importing user, the `public` schema step - below must give it `CREATE` on `public`, and step 6.4 accepts that - difference too. - -Step 6.4 compares the result. Also recreate what an export doesn't carry: the -database-level privileges (`GRANT CONNECT` and `CREATE ON DATABASE`) from the -`\l` record, and role memberships from the roles query. Table privileges, -schema privileges and default privileges (`\dp`, `\dn+`, `\ddp`) come with the -export, provided the roles exist. + application". If that user is the importing user, the "Before any import" + steps below must also grant it `CREATE` on `public`, and step 6.4 accepts + that difference too. + +**Before any import into the new database** (step 5.5's rehearsal and step +6.3), give it the record's privileges and owners. An export doesn't carry the +database-level privileges. And the importing user (normally the migration +user) can create the tables only if it may create objects in schema `public`: +on PostgreSQL 15 and later, `public` belongs to `pg_database_owner`, that is +to the database's owner, and other users no longer have `CREATE` on it +(PostgreSQL 15 release notes). On the old database the importing user had it +in one of three ways, which the `\l` and `\dn+ public` records show: it owns +the database or is a member of its owner (as every `cloudsqlsuperuser` member +is, if `cloudsqlsuperuser` owns it); it owns `public`; or it was granted +`CREATE` on `public`. The dump also sets `public`'s owner and privileges, but +as the importing user and only after creating the tables, so it can't give +the importing user what those tables need. Cloud SQL's users pages don't say +who owns a database that `gcloud sql databases create` makes, so compare +first. + +Connected to the new database as `postgres`, compare `\l "$DATABASE_NAME"` and +`\dn+ public` with the record, then apply what differs, in this order: + +1. **Privileges:** the database-level ones from `\l` (`GRANT CONNECT`, + `GRANT CREATE ON DATABASE`) and `public`'s from `\dn+ public` + (`GRANT USAGE, CREATE ON SCHEMA public TO ...`). `postgres` may grant them + while it acts as the owner, which it does through `cloudsqlsuperuser` if + that owns the new database. Grant them before changing an owner: after + that, `postgres` no longer acts as the owner. +2. **Owners:** `ALTER DATABASE ... OWNER TO ...` or + `ALTER SCHEMA public OWNER TO ...`. PostgreSQL 16's `ALTER DATABASE` page + says: "To alter the owner, you must be able to `SET ROLE` to the new owning + role, and you must have the `CREATEDB` privilege." + - **If Cloud SQL made the new owner,** connect as that user: as a + `cloudsqlsuperuser` member with `CREATEDB`, it can make itself the owner + of a database that `cloudsqlsuperuser` owns. + - **If `postgres` made it,** `postgres` holds it only as `(admin)`, without + `SET`. The "Role Attributes" page says a creator "can gain access to the + created role by simply granting that role back to themselves with the + `INHERIT` and/or `SET` options", so grant `SET` for the change and revoke + it afterwards, for example: + + ```sql + GRANT telemetry_migrator TO postgres WITH SET TRUE; + ALTER DATABASE telemetry OWNER TO telemetry_migrator; + REVOKE telemetry_migrator FROM postgres; + ``` + + The revoke removes only the grant `postgres` made; the same page says + the automatic grant "cannot be removed or changed by the `CREATEROLE` + user". The roles query's `postgres` row must match the record again. + +Table privileges and default privileges (`\dp`, `\ddp`) come with the export, +provided the roles exist. Step 6.4 compares all four records after the +import. Let both instances' service agents use the bucket. Google's export and import pages name `roles/storage.objectAdmin` for both: @@ -623,8 +670,8 @@ Before production's window, export the production database with `--offload` instance), import it into the new instance as in step 6.3, and note both durations. Then delete and re-create the new database (`gcloud sql databases delete`, then `create`), so the window's import starts -empty, and grant the database-level privileges from step 5.2 again: deleting -the database drops them. Staging's window gives the same timings for staging's +empty, and repeat step 5.2's "Before any import" steps (privileges, then +owners): deleting the database drops them. Staging's window gives the same timings for staging's size. ## 6. The window @@ -762,7 +809,8 @@ gcloud sql import sql "$NEW_INSTANCE" "$export_object" \ ``` Import as the role that owns the tables in the step 3.2 record (normally the -migration user), so the restored objects keep their owner. If #12's intake is +migration user), so the restored objects keep their owner. Step 5.2's "Before +any import" steps let it create them in `public`. If #12's intake is switched on, its bucket isn't copied: it stays in the old project (step 5.3). ### 6.4 Check the copy @@ -773,8 +821,9 @@ On both instances, as the migration user: - `SELECT count(*)` on every table in the `public` schema gives the same counts. At revision `0001` these are `telemetry_runs`, `telemetry_producers` and `telemetry_events`; later migrations add more. -- `\l`, `\dp public.*` and `\ddp` match the step 3.2 record, so the - database-level, table and default privileges all came across. +- `\l`, `\dn+ public`, `\dp public.*` and `\ddp` match the step 3.2 record, + so the database's owner and privileges, `public`'s owner and privileges, + and the table and default privileges all came across. - The roles query from step 3.2 gives the same rows, `postgres`'s included, so every role has the same attributes and the same memberships with the same options. Step 5.2 already compared them before the window; the only From 86dc17bbe20fd116bb99dc74a7e6059668364d96 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Wed, 21 Oct 2026 00:52:25 +0400 Subject: [PATCH 18/31] Restore the old project's queue, schedules and alerts on rollback The first rollback row re-enabled the old project's alert policies but didn't run the old check job, so after a rollback the old "no success for 23 hours" policy couldn't report a stopped schedule: #19's README says a metric-absence condition needs a successful run after the policy is created or changed. The row now runs the old check job once and waits for it, and pauses the new queue along with the new schedules. The "after the emitter release" row paused only the new project's queue and recovery schedule. It now also pauses the new check schedule and disables the new alert policies, and in the old project resumes the queue, the recovery schedule and the check schedule, enables the old alert policies and runs the old check job once. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 8aa01df..9c84dc1 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1158,10 +1158,10 @@ the moment the old service entered maintenance mode. | Situation | What to do | |---|---| -| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "$rollback_revision=100"`, with the revision step 6.1 recorded in this window. Set back, to the step 3.2 record, only the GitHub values step 6.5 changed in this window: in staging's window, staging's; in production's window, production's and `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` in staging's environment. Staging's other values stay pointed at the new project, where its database now is. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). If step 6.7 ran, pause the schedules it created in the new project and disable the new alert policies; for #19, resume the old check schedule and enable the old policies again (`--enabled`). Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | +| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "$rollback_revision=100"`, with the revision step 6.1 recorded in this window. Set back, to the step 3.2 record, only the GitHub values step 6.5 changed in this window: in staging's window, staging's; in production's window, production's and `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` in staging's environment. Staging's other values stay pointed at the new project, where its database now is. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). If step 6.7 ran, pause the schedules it created in the new project, with the new queue, and disable the new alert policies; for #19, resume the old check schedule, enable the old policies again (`--enabled`), then run the old check job once by hand and wait for it to succeed: `gcloud run jobs execute "$HF_CHECK_JOB" --region "$REGION" --project "$OLD_PROJECT_ID" --wait`. #19's README says the absence condition is met only after a successful run that follows the policy being "created or changed", and enabling it again changes it. Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | | A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | | After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | -| After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to the revision step 6.1 recorded in production's window (if a release with a migration has deployed in the new project since, that revision's code is older than the imported schema: set the GitHub values back first, and let the next merge deploy the current release to the old project instead), set the GitHub and dashboard settings back, and release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | +| After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to the revision step 6.1 recorded in production's window (if a release with a migration has deployed in the new project since, that revision's code is older than the imported schema: set the GitHub values back first, and let the next merge deploy the current release to the old project instead), and set the GitHub and dashboard settings back. Then reverse step 6.7: in the new project, pause #19's check schedule and disable the new alert policies (step 6.2's commands have already paused #12's queue and recovery schedule there); in the old project, resume #12's queue and recovery schedule, resume #19's check schedule, enable the old alert policies again and run the old check job once, as in the first row. Last, release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | | The hostname | D3 B: delete the mapping, or map the hostname back to the old project's service with `--force-override`. D3 C: point the `A` record back to what it named before, if anything. | ## 11. Retire the old resources From dbacc934101534474a78379d242c5757204af720 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Wed, 21 Oct 2026 00:52:57 +0400 Subject: [PATCH 19/31] Say the step 6.2 loop doesn't see a local reprocess run Step 6.2 said every reprocess run is a recovery-job execution, so the loop covers it. #12's README also allows running reprocess locally through the Cloud SQL Auth Proxy, which writes to the database without a Cloud Run execution. The step now says so and asks that nobody runs it that way during the window. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 9c84dc1..c950ad0 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -768,8 +768,10 @@ done ``` The loop prints the executions that haven't finished; repeat it until it -prints nothing. An operator's `reprocess` run is an execution of the recovery -job, so the loop covers it too. +prints nothing. It also lists an operator's `reprocess` run made as an +execution of the recovery job. #12's README also allows running `reprocess` +locally through the Cloud SQL Auth Proxy, which the loop can't see, so make +sure nobody runs it that way during the window. - **A recovery execution** only updates job rows and creates Cloud Tasks. Wait for it. From 78cf7226551eef644246ae85cbb8279f533c43cd Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Wed, 21 Oct 2026 01:09:02 +0400 Subject: [PATCH 20/31] List what is still unverified before the staging window Section 12's closing paragraph now lists the open questions plainly: whether postgres may ALTER ROLE a user Cloud SQL made (step 5.2's staging check settles it); who owns a database gcloud sql databases create makes, and the old databases' owners and grants; whether either project is inside a VPC Service Controls perimeter; whether policyengine-api has a project lien; and #12's unwritten deployment wiring. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index c950ad0..4abac32 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1230,13 +1230,33 @@ after an agreed holding period: | Dashboard settings | calibration-diagnostics GitHub `Production` environment and the Vercel project | see section 8.1 | see section 8.1 | | `PRODUCTION_COLLECTOR_URL`, `TRUSTED_COLLECTOR_ORIGINS` | emitter source (`core/constants.py`) | every emitter service module | the hostname, by an emitter release | -**Unverified:** nothing in this runbook has been run. The current deployment -account's roles, the database roles and privileges (including whether -production's `telemetry_app` owns or holds anything, and whether the old -users are members of `cloudsqlsuperuser`), and the dashboard -identity's bindings were not read for it (they are recorded in step 3.2). -Whether a workflow re-run reads changed environment values is unchecked (step -6.6). Every setting above needs checking end to end in the staging pass. +**Unverified before the staging window.** Nothing in this runbook has been +run against Cloud SQL or the new project. Still open: + +- **Whether `postgres` may change a user Cloud SQL made** with `ALTER ROLE`, + which PostgreSQL 16 allows only with `ADMIN OPTION` on that user. Step + 5.2's check on staging's new instance settles it, and gives the fallback. +- **Database owners:** who owns a database that `gcloud sql databases create` + makes, and the old databases' owners, `public` privileges, roles, grants and + memberships (including whether production's `telemetry_app` owns or holds + anything). These need a database connection; step 3.2 records the old ones, + and step 5.2 compares the new database before any import. +- **VPC Service Controls:** whether either project is inside a service + perimeter, which could block access between them: the new accounts' access + to the old intake bucket (step 5.3), and the old project's image reads from + the new Artifact Registry store (step 4.1). Checking needs + organization-level access. +- **Project liens:** whether `policyengine-api` has a lien that stops it being + deleted, which matters if the intake bucket stays there. Listing liens needs + `gcloud alpha resource-manager liens list`. +- **#12's deployment wiring,** not yet written: its GitHub variable names + (step 6.5, this table), and whether it grants storage roles on the bucket or + on the project (step 3.2). +- The current deployment account's roles and the dashboard identity's bindings + (recorded in step 3.2), and whether a workflow re-run reads changed + environment values (step 6.6). + +Every setting above needs checking end to end in the staging pass. ## 13. Code changes that go with this runbook (not yet written) From 543a8bd5df9829ad3d577e75ce913d15a3b28954 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Wed, 21 Oct 2026 01:36:40 +0400 Subject: [PATCH 21/31] Correct two overstatements about the intake bucket, and rewrap The memo said the bucket "can't move" and that the files depend on policyengine-api "for good", while the same text says a one-off change could move them. It now says the files depend on that project until such a change, not yet written, moves them. Section 12 no longer says nothing has been run: some read-only commands have been. One sentence in step 5.5 is rewrapped. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 15 ++++++++------- docs/project-move-runbook.md | 9 +++++---- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index d9a3bfc..786e4ab 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -50,7 +50,7 @@ Where the address appears: | Option | Cost | Risks and limits | |---|---|---| | **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | -| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project (except #12's intake bucket, if intake is switched on before the move). One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. **If #12's intake is switched on before the move,** its bucket can't move (see "Order and timing"), so its storage cost, its data-access audit logs and its administration stay in `policyengine-api`, and anyone who administers storage there can change or delete the collector's files. Shutting that project down, or removing its billing, would then make every submission's files unreadable: Google's "Delete and restore projects" page says "Shutting down a project stops all billing and resource usage", and that Cloud Storage "might delete resources much sooner" than the 30-day recovery period. | +| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project (except #12's intake bucket, if intake is switched on before the move). One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. **If #12's intake is switched on before the move,** its bucket stays in `policyengine-api` (see "Order and timing"), so its storage cost, its data-access audit logs and its administration stay in `policyengine-api`, and anyone who administers storage there can change or delete the collector's files. Shutting that project down, or removing its billing, would then make every submission's files unreadable: Google's "Delete and restore projects" page says "Shutting down a project stops all billing and resource usage", and that Cloud Storage "might delete resources much sooner" than the 30-day recovery period. | | **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | What a dedicated project gives: its own permissions (a deployment identity @@ -208,13 +208,14 @@ The plan's constraints: `policyengine-api`: each accepted submission's row records its files' Cloud Storage generation numbers, which a copy into a new bucket doesn't keep (the runbook's step 5.3). The collector's files then depend on that - project for good, as D2 B's row says. Moving them later needs a one-off - change, not yet written, that copies the files, checks their SHA-256 values - again and rewrites each submission's recorded generation numbers. + project, as D2 B's row says, until a one-off change moves them: one that + copies the files, checks their SHA-256 values again and rewrites each + submission's recorded generation numbers. None is written. - Recommended, if D2 is B or C: switch intake on only after the move, which - creates the bucket in the new project from the start. Intake is off today: a read-only check of - `policyengine-api` on 2026-10-20 found the Cloud Tasks and Cloud Scheduler - APIs not enabled, and none of its six buckets is an intake bucket. + creates the bucket in the new project from the start. Intake is off today: + a read-only check of `policyengine-api` on 2026-10-20 found the Cloud Tasks + and Cloud Scheduler APIs not enabled, and none of its six buckets is an + intake bucket. If both D2 and D3 are adopted, there are two orders: diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 4abac32..dc37d28 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -671,8 +671,8 @@ instance), import it into the new instance as in step 6.3, and note both durations. Then delete and re-create the new database (`gcloud sql databases delete`, then `create`), so the window's import starts empty, and repeat step 5.2's "Before any import" steps (privileges, then -owners): deleting the database drops them. Staging's window gives the same timings for staging's -size. +owners): deleting the database drops them. Staging's window gives the same +timings for staging's size. ## 6. The window @@ -1230,8 +1230,9 @@ after an agreed holding period: | Dashboard settings | calibration-diagnostics GitHub `Production` environment and the Vercel project | see section 8.1 | see section 8.1 | | `PRODUCTION_COLLECTOR_URL`, `TRUSTED_COLLECTOR_ORIGINS` | emitter source (`core/constants.py`) | every emitter service module | the hostname, by an emitter release | -**Unverified before the staging window.** Nothing in this runbook has been -run against Cloud SQL or the new project. Still open: +**Unverified before the staging window.** Apart from some read-only +commands against `policyengine-api`, nothing in this runbook has been run. +Still open: - **Whether `postgres` may change a user Cloud SQL made** with `ALTER ROLE`, which PostgreSQL 16 allows only with `ADMIN OPTION` on that user. Step From 8434460f66d83ff5aa34f2770ba2e82d68de7a8d Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:48:34 +0400 Subject: [PATCH 22/31] Give the runbook the hostname-first order's first phase The memo offers a hostname-first order, but the runbook had no step that points the hostname at today's service, releases the emitter with it, and waits for builds to leave the default address before the move; section 7 assumed the hostname was "already mapped", and section 8.2 placed the emitter release after the move. Step 3.4 now does that phase against policyengine-api, says why emitter 0.2.1 decides the wait, and section 11 retires the mapping or load balancer it created there. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 6 ++--- docs/project-move-runbook.md | 37 +++++++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 5 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 786e4ab..82453cb 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -222,9 +222,9 @@ If both D2 and D3 are adopted, there are two orders: 1. **Hostname first.** Point the custom hostname at today's service in `policyengine-api`, release the emitter with the hostname as its default, and wait until builds use it (after H2a, and after Microcosm leaves the - 0.2.1 pin). Then move the project behind the hostname: only the hostname's - target changes, and the old default address can be retired, because no - build uses it. With the load balancer (D3 C), the certificate is issued in + 0.2.1 pin), as the runbook's step 3.4 describes. Then move the project + behind the hostname: only the hostname's target changes, and the old + default address can be retired, because no build uses it. With the load balancer (D3 C), the certificate is issued in the new project before the DNS change. With domain mapping (D3 B), the mapping is re-created in the new project, and requests fail TLS checks until the new certificate is issued (the emitter retries). diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index dc37d28..4da3443 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -218,6 +218,36 @@ One for staging, one for production (choice 8). From the start of a window until its verification passes, nothing merges to this repo's `main`: every merge deploys. +### 3.4 Hostname first only: put the hostname in front of today's service + +In the hostname-first order (choice 6), builds move to the hostname before the +project moves, so that none of them still uses the default address when step +6.2 puts the old service in maintenance mode. Skip this step in the "both +together" order. Before section 4: + +1. Do section 7 for each environment against today's service: run its + commands with `--project "$OLD_PROJECT_ID"` wherever they say + `--project "$NEW_PROJECT_ID"`. For D3 C, this puts the load balancer's + resources in `policyengine-api` (its Compute Engine and Certificate Manager + APIs were already enabled on 2026-10-22) until section 11 removes them. + Then run section 7's hostname checks. +2. Do section 8.2: release the emitter with the hostname as + `PRODUCTION_COLLECTOR_URL`, keeping the default address in + `TRUSTED_COLLECTOR_ORIGINS`. +3. Optionally point the dashboard's `MICROCOSM_TELEMETRY_COLLECTOR_URL` at the + hostname now (the first two rows of section 8.1's settings table). Section + 8.1 then changes only the secret settings. +4. Wait until builds use the hostname. Emitter 0.2.1 has the default address + built in, with no setting that changes it, so Microcosm has to have left + its `microcosm-emitter==0.2.1` pin; and the machines and build images that + start the separately installed emitter need the new release. Builds that + still use the default address when a window starts are handled as choice 7 + says. + +Then continue with section 4. Section 8.2 is then already done, and section 7 +is done again in the new project inside production's window, as its "Hostname +first" notes say. + ## 4. Create the project (once, no downtime) ```bash @@ -970,8 +1000,8 @@ can take up to 24 hours." - **Both together (choice 6):** nothing uses the hostname yet, so map it any time after the first deployment, and wait for the certificate before the emitter release (section 8.2). -- **Hostname first:** the hostname is already mapped to the old project's - service. Re-create the mapping in the new project with `--force-override` +- **Hostname first:** step 3.4 already mapped the hostname to the old + project's service. Re-create the mapping in the new project with `--force-override` inside production's window, after the deployment. That moves the hostname at once, and requests fail TLS checks until the new certificate is issued; the emitter retries. @@ -1197,6 +1227,9 @@ after an agreed holding period: `roles/iam.workloadIdentityUser` binding for this repo's principal set. Keep calibration-diagnostics' provider and the shared deployment account: the dashboard's deployment still uses them. +- In the hostname-first order, the domain mapping or the load balancer's + resources that step 3.4 created in `policyengine-api`, once the hostname + points at the new project. - The export bucket and every dump in it. - calibration-diagnostics' leftover collector settings, as listed in calibration-diagnostics#233's "Manual steps". From fecbf106e870746e30f399f89c105e0ac1f70a68 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:48:57 +0400 Subject: [PATCH 23/31] Create the load balancer's DNS authorization per project The runbook created the new project's Certificate Manager DNS authorization with the default fixed record name, `_acme-challenge.HOSTNAME`. In the hostname-first order the old project's authorization already holds that name, and DNS allows one CNAME per name, so the new certificate couldn't be issued before the A record moved, as the memo says it is. The command now passes `--type=PER_PROJECT_RECORD`, whose record name carries a suffix for the project, and the text says why, quoting Certificate Manager's page. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 3 ++- docs/project-move-runbook.md | 20 ++++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 82453cb..518edee 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -225,7 +225,8 @@ If both D2 and D3 are adopted, there are two orders: 0.2.1 pin), as the runbook's step 3.4 describes. Then move the project behind the hostname: only the hostname's target changes, and the old default address can be retired, because no build uses it. With the load balancer (D3 C), the certificate is issued in - the new project before the DNS change. With domain mapping (D3 B), the + the new project before the DNS change, through a per-project DNS + authorization whose record doesn't clash with the old project's. With domain mapping (D3 B), the mapping is re-created in the new project, and requests fail TLS checks until the new certificate is issued (the emitter retries). 2. **Together.** Create the new project and the hostname, move each database diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 4da3443..45b9fa2 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1011,13 +1011,24 @@ can take up to 24 hours." ```bash gcloud compute addresses create collector-address --global --project "$NEW_PROJECT_ID" gcloud certificate-manager dns-authorizations create collector-hostname \ - --domain "$COLLECTOR_HOSTNAME" --project "$NEW_PROJECT_ID" + --domain "$COLLECTOR_HOSTNAME" --type=PER_PROJECT_RECORD --project "$NEW_PROJECT_ID" gcloud certificate-manager dns-authorizations describe collector-hostname \ --project "$NEW_PROJECT_ID" --format='yaml(dnsResourceRecord)' ``` Add that CNAME at the DNS provider. It authorizes the certificate and moves no -traffic. Then: +traffic. The authorization is per project (`--type=PER_PROJECT_RECORD`), so the +CNAME's name carries a suffix for the project +(`_acme-challenge_SUFFIX.HOSTNAME`, as `describe` shows) instead of the fixed +`_acme-challenge.HOSTNAME`. In the hostname-first order, step 3.4's +authorization in the old project and this one therefore have separate records +and can both exist, so the new certificate is issued before the `A` record +changes. A fixed-name authorization in both projects would need two different +CNAME records at one name, which DNS doesn't allow. Certificate Manager's DNS +authorization page says per-project authorizations "are self-contained and +don't interact with those in other projects", and that for them it supports +only certificates issued by Google Trust Services (`pki.goog`, which the CAA +note above allows). Then: ```bash gcloud certificate-manager certificates create collector-certificate \ @@ -1046,10 +1057,7 @@ Wait until prints `ACTIVE`. Then point the hostname's `A` record at the address (`gcloud compute addresses describe collector-address --global --project "$NEW_PROJECT_ID" --format='value(address)'`). Lower the record's TTL to 300 seconds a day before, so a rollback propagates -quickly. In the hostname-first order, the old project's certificate -authorization record may still exist at the same name; Certificate Manager's -per-project DNS authorizations (`--type=PER_PROJECT_RECORD`) let both exist at -once. Check its documentation before relying on that. +quickly. ### Check the hostname From 63689892a404e669ccbc60c05c3548eeca98b9a6 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:49:22 +0400 Subject: [PATCH 24/31] Move a domain mapping by deleting it first, not with --force-override For the hostname-first order with domain mapping, the runbook moved the hostname by creating the new project's mapping with --force-override and said this "moves the hostname at once". gcloud's help describes the flag as mapping a domain "even if it is already mapped to another service", and nothing says it reaches a mapping in another project. The runbook now deletes the old project's mapping and then creates the new one, and the rollback row does the reverse; section 11 no longer lists that mapping. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 45b9fa2..af77789 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1001,10 +1001,20 @@ can take up to 24 hours." time after the first deployment, and wait for the certificate before the emitter release (section 8.2). - **Hostname first:** step 3.4 already mapped the hostname to the old - project's service. Re-create the mapping in the new project with `--force-override` - inside production's window, after the deployment. That moves the hostname - at once, and requests fail TLS checks until the new certificate is issued; - the emitter retries. + project's service. Inside production's window, after the deployment, delete + that mapping, then create the mapping in the new project with the commands + above: + + ```bash + gcloud beta run domain-mappings delete --domain "$COLLECTOR_HOSTNAME" \ + --region "$REGION" --project "$OLD_PROJECT_ID" --quiet + ``` + + The DNS records stay as they are. Requests fail until the new certificate + is issued; the emitter retries. `domain-mappings create --force-override` + maps a domain "even if it is already mapped to another service" (its + `--help`), but neither the help nor the domain mapping page says whether + that includes a service in another project, so don't rely on it. ### D3 C: load balancer @@ -1202,7 +1212,7 @@ the moment the old service entered maintenance mode. | A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | | After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | | After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to the revision step 6.1 recorded in production's window (if a release with a migration has deployed in the new project since, that revision's code is older than the imported schema: set the GitHub values back first, and let the next merge deploy the current release to the old project instead), and set the GitHub and dashboard settings back. Then reverse step 6.7: in the new project, pause #19's check schedule and disable the new alert policies (step 6.2's commands have already paused #12's queue and recovery schedule there); in the old project, resume #12's queue and recovery schedule, resume #19's check schedule, enable the old alert policies again and run the old check job once, as in the first row. Last, release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | -| The hostname | D3 B: delete the mapping, or map the hostname back to the old project's service with `--force-override`. D3 C: point the `A` record back to what it named before, if anything. | +| The hostname | D3 B: delete the new project's mapping; in the hostname-first order, then create the mapping in the old project again (section 7's commands with `--project "$OLD_PROJECT_ID"`) and wait for its certificate. D3 C: point the `A` record back to what it named before, if anything. | ## 11. Retire the old resources @@ -1235,9 +1245,9 @@ after an agreed holding period: `roles/iam.workloadIdentityUser` binding for this repo's principal set. Keep calibration-diagnostics' provider and the shared deployment account: the dashboard's deployment still uses them. -- In the hostname-first order, the domain mapping or the load balancer's - resources that step 3.4 created in `policyengine-api`, once the hostname - points at the new project. +- In the hostname-first order with D3 C, the load balancer's resources that + step 3.4 created in `policyengine-api`, once the hostname points at the new + project. (With D3 B, section 7 already deleted the old project's mapping.) - The export bucket and every dump in it. - calibration-diagnostics' leftover collector settings, as listed in calibration-diagnostics#233's "Manual steps". From 2d69316cc4ec6eb457ee39009eeec670d0b7d318 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:49:48 +0400 Subject: [PATCH 25/31] Record and retire production's unused older database URL secret A read-only check of policyengine-api on 2026-10-22 found a fifth production collector secret, `microcosm-telemetry-database-url`, made four days before the runtime database URL secret. No GitHub variable names it and no service or job mounts it, but production's runtime account may read it. Step 3.2 recorded secrets only from the deployment's settings, so section 11 would have left it in policyengine-api. Step 3.2 now lists the old project's secrets and notes unused ones, section 11 deletes them, and the memo says what the 2026-10-22 check found. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 10 ++++++---- docs/project-move-runbook.md | 12 ++++++++++-- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 518edee..e1ae124 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -19,14 +19,16 @@ end. Taken from this repo's deployment code, the GitHub environment settings (names compared, values not printed), the Vercel project settings (names only), and -calibration-diagnostics#233's read-only checks of the live resources. The live -Google Cloud resources were not re-read for this memo, except for whether -#12's intake is switched on (see "Order and timing"). +calibration-diagnostics#233's read-only checks of the live resources. On +2026-10-22 a read-only check of `policyengine-api` (`describe`, `list` and +`get-iam-policy`) found the services, migration jobs, instances, accounts, +secret names, image store and Workload Identity providers as described here, +and #12's intake still off (see "Order and timing"). | What | Where | |---|---| | Google Cloud project | `policyengine-api`, which also hosts the Public API. Both GitHub environments, `staging` and `production`, name it. | -| Per environment | a Cloud Run service, a Cloud Run migration job, a Cloud SQL for PostgreSQL instance, a runtime service account, a migration service account, and four Secret Manager secrets: the runtime database URL, the migration database URL, the session-token signing secret and the dashboard's read token | +| Per environment | a Cloud Run service, a Cloud Run migration job, a Cloud SQL for PostgreSQL instance, a runtime service account, a migration service account, and four Secret Manager secrets: the runtime database URL, the migration database URL, the session-token signing secret and the dashboard's read token. Production also keeps an older database URL secret that no deployment uses. | | Shared by both environments | one Artifact Registry Docker store and image name; the deployment service account, which calibration-diagnostics also uses; the Workload Identity providers for calibration-diagnostics and for this repo | | The dashboard's Google identity | a Workload Identity provider for Vercel's OIDC tokens, in the same project, named by its project number | | Address | Cloud Run's default address for the production service (`https://-..run.app`). The project number in it is `policyengine-api`'s. | diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index af77789..1094a89 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -180,6 +180,13 @@ For each environment (read-only): for each secret the environment uses: the four named above, and, if merged, #16's signing-key secret and S1's Hugging Face token secret. The read secret's policy names the dashboard's Vercel identity. +- Collector secrets that no deployment uses. On 2026-10-22, production also + had `microcosm-telemetry-database-url`, a database URL secret made four days + before the runtime one that the service mounts. No GitHub variable names it + and no service or job mounts it, but production's runtime account may read + it. List the old project's secrets + (`gcloud secrets list --project "$OLD_PROJECT_ID"`), note any such secret, + and don't copy it; section 11 deletes it. - What the open collector PRs added, if merged (step 3.1), each command with `--project "$OLD_PROJECT_ID"`: - #12: `gcloud tasks queues describe WORKER_QUEUE --location "$REGION"`; @@ -1224,8 +1231,9 @@ after an agreed holding period: then `gcloud sql instances patch "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --no-deletion-protection` and `gcloud sql instances delete`. - The old Cloud Run services and migration jobs, every secret step 3.2 - recorded (the four per environment, and #16's signing key and S1's token - if merged), and the old runtime, migration and worker service accounts. + recorded (the four per environment, the unused ones such as production's + older database URL secret, and #16's signing key and S1's token if merged), + and the old runtime, migration and worker service accounts. - If #12 merged: the queue (with any tasks still in it), the worker and recovery jobs, the recovery schedule and the account it used, the invoker account, and the old accounts' roles on the intake bucket, or on the From a3c0eb1f4677f31098f7963078485b66b7400517 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:50:04 +0400 Subject: [PATCH 26/31] Say which instance settings differ from the record on purpose Step 5.2 said to create each instance with the recorded settings and to "keep" deletion protection on, but on 2026-10-22 production's instance had deletion protection off, and its command always passed --backup-start-time, which turns on automated backups that staging's instance doesn't have. It now says deletion protection is on for both new instances, and to use --no-backup where the record shows backups off. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 1094a89..8875a47 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -375,8 +375,12 @@ example, an IAM condition on the instance name), repeat that here. ### 5.2 Cloud SQL -Create the instance with the settings recorded in step 3.2. Keep deletion -protection on. +Create the instance with the settings recorded in step 3.2, except deletion +protection, which is on for both new instances: on 2026-10-22 staging's +instance had it, but production's didn't. Where the record shows automated +backups off, as staging's did on 2026-10-22, use `--no-backup` instead of +`--backup-start-time`; only production's deployment takes a backup +(`create-database-backup`). ```bash gcloud sql instances create "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --region "$REGION" \ From 7b5a9438ac3529d0fef430e3ae798c6debacf339 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:50:15 +0400 Subject: [PATCH 27/31] List organization policies as unverified before the staging window The new project inherits its parent's organization policies. A policy such as domain-restricted sharing would refuse the allUsers invoker binding that --allow-unauthenticated adds, and the deployment's maintenance check would then fail. Whether one applies couldn't be read: the Organization Policy API isn't enabled in policyengine-api. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 8875a47..c7be203 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1310,6 +1310,13 @@ Still open: to the old intake bucket (step 5.3), and the old project's image reads from the new Artifact Registry store (step 4.1). Checking needs organization-level access. +- **Organization policies:** the new project inherits its parent's. + `policyengine-api` belongs to the same organization and lets `allUsers` + invoke the collector, which `--allow-unauthenticated` needs, but whether + the chosen parent or `policyengine-api` itself sets a different policy, + such as domain-restricted sharing, wasn't checked: the Organization Policy + API isn't enabled in `policyengine-api`. If the new project refuses + `allUsers`, step 6.6's deployment can't reach its maintenance revision. - **Project liens:** whether `policyengine-api` has a lien that stops it being deleted, which matters if the intake bucket stays there. Listing liens needs `gcloud alpha resource-manager liens list`. From 5806a65ff9f162a1221776e3e126d820b2e888a7 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:51:38 +0400 Subject: [PATCH 28/31] Rewrap the memo's order paragraph and state the invoker check plainly Second review pass over this round's fixes: the memo's hostname-first paragraph had overlong lines and still said the D3 B mapping is "re-created" rather than deleted and created again; the organization policy note implied the new project shares policyengine-api's organization, which choice 2 decides, and now names the checked invoker binding. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 12 +++++++----- docs/project-move-runbook.md | 15 ++++++++------- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index e1ae124..9a5b2f2 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -226,11 +226,13 @@ If both D2 and D3 are adopted, there are two orders: and wait until builds use it (after H2a, and after Microcosm leaves the 0.2.1 pin), as the runbook's step 3.4 describes. Then move the project behind the hostname: only the hostname's target changes, and the old - default address can be retired, because no build uses it. With the load balancer (D3 C), the certificate is issued in - the new project before the DNS change, through a per-project DNS - authorization whose record doesn't clash with the old project's. With domain mapping (D3 B), the - mapping is re-created in the new project, and requests fail TLS checks - until the new certificate is issued (the emitter retries). + default address can be retired, because no build uses it. With the load + balancer (D3 C), the certificate is issued in the new project before the + DNS change, through a per-project DNS authorization whose record doesn't + clash with the old project's. With domain mapping (D3 B), the old project's + mapping is deleted and the mapping created again in the new project, and + requests fail TLS checks until the new certificate is issued (the emitter + retries). 2. **Together.** Create the new project and the hostname, move each database in one window, then release the emitter. Builds keep the old address until they start a newer emitter, and builds pinned to 0.2.1 keep it until diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index c7be203..329329b 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1310,13 +1310,14 @@ Still open: to the old intake bucket (step 5.3), and the old project's image reads from the new Artifact Registry store (step 4.1). Checking needs organization-level access. -- **Organization policies:** the new project inherits its parent's. - `policyengine-api` belongs to the same organization and lets `allUsers` - invoke the collector, which `--allow-unauthenticated` needs, but whether - the chosen parent or `policyengine-api` itself sets a different policy, - such as domain-restricted sharing, wasn't checked: the Organization Policy - API isn't enabled in `policyengine-api`. If the new project refuses - `allUsers`, step 6.6's deployment can't reach its maintenance revision. +- **Organization policies:** the new project inherits the policies of the + parent chosen in choice 2. In `policyengine-api`, which sits directly under + an organization, both collector services grant `roles/run.invoker` to + `allUsers`, as `--allow-unauthenticated` needs. Whether the chosen parent + sets a different policy, such as domain-restricted sharing, wasn't checked: + the Organization Policy API isn't enabled in `policyengine-api`. If the new + project refuses `allUsers`, step 6.6's deployment can't reach its + maintenance revision. - **Project liens:** whether `policyengine-api` has a lien that stops it being deleted, which matters if the intake bucket stays there. Listing liens needs `gcloud alpha resource-manager liens list`. From 8e10996b758c87afb508a41cdaf1ae9fdbf58fa6 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 27 Oct 2026 18:43:53 +0400 Subject: [PATCH 29/31] Record the project and hostname decisions of 2026-10-27 The memo now records Anthony's decisions: a dedicated project in the same organization and billing account (D2 B), a global external load balancer with one hostname per environment (D3 C), DNS records added by hand outside Google Cloud, hostname first with a cross-project backend service until each move, no scheduled windows, staging copied as the rehearsal, the dashboard's identity kept in the current project, and microcosm-runs left until O7 and O8. Plan codes follow the 2026-10-27 renumbering (C2 to C8). Concrete values, such as project IDs, domains and database user names, are replaced by named placeholders. Co-Authored-By: Claude Opus 5.5 --- .../project-and-domain-decision.added.md | 2 +- docs/project-and-domain-decision.md | 333 ++++++++++-------- 2 files changed, 181 insertions(+), 154 deletions(-) diff --git a/changelog.d/project-and-domain-decision.added.md b/changelog.d/project-and-domain-decision.added.md index 38fe1c4..ea56409 100644 --- a/changelog.d/project-and-domain-decision.added.md +++ b/changelog.d/project-and-domain-decision.added.md @@ -1 +1 @@ -Document the choices for moving the collector to its own Google Cloud project and a custom hostname (plan decisions D2 and D3) in `docs/project-and-domain-decision.md`, and the steps for the move, including the Postgres dump and restore and rollback, in `docs/project-move-runbook.md`. +Record the decisions on the collector's Google Cloud project and hostname (plan decisions D2 and D3: a dedicated project, behind a load balancer with one hostname per environment) in `docs/project-and-domain-decision.md`, and the steps for the hostname and then the move, including the Postgres dump and restore and rollback, in `docs/project-move-runbook.md`. diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 9a5b2f2..2db1769 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -1,8 +1,10 @@ # The collector's Google Cloud project and address: decision memo -Status: **for decision.** Decisions D2 and D3 of the collector plan are not -made, and nothing described here has been created. Written 2026-10-20. The -steps that carry out the decisions are in +Status: **decided.** Anthony made decisions D2 and D3 of the collector plan, +and the choices that follow from them, on 2026-10-27. Nothing described here +has been created yet. The memo was written on 2026-10-20 to set out the +options; they are kept below as the record of what was weighed. The steps +that carry out the decisions are in [`project-move-runbook.md`](project-move-runbook.md). - **D2:** does the collector stay in the Google Cloud project it shares with @@ -10,28 +12,97 @@ steps that carry out the decisions are in - **D3:** do builds and consumers keep using Cloud Run's default address for the collector, or a hostname of our own? -The plan recommends both a dedicated project and a custom hostname, and pairs -them: moving the project changes the default address, so without D3 every -consumer changes twice. The [choices](#the-choices-to-make) are listed at the -end. +This memo and the runbook contain no hostnames, domain names, IP addresses, +project IDs or numbers, account IDs, instance, database or user names, +connection names, service-account emails or secret values. Where one is +needed they use a named placeholder, such as `` or +``, and say where the value is kept. + +## The decisions + +Numbered as the choices were offered (the last section of the 2026-10-20 +version of this memo). + +| Choice | Decision | +|---|---| +| 1. D2, the project | **B: a dedicated project,** holding both environments, each moved in one database copy. | +| 2. The project's details | It uses the **same organization and billing account as the current project.** It has **one owner.** Its ID (``) is chosen when it is created and recorded by the operator; after each environment's copy it is also the value of that environment's `TELEMETRY_GCP_PROJECT_ID` variable. | +| 3. D3, the address | **C: a global external Application Load Balancer,** with a Google-managed certificate from Certificate Manager. | +| 4. Hostnames | **One load balancer and one hostname per environment,** `` and ``, both under a PolicyEngine-owned domain (``). | +| 5. DNS | The domain's DNS is **managed outside Google Cloud and edited by hand by Anthony.** Each environment needs two records: an `A` record for its load balancer's static IP, and the `CNAME` for its Certificate Manager DNS authorization. The runbook lists them (step 5.3). | +| 6. Order | **Hostname first.** Each environment's load balancer is built in the new project from the start. Until that environment moves, it sends requests across projects to today's service, through a backend service and a serverless network endpoint group left in the current project. Clients switch to the hostname; each environment's move is then one change to its load balancer's URL map. | +| 7. The old address | **Maintenance mode after production's copy.** The old production service is deleted once no build uses emitter 0.2.1 and it has had no requests for seven days. | +| 8. Windows | **No scheduled maintenance windows.** Each environment's database copy still puts that environment's service in maintenance mode while it runs. | +| 9. Staging data | **Copied with a dump and restore,** as the rehearsal for production's copy. | +| 10. The dashboard's Google identity | **It stays in the current project,** and is granted read access to the new project's read secrets. That access is removed when the dashboard switches to its Vercel OIDC token (C8). | +| 11. microcosm-runs | **Its address is left unchanged** until the emitter's direct upload (O7) and microcosm-runs' upload API (O8) are removed. | + +### What follows from them + +The plan's items for this work, as renumbered on 2026-10-27: + +1. **C2, the hostname:** each environment's load balancer, certificate and + DNS records, in front of today's service. The dashboard then reads the + collector through the hostname. +2. **C3, the emitter's default:** an emitter release whose fixed set of + trusted origins (PolicyEngine/microcosm-emitter#29) includes the + production hostname, which is also its default. +3. **C4, builds run that release,** including the Modal image. Microcosm has + to leave its `microcosm-emitter==0.2.1` pin first (H2a). +4. **C5, the move:** prepare the new project, then copy staging, then + production. Production's copy waits for C4. +5. **After the move, in this order:** the collector's own Hugging Face token + and its daily check (S1, S2); switching artifact intake on; read API v2 + (C6); ES256 session-token signing (C7). Each creates its cloud resources + once, in the new project. + +### Why + +- **The same organization** is what lets the new project's load balancer use + a backend service in the current project. Google's external Application + Load Balancer overview says the frontend and URL map "can reference backend + services or backend buckets from any project within the same + organization". +- **A serverless network endpoint group can't point at a Cloud Run service in + another project** ("the serverless NEG must be created in the same project + as the backing Cloud Run" service, in Google's serverless NEG overview). So + the endpoint group and backend service for today's service stay in the + current project, and only the frontend is in the new project. +- **The load balancer's certificate is issued before any client uses the + hostname,** through the DNS authorization, so there is never a period + without a valid certificate. Because the static IP is in the new project + from the start, **the DNS records never change during the move**: each + move, and each rollback, is one URL map change, with no DNS caching delay. +- **Hostname first** means builds use the hostname before production's copy, + so production's old service can stay in maintenance mode afterwards + instead of serving the new database across projects. The cost is the wait + for C4. +- **Keeping the dashboard's identity** avoids building a provider in the new + project that C8 would then remove, and avoids changing + `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`, one Vercel setting that + Production and Preview share. +- **Switching intake on after the move** creates its bucket in the new + project from the start. Moving a bucket later would need a one-off change, + because each accepted submission records its files' Cloud Storage + generation numbers, and a copy into another bucket gets new ones. ## Today Taken from this repo's deployment code, the GitHub environment settings (names compared, values not printed), the Vercel project settings (names only), and calibration-diagnostics#233's read-only checks of the live resources. On -2026-10-22 a read-only check of `policyengine-api` (`describe`, `list` and +2026-10-22 a read-only check of the current project (`describe`, `list` and `get-iam-policy`) found the services, migration jobs, instances, accounts, -secret names, image store and Workload Identity providers as described here, -and #12's intake still off (see "Order and timing"). +secret names, image store and Workload Identity providers as described here. | What | Where | |---|---| -| Google Cloud project | `policyengine-api`, which also hosts the Public API. Both GitHub environments, `staging` and `production`, name it. | -| Per environment | a Cloud Run service, a Cloud Run migration job, a Cloud SQL for PostgreSQL instance, a runtime service account, a migration service account, and four Secret Manager secrets: the runtime database URL, the migration database URL, the session-token signing secret and the dashboard's read token. Production also keeps an older database URL secret that no deployment uses. | +| Google Cloud project | The current project, ``, which also hosts the Public API. Both GitHub environments, `staging` and `production`, name it in `TELEMETRY_GCP_PROJECT_ID`. | +| Per environment | a Cloud Run service, a Cloud Run migration job, a Cloud SQL for PostgreSQL instance, a runtime service account, a migration service account, and four Secret Manager secrets: the runtime database URL, the migration database URL, the session-token signing secret and the dashboard's read token. Production also keeps an older database URL secret that no deployment uses; its only version was disabled on 2026-10-27. | +| Database users | the migration user, which owns the tables, and the runtime user, which the service connects as, besides Cloud SQL's built-in admin user. Production's unused third user was dropped on 2026-10-27. | | Shared by both environments | one Artifact Registry Docker store and image name; the deployment service account, which calibration-diagnostics also uses; the Workload Identity providers for calibration-diagnostics and for this repo | | The dashboard's Google identity | a Workload Identity provider for Vercel's OIDC tokens, in the same project, named by its project number | -| Address | Cloud Run's default address for the production service (`https://-..run.app`). The project number in it is `policyengine-api`'s. | +| Address | Cloud Run's default address for the production service, which contains the service name, the project number and the region. The project number in it is the current project's. | Where the address appears: @@ -51,57 +122,57 @@ Where the address appears: | Option | Cost | Risks and limits | |---|---|---| -| **A. Stay in `policyengine-api`** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C5's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | -| **B. A dedicated project, moved in one maintenance window per environment** (the plan's recommendation) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project (except #12's intake bucket, if intake is switched on before the move). One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed, and the dashboard's reads stay down until its own settings change is deployed. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 6 gives the measurements). The staging rehearsal measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; up to three Vercel settings; and the dashboard's read-secret check. **The default address changes,** so without D3 the emitter needs a release and builds on older emitters lose delivery. **If #12's intake is switched on before the move,** its bucket stays in `policyengine-api` (see "Order and timing"), so its storage cost, its data-access audit logs and its administration stay in `policyengine-api`, and anyone who administers storage there can change or delete the collector's files. Shutting that project down, or removing its billing, would then make every submission's files unreadable: Google's "Delete and restore projects" page says "Shutting down a project stops all billing and resource usage", and that Cloud Storage "might delete resources much sooner" than the 30-day recovery period. | -| **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two windows. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first window but adds a second one, and leaves the collector split over two projects in between. Worth it only if one window of B is too long. | +| **A. Stay in the current project** | Nothing new. | The collector's deployment identity is the service account calibration-diagnostics also uses, in the project that runs the Public API, so a mistake in the collector's permissions, quotas or deployments can reach the Public API, and the other way round. Billing and audit logs stay mixed. Everything the collector adds next goes into that project too: S1's Hugging Face read token (#13), C7's signing key, and #12's intake bucket, Cloud Tasks queue and worker jobs. The address doesn't change, so D3 becomes optional. | +| **B. A dedicated project, moved in one database copy per environment** (**chosen**) | Recurring cost stays about the same: the same Cloud Run, Cloud SQL, Secret Manager and image storage, billed to the new project, plus the load balancers (D3 C). One-off: until the old resources are deleted, both projects run a Cloud SQL instance for each environment, and the old Cloud Run services each keep one instance running (`--min 1`). A small export bucket. People's time. | **Downtime:** telemetry ingestion and the dashboard's collector reads pause while each database is exported and imported and the service is redeployed. With the hostname first, the dashboard's reads recover at the URL map change, with no dashboard deployment. The export and import take minutes; the redeploy is a full release (two Push runs, then staging's and production's deployments), whose length depends on the queue for GitHub's runners. In all, about 40 minutes on a quiet day, 2 to 3 hours when runners are queued, as on 2026-10-19 and 2026-10-20 (the runbook's section 8 gives the measurements). Staging's copy measures it. **The database move:** the database roles and their privileges were provisioned by scripts kept outside this repo, so they have to be read from the old instance and recreated by hand before the import, and the migration user's default privileges set again. **Every project-specific setting changes:** in each GitHub environment, five of the 15 variables (the project, both Cloud SQL connections and both service accounts) and both secrets, while the other ten can keep their values if the new resources reuse the old names; later, one Vercel setting and the dashboard's read-secret check. | +| **C. A dedicated project, with the database moved later** (Cloud Run in the new project connects to Cloud SQL in the old one for a while) | As B, spread over two copies. | Needs deployment-script changes first: `create-database-backup` refuses a Cloud SQL connection whose project isn't `PROJECT_ID`. Needs cross-project grants for Cloud SQL and Secret Manager. It shortens the first copy but adds a second one, and leaves the collector split over two projects in between. Worth it only if one copy of B is too long. | What a dedicated project gives: its own permissions (a deployment identity that can reach only the collector), its own quotas and billing, audit logs that contain only the collector, and room for the intake and worker resources -without adding them to the Public API's project. For intake's bucket this holds -only if intake is switched on after the move; otherwise the bucket and the -collector's stored files stay in the Public API's project. +without adding them to the Public API's project. Intake is switched on only +after the move, so its bucket is in the new project from the start. ## D3: which address | Option | Cost | Risks and limits | |---|---|---| | **A. Keep Cloud Run's default address** | Nothing. | The address contains the project number, so D2 B or C changes it. Every change needs an emitter release, and builds on older emitters lose delivery until they move. | -| **B. Cloud Run domain mapping** | No charge. | Google lists it as Preview: "not production-ready", and "not recommended for production services" because of latency. TLS 1.0 and 1.1 can't be switched off, and only Google's certificate can be used. The certificate is issued after the mapping exists and DNS points at it: "usually about 15 minutes but can take up to 24 hours." A hostname has one active mapping, so moving the project means re-creating the mapping there and waiting for a new certificate. `us-central1` supports mappings. | -| **C. Global external Application Load Balancer** with a serverless network endpoint group and a Certificate Manager certificate | About $0.025 an hour for the forwarding rule (about $18 a month) plus about $0.008 per GB processed; Certificate Manager's first 100 certificates are free. Check Google's current price lists. | Generally available, and Google's recommended option. With DNS authorization the certificate is issued before any traffic moves, so a later project move is a DNS change with no certificate gap. More resources to keep: a static IP, a network endpoint group, a backend service, a URL map, an HTTPS proxy, a forwarding rule and a certificate map. Cloud Armor can be added later. | +| **B. Cloud Run domain mapping** | No charge. | Google lists it as Preview: "not production-ready", and "not recommended for production services" because of latency. TLS 1.0 and 1.1 can't be switched off, and only Google's certificate can be used. The certificate is issued after the mapping exists and DNS points at it: "usually about 15 minutes but can take up to 24 hours." A hostname has one active mapping, so moving the project means re-creating the mapping there and waiting for a new certificate. | +| **C. Global external Application Load Balancer** with a serverless network endpoint group and a Certificate Manager certificate (**chosen**) | About $0.025 an hour for up to five forwarding rules in a project (about $18 a month), so a second load balancer in the same project adds no forwarding-rule charge, plus about $0.008 per GB processed; Certificate Manager's first 100 certificates are free. Check Google's current price lists. | Generally available, and Google's recommended option. With DNS authorization the certificate is issued before any traffic moves. More resources to keep: a static IP, a network endpoint group, a backend service, a URL map, an HTTPS proxy, a forwarding rule and a certificate map per environment. Cloud Armor and an SSL policy (for example TLS 1.2 as the minimum version) can be added later. | | **D. Route through Vercel or Cloudflare** | Varies. | Not recommended: another provider would end TLS for requests that carry builders' Hugging Face tokens. | -Which DNS zone (from public name-server records, 2026-10-20): +DNS, as decided: -- **`microcosm.institute`** is served by Cloudflare's name servers; its apex - points at Vercel, which serves the dashboard. A collector record there must - be "DNS only", not proxied, so that Cloudflare doesn't end TLS and doesn't - interfere with Google's certificate checks. -- **`policyengine.org`** is served by Google Cloud DNS name servers. Which - project holds that zone wasn't checked. -- Neither zone has CAA records today. If one is added, it must allow - `pki.goog` and `letsencrypt.org`. -- Domain mapping (B) needs the base domain verified in Google Search Console - by the account that creates the mapping. The load balancer's certificate (C) - doesn't: Certificate Manager's DNS authorization needs only a `CNAME` record - in the zone. +- **The records are plain DNS records** at the domain's provider, outside + Google Cloud, with no proxying or forwarding in front of them, so that no + other provider ends TLS and Google can check the authorization record. +- **The authorization `CNAME` stays for good.** Certificate Manager uses it + for renewals as well as the first issue. +- **CAA records:** the domain had none when checked. The runbook uses + per-project DNS authorizations, for which Certificate Manager issues only + Google Trust Services certificates, so any CAA record added later must allow + Google Trust Services. A CAA record at the domain's apex applies to every + hostname under it. +- Domain mapping (B) would have needed the domain verified in Google Search + Console. The load balancer's certificate doesn't: the DNS authorization + needs only the `CNAME` record. -Whatever is chosen, three things stay as they are: +Three things stay as they are: - **Cloud Run's default addresses stay on** (`--default-url` in `scripts/deploy.sh`). The deployment checks the maintenance and candidate revisions through their tagged default addresses, and builds on older emitters use the production default address. -- **Ingress stays open to the internet.** Build machines and Modal containers - reach the collector directly. Allowing only load-balancer traffic would - break the tagged-address checks and older emitters. +- **Ingress stays open to the internet.** Allowing only load-balancer traffic + would break the tagged-address checks and older emitters. - **The old address can't redirect to the new one.** The emitter doesn't follow redirects on requests that carry tokens (`_NoRedirectHandler` in - `core/auth.py`). After a move, the old service either keeps serving or stops. + `core/auth.py`). After a move, the old service either keeps serving or stops; + choice 7 keeps it in maintenance mode. ## What changes for each consumer -### The emitter +### The emitter (C3, C4) - **Where:** `PRODUCTION_COLLECTOR_URL`, and after PolicyEngine/microcosm-emitter#29 also `TRUSTED_COLLECTOR_ORIGINS`, both in @@ -110,17 +181,20 @@ Whatever is chosen, three things stay as they are: module, `huggingface/service/module.py` (added by #23), send submissions to the collector; it chooses the origin with its own copy of the old expression and should use the same resolver. -- **The change:** one emitter PR adds the new origin to - `TRUSTED_COLLECTOR_ORIGINS` and points `PRODUCTION_COLLECTOR_URL` at it. - Merging it publishes a release to PyPI. -- **Who picks it up:** +- **The change (C3):** one emitter PR adds `https://` to + `TRUSTED_COLLECTOR_ORIGINS` and points `PRODUCTION_COLLECTOR_URL` at it, + keeping the old default address in the trusted set until the old service is + deleted. Merging it publishes a release to PyPI. It needs only C2: the + hostname already reaches today's service. +- **Who picks it up (C4):** - Builds that start the separately installed emitter (emitter #21 and #22, and Microcosm's H2a) use the new release from their next start once it is - installed on that machine or in that build image. They need no Microcosm - change. A running build keeps the emitter it started with. + installed on that machine or in that build image, the Modal image + included. A running build keeps the emitter it started with. - Builds pinned to `microcosm-emitter==0.2.1` (Microcosm's `packages/microcosm-build/pyproject.toml` today) keep the old address - until Microcosm moves off that pin. + until Microcosm moves off that pin. 0.2.1 has the address built in, with + no setting that changes it. - **If the old address stops answering,** a build that still uses it keeps its telemetry on its own machine: the emitter retries with a backoff of up to 60 seconds and keeps undelivered events for seven days (up to 100 MiB), @@ -130,30 +204,29 @@ Whatever is chosen, three things stay as they are: ### The calibration dashboard (calibration-diagnostics) -| Setting | Provider | Read by | Changes under D2 B | Changes under D3 B or C | -|---|---|---|---|---| -| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | GitHub `Production` environment variable; also set in the Vercel project for Production and Preview | `deploy.yml` passes the GitHub value to `.github/scripts/deploy-dashboard.sh`, which requires it and sets it on the production deployment with `vercel deploy --env`. Preview deployments use the Vercel value. The server reads it in `frontend/lib/server/microcosm/telemetry-collector-client.ts`. | the new default address (if D3 is A) | the custom hostname, in all three places (GitHub Production, Vercel Production, and Vercel Preview with the staging address) | -| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel project setting, Production and Preview | `telemetry-collector-client.ts`: the full Secret Manager version resource, which names the project | the new project's read secrets | no | -| `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` | Vercel project setting, one entry for Production and Preview | `frontend/lib/server/secret-manager.ts`, on Vercel: the Workload Identity provider resource, named by project number | only if the provider moves (see choice 10) | no | -| `TELEMETRY_GCP_PROJECT_ID`, `TELEMETRY_READ_SECRET_NAME` | GitHub `Production` environment variables | `deploy.yml` step "Check collector read secret exists" | the new project | no | -| `TELEMETRY_GCP_WIF_PROVIDER`, `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | GitHub `Production` environment secrets | the same step's Google authentication | no, if the dashboard's own identity in `policyengine-api` is granted metadata read on the new read secret | no | +| Setting | Provider | Read by | Changes | +|---|---|---|---| +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | GitHub `Production` environment variable; also set in the Vercel project for Production and Preview | `deploy.yml` passes the GitHub value to `.github/scripts/deploy-dashboard.sh`, which requires it and sets it on the production deployment with `vercel deploy --env`. Preview deployments use the Vercel value. The server reads it in `frontend/lib/server/microcosm/telemetry-collector-client.ts`. | In C2, to `https://` (GitHub `Production` and Vercel Production) and `https://` (Vercel Preview). Not again. | +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel project setting, Production and Preview | `telemetry-collector-client.ts`: the full Secret Manager version resource, which names the project | After each environment's copy, in an ordinary dashboard deployment, to the new project's read secret. The old project's copy holds the same value, so the dashboard keeps working until then. | +| `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` | Vercel project setting, one entry for Production and Preview | `frontend/lib/server/secret-manager.ts`, on Vercel: the Workload Identity provider resource, named by project number | No: the provider stays in the current project (choice 10). | +| `TELEMETRY_GCP_PROJECT_ID`, `TELEMETRY_READ_SECRET_NAME` | GitHub `Production` environment variables | `deploy.yml` step "Check collector read secret exists" | With `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` for production: the new project, and the secret name if it changed. | +| `TELEMETRY_GCP_WIF_PROVIDER`, `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | GitHub `Production` environment secrets | the same step's Google authentication | No: the dashboard's deployment identity in the current project is granted metadata read on the new read secret. | Two things found while checking: - **Preview reads staging's collector.** The identity-provider setting covers Production and Preview, the provider accepts both environments' Vercel identities, and staging's read secret grants the Preview identity access. So - under choice 10 both identities need access to the new project's read - secrets, each to its own environment's. -- **After C6,** the dashboard authenticates to the collector with its own + both identities get access to the new project's read secrets, each to its + own environment's. +- **After C8,** the dashboard authenticates to the collector with its own Vercel OIDC token instead of a shared read token, so it no longer reads - Secret Manager for the collector. If C6 lands before the move, only - `MICROCOSM_TELEMETRY_COLLECTOR_URL` changes. + Secret Manager for the collector, and the grants above are removed. Settings take effect on the next deployment: a merge to calibration-diagnostics `main`, or a re-run of its latest "Deploy production" run. Whether a re-run reads a changed GitHub variable such as `MICROCOSM_TELEMETRY_COLLECTOR_URL` -hasn't been checked (the runbook's step 6.6 has the same question), so a merge +hasn't been checked (the runbook's step 8.7 has the same question), so a merge is the reliable way. ### microcosm-runs @@ -165,31 +238,30 @@ is the reliable way. the tracked default. - **What uses it:** only the upload API's authorization call, `POST /v1/auth/graph-publication/authorize`. The collector doesn't implement - that endpoint (calibration-diagnostics#208 was closed), and O7 removes the - upload API. -- **So:** the move breaks nothing there. Either update the constant and - `.env.example` in a small PR when the emitter's default changes, or leave - them for O7 to delete (choice 11). If O6 later reads collector outputs - through its API, it should use the custom hostname from the start. + that endpoint (calibration-diagnostics#208 was closed). +- **So:** the move breaks nothing there, even after the old service is + deleted. The constant and `.env.example` stay as they are until O7 removes + the emitter's direct upload and O8 removes microcosm-runs' upload API + (choice 11). If O6 later reads collector outputs through the collector's + API, it uses the hostname from the start. ### Microcosm -No change. It doesn't name the address, and builds don't set -`collector_origin`. Its emitter pin decides how long builds keep the old -address. +It doesn't name the address, and builds don't set `collector_origin`. Its +emitter pin decides how long builds keep the old address, so C4 needs H2a, +which runs the separately installed emitter, including in the Modal image. ### The collector itself - **GitHub `staging` and `production` environments:** every project-specific - value changes. The runbook lists them. -- **Session tokens:** their issuer and audience are names - (`TELEMETRY_JWT_ISSUER` and `TELEMETRY_JWT_AUDIENCE` default to - `policyengine-telemetry` and `microcosm-telemetry-collector`), not - addresses, so they don't change. C5 publishes a JWKS; whoever fetches it - will use the collector's address, which is one more reason to fix the - address before consumers depend on it. + value changes in each environment's copy. The runbook lists them. +- **Session tokens:** their issuer and audience are fixed names + (`TELEMETRY_JWT_ISSUER` and `TELEMETRY_JWT_AUDIENCE`), not addresses, so + they don't change. C7 publishes a JWKS; whoever fetches it will use the + hostname. - **The README** says the production address isn't repeated in - documentation. A custom hostname is stable, so it could be documented. + documentation. A runbook PR (its section 16) updates the README for the new + project, without naming the hostname. ## Order and timing @@ -197,79 +269,34 @@ The plan's constraints: - The move happens only after the collector deploy switch-over (C1) is verified. From then on, every merge to this repo's `main` deploys - production, so nothing else merges during a window. -- calibration-diagnostics' `telemetry-service.yml` can still redeploy the - collector for rollback until calibration-diagnostics#234 merges. Its - settings name the old project, so merge #234 before the move, or never - dispatch that workflow afterwards. -- The new project has to include whatever has merged by then: #12's intake - resources (including its recovery job and schedule), S1's secret and its - mount, C5's signing key, and S2's daily check job, schedule and alert - policies (#19). -- If #12's intake is switched on before the move, its bucket stays in - `policyengine-api`: each accepted submission's row records its files' - Cloud Storage generation numbers, which a copy into a new bucket doesn't - keep (the runbook's step 5.3). The collector's files then depend on that - project, as D2 B's row says, until a one-off change moves them: one that - copies the files, checks their SHA-256 values again and rewrites each - submission's recorded generation numbers. None is written. -- Recommended, if D2 is B or C: switch intake on only after the move, which - creates the bucket in the new project from the start. Intake is off today: - a read-only check of `policyengine-api` on 2026-10-20 found the Cloud Tasks - and Cloud Scheduler APIs not enabled, and none of its six buckets is an - intake bucket. - -If both D2 and D3 are adopted, there are two orders: - -1. **Hostname first.** Point the custom hostname at today's service in - `policyengine-api`, release the emitter with the hostname as its default, - and wait until builds use it (after H2a, and after Microcosm leaves the - 0.2.1 pin), as the runbook's step 3.4 describes. Then move the project - behind the hostname: only the hostname's target changes, and the old - default address can be retired, because no build uses it. With the load - balancer (D3 C), the certificate is issued in the new project before the - DNS change, through a per-project DNS authorization whose record doesn't - clash with the old project's. With domain mapping (D3 B), the old project's - mapping is deleted and the mapping created again in the new project, and - requests fail TLS checks until the new certificate is issued (the emitter - retries). -2. **Together.** Create the new project and the hostname, move each database - in one window, then release the emitter. Builds keep the old address until - they start a newer emitter, and builds pinned to 0.2.1 keep it until - Microcosm moves. Meanwhile the old service either keeps serving, connected - to the new database across projects, or stays in maintenance mode, and - those builds' telemetry stays on their machines (choice 7). - -## The choices to make - -1. **D2:** A (stay), B (a dedicated project, one window per environment) or C - (a dedicated project, database later). The plan recommends B. -2. **If B or C:** the project ID and display name; its parent organization or - folder; the billing account; who holds the owner and IAM-admin roles. -3. **D3:** A (default address), B (Cloud Run domain mapping) or C (load - balancer). The plan recommends a custom domain without naming a mechanism. -4. **Hostnames:** the production hostname, for example - `collector.microcosm.institute`; and whether staging gets one, for example - `collector-staging.microcosm.institute`, or keeps its default address. -5. **DNS:** which zone (`microcosm.institute` on Cloudflare, or - `policyengine.org` on Cloud DNS); who adds the records; for D3 B, which - Google account verifies the base domain in Search Console. -6. **Order:** hostname first, or both together. -7. **The old address after the move:** keep the old service serving, - connected to the new database, until no build uses emitter 0.2.1; or leave - it in maintenance mode, so builds still on it keep their telemetry locally; - or delete it. A redirect isn't possible. -8. **The window:** when, and how long is acceptable. During it, running builds - keep their events locally and deliver them afterwards, except on a machine - discarded when its build ends, such as a Modal container, which loses the - events not sent by then. The build monitor shows only Hugging Face runs. -9. **Staging data:** dump and restore it like production (which also rehearses - the production move), or start staging empty. -10. **The dashboard's Google identity:** create a new Vercel Workload Identity - provider in the new project (changes - `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`), or keep it in - `policyengine-api` and grant its identities access to the new secrets - across projects (no Vercel change, but a lasting dependency on the shared - project), or finish C6 first so the dashboard needs neither. -11. **microcosm-runs:** update its default address with the emitter release, - or leave it for O7. + production, so nothing else merges while a copy runs. +- calibration-diagnostics' `telemetry-service.yml` must never be run after the + move: its settings name the old project, and its code predates migration + `0002_artifact_intake`, so it can't run against either database. + calibration-diagnostics#234 removes it. +- #12 merged with intake off: migration `0002_artifact_intake` has added the + two intake tables, which stay empty, and nothing else of intake exists in + either project. Its bucket, queue, jobs and schedule are created in the new + project after the move. +- The migration user's default privileges give the runtime user access to + every table a migration creates; the migrations grant nothing (#12). Both + environments have them today. The copy recreates the database users, so the + runbook sets them again on each new instance before any migration runs + there. + +In order: + +1. **C2:** each environment's load balancer in the new project, in front of + today's service; the DNS records; the dashboard reads through the + hostname. No downtime. +2. **C3 and C4:** the emitter release with the hostname as its default, and + builds running it. +3. **C5:** the new project's resources for each environment (no downtime); + then staging's copy; then, once C4 holds, production's copy. Each copy puts + that environment's service in maintenance mode, copies the database, + deploys into the new project, and changes the URL map. Nothing is + scheduled or announced in advance. +4. **After the move:** S1 and S2, switching intake on, C6, then C7, each + created once in the new project. +5. **Retiring the old resources:** once no build uses emitter 0.2.1 and the + old production service has had no requests for seven days. From c4ca2ff1d73443bc11ce1a7be7d0809983474d96 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 27 Oct 2026 18:43:53 +0400 Subject: [PATCH 30/31] Rewrite the runbook for hostname first and a URL map move C2 builds each environment's load balancer in the new project, sending requests to today's service through a backend service left in the current project, and lists the four DNS records to add. C5 copies each database with the service in maintenance mode, sets the migration user's default privileges before any migration runs, deploys into the new project and changes the URL map. Rollback is the reverse URL map change. Intake, S1, S2, C6 and C7 are created once in the new project after the move. The third database user, domain mapping and the "both together" order are removed, and no concrete values remain. Co-Authored-By: Claude Opus 5.5 --- docs/project-move-runbook.md | 1637 ++++++++++++++++------------------ 1 file changed, 784 insertions(+), 853 deletions(-) diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index 329329b..f15fe13 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -1,47 +1,61 @@ -# Moving the collector to its own project and hostname (runbook) - -This is plan item C2. Use it only when both of these hold: - -- **The collector deploy switch-over (C1) is verified:** this repo deploys - staging and production, and every check in the "Verify the deployment from - microcosm-collector" section of calibration-diagnostics#233 has passed. -- **Decisions D2 and D3 are made.** See - [`project-and-domain-decision.md`](project-and-domain-decision.md); the - numbered choices there are referred to below as "choice N". - -The runbook assumes D2 option B (a dedicated project, one maintenance window -per environment) and a custom hostname (D3 B, Cloud Run domain mapping, or D3 -C, a load balancer). Steps that depend on a choice say so. It hasn't been -rehearsed. Moving staging first is the rehearsal, so check each command against -the decided names there before running it for production. - -No command here prints a secret value. Values are generated in the shell and -piped into Secret Manager, or piped from one Secret Manager secret into -another. +# The collector's hostname and project move (runbook) + +This runbook carries out the decisions recorded in +[`project-and-domain-decision.md`](project-and-domain-decision.md) (the +numbered choices there are called "choice N" below). It covers two plan +items: + +- **C2, the hostname:** each environment gets a global external Application + Load Balancer, built in the new project from the start, with a + Google-managed certificate. Until the environment moves, the load balancer + sends requests across projects to today's service, through a backend + service and a serverless network endpoint group left in the current + project. The dashboard then reads the collector through the hostname. +- **C5, the move:** each environment's database is copied into the new + project with a dump and restore, staging first as the rehearsal. The + service runs in the new project, and the load balancer's URL map is changed + to send requests to it. The DNS records don't change. + +Between them come C3 (the emitter release with the production hostname as its +default) and C4 (builds run that release). Production's copy waits for C4. +After the move come S1 and S2, switching intake on, C6 and C7, in that order, +each created once in the new project (section 12). + +Use it only when the collector deploy switch-over (C1) is verified: this repo +deploys staging and production, and every check in the "Verify the +deployment from microcosm-collector" section of calibration-diagnostics#233 +has passed. + +Nothing here has been run. Staging goes first in every section, so check each +command there before running it for production. + +**No concrete values.** This runbook names no hostname, domain, IP address, +project ID or number, organization or billing account ID, instance, database +or user name, connection name, service-account email or secret value. Section +2 sets each one from where it is kept, or from a placeholder in angle brackets +that the operator replaces and records. No command here prints a secret +value: values are generated in the shell and piped into Secret Manager, or +piped from one Secret Manager secret into another. ## 1. People and access - **Google Cloud, the new project:** permission to create a project under the - chosen parent and link the billing account; then the owner role on the new - project. -- **Google Cloud, `policyengine-api`:** the steps change resources there, not - only read them. The project owner role covers all of these, if that is - simpler for the windows: + current project's organization (Project Creator) and to link its billing + account (Billing Account User). The new project has one owner. +- **Google Cloud, the current project:** the steps change resources there, + not only read them. The project owner role covers all of these: - | Steps | What they do in `policyengine-api` | Role | + | Steps | What they do in the current project | Role | |---|---|---| - | 2, 3.2, 6.1, 9 | read resources and their IAM policies | `roles/viewer` and `roles/iam.securityReviewer` | - | 3.2, 5.3 | read the collector's secret values (step 3.2's database connection uses the migration user's password) | `roles/secretmanager.secretAccessor` on the collector's secrets | - | 3.2, 5.2, 5.5, 6.3, 11 | connect to and export the old databases; delete the old instances | `roles/cloudsql.admin` | - | 5.3, 11 | change the intake bucket's IAM policy (#12) | `roles/storage.admin` on that bucket | - | 6.2, 8.4, 10, 11 | update the old service and its traffic; run and cancel job executions; delete the old services and jobs | `roles/run.admin`, and `roles/iam.serviceAccountUser` on the old runtime account, because a service update deploys a revision that runs as it | - | 6.2, 10, 11 | pause, resume and delete #12's queue | `roles/cloudtasks.queueAdmin` | - | 6.2, 6.7, 10, 11 | pause, resume and delete #12's and #19's schedules | `roles/cloudscheduler.admin` | - | 6.7, 10, 11 | disable, enable and delete #19's alert policies; delete its notification channel | `roles/monitoring.alertPolicyEditor` and `roles/monitoring.notificationChannelEditor` | - | 11 | delete secrets, service accounts, this repo's Workload Identity provider and the collector's images | `roles/secretmanager.admin`, `roles/iam.serviceAccountAdmin`, `roles/iam.workloadIdentityPoolAdmin` and `roles/artifactregistry.repoAdmin` | -- **DNS:** edit access to the chosen zone (choice 5). For D3 B only, a Google - account that is a verified owner of the base domain in Search Console; D3 - C's certificate needs only a DNS record. + | 3, 8.1, 10 | read resources and their IAM policies | `roles/viewer` and `roles/iam.securityReviewer` | + | 3, 7.3 | read the collector's secret values (step 3's database connection uses the migration user's password; step 7.3 copies two secrets) | `roles/secretmanager.secretAccessor` on the collector's secrets | + | 3, 7.2, 8.3, 13 | connect to and export the old databases; delete the old instances | `roles/cloudsql.admin` | + | 5.1, 13 | create and delete the serverless network endpoint group and backend service for today's service, and grant a role on that backend service | `roles/compute.loadBalancerAdmin`, and `compute.backendServices.setIamPolicy` for the grant | + | 5.2, 8.8, 11 | use that backend service from the new project's URL map | `roles/compute.loadBalancerServiceUser` on that backend service (step 5.1 grants it) | + | 8.2, 11, 13 | update the old service and its traffic; delete the old services and jobs | `roles/run.admin`, and `roles/iam.serviceAccountUser` on the old runtime account, because a service update deploys a revision that runs as it | + | 13 | delete secrets, service accounts, this repo's Workload Identity provider and the collector's images | `roles/secretmanager.admin`, `roles/iam.serviceAccountAdmin`, `roles/iam.workloadIdentityPoolAdmin` and `roles/artifactregistry.repoAdmin` | +- **DNS:** the domain's DNS is managed outside Google Cloud. Anthony adds the + records by hand (step 5.3). - **GitHub:** an admin of PolicyEngine/microcosm-collector (environment settings), and someone who can change the calibration-diagnostics `Production` environment and merge in both repos. @@ -51,16 +65,20 @@ another. ## 2. Names used below -Set these once per environment, in a `bash` shell. Every name a choice decides -is marked. +Set these once per environment, in a `bash` shell, before that environment's +part of each section. Step 8.6 changes some of the GitHub variables read +here, so keep the shell open through a copy, or write the values in the +copy's notes, which rollback (section 11) needs. ```bash -OLD_PROJECT_ID=policyengine-api +OLD_PROJECT_ID='' # TELEMETRY_GCP_PROJECT_ID in both environments until their copies; recorded by the operator OLD_PROJECT_NUMBER=$(gcloud projects describe "$OLD_PROJECT_ID" --format='value(projectNumber)') -OLD_DEPLOY_SERVICE_ACCOUNT=... # the current deployment account, from step 3.2 -NEW_PROJECT_ID=... # choice 2 -ENVIRONMENT_NAME=staging # then production -COLLECTOR_HOSTNAME=... # choice 4, for this environment +OLD_DEPLOY_SERVICE_ACCOUNT='' # the TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT secret; found in step 3 +NEW_PROJECT_ID='' # chosen in section 4; recorded by the operator +NEW_PROJECT_NUMBER=$(gcloud projects describe "$NEW_PROJECT_ID" --format='value(projectNumber)') # after section 4 creates it +OPERATOR_ACCOUNT='' # the account that runs these steps +ENVIRONMENT_NAME=staging # then production +COLLECTOR_HOSTNAME='' # '' for production; recorded by the operator, and in the dashboard's MICROCOSM_TELEMETRY_COLLECTOR_URL after step 5.5 collector_variable() { gh api "repos/PolicyEngine/microcosm-collector/environments/$ENVIRONMENT_NAME/variables/$1" --jq .value @@ -69,13 +87,18 @@ REGION=$(collector_variable TELEMETRY_GCP_REGION) SERVICE=$(collector_variable TELEMETRY_SERVICE) MIGRATION_JOB=$(collector_variable TELEMETRY_MIGRATION_JOB) ARTIFACT_REGISTRY_NAME=$(collector_variable TELEMETRY_ARTIFACT_REGISTRY_NAME) -OLD_CLOUD_SQL_CONNECTION=$(collector_variable TELEMETRY_CLOUD_SQL_CONNECTION) +OLD_CLOUD_SQL_CONNECTION=$(collector_variable TELEMETRY_CLOUD_SQL_CONNECTION) # read before step 8.6 OLD_INSTANCE=${OLD_CLOUD_SQL_CONNECTION##*:} DATABASE_SECRET_NAME=$(collector_variable TELEMETRY_DATABASE_SECRET_NAME) MIGRATION_DATABASE_SECRET_NAME=$(collector_variable TELEMETRY_MIGRATION_DATABASE_SECRET_NAME) JWT_SECRET_NAME=$(collector_variable TELEMETRY_JWT_SECRET_NAME) READ_SECRET_NAME=$(collector_variable TELEMETRY_READ_SECRET_NAME) -HF_CHECK_JOB=$(collector_variable TELEMETRY_HF_CHECK_JOB) # production, once #19 has merged + +# From step 3's record: the names in the database URL secrets, which `gcloud sql users list` also lists. +DATABASE_NAME='' +MIGRATION_DATABASE_USER='' +RUNTIME_DATABASE_USER='' +ADMIN_DATABASE_USER='' # Cloud SQL's built-in admin user # Names are scoped to a project, so the new resources can reuse the old ones. NEW_INSTANCE=$OLD_INSTANCE @@ -84,46 +107,33 @@ RUNTIME_SERVICE_ACCOUNT="collector-runtime-$ENVIRONMENT_NAME@$NEW_PROJECT_ID.iam MIGRATION_SERVICE_ACCOUNT="collector-migration-$ENVIRONMENT_NAME@$NEW_PROJECT_ID.iam.gserviceaccount.com" DEPLOY_SERVICE_ACCOUNT="collector-deploy@$NEW_PROJECT_ID.iam.gserviceaccount.com" EXPORT_BUCKET="gs://$NEW_PROJECT_ID-collector-move" + +# Load balancer resources, one set per environment. Google recommends that a +# backend service's name differs between projects, so the console tells them apart. +OLD_ENDPOINT_GROUP="collector-current-endpoints-$ENVIRONMENT_NAME" # in the current project +OLD_BACKEND="collector-current-backend-$ENVIRONMENT_NAME" # in the current project +OLD_BACKEND_PATH="projects/$OLD_PROJECT_ID/global/backendServices/$OLD_BACKEND" +NEW_ENDPOINT_GROUP="collector-endpoints-$ENVIRONMENT_NAME" +NEW_BACKEND="collector-backend-$ENVIRONMENT_NAME" +ADDRESS="collector-address-$ENVIRONMENT_NAME" +DNS_AUTHORIZATION="collector-hostname-$ENVIRONMENT_NAME" +CERTIFICATE="collector-certificate-$ENVIRONMENT_NAME" +CERTIFICATE_MAP="collector-certificates-$ENVIRONMENT_NAME" +CERTIFICATE_MAP_ENTRY="collector-hostname-$ENVIRONMENT_NAME" +URL_MAP="collector-routes-$ENVIRONMENT_NAME" +HTTPS_PROXY="collector-proxy-$ENVIRONMENT_NAME" +FORWARDING_RULE="collector-https-$ENVIRONMENT_NAME" ``` -`DATABASE_NAME`, `RUNTIME_DATABASE_USER`, `MIGRATION_DATABASE_USER` and -`OTHER_DATABASE_USERS` come from the old instance in step 3.2. - -## 3. Before the windows - -### 3.1 Check the prerequisites - -- `COLLECTOR_DEPLOY_ENABLED` is `true`, and C1 is verified. -- calibration-diagnostics#234 has merged, or everyone has agreed never to - dispatch calibration-diagnostics' `telemetry-service.yml` again. Its settings - name the old project. -- PolicyEngine/microcosm-emitter#29 has merged, so changing the emitter's - default is a constant change (section 8.2). -- List what has merged here that needs its own resources in the new project, - and add those steps where this runbook marks them: - - #12's artifact intake: a Cloud Tasks queue, the worker job, the recovery - job and its Cloud Scheduler job, the invoker account and their - permissions, as in the README's "Artifact intake" section. Its bucket - stays in the old project (step 5.3), which ties the collector's files to - `policyengine-api` (the memo's D2 B). Intake was off on 2026-10-20: the - Cloud Tasks and Cloud Scheduler APIs weren't enabled there, and no intake - bucket existed. Switching it on only after the move avoids this; - - S1's Hugging Face read token (#13, PR #18); - - C5's session signing key (#16), a fifth Secret Manager secret per - environment; - - S2's daily check (#19): production's check job, its Cloud Scheduler job - and the account that job runs as, a notification channel and two Cloud - Monitoring alert policies, and three production variables. - -### 3.2 Record the current state - -For each environment (read-only): +## 3. Record the current state (read-only) + +For each environment, before section 5. Write everything in the operator's +notes; the later steps compare against them. - The service's URL, serving revision, image, Cloud SQL connection and secret names: the `gcloud run services describe` command in step 0 of calibration-diagnostics#233. This serving revision isn't the rollback - target: production redeploys between the windows, so step 6.1 records the - target again just before each window's changes. + target: step 8.1 records that just before each copy. - The instance's settings: `gcloud sql instances describe "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --format=yaml`. Keep the database version, edition, tier, disk size, availability type, @@ -131,37 +141,25 @@ For each environment (read-only): flags, and IP and SSL configuration. - The database and its users: `gcloud sql databases list` and `gcloud sql users list` with - `--instance "$OLD_INSTANCE" --project "$OLD_PROJECT_ID"`. On 2026-10-20 - both instances held the databases `postgres` and `telemetry`, and listed - these users: - - | Environment | Users | - |---|---| - | staging | `postgres`, `telemetry_migrator`, `telemetry_runtime` | - | production | `postgres`, `telemetry_app`, `telemetry_migrator`, `telemetry_runtime` | - - Set `DATABASE_NAME`, `RUNTIME_DATABASE_USER` and `MIGRATION_DATABASE_USER` - from them. Set `OTHER_DATABASE_USERS` to every other role the roles query - below lists, apart from `postgres`; as on 2026-10-20: - - ```bash - case "$ENVIRONMENT_NAME" in - production) OTHER_DATABASE_USERS=(telemetry_app) ;; - *) OTHER_DATABASE_USERS=() ;; - esac - ``` - - Change it if the record shows otherwise. + `--instance "$OLD_INSTANCE" --project "$OLD_PROJECT_ID"`. Expect Cloud SQL's + built-in admin user, the migration user and the runtime user, and no other: + production's unused third user was dropped on 2026-10-27. The copy + recreates only the migration and runtime users. Set `DATABASE_NAME`, + `MIGRATION_DATABASE_USER`, `RUNTIME_DATABASE_USER` and + `ADMIN_DATABASE_USER` from the record. - **The database's roles and privileges.** Connect as the migration user through Cloud SQL Studio, or with `psql` through the Cloud SQL Auth Proxy, and save the output of `\l`, `\dn+ public`, `\dp public.*` and `\ddp`, and of the roles query below. Note which role owns the tables and what the runtime user is granted. These were provisioned by scripts kept outside - this repo, so this record is the only specification of them. The roles - query lists every role's attributes and memberships in one result, since - psql 16 and later no longer show memberships in `\du`. Each membership - carries its PostgreSQL 16 options (`admin`, `inherit`, `set`, from - `pg_auth_members`), which show how each role was made (step 5.2): + this repo, so this record is the only specification of them. `\ddp` must + show the migration user's default privileges for the runtime user (the + README's "Database access" section); both environments had them on + 2026-10-27. The roles query lists every role's attributes and memberships + in one result, since psql 16 and later no longer show memberships in + `\du`. Each membership carries its PostgreSQL 16 options (`admin`, + `inherit`, `set`, from `pg_auth_members`), which show how each role was + made (step 7.2): ```sql SELECT r.rolname, r.rolcanlogin, r.rolcreaterole, r.rolcreatedb, r.rolinherit, @@ -177,33 +175,14 @@ For each environment (read-only): ``` - Who may read each secret: `gcloud secrets get-iam-policy SECRET --project "$OLD_PROJECT_ID" --format=yaml` - for each secret the environment uses: the four named above, and, if - merged, #16's signing-key secret and S1's Hugging Face token secret. The - read secret's policy names the dashboard's Vercel identity. -- Collector secrets that no deployment uses. On 2026-10-22, production also - had `microcosm-telemetry-database-url`, a database URL secret made four days - before the runtime one that the service mounts. No GitHub variable names it - and no service or job mounts it, but production's runtime account may read - it. List the old project's secrets - (`gcloud secrets list --project "$OLD_PROJECT_ID"`), note any such secret, - and don't copy it; section 11 deletes it. -- What the open collector PRs added, if merged (step 3.1), each command with - `--project "$OLD_PROJECT_ID"`: - - #12: `gcloud tasks queues describe WORKER_QUEUE --location "$REGION"`; - `gcloud run jobs describe` with `--region "$REGION"` for the worker job - and the recovery job; `gcloud scheduler jobs describe RECOVERY_SCHEDULE --location "$REGION"`; - the bucket's roles with `gcloud storage buckets get-iam-policy gs://INTAKE_BUCKET`; - the invoker account's with `gcloud iam service-accounts get-iam-policy`; - and the project-level roles of the old runtime, worker and recovery - accounts, with the `gcloud projects get-iam-policy` command shown below for - the deployment account. #12's deployment wiring isn't written, so its - storage grants may be on the project rather than on the bucket, and the - bucket's policy wouldn't show them. - - #19: `gcloud run jobs describe "$HF_CHECK_JOB" --region "$REGION"`; - `gcloud scheduler jobs describe HF_CHECK_SCHEDULE --location "$REGION"`, - which shows the account it starts the job as; the two alert policies' - names with `gcloud monitoring policies list --format='value(name,displayName,enabled)'`; - the notification channel with `gcloud beta monitoring channels list`. + for each of the four secrets named above. The read secret's policy names + the dashboard's Vercel identity for that environment (step 7.3 grants it + the same access in the new project). +- Collector secrets that no deployment uses: list the current project's + secrets (`gcloud secrets list --project "$OLD_PROJECT_ID"`). Production + has an older database URL secret that no GitHub variable names and no + service or job mounts; its only version was disabled on 2026-10-27. Note + any such secret and don't copy it; section 13 deletes it. - The deployment account's roles. Its email is the `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` secret, which GitHub can't show. It is the account whose IAM policy grants `roles/iam.workloadIdentityUser` to this @@ -211,67 +190,44 @@ For each environment (read-only): `gcloud iam service-accounts get-iam-policy ACCOUNT --project "$OLD_PROJECT_ID"` shows it. Set `OLD_DEPLOY_SERVICE_ACCOUNT` to it. Then list its roles on the project with - `gcloud projects get-iam-policy "$OLD_PROJECT_ID" --flatten=bindings --filter="bindings.members:serviceAccount:DEPLOY_ACCOUNT" --format="value(bindings.role)"`, + `gcloud projects get-iam-policy "$OLD_PROJECT_ID" --flatten=bindings --filter="bindings.members:serviceAccount:$OLD_DEPLOY_SERVICE_ACCOUNT" --format="value(bindings.role)"`, and its roles on individual resources (the runtime and migration accounts, the secrets and the Artifact Registry store) from their IAM policies. +- The dashboard's settings: the current values of + `MICROCOSM_TELEMETRY_COLLECTOR_URL` (calibration-diagnostics' GitHub + `Production` environment, and Vercel Production and Preview) and + `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` (Vercel Production and Preview). + Rolling back step 5.5 or section 9 sets them back. - The dashboard's Vercel identity provider: the resource named by the Vercel setting `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`, read with `gcloud iam workload-identity-pools providers describe` (issuer, allowed - audiences, attribute mapping and condition). - -### 3.3 Agree the windows - -One for staging, one for production (choice 8). From the start of a window -until its verification passes, nothing merges to this repo's `main`: every -merge deploys. - -### 3.4 Hostname first only: put the hostname in front of today's service - -In the hostname-first order (choice 6), builds move to the hostname before the -project moves, so that none of them still uses the default address when step -6.2 puts the old service in maintenance mode. Skip this step in the "both -together" order. Before section 4: - -1. Do section 7 for each environment against today's service: run its - commands with `--project "$OLD_PROJECT_ID"` wherever they say - `--project "$NEW_PROJECT_ID"`. For D3 C, this puts the load balancer's - resources in `policyengine-api` (its Compute Engine and Certificate Manager - APIs were already enabled on 2026-10-22) until section 11 removes them. - Then run section 7's hostname checks. -2. Do section 8.2: release the emitter with the hostname as - `PRODUCTION_COLLECTOR_URL`, keeping the default address in - `TRUSTED_COLLECTOR_ORIGINS`. -3. Optionally point the dashboard's `MICROCOSM_TELEMETRY_COLLECTOR_URL` at the - hostname now (the first two rows of section 8.1's settings table). Section - 8.1 then changes only the secret settings. -4. Wait until builds use the hostname. Emitter 0.2.1 has the default address - built in, with no setting that changes it, so Microcosm has to have left - its `microcosm-emitter==0.2.1` pin; and the machines and build images that - start the separately installed emitter need the new release. Builds that - still use the default address when a window starts are handled as choice 7 - says. - -Then continue with section 4. Section 8.2 is then already done, and section 7 -is done again in the new project inside production's window, as its "Hostname -first" notes say. + audiences, attribute mapping and condition). It stays where it is (choice + 10). ## 4. Create the project (once, no downtime) +The new project goes directly under the organization that holds the current +project, as the current project does, and uses the same billing account +(choice 2). Read +both from the current project, so neither value is typed: + ```bash -gcloud projects create "$NEW_PROJECT_ID" --name "DISPLAY_NAME" --organization ORGANIZATION_ID # or --folder FOLDER_ID -gcloud billing projects link "$NEW_PROJECT_ID" --billing-account BILLING_ACCOUNT_ID -NEW_PROJECT_NUMBER=$(gcloud projects describe "$NEW_PROJECT_ID" --format='value(projectNumber)') +gcloud projects describe "$OLD_PROJECT_ID" --format='value(parent.type)' # must print organization +ORGANIZATION_ID=$(gcloud projects describe "$OLD_PROJECT_ID" --format='value(parent.id)') +BILLING_ACCOUNT=$(gcloud billing projects describe "$OLD_PROJECT_ID" --format='value(billingAccountName)') +gcloud projects create "$NEW_PROJECT_ID" --name '' --organization "$ORGANIZATION_ID" +gcloud billing projects link "$NEW_PROJECT_ID" --billing-account "${BILLING_ACCOUNT#billingAccounts/}" gcloud services enable run.googleapis.com sqladmin.googleapis.com \ secretmanager.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \ iamcredentials.googleapis.com sts.googleapis.com storage.googleapis.com \ + compute.googleapis.com certificatemanager.googleapis.com \ --project "$NEW_PROJECT_ID" ``` -Also enable `compute.googleapis.com` and `certificatemanager.googleapis.com` -for a load balancer (D3 C); `cloudtasks.googleapis.com` and -`cloudscheduler.googleapis.com` if #12's intake is switched on; and -`cloudscheduler.googleapis.com` and `monitoring.googleapis.com` if #19's daily -check has merged. +A project ID can't be changed later, and can't be reused even after its +project is deleted. Record it. A budget alert on the new project, at the +collector's current cost plus the load balancers, shows an unexpected cost +early. ### 4.1 Artifact Registry @@ -285,16 +241,17 @@ for the store, which this repo's vocabulary test keeps out of documentation. `${REGION}-docker.pkg.dev/${PROJECT_ID}/${ARTIFACT_REGISTRY_NAME}/${IMAGE_NAME}`, so the store must be in the new project. -**Let the old project's production deployments read it.** `deploy-staging` -builds the image with staging's `PROJECT_ID`, and `deploy-production` deploys -that same image. So from staging's window until production's window, production -deploys an image from the new project's store into the old project. Cloud Run -then needs both the deploying account and the deploying project's Cloud Run -service agent to hold Artifact Registry Reader on that store. Without it, -production's "Deploy an isolated maintenance revision" step fails (no traffic -moves) in staging's window and on every merge until production's window. The -new project holds only the collector's store, so grant the role on the project -before staging's window: +**Let the current project's production deployments read it.** +`deploy-staging` builds the image with staging's `PROJECT_ID`, and +`deploy-production` deploys that same image. So from staging's copy until +production's copy, which waits for C4 and may be weeks later, production +deploys an image from the new project's store into the current project. +Cloud Run then needs both the deploying account and the deploying project's +Cloud Run service agent to hold Artifact Registry Reader on that store. +Without it, production's "Deploy an isolated maintenance revision" step fails +(no traffic moves) on every merge from staging's copy on. The new project +holds only the collector's store, so grant the role on the project before +staging's copy: ```bash for old_production_reader in "serviceAccount:$OLD_DEPLOY_SERVICE_ACCOUNT" \ @@ -304,12 +261,12 @@ for old_production_reader in "serviceAccount:$OLD_DEPLOY_SERVICE_ACCOUNT" \ done ``` -Remove both bindings in section 11. +Section 13 removes both bindings. ### 4.2 The deployment account and Workload Identity Federation for this repo The new project gets a deployment account used only by this repo. The dashboard -keeps its own deployment identity in `policyengine-api`. +keeps its own deployment identity in the current project. ```bash gcloud iam service-accounts create collector-deploy --project "$NEW_PROJECT_ID" \ @@ -332,31 +289,212 @@ secret, is `projects/$NEW_PROJECT_NUMBER/locations/global/workloadIdentityPools/github/providers/microcosm-collector`. What `scripts/deploy.sh` needs the deployment account to do, and the role that -allows it. Compare with the record from step 3.2 before granting: +allows it. Compare with the record from section 3 before granting: | `deploy.sh` command | Needs | Grant | |---|---|---| | `gcloud run deploy`, `run jobs deploy` and `execute`, `services describe`, `update-traffic`, `--allow-unauthenticated` | Cloud Run administration | `roles/run.admin` on the project | -| `--service-account` for the service and the migration job | acting as those accounts | `roles/iam.serviceAccountUser` on each runtime and migration account (step 5.1) | +| `--service-account` for the service and the migration job | acting as those accounts | `roles/iam.serviceAccountUser` on each runtime and migration account (step 7.1) | | `build-and-push-image` | pushing images | `roles/artifactregistry.writer` on the store | -| `check-secrets-exist` (`gcloud secrets describe`) | reading secret metadata | `roles/secretmanager.viewer` on each secret (step 5.3) | +| `check-secrets-exist` (`gcloud secrets describe`) | reading secret metadata | `roles/secretmanager.viewer` on each secret (step 7.3) | | `create-database-backup` (production) | `cloudsql.backupRuns.create` | `roles/cloudsql.editor` on the project, or a custom role with `cloudsql.backupRuns.create` and `cloudsql.instances.get` | +The deployment account needs nothing on the load balancers: the operator +creates and changes them. + ### 4.3 The export bucket It will hold complete copies of the telemetry databases, so it is private and -is deleted after the move (section 11). +is deleted after the move (section 13). ```bash gcloud storage buckets create "$EXPORT_BUCKET" --project "$NEW_PROJECT_ID" \ --location "$REGION" --uniform-bucket-level-access --public-access-prevention ``` -## 5. Create each environment's resources (no downtime) +## 5. C2: put the hostname in front of today's service (no downtime) + +Staging first: its load balancer is the first test of the cross-project +backend service (section 15). Production's follows once staging's hostname +checks pass. + +### 5.1 In the current project: a backend service for today's service + +A serverless network endpoint group has to be in the same project as its +Cloud Run service, so these two resources stay in the current project until +section 13 deletes them. Its Compute Engine API was already enabled on +2026-10-22. + +```bash +gcloud compute network-endpoint-groups create "$OLD_ENDPOINT_GROUP" --region "$REGION" \ + --network-endpoint-type serverless --cloud-run-service "$SERVICE" --project "$OLD_PROJECT_ID" +gcloud compute backend-services create "$OLD_BACKEND" --global \ + --load-balancing-scheme EXTERNAL_MANAGED --project "$OLD_PROJECT_ID" +gcloud compute backend-services add-backend "$OLD_BACKEND" --global \ + --network-endpoint-group "$OLD_ENDPOINT_GROUP" --network-endpoint-group-region "$REGION" \ + --project "$OLD_PROJECT_ID" +gcloud compute backend-services add-iam-policy-binding "$OLD_BACKEND" --global \ + --member "user:$OPERATOR_ACCOUNT" --role roles/compute.loadBalancerServiceUser \ + --project "$OLD_PROJECT_ID" +``` + +The last command lets the operator's account use this backend service from +another project's URL map. Google's external Application Load Balancer +overview says the role "can be granted at the project level or at the +resource level"; this grants it on the backend service only. Keep it until +section 13: step 8.8's rollback and section 11 use it to point the URL map +back here. + +### 5.2 In the new project: the load balancer and its certificate + +```bash +gcloud compute addresses create "$ADDRESS" --global --project "$NEW_PROJECT_ID" +gcloud certificate-manager dns-authorizations create "$DNS_AUTHORIZATION" \ + --domain "$COLLECTOR_HOSTNAME" --type per-project-record --project "$NEW_PROJECT_ID" +gcloud certificate-manager certificates create "$CERTIFICATE" \ + --domains "$COLLECTOR_HOSTNAME" --dns-authorizations "$DNS_AUTHORIZATION" --project "$NEW_PROJECT_ID" +gcloud certificate-manager maps create "$CERTIFICATE_MAP" --project "$NEW_PROJECT_ID" +gcloud certificate-manager maps entries create "$CERTIFICATE_MAP_ENTRY" --map "$CERTIFICATE_MAP" \ + --certificates "$CERTIFICATE" --hostname "$COLLECTOR_HOSTNAME" --project "$NEW_PROJECT_ID" +gcloud compute url-maps create "$URL_MAP" --global \ + --default-service "$OLD_BACKEND_PATH" --project "$NEW_PROJECT_ID" +gcloud compute target-https-proxies create "$HTTPS_PROXY" --url-map "$URL_MAP" \ + --certificate-map "$CERTIFICATE_MAP" --global --project "$NEW_PROJECT_ID" +gcloud compute forwarding-rules create "$FORWARDING_RULE" --global \ + --load-balancing-scheme EXTERNAL_MANAGED --address "$ADDRESS" \ + --target-https-proxy "$HTTPS_PROXY" --ports 443 --project "$NEW_PROJECT_ID" +``` + +The URL map's default service is the current project's backend service, named +by its full path. Google: for global external Application Load Balancers, the +frontend and URL map can reference backend services "from any project within +the same organization. No VPC network restrictions apply." The load +balancer's charges go to the new project, where the forwarding rule is. + +The DNS authorization is per project: its `CNAME` has a name with a suffix +for the project, instead of a fixed name, so it never clashes with an +authorization for the same hostname in another project. For per-project +authorizations Certificate Manager issues only Google Trust Services +certificates. + +### 5.3 The DNS records + +Print what Anthony adds for this environment: + +```bash +gcloud certificate-manager dns-authorizations describe "$DNS_AUTHORIZATION" \ + --project "$NEW_PROJECT_ID" --format='value(dnsResourceRecord.name,dnsResourceRecord.type,dnsResourceRecord.data)' +gcloud compute addresses describe "$ADDRESS" --global --project "$NEW_PROJECT_ID" --format='value(address)' +``` + +Anthony adds these records by hand at the domain's DNS provider, outside +Google Cloud. Four in all, two per environment: + +| Environment | Type | Name | Value | +|---|---|---|---| +| staging | `CNAME` | ``: the `dnsResourceRecord.name` printed above, which starts with `_acme-challenge_` and ends with the staging hostname | ``: the `dnsResourceRecord.data` printed above | +| staging | `A` | `` | ``: the address printed above | +| production | `CNAME` | ``, as for staging | `` | +| production | `A` | `` | `` | + +- **Plain records:** no proxying, forwarding or redirection in front of + them. A proxy would end TLS for requests that carry Hugging Face tokens, + and would hide the `CNAME` from Google. +- **Names:** if the provider's form adds the domain itself, enter only the + part of each name before `.`. A trailing dot at the + end of a printed name marks it as complete; leave it out if the form + doesn't accept it. +- **Both records at once:** nothing uses the hostname until step 5.5, so the + `A` record can go in before the certificate is issued. +- **They stay for good.** Certificate Manager uses the `CNAME` for renewals + as well as the first issue, and the `A` record doesn't change during the + move, because the static IP is already in the new project. The default TTL + is fine. +- **CAA records:** the domain had none when checked. Any added later must + allow Google Trust Services. + +Then wait until the certificate is issued: + +```bash +gcloud certificate-manager certificates describe "$CERTIFICATE" --project "$NEW_PROJECT_ID" \ + --format='value(managed.state)' +``` + +It must print `ACTIVE`. While it doesn't, `--format=yaml` shows +`managed.authorizationAttemptInfo`, which says whether Google found the +`CNAME`. + +### 5.4 Check the hostname + +```bash +curl -fsS "https://$COLLECTOR_HOSTNAME/ready" +curl --silent --output /dev/null --write-out '%{http_code}\n' \ + --request POST "https://$COLLECTOR_HOSTNAME/v1/auth/huggingface/exchange" +``` + +The first must succeed. The second must print `401` (no credential sent), +which shows the request reached the application through the load balancer +and its certificate. A new load balancer can take a few minutes to answer +after its certificate becomes active. -Staging first. Production's can be created days before its window. +### 5.5 Point the dashboard at the hostname -### 5.1 Service accounts +Once both environments pass step 5.4: + +| Setting | Where | New value | +|---|---|---| +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | calibration-diagnostics GitHub environment `Production` (variable) | `https://` | +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | Vercel project calibration-diagnostics, Production | `https://` | +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | Vercel project calibration-diagnostics, Preview | `https://` | + +Settings take effect on the next deployment: merge calibration-diagnostics' +next PR, and open a new Preview deployment. The build monitor should then +list collector runs as before. The read token doesn't change. To undo, set +the three values back to section 3's record and deploy again. + +`MICROCOSM_TELEMETRY_COLLECTOR_URL` doesn't change again: the move changes only +what the hostname reaches. + +## 6. C3 and C4: the emitter and builds + +Done in the emitter and Microcosm, not here; this section says what must hold +before production's copy. + +- **C3:** an emitter release from PolicyEngine/microcosm-emitter#29 and one + constant change, in `src/microcosm_emitter/core/constants.py`: + `https://` is added to `TRUSTED_COLLECTOR_ORIGINS` and + becomes `PRODUCTION_COLLECTOR_URL`. The old default address stays in + `TRUSTED_COLLECTOR_ORIGINS` until the old service is deleted (section 13). + It needs only C2: the hostname already reaches today's service. +- **C4:** builds run that release. Builds that start the separately + installed emitter pick it up at their next start, once it is installed on + that machine or in that build image, the Modal image included. Builds + pinned to `microcosm-emitter==0.2.1` keep the old address, which 0.2.1 has + built in with no setting to change it, until Microcosm leaves that pin + (H2a). + +**Production's copy waits for C4.** Check that Microcosm's +`packages/microcosm-build/pyproject.toml` on `main` no longer pins 0.2.1, and +that the Modal image carries the release. In Cloud Monitoring, compare the +production service's request count (`run.googleapis.com/request_count`) with +its load balancer's (`loadbalancing.googleapis.com/https/request_count`): +once builds use the hostname, nearly every request to the service comes +through the load balancer. Requests still on the default address after +production's copy get 503 (step 8.9). + +Staging's copy doesn't wait for C4: no build sends to staging by default. + +## 7. C5: prepare each environment in the new project (no downtime) + +Staging first. Production's resources can be created any time before its +copy. + +**Before production's copy, merge none of these:** S1's and S2's PRs (#18, +#19), intake's deployment wiring, C6, or C7's PR (#16). Each needs resources +of its own, which section 12 creates once, in the new project, after the +move. Merged earlier, each would need them in the current project too. + +### 7.1 Service accounts ```bash gcloud iam service-accounts create "collector-runtime-$ENVIRONMENT_NAME" --project "$NEW_PROJECT_ID" @@ -370,12 +508,12 @@ for service_account in "$RUNTIME_SERVICE_ACCOUNT" "$MIGRATION_SERVICE_ACCOUNT"; done ``` -If the record from step 3.2 shows these accounts limited further today (for +If the record from section 3 shows these accounts limited further today (for example, an IAM condition on the instance name), repeat that here. -### 5.2 Cloud SQL +### 7.2 Cloud SQL -Create the instance with the settings recorded in step 3.2, except deletion +Create the instance with the settings recorded in section 3, except deletion protection, which is on for both new instances: on 2026-10-22 staging's instance had it, but production's didn't. Where the record shows automated backups off, as staging's did on 2026-10-22, use `--no-backup` instead of @@ -394,18 +532,16 @@ Match the record's IP configuration (Cloud Run connects through the Cloud SQL connector, so a public IP needs no authorized networks), SSL mode, database flags, maintenance window and point-in-time recovery. -Create every role the step 3.2 roles query lists, except `postgres`, with the -**same names** as on the old instance and new random passwords. Google's import -guide says every user that owns objects or was granted permissions on objects -in the dumped database must exist on the target instance, or the import -fails. Besides the migration and runtime users, create each role in -`OTHER_DATABASE_USERS` (step 3.2: `telemetry_app` for production today, which -the collector doesn't use, and none for staging), with a random password that -isn't kept. +**Create the migration user and the runtime user,** with the **same names** +as on the old instance and new random passwords. Google's import guide says +every user that owns objects or was granted permissions on objects in the +dumped database must exist on the target instance, or the import fails. These +two are the only ones: production's unused third user was dropped on +2026-10-27, and the admin user exists on every new instance. -**Make each role the way its old one was made.** An export carries no roles, +**Make each user the way its old one was made.** An export carries no roles, so nothing corrects a role's attributes or memberships later. Made the same -way, each role's row and its creator's row match the record, so step 6.4's +way, each user's row and its creator's row match the record, so step 8.5's comparison of the roles query shows real differences only. The record shows how each was made: @@ -413,25 +549,27 @@ how each was made: "About PostgreSQL users and roles" and "Create and manage users" say a user created with `gcloud sql users create` is a member of `cloudsqlsuperuser`, with the `CREATEROLE`, `CREATEDB` and `LOGIN` attributes. Create it that way. - If its row lacks one of those attributes, see the staging check below. + If its row lacks one of those attributes, see "If a recorded user lacks an + attribute" below. - **Its row doesn't list `cloudsqlsuperuser`:** it was made in SQL, or its membership was taken away later. On PostgreSQL 16, a role made by a role - that isn't a superuser, such as Cloud SQL's `postgres`, "is automatically + that isn't a superuser, such as Cloud SQL's admin user, "is automatically granted back to the creating user, just as if the bootstrap superuser had executed the command `GRANT created_user TO creating_user WITH ADMIN TRUE, SET FALSE, INHERIT FALSE`" (PostgreSQL 16, "Role Attributes"). The roles query shows that grant as `ROLE (admin)` in the creator's `member_of`, - normally `postgres`'s. Create the role in SQL, connected as that creator, - with exactly the attributes its row shows. PostgreSQL then makes the same - grant, so both rows match the record, and nothing has to be removed - afterwards. If no role lists it as `(admin)`, create it as `postgres`; - step 6.4 then accepts that one extra entry in `postgres`'s row. + normally the admin user's. Create the user in SQL, connected as that + creator, with exactly the attributes its row shows. PostgreSQL then makes + the same grant, so both rows match the record, and nothing has to be + removed afterwards. If no role lists it as `(admin)`, create it as the + admin user; step 8.5 then accepts that one extra entry in the admin user's + row. For the SQL, run `psql` through the Cloud SQL Auth Proxy (`cloud-sql-proxy "$NEW_CLOUD_SQL_CONNECTION"` in another shell, listening on -127.0.0.1:5432). First set `postgres`'s password on the new instance, typed -rather than printed: -`gcloud sql users set-password postgres --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --prompt-for-password`. +port 5432 on this machine). First set the admin user's password on the new +instance, typed rather than printed: +`gcloud sql users set-password "$ADMIN_DATABASE_USER" --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --prompt-for-password`. ```bash migration_password="$(openssl rand -hex 32)" @@ -440,19 +578,15 @@ create_user_with_cloud_sql() { # USER PASSWORD gcloud sql users create "$1" --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --password "$2" } create_user_in_sql() { # CREATOR USER PASSWORD 'ATTRIBUTES' - psql "host=127.0.0.1 port=5432 dbname=postgres user=$1" --set ON_ERROR_STOP=1 \ + psql "host=localhost port=5432 dbname=$DATABASE_NAME user=$1" --set ON_ERROR_STOP=1 \ --set user_name="$2" --set user_password="$3" --set attributes="$4" <<'SQL' CREATE ROLE :"user_name" WITH :attributes PASSWORD :'user_password'; SQL } -# The calls below are examples: for each role, make the one call its step 3.2 row -# calls for, with the attributes its row shows. Until the placeholder in the loop -# is replaced, PostgreSQL refuses each call there and creates nothing. +# Examples: for each user, make the one call its section 3 row calls for, with +# the attributes its row shows. create_user_with_cloud_sql "$MIGRATION_DATABASE_USER" "$migration_password" -create_user_in_sql postgres "$RUNTIME_DATABASE_USER" "$runtime_password" 'LOGIN NOCREATEROLE NOCREATEDB INHERIT' -for other_database_user in "${OTHER_DATABASE_USERS[@]}"; do # none for staging - create_user_in_sql postgres "$other_database_user" "$(openssl rand -hex 32)" 'ATTRIBUTES FROM ITS ROW' -done +create_user_in_sql "$ADMIN_DATABASE_USER" "$RUNTIME_DATABASE_USER" "$runtime_password" 'LOGIN NOCREATEROLE NOCREATEDB INHERIT' ``` psql puts the variables' values into the statement, quoted as a name @@ -460,58 +594,59 @@ psql puts the variables' values into the statement, quoted as a name (`:attributes`), so neither the shell's history nor psql's holds a password. PostgreSQL's `CREATE ROLE` page warns that a password in the statement "might also be logged in ... the server log": the server logs it if the statement -fails, or if the `log_statement` flag recorded in step 3.2 is `ddl` or `all`. -Keep the two password variables in this shell for step 5.3. Then grant the -other memberships the rows show, and run the roles query on the new instance: -it must already match the record, apart from the differences this step names. - -**Check once, on staging's new instance, whether `postgres` can change a user -Cloud SQL made.** PostgreSQL 16's `ALTER ROLE` page says a role with -`CREATEROLE` that isn't a superuser, as Cloud SQL's `postgres` is, can change -another role's attributes "only for non-superuser and non-replication roles -for which they have been granted `ADMIN OPTION`". Cloud SQL's pages say "You -can change the attributes of any user using the ALTER ROLE command", but not -whether `postgres` holds `ADMIN OPTION` on users Cloud SQL makes. So don't -rely on it until this check has passed, in staging's pass and before -production's step 5.2: +fails, or if the `log_statement` flag recorded in section 3 is `ddl` or +`all`. Keep the two password variables in this shell for step 7.3. Then grant +the other memberships the rows show, and run the roles query on the new +instance: it must already match the record, apart from the differences this +step names. + +**If a recorded user lacks an attribute.** This matters only for a row that +lists `cloudsqlsuperuser` but lacks `CREATEROLE`, `CREATEDB` or `LOGIN`. The +admin user would then have to remove that attribute with `ALTER ROLE`. +PostgreSQL 16's `ALTER ROLE` page allows that, for a role with `CREATEROLE` +that isn't a superuser, "only for non-superuser and non-replication roles for +which they have been granted `ADMIN OPTION`". Cloud SQL's pages say "You can +change the attributes of any user using the ALTER ROLE command", but not +whether the admin user holds `ADMIN OPTION` on users Cloud SQL makes. So check +it once, on staging's new instance, with a throwaway user, before relying on +it: ```bash -gcloud sql users create move_check --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" \ +throwaway_user='' # any name the record doesn't use +gcloud sql users create "$throwaway_user" --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" \ --password "$(openssl rand -hex 32)" -psql "host=127.0.0.1 port=5432 dbname=postgres user=postgres" <<'SQL' +psql "host=localhost port=5432 dbname=$DATABASE_NAME user=$ADMIN_DATABASE_USER" \ + --set user_name="$throwaway_user" <<'SQL' SELECT g.rolname AS holder, m.admin_option, m.inherit_option, m.set_option FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.member -WHERE m.roleid = 'move_check'::regrole; -ALTER ROLE move_check NOCREATEROLE NOCREATEDB NOLOGIN; +WHERE m.roleid = :'user_name'::regrole; +ALTER ROLE :"user_name" NOCREATEROLE NOCREATEDB NOLOGIN; SQL -gcloud sql users delete move_check --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --quiet +gcloud sql users delete "$throwaway_user" --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --quiet ``` -Write the result in the window notes. It matters only for a row that lists -`cloudsqlsuperuser` but lacks `CREATEROLE`, `CREATEDB` or `LOGIN`: - - **`ALTER ROLE` succeeds:** create that user with `gcloud sql users create`, - then remove what its row lacks with `ALTER ROLE` as `postgres`, for example - `ALTER ROLE telemetry_app NOLOGIN;`. + then remove what its row lacks with `ALTER ROLE` as the admin user. - **It fails** with "permission denied to alter role": create that user in - SQL as `postgres` instead, with the attributes its row shows, so that - `postgres` holds `ADMIN OPTION` on it. Its row then lacks - `cloudsqlsuperuser`, and `postgres`'s row lists it as `(admin)`; step 6.4 - accepts exactly these two differences for that user. It then has less - privilege than its record. The README asks no more of the two users the - collector uses: the migration credential needs "schema-modification - access", and the runtime credential "only the data access required by the - application". If that user is the importing user, the "Before any import" - steps below must also grant it `CREATE` on `public`, and step 6.4 accepts + SQL as the admin user instead, with the attributes its row shows. Its row + then lacks `cloudsqlsuperuser`, and the admin user's row lists it as + `(admin)`; step 8.5 accepts exactly these two differences for that user. It + then has less privilege than its record. The README asks no more of the + two users: the migration credential needs "schema-modification access", + and the runtime credential "only the data access required by the + application". If that user is the migration user, the "Before any import" + steps below must also grant it `CREATE` on `public`, and step 8.5 accepts that difference too. -**Before any import into the new database** (step 5.5's rehearsal and step -6.3), give it the record's privileges and owners. An export doesn't carry the -database-level privileges. And the importing user (normally the migration -user) can create the tables only if it may create objects in schema `public`: -on PostgreSQL 15 and later, `public` belongs to `pg_database_owner`, that is -to the database's owner, and other users no longer have `CREATE` on it -(PostgreSQL 15 release notes). On the old database the importing user had it +Write the result in the copy's notes. + +**Before any import into the new database** (step 7.6's timing run and step +8.3), give it the record's privileges and owners. An export doesn't carry the +database-level privileges. And the importing user (the migration user) can +create the tables only if it may create objects in schema `public`: on +PostgreSQL 15 and later, `public` belongs to `pg_database_owner`, that is to +the database's owner, and other users no longer have `CREATE` on it +(PostgreSQL 15 release notes). On the old database the migration user had it in one of three ways, which the `\l` and `\dn+ public` records show: it owns the database or is a member of its owner (as every `cloudsqlsuperuser` member is, if `cloudsqlsuperuser` owns it); it owns `public`; or it was granted @@ -521,15 +656,16 @@ the importing user what those tables need. Cloud SQL's users pages don't say who owns a database that `gcloud sql databases create` makes, so compare first. -Connected to the new database as `postgres`, compare `\l "$DATABASE_NAME"` and -`\dn+ public` with the record, then apply what differs, in this order: +Connected to the new database as the admin user, compare +`\l "$DATABASE_NAME"` and `\dn+ public` with the record, then apply what +differs, in this order: 1. **Privileges:** the database-level ones from `\l` (`GRANT CONNECT`, `GRANT CREATE ON DATABASE`) and `public`'s from `\dn+ public` - (`GRANT USAGE, CREATE ON SCHEMA public TO ...`). `postgres` may grant them - while it acts as the owner, which it does through `cloudsqlsuperuser` if - that owns the new database. Grant them before changing an owner: after - that, `postgres` no longer acts as the owner. + (`GRANT USAGE, CREATE ON SCHEMA public TO ...`). The admin user may grant + them while it acts as the owner, which it does through `cloudsqlsuperuser` + if that owns the new database. Grant them before changing an owner: after + that, the admin user no longer acts as the owner. 2. **Owners:** `ALTER DATABASE ... OWNER TO ...` or `ALTER SCHEMA public OWNER TO ...`. PostgreSQL 16's `ALTER DATABASE` page says: "To alter the owner, you must be able to `SET ROLE` to the new owning @@ -537,25 +673,28 @@ Connected to the new database as `postgres`, compare `\l "$DATABASE_NAME"` and - **If Cloud SQL made the new owner,** connect as that user: as a `cloudsqlsuperuser` member with `CREATEDB`, it can make itself the owner of a database that `cloudsqlsuperuser` owns. - - **If `postgres` made it,** `postgres` holds it only as `(admin)`, without - `SET`. The "Role Attributes" page says a creator "can gain access to the - created role by simply granting that role back to themselves with the - `INHERIT` and/or `SET` options", so grant `SET` for the change and revoke - it afterwards, for example: + - **If the admin user made it,** the admin user holds it only as + `(admin)`, without `SET`. The "Role Attributes" page says a creator "can + gain access to the created role by simply granting that role back to + themselves with the `INHERIT` and/or `SET` options", so grant `SET` for + the change and revoke it afterwards. With the names from section 2 in + place of the placeholders: ```sql - GRANT telemetry_migrator TO postgres WITH SET TRUE; - ALTER DATABASE telemetry OWNER TO telemetry_migrator; - REVOKE telemetry_migrator FROM postgres; + GRANT TO WITH SET TRUE; + ALTER DATABASE OWNER TO ; + REVOKE FROM ; ``` - The revoke removes only the grant `postgres` made; the same page says - the automatic grant "cannot be removed or changed by the `CREATEROLE` - user". The roles query's `postgres` row must match the record again. + The revoke removes only the grant the admin user made; the same page + says the automatic grant "cannot be removed or changed by the + `CREATEROLE` user". The roles query's admin user row must match the + record again. -Table privileges and default privileges (`\dp`, `\ddp`) come with the export, -provided the roles exist. Step 6.4 compares all four records after the -import. +Table privileges come with the export, provided the users exist. The +migration user's default privileges are set after the import (step 8.4), so +that they don't add privileges to the tables the import creates. Step 8.5 +compares all four records. Let both instances' service agents use the bucket. Google's export and import pages name `roles/storage.objectAdmin` for both: @@ -569,7 +708,7 @@ for sql_service_agent in \ done ``` -### 5.3 Secret Manager +### 7.3 Secret Manager ```bash for secret_name in "$DATABASE_SECRET_NAME" "$MIGRATION_DATABASE_SECRET_NAME" "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do @@ -578,7 +717,8 @@ done ``` **The two database URLs are new:** the new instance and the new passwords. -First see the form of the old value with its credentials hidden: +First see the form of the old value with everything up to the last `@` +hidden: ```bash gcloud secrets versions access latest --secret "$DATABASE_SECRET_NAME" --project "$OLD_PROJECT_ID" \ @@ -598,8 +738,9 @@ unset runtime_password migration_password ``` **The signing secret and the read token are copied,** so session tokens issued -before the move stay valid afterwards (which also lets the old service keep -serving, choice 7), and the dashboard's credential doesn't change: +before the copy stay valid afterwards, and the dashboard's credential doesn't +change. The dashboard keeps reading the current project's copy of the read +token until section 9: ```bash for secret_name in "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do @@ -608,6 +749,8 @@ for secret_name in "$JWT_SECRET_NAME" "$READ_SECRET_NAME"; do done ``` +Production's older, unused database URL secret (section 3) isn't copied. + Who may read them: ```bash @@ -623,77 +766,28 @@ for secret_name in "$DATABASE_SECRET_NAME" "$MIGRATION_DATABASE_SECRET_NAME" "$J done ``` -The dashboard's access to the read secret is set in section 8.1. - -**If merged by now:** - -- **S1's Hugging Face read token (#13):** create the secret under the name S1 - records, and copy its value with the same `versions access | versions add` - pipe, or rotate it: create a new fine-grained token with the same scope in - Hugging Face, store that, and revoke the old token in section 11. Grant - `roles/secretmanager.secretAccessor` only to the account the worker jobs run - as, and mount it into the worker jobs as `COLLECTOR_HF_READ_TOKEN`. -- **C5's session signing key (#16):** create the secret named by - `TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME` in the new project and copy its - value the same way, so tokens signed before the move verify afterwards and - the key ID published in the JWKS doesn't change. Add it to both grant loops - above: `roles/secretmanager.secretAccessor` for the runtime account and - `roles/secretmanager.viewer` for the deployment account. The deployment - checks that it exists and mounts it as `TELEMETRY_JWT_SIGNING_KEY` in both - environments, so without it that environment's deployment stops at "Check - required Secret Manager entries exist". -- **#12's intake:** in the new project, the Cloud Tasks queue, the worker - job, the recovery job (its three `COLLECTOR_WORKER_*` settings naming the - new queue, worker job and invoker account), the invoker account, the - account Cloud Scheduler starts the recovery job as, and the permissions - from the README's "Artifact intake" section. Create the recovery job's - Cloud Scheduler job only in step 6.7: until the window's deployment, the - new database is empty or holds step 5.5's rehearsal copy, and a recovery - run would dispatch jobs from it. - - **Keep the intake bucket in the old project.** When a submission is - completed, the collector records each stored object's generation number in - the submission's row, and the worker reads every file at that generation. - Cloud Storage assigns generation numbers, and they can't be set, so a copy - in another bucket has different ones, and the worker would reject every - accepted submission as `object_missing`. On the old bucket, grant the new - runtime account `storage.objects.create` (its signed upload URLs create - the objects) and `storage.objects.list` (completion lists them), and the - new worker job's account `storage.objects.get` (it reads each file); the - recovery job doesn't use the bucket. Use the roles step 3.2 recorded for the - old accounts, whether they were granted on the bucket or on the project, - and grant them on the bucket, for example `roles/storage.objectCreator` and - `roles/storage.objectViewer` for the runtime account and - `roles/storage.objectViewer` for the worker's: - - ```bash - gcloud storage buckets add-iam-policy-binding gs://INTAKE_BUCKET --project "$OLD_PROJECT_ID" \ - --member "serviceAccount:$RUNTIME_SERVICE_ACCOUNT" --role ROLE - ``` +**The dashboard's access to the new read secret** (choice 10). Its Workload +Identity provider stays in the current project. Grant the principal (or +principals) that section 3's record lists on the old read secret the same +role on the new one, with the principal copied from the record: + +```bash +gcloud secrets add-iam-policy-binding "$READ_SECRET_NAME" --project "$NEW_PROJECT_ID" \ + --member '' --role roles/secretmanager.secretAccessor +``` + +Production's Vercel identity gets only production's read secret, and Preview's +only staging's. For production, also grant the dashboard's own deployment +identity, the account in calibration-diagnostics' `Production` secret +`TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT`, `roles/secretmanager.viewer` on the +new read secret, for its "Check collector read secret exists" step. These +grants are removed when the dashboard switches to its Vercel OIDC token (C8). - `COLLECTOR_INTAKE_BUCKET` keeps its value. This ties the collector's files - to `policyengine-api`: their storage cost, data-access audit logs and - administration stay there, and shutting that project down, or removing its - billing, makes them unreadable (the memo's D2 B). Moving the bucket later - needs a one-off change that copies the files, checks their SHA-256 values - again and rewrites each submission's recorded generation numbers; none is - written. -- **S2's daily check (#19), production only:** the worker service account - (the one that may read S1's token), with `roles/iam.serviceAccountUser` on - it for the deployment account; `roles/secretmanager.viewer` on S1's secret - for the deployment account; the account Cloud Scheduler starts the check - as; the notification channel and both alert policies, as in the README. - Production's deployment configures the check job itself, before it changes - the service, and stops at that step without S1's secret, the worker - account, or either of the deployment account's two roles. The channel and - the policies don't affect the deployment. Create the schedule in step 6.7, - once the deployment has created the job. - -### 5.4 Create the Cloud Run service once +### 7.4 Create the Cloud Run service once `deploy-maintenance` deploys with `--no-traffic`, which Cloud Run refuses when it creates a service. So create the service once with Google's sample image. -Nothing uses its address yet, and the first deployment replaces the revision. +Nothing reaches it yet, and the first deployment replaces the revision. ```bash gcloud run deploy "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" \ @@ -704,53 +798,69 @@ gcloud run deploy "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" \ The migration job needs no step: `configure-migration-job` runs `gcloud run jobs deploy`, which creates it. -### 5.5 Production only: time the database copy +### 7.5 A backend service for the new service + +Created now, used by the URL map only in step 8.8: + +```bash +gcloud compute network-endpoint-groups create "$NEW_ENDPOINT_GROUP" --region "$REGION" \ + --network-endpoint-type serverless --cloud-run-service "$SERVICE" --project "$NEW_PROJECT_ID" +gcloud compute backend-services create "$NEW_BACKEND" --global \ + --load-balancing-scheme EXTERNAL_MANAGED --project "$NEW_PROJECT_ID" +gcloud compute backend-services add-backend "$NEW_BACKEND" --global \ + --network-endpoint-group "$NEW_ENDPOINT_GROUP" --network-endpoint-group-region "$REGION" \ + --project "$NEW_PROJECT_ID" +``` + +### 7.6 Production only: time the database copy -Before production's window, export the production database with `--offload` +Before production's copy, export the production database with `--offload` (a serverless export, billed separately, which doesn't load the serving -instance), import it into the new instance as in step 6.3, and note both +instance), import it into the new instance as in step 8.3, and note both durations. Then delete and re-create the new database -(`gcloud sql databases delete`, then `create`), so the window's import starts -empty, and repeat step 5.2's "Before any import" steps (privileges, then -owners): deleting the database drops them. Staging's window gives the same +(`gcloud sql databases delete`, then `create`), so the copy's import starts +empty, and repeat step 7.2's "Before any import" steps (privileges, then +owners): deleting the database drops them. Staging's copy gives the same timings for staging's size. -## 6. The window +## 8. C5: copy each environment -Do staging end to end, verify it (section 9), then production. +Staging first, end to end, verified (section 10). Production's copy follows +once C4 holds (section 6). Nothing is scheduled or announced in advance: the +operator starts each copy when ready. From step 8.1 until step 8.8's checks +pass, nothing else merges to this repo's `main`, because every merge +deploys. -**How long it lasts:** from step 6.2 until the deployment in step 6.6 -finishes. Exporting and importing take minutes at today's size (10 GB disks). -Most of the time is the release that step 6.6 starts: the merge's Push run -(versioning), then the release commit's Push run (checks, tag, staging's -deployment, then production's). From merge to tag this took 25 minutes on -2026-10-15 and 88 minutes on 2026-10-19 and 2026-10-20, when the +**How long the service is in maintenance mode:** from step 8.2 until step +8.8's URL map change. Exporting and importing take minutes at today's size +(10 GB disks). Most of the time is the release that step 8.7 starts: the +merge's Push run (versioning), then the release commit's Push run (checks, +tag, staging's deployment, then production's). From merge to tag this took +25 minutes on 2026-10-15 and 88 minutes on 2026-10-19 and 2026-10-20, when the organization's runners were queued; on 2026-10-20, calibration-diagnostics' last collector deploy run took 2 hours 25 minutes in all, of which its staging -and production deploy jobs ran for 4 and 6 minutes. So plan for about 40 +and production deploy jobs ran for 4 and 6 minutes. So expect about 40 minutes on a quiet day (25 minutes to the tag, about 10 for the two deploy -jobs, and the export and import) and 2 to 3 hours when runners are queued, -and measure it in staging's window. If #12's intake is switched on, add the -wait for its worker executions in step 6.2, up to about 3 hours, unless they -are cancelled or the queue is paused early. The dashboard's collector reads -stay down until section 8.1's dashboard deployment finishes, unless the old -service keeps serving (section 8.4). - -While the window lasts: - -- **Builds:** the emitter gets HTTP 503 from the collector and retries, with - a backoff of up to 60 seconds. Events wait on the build machine (kept for - seven days, up to 100 MiB). A machine discarded when its build ends, such as - a Modal container, loses the events not sent by then. A build never fails - because of telemetry. +jobs, and the export and import) and 2 to 3 hours when runners are queued. +Staging's copy measures it. + +While it lasts, requests to that environment, through the hostname or the +default address, get HTTP 503: + +- **Builds** (production only): the emitter retries, with a backoff of up to + 60 seconds. Events wait on the build machine (kept for seven days, up to + 100 MiB). A machine discarded when its build ends, such as a Modal + container, loses the events not sent by then. A build never fails because + of telemetry. - **The dashboard:** the build monitor lists `collector` under "Runs that - could not be read" and still shows the Hugging Face runs. + could not be read" and still shows the Hugging Face runs. Its reads + recover at the URL map change, with no dashboard deployment. - **Orrery uploads:** the emitter gets their credential from the collector's - token exchange, so they wait during the window. They already wait today, - because microcosm-runs' authorization call needs a collector endpoint that - doesn't exist, so nothing changes for them. + token exchange, so they wait. They already wait today, because + microcosm-runs' authorization call needs a collector endpoint that doesn't + exist, so nothing changes for them. -### 6.1 Check that no deployment is running, and record the rollback target +### 8.1 Check that no deployment is running, and record the rollback target ```bash gh run list -R PolicyEngine/microcosm-collector --workflow push.yaml --status in_progress @@ -758,7 +868,7 @@ gh run list -R PolicyEngine/microcosm-collector --workflow push.yaml --status qu ``` Both must list nothing. Then record the revision the old service serves now. -It is the rollback target (section 10). Every merge since step 3.2 has +It is the rollback target (section 11). Every merge since section 3 has redeployed production, possibly with migrations, so an earlier revision may not match the database schema. @@ -769,9 +879,9 @@ rollback_revision=$(gcloud run services describe "$SERVICE" --project "$OLD_PROJ echo "$rollback_revision" ``` -It must print one revision name. Write it in the window's notes too. +It must print one revision name. Write it in the copy's notes too. -### 6.2 Put the old service in maintenance mode +### 8.2 Put the old service in maintenance mode `services update` makes a new revision from the serving one with only `TELEMETRY_MAINTENANCE_MODE` changed, so it keeps the same image, secrets and @@ -782,66 +892,20 @@ gcloud run services update "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REG --update-env-vars TELEMETRY_MAINTENANCE_MODE=1 --no-traffic --tag project-move --quiet gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" \ --to-tags project-move=100 --quiet -old_service_url=$(gcloud run services describe "$SERVICE" --project "$OLD_PROJECT_ID" \ - --region "$REGION" --format='value(status.url)') curl --silent --output /dev/null --write-out '%{http_code}\n' \ - --request POST "$old_service_url/v1/auth/huggingface/exchange" + --request POST "https://$COLLECTOR_HOSTNAME/v1/auth/huggingface/exchange" sleep 35 ``` -The `curl` must print `503`, as `resolve-and-verify-maintenance` checks. The -pause outlasts the 30-second request timeout, so no request to the service is -still writing. +The `curl` must print `503`, as `resolve-and-verify-maintenance` checks, here +through the hostname. The pause outlasts the 30-second request timeout, so no +request to the service is still writing. -**If #12's intake is switched on,** its worker job and its recovery job also -write to the database, and maintenance mode doesn't stop them. In the old -project, pause the queue and the recovery schedule together, then make sure -no execution of either job is still running: - -```bash -gcloud tasks queues pause WORKER_QUEUE --location "$REGION" --project "$OLD_PROJECT_ID" -gcloud scheduler jobs pause RECOVERY_SCHEDULE --location "$REGION" --project "$OLD_PROJECT_ID" -for intake_job in WORKER_JOB RECOVERY_JOB; do - gcloud run jobs executions list --job "$intake_job" --region "$REGION" \ - --project "$OLD_PROJECT_ID" --format=json \ - | jq -r '.[] | select(.status.completionTime == null) | .metadata.name' -done -``` +Intake is off (#12 merged with it off, and its deployment wiring doesn't +exist yet), so nothing else writes to the database. From here the old +database doesn't change. -The loop prints the executions that haven't finished; repeat it until it -prints nothing. It also lists an operator's `reprocess` run made as an -execution of the recovery job. #12's README also allows running `reprocess` -locally through the Cloud SQL Auth Proxy, which the loop can't see, so make -sure nobody runs it that way during the window. - -- **A recovery execution** only updates job rows and creates Cloud Tasks. - Wait for it. -- **A worker execution** can run for about 3 hours: three attempts - (`--max-retries 2`) of up to 1 hour each (`--task-timeout 1h`). Either wait, - which lengthens the window, or cancel it with - `gcloud run jobs executions cancel EXECUTION --region "$REGION" --project "$OLD_PROJECT_ID" --quiet`. - A cancelled attempt records no result. Its job stays `running` in the - copied database until its lease expires, up to 65 minutes after the attempt - started; then the new project's recovery job dispatches it again, or fails - it if it has used its three attempts or three dispatches. -- **To keep the window short,** pause the queue and the schedule up to 3 - hours before it starts, so worker executions finish first (resume both if - the window is called off). Submissions completed in the meantime wait in - the paused queue, and the new project's recovery job dispatches them after - the move (step 6.7). -- **Never pause the queue alone.** Tasks can still be added to a paused - queue, and recovery dispatches a job again each time it has waited 30 - minutes without a claim, and fails it after its third dispatch. With the - queue paused and recovery running, queued jobs fail about 90 minutes after - their first dispatch. - -Uploads in flight need nothing. A signed upload URL (valid for 30 minutes) -names the bucket, which stays where it is (step 5.3), so a build's `complete` -request to the new service finds the files there. - -From here the old database doesn't change. - -### 6.3 Export and import +### 8.3 Export and import ```bash export_object="$EXPORT_BUCKET/$ENVIRONMENT_NAME-$(date -u +%Y%m%dT%H%M%SZ).sql.gz" @@ -851,31 +915,63 @@ gcloud sql import sql "$NEW_INSTANCE" "$export_object" \ --database "$DATABASE_NAME" --user "$MIGRATION_DATABASE_USER" --project "$NEW_PROJECT_ID" --quiet ``` -Import as the role that owns the tables in the step 3.2 record (normally the -migration user), so the restored objects keep their owner. Step 5.2's "Before -any import" steps let it create them in `public`. If #12's intake is -switched on, its bucket isn't copied: it stays in the old project (step 5.3). +Import as the migration user, which owns the tables in section 3's record, so +the restored objects keep their owner. Step 7.2's "Before any import" steps +let it create them in `public`. + +### 8.4 Set the migration user's default privileges + +The migrations grant nothing: the runtime user gets access to every table a +migration creates from the migration user's default privileges (the README's +"Database access" section). Both environments have them today, but an entry +belongs to the role, not to its name, and this one is new. Set them on the new +database now, before step 8.7's deployment runs any migration there. If the +import already carried the entry, running the statement again changes nothing. + +Connected to the new database as the admin user. On PostgreSQL 16 the admin +user needs the `INHERIT` option on the migration user first, so grant it, set +the default privileges, then revoke the grant. With the names from section 2 +in place of the placeholders: + +```sql +GRANT TO WITH INHERIT TRUE; +ALTER DEFAULT PRIVILEGES FOR ROLE IN SCHEMA public + GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO ; +REVOKE FROM ; +``` -### 6.4 Check the copy +The grant needs `ADMIN` on the migration user, which the admin user holds if +it created that user in SQL (its row lists the migration user as `(admin)`). +If it doesn't, connect as the migration user instead, whose password is in +the new `$MIGRATION_DATABASE_SECRET_NAME` secret, and run only the +`ALTER DEFAULT PRIVILEGES` statement. -On both instances, as the migration user: +Then check with `\ddp`: it lists an entry owned by the migration user for +schema `public` and type `table`, with +`=arwd/`, as in section 3's record. -- `SELECT version_num FROM alembic_version;` gives the same revision. -- `SELECT count(*)` on every table in the `public` schema gives the same - counts. At revision `0001` these are `telemetry_runs`, `telemetry_producers` - and `telemetry_events`; later migrations add more. -- `\l`, `\dn+ public`, `\dp public.*` and `\ddp` match the step 3.2 record, - so the database's owner and privileges, `public`'s owner and privileges, - and the table and default privileges all came across. -- The roles query from step 3.2 gives the same rows, `postgres`'s included, - so every role has the same attributes and the same memberships with the - same options. Step 5.2 already compared them before the window; the only - accepted differences are the ones it names for a role it had to make - differently, as written in the window notes. +### 8.5 Check the copy -If anything else differs, stop and roll back (section 10, first row). +On both instances, as the migration user: -### 6.5 Point the GitHub environment at the new project +- `SELECT version_num FROM alembic_version;` gives the same revision + (`0002_artifact_intake` since #12). +- `SELECT count(*)` on every table in the `public` schema gives the same + counts. At revision `0002_artifact_intake` these are `telemetry_runs`, + `telemetry_producers`, `telemetry_events`, `artifact_submissions` and + `artifact_jobs`; the two intake tables are empty while intake is off. +- `\l`, `\dn+ public`, `\dp public.*` and `\ddp` match section 3's record, so + the database's owner and privileges, `public`'s owner and privileges, and + the table and default privileges all came across. +- The roles query from section 3 gives the same rows, the admin user's + included, so every role has the same attributes and the same memberships + with the same options. Step 7.2 already compared them before the copy; the + only accepted differences are the ones it names for a user it had to make + differently, as written in the copy's notes. + +If anything else differs, stop and roll back (section 11, first row). + +### 8.6 Point the GitHub environment at the new project ```bash collector_settings=(--repo PolicyEngine/microcosm-collector --env "$ENVIRONMENT_NAME") @@ -888,43 +984,33 @@ gh secret set TELEMETRY_GCP_WIF_PROVIDER "${collector_settings[@]}" \ gh secret set TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT "${collector_settings[@]}" --body "$DEPLOY_SERVICE_ACCOUNT" ``` -In production's window, also set `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` -to production's new connection in **both** environments, since staging's -`validate-environment-target` compares against it. If #19 has merged, also -set production's `TELEMETRY_WORKER_SERVICE_ACCOUNT` to the new worker account. -The other variables keep their values if the new resources reused the old -names (section 12). - -If #12's intake is switched on, point its settings at the new project, through -whichever environment variables #12's deployment wiring reads (not yet -written): - -- `COLLECTOR_WORKER_QUEUE` and `COLLECTOR_WORKER_JOB`: - `projects/$NEW_PROJECT_ID/locations/$REGION/queues/...` and `.../jobs/...`; -- `COLLECTOR_INTAKE_SIGNING_SERVICE_ACCOUNT`: the new runtime account; -- `COLLECTOR_WORKER_INVOKER_SERVICE_ACCOUNT`: the new invoker account. - -`COLLECTOR_INTAKE_BUCKET` keeps its value (step 5.3). With the old values, the -new service would try to sign uploads as the old runtime account and enqueue -into the old project's paused queue, so intake would stop. +In production's copy, also set `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` to +production's new connection in **both** environments, since staging's +`validate-environment-target` compares against it. The other variables keep +their values, because the new resources reuse the old names (section 14). +If section 16's `TELEMETRY_PUBLIC_ORIGIN` check exists, unset that variable in +this environment now: until step 8.8 the hostname reaches the old service, +whose `/ready` answers 503 in maintenance mode. -### 6.6 Deploy +### 8.7 Deploy -Each window needs its own merge. Its release runs the Push workflow, which +Each copy needs its own merge. Its release runs the Push workflow, which deploys staging and then production with the environments' current settings. +Prepare one PR for each copy in advance (section 16). This repo requires +branches to be up to date with `main`, so update it and let its checks pass +before step 8.1. -- **Staging's window:** merge the C2 code PR (section 13). Production then - redeploys in the old project with unchanged runtime code, from the image - staging built in the new project's store. That works only with the +- **Staging's copy:** staging deploys into the new project. Production then + redeploys in the current project with unchanged runtime code, from the + image staging built in the new project's store. That works only with the Artifact Registry Reader grant in step 4.1; it is otherwise a normal deployment, with its usual short maintenance phase. -- **Production's window:** merge a second PR prepared for it, for example the - README update in section 13. This repo requires branches to be up to date - with `main`, so update it and let its checks pass before the window starts. +- **Production's copy:** staging redeploys in the new project, then + production deploys into it. Re-running the deployment jobs of the latest Push run might do the same without a merge, but whether a re-run reads the environments' changed values -hasn't been checked. In staging's window, confirm it from the run log (the +hasn't been checked. In staging's copy, confirm it from the run log (the "Require deployment configuration" step shows `PROJECT_ID`) before relying on it for production. In production, a re-run that used the old values would take the old service out of maintenance mode, on the old database. @@ -932,416 +1018,261 @@ the old service out of maintenance mode, on the old database. The first deployment replaces the sample revision with a maintenance revision, runs the migration job (the restored database is already at the current revision, and the job checks the schema against the SQLAlchemy metadata), -deploys and checks a candidate revision, and moves traffic to it. In +deploys and checks a candidate revision, moves traffic to it, and checks the +service's default address ("Verify the stable service address"). In production it also takes a backup of the new instance first. -### 6.7 Start the scheduled jobs in the new project - -After the deployment in step 6.6 has finished. - -**#12's intake, if switched on:** create the recovery job's schedule in the -new project, every 10 minutes as the README says, as the account created for -it in step 5.3: - -```bash -gcloud scheduler jobs create http RECOVERY_SCHEDULE --location "$REGION" --project "$NEW_PROJECT_ID" \ - --schedule '*/10 * * * *' \ - --uri "https://run.googleapis.com/v2/projects/$NEW_PROJECT_ID/locations/$REGION/jobs/RECOVERY_JOB:run" \ - --http-method post --oauth-service-account-email RECOVERY_SCHEDULER_ACCOUNT -``` - -Compare its settings with the old schedule's record from step 3.2. Jobs that -the old service queued before step 6.2 are still in the old, paused queue. -The new recovery job dispatches each one again into the new queue once 30 -minutes have passed since its last dispatch, or fails it if it has already -been dispatched three times. The old queue and schedule stay paused until -section 11. - -**S2's daily check (#19), in production's window:** production's deployment -has created the check job in the new project. Create its schedule, then run -the job once by hand and wait for it: - -```bash -gcloud scheduler jobs create http HF_CHECK_SCHEDULE --location "$REGION" --project "$NEW_PROJECT_ID" \ - --schedule '0 6,18 * * *' --time-zone Etc/UTC \ - --uri "https://run.googleapis.com/v2/projects/$NEW_PROJECT_ID/locations/$REGION/jobs/$HF_CHECK_JOB:run" \ - --http-method post --oauth-service-account-email HF_CHECK_SCHEDULER_ACCOUNT -gcloud run jobs execute "$HF_CHECK_JOB" --region "$REGION" --project "$NEW_PROJECT_ID" --wait -``` - -The execution must succeed. The README says a metric-absence condition "is -met only after the job writes at least one data point after the policy is -created", so until this run the new "no successful execution for 23 hours" -policy can't report a stopped schedule. - -Then stop the old project's check, alerts first: - -```bash -for old_alert_policy in OLD_FAILURE_POLICY OLD_ABSENCE_POLICY; do - gcloud monitoring policies update "$old_alert_policy" --no-enabled --project "$OLD_PROJECT_ID" -done -gcloud scheduler jobs pause HF_CHECK_SCHEDULE --location "$REGION" --project "$OLD_PROJECT_ID" -``` - -The two policy names are in the step 3.2 record. Disable the policies before -pausing the schedule, and do both before section 11 removes the old check job -or revokes S1's old token. Otherwise the old project emails false alerts: -the failure policy at the next failed run, and the absence policy 23 hours -after the last successful one. - -## 7. The hostname and DNS - -Records at Cloudflare must be "DNS only". If the zone ever gets CAA records, -they must allow `pki.goog` and `letsencrypt.org`. - -### D3 B: Cloud Run domain mapping - -```bash -gcloud domains verify BASE_DOMAIN # once, by the account that creates mappings -gcloud beta run domain-mappings create --service "$SERVICE" --domain "$COLLECTOR_HOSTNAME" \ - --region "$REGION" --project "$NEW_PROJECT_ID" -gcloud beta run domain-mappings describe --domain "$COLLECTOR_HOSTNAME" --region "$REGION" \ - --project "$NEW_PROJECT_ID" --format='yaml(status.resourceRecords,status.conditions)' -``` - -Add the records `describe` lists at the DNS provider, then wait for the -`CertificateProvisioned` condition to be `True`: "usually about 15 minutes but -can take up to 24 hours." - -- **Both together (choice 6):** nothing uses the hostname yet, so map it any - time after the first deployment, and wait for the certificate before the - emitter release (section 8.2). -- **Hostname first:** step 3.4 already mapped the hostname to the old - project's service. Inside production's window, after the deployment, delete - that mapping, then create the mapping in the new project with the commands - above: - - ```bash - gcloud beta run domain-mappings delete --domain "$COLLECTOR_HOSTNAME" \ - --region "$REGION" --project "$OLD_PROJECT_ID" --quiet - ``` - - The DNS records stay as they are. Requests fail until the new certificate - is issued; the emitter retries. `domain-mappings create --force-override` - maps a domain "even if it is already mapped to another service" (its - `--help`), but neither the help nor the domain mapping page says whether - that includes a service in another project, so don't rely on it. +### 8.8 Send the hostname to the new service -### D3 C: load balancer +After the deployment has finished, change the URL map's default service to +the new project's backend service: ```bash -gcloud compute addresses create collector-address --global --project "$NEW_PROJECT_ID" -gcloud certificate-manager dns-authorizations create collector-hostname \ - --domain "$COLLECTOR_HOSTNAME" --type=PER_PROJECT_RECORD --project "$NEW_PROJECT_ID" -gcloud certificate-manager dns-authorizations describe collector-hostname \ - --project "$NEW_PROJECT_ID" --format='yaml(dnsResourceRecord)' +gcloud compute url-maps set-default-service "$URL_MAP" --global \ + --default-service "$NEW_BACKEND" --project "$NEW_PROJECT_ID" ``` -Add that CNAME at the DNS provider. It authorizes the certificate and moves no -traffic. The authorization is per project (`--type=PER_PROJECT_RECORD`), so the -CNAME's name carries a suffix for the project -(`_acme-challenge_SUFFIX.HOSTNAME`, as `describe` shows) instead of the fixed -`_acme-challenge.HOSTNAME`. In the hostname-first order, step 3.4's -authorization in the old project and this one therefore have separate records -and can both exist, so the new certificate is issued before the `A` record -changes. A fixed-name authorization in both projects would need two different -CNAME records at one name, which DNS doesn't allow. Certificate Manager's DNS -authorization page says per-project authorizations "are self-contained and -don't interact with those in other projects", and that for them it supports -only certificates issued by Google Trust Services (`pki.goog`, which the CAA -note above allows). Then: - -```bash -gcloud certificate-manager certificates create collector-certificate \ - --domains "$COLLECTOR_HOSTNAME" --dns-authorizations collector-hostname --project "$NEW_PROJECT_ID" -gcloud certificate-manager maps create collector-certificates --project "$NEW_PROJECT_ID" -gcloud certificate-manager maps entries create collector-hostname --map collector-certificates \ - --certificates collector-certificate --hostname "$COLLECTOR_HOSTNAME" --project "$NEW_PROJECT_ID" -gcloud compute network-endpoint-groups create collector-endpoints --region "$REGION" \ - --network-endpoint-type serverless --cloud-run-service "$SERVICE" --project "$NEW_PROJECT_ID" -gcloud compute backend-services create collector-backend --global \ - --load-balancing-scheme EXTERNAL_MANAGED --project "$NEW_PROJECT_ID" -gcloud compute backend-services add-backend collector-backend --global \ - --network-endpoint-group collector-endpoints --network-endpoint-group-region "$REGION" \ - --project "$NEW_PROJECT_ID" -gcloud compute url-maps create collector-routes --default-service collector-backend \ - --global --project "$NEW_PROJECT_ID" -gcloud compute target-https-proxies create collector-proxy --url-map collector-routes \ - --certificate-map collector-certificates --global --project "$NEW_PROJECT_ID" -gcloud compute forwarding-rules create collector-https --global \ - --load-balancing-scheme EXTERNAL_MANAGED --address collector-address \ - --target-https-proxy collector-proxy --ports 443 --project "$NEW_PROJECT_ID" -``` +Then run step 5.4's checks again: `/ready` must succeed and the token exchange +must print `401`. A URL map change can take a few minutes to apply +everywhere; until then some requests still reach the old service and get 503, +which the emitter retries. The DNS records don't change. -Wait until -`gcloud certificate-manager certificates describe collector-certificate --project "$NEW_PROJECT_ID" --format='value(managed.state)'` -prints `ACTIVE`. Then point the hostname's `A` record at the address -(`gcloud compute addresses describe collector-address --global --project "$NEW_PROJECT_ID" --format='value(address)'`). -Lower the record's TTL to 300 seconds a day before, so a rollback propagates -quickly. +If section 16's `TELEMETRY_PUBLIC_ORIGIN` check exists, set that variable +again in this environment. -### Check the hostname +**To undo this step** (section 11), point the URL map back: ```bash -curl -fsS "https://$COLLECTOR_HOSTNAME/ready" -curl --silent --output /dev/null --write-out '%{http_code}\n' \ - --request POST "https://$COLLECTOR_HOSTNAME/v1/auth/huggingface/exchange" +gcloud compute url-maps set-default-service "$URL_MAP" --global \ + --default-service "$OLD_BACKEND_PATH" --project "$NEW_PROJECT_ID" ``` -The first must succeed. The second must print `401` (no credential sent), which -shows the request reached the application rather than a maintenance revision. - -## 8. Point the consumers at the new collector +### 8.9 The old service after the copy -### 8.1 The calibration dashboard +It stays in maintenance mode (choice 7), so it answers 503 at its default +address. After production's copy, builds still on that address, on emitter +0.2.1 or on stale machines, keep their events locally and send them through +a later emitter on the same machine; events on discarded machines, such as +Modal containers, are lost. The old service keeps one instance running +(`--min 1`) until section 13 deletes it. -**Give the dashboard's identity access to the new read secrets first** (choice -10): +**From the move on, never run calibration-diagnostics' +`telemetry-service.yml`,** for rollback or anything else. Its settings name +the old project, and its code predates migration `0002_artifact_intake`, so +it can't run against either database. calibration-diagnostics#234 removes +it. -- **Keep its provider in `policyengine-api`:** grant the principal (or - principals) that the step 3.2 record lists on the old read secret the same - role on the new one: - `gcloud secrets add-iam-policy-binding "$READ_SECRET_NAME" --project "$NEW_PROJECT_ID" --member "PRINCIPAL" --role roles/secretmanager.secretAccessor`. - Production's identity gets only production's read secret, and Preview's only - staging's. -- **A new provider in the new project:** create a pool and provider with the - issuer, allowed audiences, attribute mapping and condition recorded in step - 3.2, grant its principals as above, and set the Vercel setting - `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` to the new provider's - resource. -- **C6 done:** nothing; the dashboard no longer reads Secret Manager for the - collector. +## 9. After each copy: the dashboard reads the new project's read secret -The dashboard's deployment checks that its read secret exists ("Check -collector read secret exists" in `deploy.yml`). Grant its own deployment -identity, the account in calibration-diagnostics' `Production` secret -`TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT`, `roles/secretmanager.viewer` on -production's new read secret. - -**Then change the settings:** +In an ordinary dashboard deployment, not during a copy. Until then the +dashboard reads the current project's copy of the read token, which has the +same value. Step 7.3 has already granted the access this needs. -| Setting | Where | New value | -|---|---|---| -| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | calibration-diagnostics GitHub environment `Production` | `https://$COLLECTOR_HOSTNAME` for production | -| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | Vercel project calibration-diagnostics, Production and Preview | production's and staging's addresses | -| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Production and Preview | `projects/$NEW_PROJECT_ID/secrets/READ_SECRET/versions/VERSION`, for production's and staging's read secrets, in the form used today | -| `TELEMETRY_GCP_PROJECT_ID` | calibration-diagnostics GitHub environment `Production` | `$NEW_PROJECT_ID` | -| `TELEMETRY_READ_SECRET_NAME` | calibration-diagnostics GitHub environment `Production` | unchanged if the name was reused | - -Settings take effect on the next deployment. Merge calibration-diagnostics' -next PR, and open a new Preview deployment. Re-running its latest "Deploy -production" run may also work, but whether a re-run reads the changed GitHub -variable is the unchecked question from step 6.6. The build monitor should -then list collector runs from before the move. - -### 8.2 The emitter - -In PolicyEngine/microcosm-emitter, one PR: add `https://COLLECTOR_HOSTNAME` -(production's) to `TRUSTED_COLLECTOR_ORIGINS` and point -`PRODUCTION_COLLECTOR_URL` at it in `src/microcosm_emitter/core/constants.py`. -Keep the old origin in the set while the old address serves. Merging it -publishes the release. - -- Builds that start the separately installed emitter pick it up at their next - start, once it is installed on that machine or in that build image. A - running build keeps the emitter it started with. -- Builds pinned to `microcosm-emitter==0.2.1` keep the old address until - Microcosm moves off that pin. - -### 8.3 microcosm-runs (choice 11) - -Either change `PRODUCTION_COLLECTOR_ORIGIN` in `lib/auth.ts` and the value in -`.env.example`, or leave both for O7, which removes the upload API that uses -them. Its Vercel project sets no `ORRERY_COLLECTOR_ORIGIN`. - -### 8.4 The old service (choice 7) - -- **Keep serving, connected to the new database:** grant the old runtime - account `roles/cloudsql.client` in the new project and - `roles/secretmanager.secretAccessor` on the new runtime database secret. - Then update the old service, which keeps the image it ran before the move: - - ```bash - gcloud run services update "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" \ - --set-cloudsql-instances "$NEW_CLOUD_SQL_CONNECTION" \ - --update-secrets "DATABASE_URL=projects/$NEW_PROJECT_NUMBER/secrets/$DATABASE_SECRET_NAME:latest" \ - --update-env-vars TELEMETRY_MAINTENANCE_MODE=0 --no-traffic --tag new-database --quiet - ``` +| Setting | Where | New value | When | +|---|---|---|---| +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Preview | staging's new read secret version, `projects//secrets//versions/`, naming the project as section 3's recorded value does | after staging's copy | +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Production | the same, for production's read secret | after production's copy | +| `TELEMETRY_GCP_PROJECT_ID` | calibration-diagnostics GitHub environment `Production` (variable) | `` | with Vercel Production | +| `TELEMETRY_READ_SECRET_NAME` | calibration-diagnostics GitHub environment `Production` (variable) | unchanged, because the name was reused | — | - Check `/ready` on the `new-database` tag's address, then route traffic to it - with `update-traffic --to-tags new-database=100`. Retire it before any - migration that its older code can't work with, and as soon as no build uses - emitter 0.2.1. -- **Stay in maintenance mode:** nothing to do. Builds still on the old address - keep their telemetry locally. -- **Delete it:** see section 11. +The read secrets' names are this repo's `TELEMETRY_READ_SECRET_NAME` +variables. Merge calibration-diagnostics' next PR, and open a new Preview +deployment; the build monitor should still list collector runs. +`MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`, `TELEMETRY_GCP_WIF_PROVIDER` +and `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` don't change. To undo, set the +values back to section 3's record. Until this is done, section 13 keeps the +old read secrets. -## 9. Verify +## 10. Verify -For each environment: +For each environment, after its copy: - `gcloud run services describe "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" --format=json` shows an image in the new store, tagged with this repo's release commit (whose message is "Update microcosm-collector"), the new Cloud SQL connection, and - the four secret names (five once #16 has merged). + the four secret names. - `curl -fsS "$(gcloud run services describe "$SERVICE" --project "$NEW_PROJECT_ID" --region "$REGION" --format='value(status.url)')/ready"` - succeeds, and so do the hostname checks in section 7. + succeeds, and so do step 5.4's hostname checks. +- `gcloud compute url-maps describe "$URL_MAP" --global --project "$NEW_PROJECT_ID" --format='value(defaultService)'` + ends with `$NEW_BACKEND`. - `gcloud run jobs executions list --job "$MIGRATION_JOB" --project "$NEW_PROJECT_ID" --region "$REGION" --limit 1` shows a successful execution. - Production: `gcloud sql backups list --instance "$NEW_INSTANCE" --project "$NEW_PROJECT_ID" --limit 1 --format='table(id,windowStartTime,status,description)'` shows `telemetry-release-RUN_ID-ATTEMPT` for the Push run. The default columns leave out the description. -- The row counts from step 6.4 still match. -- After section 8.1: the build monitor lists runs from before the move and - doesn't list `collector` under "Runs that could not be read". -- After section 8.2: the next build that starts the new emitter shows its - events in the build monitor. -- The old service answers as choice 7 says: `503` in maintenance mode, or - `/ready` when kept serving. -- If #19 has merged: step 6.7's execution of the check job succeeded, the - old project's two alert policies are disabled, and its check schedule is - paused. -- If #12's intake is switched on: - `gcloud storage buckets get-iam-policy gs://INTAKE_BUCKET --project "$OLD_PROJECT_ID"` - lists the new runtime and worker accounts with the roles from step 5.3, and - one small test submission, made through the flow in #12's - `docs/artifact-submissions.md` (token exchange for `artifacts:submit`, - create, upload, complete), reaches the status `succeeded` at - `GET /v1/submissions/{submission_id}`. That shows the new runtime account - can sign uploads and list the bucket, the new queue starts the new worker - job, and that job can read the files. The test files stay in the bucket, - which has no lifecycle deletion. - -## 10. Rollback - -Keep every old resource until section 11. The old database is unchanged from -the moment the old service entered maintenance mode. +- The row counts from step 8.5 still match, and `\ddp` still shows step + 8.4's entry. +- The build monitor lists runs from before the copy and doesn't list + `collector` under "Runs that could not be read". +- After production's copy: the next build that starts the C3 emitter shows + its events in the build monitor. +- The old service answers `503` at its default address. -| Situation | What to do | -|---|---| -| In the window, before section 8 | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "$rollback_revision=100"`, with the revision step 6.1 recorded in this window. Set back, to the step 3.2 record, only the GitHub values step 6.5 changed in this window: in staging's window, staging's; in production's window, production's and `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` in staging's environment. Staging's other values stay pointed at the new project, where its database now is. If step 6.2 paused #12's queue and recovery schedule, resume them (`gcloud tasks queues resume`, `gcloud scheduler jobs resume`). If step 6.7 ran, pause the schedules it created in the new project, with the new queue, and disable the new alert policies; for #19, resume the old check schedule, enable the old policies again (`--enabled`), then run the old check job once by hand and wait for it to succeed: `gcloud run jobs execute "$HF_CHECK_JOB" --region "$REGION" --project "$OLD_PROJECT_ID" --wait`. #19's README says the absence condition is met only after a successful run that follows the policy being "created or changed", and enabling it again changes it. Builds kept their events and deliver them to the old service, except builds on a machine discarded when the build ended, such as a Modal container, which lost the events not sent by then. | -| A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The new service isn't in use yet, so the first row also applies. | -| After the dashboard switched (8.1), before the emitter release (8.2) | Builds still use only the old address. If the old service was in maintenance mode the whole time, nothing new is in the new database: do the first row, then set the dashboard's settings back and redeploy it. If the old service was kept serving against the new database (8.4), new events are in the new database: do the next row. | -| After the emitter release | New events are in the new database. Put the new service in maintenance mode (step 6.2's commands with the new project), export the new database, import it into a fresh database on the old instance (drop and re-create the old database first, or import under a new name and update the old runtime database secret), route the old service back to the revision step 6.1 recorded in production's window (if a release with a migration has deployed in the new project since, that revision's code is older than the imported schema: set the GitHub values back first, and let the next merge deploy the current release to the old project instead), and set the GitHub and dashboard settings back. Then reverse step 6.7: in the new project, pause #19's check schedule and disable the new alert policies (step 6.2's commands have already paused #12's queue and recovery schedule there); in the old project, resume #12's queue and recovery schedule, resume #19's check schedule, enable the old alert policies again and run the old check job once, as in the first row. Last, release the emitter with the old `PRODUCTION_COLLECTOR_URL` (the old origin is still trusted, so only that constant changes). Builds that started on the new default keep it until they finish, and keep their events locally while the new service is in maintenance mode. | -| The hostname | D3 B: delete the new project's mapping; in the hostname-first order, then create the mapping in the old project again (section 7's commands with `--project "$OLD_PROJECT_ID"`) and wait for its certificate. D3 C: point the `A` record back to what it named before, if anything. | +## 11. Rollback -## 11. Retire the old resources +Keep every old resource until section 13. The old database is unchanged from +the moment the old service entered maintenance mode. Rollback never changes +the DNS records or the dashboard's `MICROCOSM_TELEMETRY_COLLECTOR_URL`. -Only when no build uses the old address (Microcosm has left the 0.2.1 pin, and -the old service's request count in Cloud Monitoring has stayed at zero), and -after an agreed holding period: +| Situation | What to do | +|---|---| +| During a copy, before step 8.8's URL map change | Route the old service back: `gcloud run services update-traffic "$SERVICE" --project "$OLD_PROJECT_ID" --region "$REGION" --to-revisions "$rollback_revision=100"`, with the revision step 8.1 recorded in this copy. Set back, to section 3's record, only the GitHub values step 8.6 changed in this copy: in staging's copy, staging's; in production's copy, production's and `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` in staging's environment. Staging's other values stay pointed at the new project, where its database now is. The hostname still reaches the old service, so builds and the dashboard recover with no other change. Builds kept their events and deliver them, except builds on a machine discarded when the build ended, such as a Modal container. | +| A deployment into the new project fails | As in calibration-diagnostics#233: before the migration step, fix and re-run the failed jobs. The URL map hasn't changed, so the first row also applies. | +| After step 8.8's URL map change | New events are in the new database, so prefer fixing forward. To go back: put the new service in maintenance mode (step 8.2's commands with `--project "$NEW_PROJECT_ID"`); export the new database; on the old instance, delete and re-create the old database, repeat step 7.2's "Before any import" steps there, import, and repeat step 8.4 there (deleting a database drops its default privileges); route the old service back to the revision step 8.1 recorded; set the GitHub values back as in the first row; then point the URL map back to `$OLD_BACKEND_PATH` (step 8.8's undo command). If a release with a migration has deployed in the new project since, the recorded revision's code is older than the imported schema: set the GitHub values back first, and let the next merge deploy the current release to the current project instead. If section 9 has moved the dashboard's read secret setting, it can stay: both projects' read tokens have the same value. | + +Whatever the situation, never use calibration-diagnostics' +`telemetry-service.yml` (step 8.9). + +## 12. After the move + +Once production's copy is verified, in this order. Each creates its cloud +resources once, in the new project, and needs nothing in the current project. + +1. **The collector's own Hugging Face token and its daily check (S1, S2; + #13, #18, #19).** Before merging each PR, create in the new project what + its README section names. For S1: the secret, readable only by the + account the worker jobs run as, which they read as + `COLLECTOR_HF_READ_TOKEN`. For + S2: production's check job, its Cloud Scheduler job and the account it + starts the job as, a notification channel and two Cloud Monitoring alert + policies, and its production variables (`TELEMETRY_WORKER_SERVICE_ACCOUNT`, + `TELEMETRY_HF_CHECK_JOB`, `TELEMETRY_HF_READ_SECRET_NAME`). Enable + `cloudscheduler.googleapis.com` and `monitoring.googleapis.com`. +2. **Switching intake on.** Everything in the README's "Artifact intake" + section: the bucket, the Cloud Tasks queue, the worker and recovery jobs, + the recovery job's schedule, the accounts and permissions, and the + deployment wiring that sets `COLLECTOR_INTAKE_ENABLED`. Enable + `cloudtasks.googleapis.com`. The runtime user's access to the intake + tables comes from the default privileges (step 8.4). +3. **Read API v2 (C6).** +4. **ES256 session-token signing (C7, #16).** Before merging it, create each + environment's signing-key secret, named by + `TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME`, readable by the runtime account + and visible to the deployment account (`roles/secretmanager.viewer`). The + deployment checks that it exists and mounts it as + `TELEMETRY_JWT_SIGNING_KEY`; without it, that environment's deployment + stops at "Check required Secret Manager entries exist". + +## 13. Retire the old resources + +When no build uses emitter 0.2.1 (Microcosm's `main` no longer pins it) and +the old production service's request count in Cloud Monitoring +(`run.googleapis.com/request_count`) has been zero for seven days, the +emitter's local retention period (choice 7). Then, in the current project +unless stated: - The old Cloud SQL instances: export a last copy into the bucket if wanted, then `gcloud sql instances patch "$OLD_INSTANCE" --project "$OLD_PROJECT_ID" --no-deletion-protection` and `gcloud sql instances delete`. -- The old Cloud Run services and migration jobs, every secret step 3.2 - recorded (the four per environment, the unused ones such as production's - older database URL secret, and #16's signing key and S1's token if merged), - and the old runtime, migration and worker service accounts. -- If #12 merged: the queue (with any tasks still in it), the worker and - recovery jobs, the recovery schedule and the account it used, the invoker - account, and the old accounts' roles on the intake bucket, or on the - project if step 3.2 found them there. Keep the bucket: it holds every - submission's files and is still in use (step 5.3). So `policyengine-api` - has to stay active, and billed, for as long as the collector reads them. -- If #19 merged: the old check job, its schedule and the account the - schedule used, both alert policies and the notification channel. Step 6.7 - already disabled the policies and paused the schedule. +- The old Cloud Run services and migration jobs, and the old runtime and + migration service accounts. +- Every collector secret section 3 recorded: the four per environment, and + production's older, unused database URL secret. Delete the old read secrets + only once section 9 has moved the dashboard to the new ones. +- Each environment's backend service and serverless network endpoint group + from step 5.1, and the operator's `roles/compute.loadBalancerServiceUser` + grant on them. After this, the URL map can no longer be pointed back. - The collector's images in the old Artifact Registry store (only the collector's image name, unless the store holds nothing else). - The two `roles/artifactregistry.reader` bindings on the new project from - step 4.1 (the old deployment account and the old project's Cloud Run service - agent). -- This repo's Workload Identity provider in `policyengine-api` + step 4.1 (the old deployment account and the current project's Cloud Run + service agent). +- This repo's Workload Identity provider in the current project (`github/microcosm-collector`) and the old deployment account's `roles/iam.workloadIdentityUser` binding for this repo's principal set. Keep calibration-diagnostics' provider and the shared deployment account: - the dashboard's deployment still uses them. -- In the hostname-first order with D3 C, the load balancer's resources that - step 3.4 created in `policyengine-api`, once the hostname points at the new - project. (With D3 B, section 7 already deleted the old project's mapping.) -- The export bucket and every dump in it. + the dashboard's deployment still uses them. Keep the dashboard's Vercel + identity provider until C8. +- The export bucket and every dump in it (in the new project). - calibration-diagnostics' leftover collector settings, as listed in calibration-diagnostics#233's "Manual steps". -- S1's old Hugging Face token, if it was rotated. If #19 merged, revoke it - only after step 6.7 has paused the old check, which uses it. +- In a later emitter release, the old default address from + `TRUSTED_COLLECTOR_ORIGINS`. -## 12. Settings this runbook changes +## 14. Settings this runbook changes -| Setting | Provider | Read by | After the move | +| Setting | Provider | Read by | Changes | |---|---|---|---| -| `TELEMETRY_GCP_PROJECT_ID` | this repo's GitHub environments `staging` and `production` (variable) | `push.yaml` job `env:` as `PROJECT_ID`, then every `scripts/deploy.sh` command | the new project ID | -| `TELEMETRY_CLOUD_SQL_CONNECTION` | same (variable) | `CLOUD_SQL_CONNECTION`: `--add-cloudsql-instances`, `--set-cloudsql-instances`, `create-database-backup` | `NEW_PROJECT_ID:REGION:INSTANCE` | -| `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` | same, in both environments (variable) | `validate-environment-target` | production's new connection | -| `TELEMETRY_RUNTIME_SERVICE_ACCOUNT`, `TELEMETRY_MIGRATION_SERVICE_ACCOUNT` | same (variables) | `--service-account` for the service and the migration job | the new accounts | -| `TELEMETRY_GCP_REGION`, `TELEMETRY_ARTIFACT_REGISTRY_NAME`, `TELEMETRY_IMAGE_NAME`, `TELEMETRY_SERVICE`, `TELEMETRY_PRODUCTION_SERVICE`, `TELEMETRY_MIGRATION_JOB`, and the four `TELEMETRY_*_SECRET_NAME` variables | same (variables) | image path, Cloud Run names, `--set-secrets`, `check-secrets-exist` | unchanged when the new resources reuse the names | -| `TELEMETRY_GCP_WIF_PROVIDER` | same (secret) | `google-github-actions/auth`, `workload_identity_provider` | the new provider's resource | -| `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | same (secret) | `google-github-actions/auth`, `service_account` | the new deployment account | +| `TELEMETRY_GCP_PROJECT_ID` | this repo's GitHub environments `staging` and `production` (variable) | `push.yaml` job `env:` as `PROJECT_ID`, then every `scripts/deploy.sh` command | the new project ID, in step 8.6 | +| `TELEMETRY_CLOUD_SQL_CONNECTION` | same (variable) | `CLOUD_SQL_CONNECTION`: `--add-cloudsql-instances`, `--set-cloudsql-instances`, `create-database-backup` | the new connection, `::`, in step 8.6 | +| `TELEMETRY_PRODUCTION_CLOUD_SQL_CONNECTION` | same, in both environments (variable) | `validate-environment-target` | production's new connection, in production's copy | +| `TELEMETRY_RUNTIME_SERVICE_ACCOUNT`, `TELEMETRY_MIGRATION_SERVICE_ACCOUNT` | same (variables) | `--service-account` for the service and the migration job | the new accounts, in step 8.6 | +| `TELEMETRY_GCP_REGION`, `TELEMETRY_ARTIFACT_REGISTRY_NAME`, `TELEMETRY_IMAGE_NAME`, `TELEMETRY_SERVICE`, `TELEMETRY_PRODUCTION_SERVICE`, `TELEMETRY_MIGRATION_JOB`, and the four `TELEMETRY_*_SECRET_NAME` variables | same (variables) | image path, Cloud Run names, `--set-secrets`, `check-secrets-exist` | unchanged, because the new resources reuse the names | +| `TELEMETRY_GCP_WIF_PROVIDER` | same (secret) | `google-github-actions/auth`, `workload_identity_provider` | the new provider's resource, in step 8.6 | +| `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | same (secret) | `google-github-actions/auth`, `service_account` | the new deployment account, in step 8.6 | +| `TELEMETRY_PUBLIC_ORIGIN` (only once section 16's check exists) | same (variable) | the proposed deployment check | `https://` after step 5.4; unset in step 8.6 and set again in step 8.8 | | `COLLECTOR_DEPLOY_ENABLED` | this repo's variable | the deployment jobs' `if:` | unchanged (`true`) | -| `roles/artifactregistry.reader` on the new project for the old deployment account and the old project's Cloud Run service agent | Google Cloud IAM (step 4.1) | Cloud Run, when `deploy-production` deploys staging's image into the old project | removed in section 11 | -| `TELEMETRY_JWT_SIGNING_KEY_SECRET_NAME` (if #16 merged) | same, in both environments (variable) | `check-secrets-exist`, `--set-secrets` | unchanged when the new secret reuses the name | -| `TELEMETRY_WORKER_SERVICE_ACCOUNT` (if #19 merged) | `production` (variable) | `configure-huggingface-check-job`, `--service-account` | the new worker account | -| `TELEMETRY_HF_CHECK_JOB`, `TELEMETRY_HF_READ_SECRET_NAME` (if #19 merged) | `production` (variables) | `configure-huggingface-check-job` | unchanged when the new resources reuse the names | -| `COLLECTOR_WORKER_QUEUE`, `COLLECTOR_WORKER_JOB` (if #12's intake is switched on) | the environment variables #12's deployment wiring will read (not yet written); set by hand on the recovery job | the service and the recovery job | the new project's queue and worker job | -| `COLLECTOR_INTAKE_SIGNING_SERVICE_ACCOUNT`, `COLLECTOR_WORKER_INVOKER_SERVICE_ACCOUNT` (same) | same | the service signs upload URLs as the first; the service and the recovery job dispatch as the second | the new runtime account and the new invoker account | -| `COLLECTOR_INTAKE_BUCKET` (same) | same | the service and the worker job | unchanged: the bucket stays in the old project (step 5.3) | +| `roles/artifactregistry.reader` on the new project for the old deployment account and the current project's Cloud Run service agent | Google Cloud IAM (step 4.1) | Cloud Run, when `deploy-production` deploys staging's image into the current project | removed in section 13 | | `DATABASE_URL`, `TELEMETRY_JWT_SECRET`, `TELEMETRY_READ_TOKEN` | Google Secret Manager in the new project | the service, through `--set-secrets` | new database URL; copied signing secret and read token | -| `TELEMETRY_JWT_SIGNING_KEY` (if #16 merged) | Secret Manager in the new project | the service, through `--set-secrets` | copied key | | `DATABASE_URL` (migration job) | Secret Manager in the new project | the migration job, through `--set-secrets` | new migration database URL | -| `COLLECTOR_HF_READ_TOKEN` (if S1 merged) | Secret Manager in the new project | the worker jobs, through a secret mount | copied or rotated | -| `TELEMETRY_MAINTENANCE_MODE` | `scripts/deploy.sh` (`--set-env-vars`), and step 6.2 on the old service | the service | unchanged | -| Dashboard settings | calibration-diagnostics GitHub `Production` environment and the Vercel project | see section 8.1 | see section 8.1 | -| `PRODUCTION_COLLECTOR_URL`, `TRUSTED_COLLECTOR_ORIGINS` | emitter source (`core/constants.py`) | every emitter service module | the hostname, by an emitter release | - -**Unverified before the staging window.** Apart from some read-only -commands against `policyengine-api`, nothing in this runbook has been run. -Still open: - -- **Whether `postgres` may change a user Cloud SQL made** with `ALTER ROLE`, - which PostgreSQL 16 allows only with `ADMIN OPTION` on that user. Step - 5.2's check on staging's new instance settles it, and gives the fallback. +| `TELEMETRY_MAINTENANCE_MODE` | `scripts/deploy.sh` (`--set-env-vars`), and step 8.2 on the old service | the service | unchanged by deployments; `1` on the old service from step 8.2 | +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | calibration-diagnostics GitHub `Production` environment (variable), and the Vercel project's Production and Preview settings | `deploy.yml` and `deploy-dashboard.sh` (production); Vercel (Preview); `telemetry-collector-client.ts` | the hostnames, in step 5.5 | +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel project, Production and Preview | `telemetry-collector-client.ts` | the new project's read secrets, in section 9 | +| `TELEMETRY_GCP_PROJECT_ID`, `TELEMETRY_READ_SECRET_NAME` (dashboard) | calibration-diagnostics GitHub `Production` environment (variables) | `deploy.yml` "Check collector read secret exists" | the new project, in section 9; the name is unchanged | +| `PRODUCTION_COLLECTOR_URL`, `TRUSTED_COLLECTOR_ORIGINS` | emitter source (`core/constants.py`) | every emitter service module | the production hostname, by the C3 release | + +Settings that section 12's work adds (S1, S2, intake, C7) are set up then, in +the new project only. + +## 15. Unverified before staging's work + +Apart from some read-only commands against the current project, nothing in +this runbook has been run. Still open: + +- **The cross-project backend service with a Cloud Run backend.** Google's + overview says cross-project service referencing works with serverless + network endpoint groups and needs no Shared VPC for global external + Application Load Balancers, but its step-by-step cross-project example uses + an instance group. Staging's load balancer (section 5) tests it first. If + it fails, staging's and production's hostname phase would need the whole + load balancer in the current project, and each move a change of the `A` + record: a change to these decisions, to agree before going further. +- **Organization policies:** the new project inherits the organization's + policies. In the current project, which sits directly under the + organization, both collector services grant `roles/run.invoker` to + `allUsers`, as `--allow-unauthenticated` needs, so the organization + probably doesn't enforce domain-restricted sharing. Whether it restricts + cross-project backend services wasn't checked: the Organization Policy API + isn't enabled in the current project. If the new project refuses + `allUsers`, step 8.7's deployment can't reach its maintenance revision. +- **VPC Service Controls:** whether either project is inside a service + perimeter, which could block the URL map's use of the current project's + backend service (step 5.2), and the current project's image reads from the + new Artifact Registry store (step 4.1). Checking needs organization-level + access. +- **Whether the admin user may change a user Cloud SQL made** with + `ALTER ROLE`, which PostgreSQL 16 allows only with `ADMIN OPTION` on that + user. Step 7.2's check settles it, if a recorded user needs it. - **Database owners:** who owns a database that `gcloud sql databases create` makes, and the old databases' owners, `public` privileges, roles, grants and - memberships (including whether production's `telemetry_app` owns or holds - anything). These need a database connection; step 3.2 records the old ones, - and step 5.2 compares the new database before any import. -- **VPC Service Controls:** whether either project is inside a service - perimeter, which could block access between them: the new accounts' access - to the old intake bucket (step 5.3), and the old project's image reads from - the new Artifact Registry store (step 4.1). Checking needs - organization-level access. -- **Organization policies:** the new project inherits the policies of the - parent chosen in choice 2. In `policyengine-api`, which sits directly under - an organization, both collector services grant `roles/run.invoker` to - `allUsers`, as `--allow-unauthenticated` needs. Whether the chosen parent - sets a different policy, such as domain-restricted sharing, wasn't checked: - the Organization Policy API isn't enabled in `policyengine-api`. If the new - project refuses `allUsers`, step 6.6's deployment can't reach its - maintenance revision. -- **Project liens:** whether `policyengine-api` has a lien that stops it being - deleted, which matters if the intake bucket stays there. Listing liens needs - `gcloud alpha resource-manager liens list`. -- **#12's deployment wiring,** not yet written: its GitHub variable names - (step 6.5, this table), and whether it grants storage roles on the bucket or - on the project (step 3.2). + memberships. These need a database connection; section 3 records the old + ones, and step 7.2 compares the new database before any import. +- **Whether the export carries the default privileges.** Step 8.4 sets them + either way. +- **How long a URL map change takes** to reach every Google front end (step + 8.8); staging's copy measures it. - The current deployment account's roles and the dashboard identity's bindings - (recorded in step 3.2), and whether a workflow re-run reads changed - environment values (step 6.6). + (recorded in section 3), and whether a workflow re-run reads changed + environment values (step 8.7). -Every setting above needs checking end to end in the staging pass. +Every setting above needs checking end to end in staging's pass. -## 13. Code changes that go with this runbook (not yet written) +## 16. Code changes that go with this runbook (not yet written) -- **This repo:** a step after "Verify the stable service address" that checks - `https://$PUBLIC_ORIGIN/ready`, from a new environment variable +- **This repo, optional:** a step after "Verify the stable service address" + that checks `https://$PUBLIC_ORIGIN/ready`, from a new environment variable `TELEMETRY_PUBLIC_ORIGIN`, so every deployment proves the hostname reaches - the new revision. The step skips when the variable is unset: in the "both - together" order the hostname reaches the new service only after section 7, - so set the variable once section 7's checks pass. Its merge is the release - that deploys staging into the new project (step 6.6). -- **This repo, for production's window:** the README's "Cloud Run deployment" + the deployed service. The step skips when the variable is unset. It can be + set once step 5.4 passes, because the hostname then reaches the service + this repo deploys; steps 8.6 and 8.8 unset and set it around each copy. + Its merge can be the one staging's copy needs (step 8.7). +- **This repo, for production's copy:** the README's "Cloud Run deployment" and "Google Cloud access" sections updated to describe the new project and - the hostname. Its merge is the release that deploys production into the new - project (step 6.6). -- **The emitter:** the release in section 8.2. -- **microcosm-runs:** section 8.3, if chosen. + the load balancer, without naming the hostname. Its merge is the one + production's copy needs (step 8.7). +- **The emitter:** C3 (section 6), and later the removal of the old default + address from `TRUSTED_COLLECTOR_ORIGINS` (section 13). +- **calibration-diagnostics#234:** removes `telemetry-service.yml` (step + 8.9). +- **microcosm-runs:** nothing until O7 and O8 remove the code that uses the + old address (choice 11). From 7a2b3482e9d051db61b310481d19fce9c6c04140 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 27 Oct 2026 19:00:39 +0400 Subject: [PATCH 31/31] Say where each placeholder's value is kept, and check the rollback revision The memo says where the hostnames and the domain are kept, and states that the copy recreates only the migration and runtime users. The runbook sets the new project's number right after creating the project (step 4.2's binding needs it), reads the operator's account from gcloud's configuration, and in step 8.1 checks that the serving revision is the latest created one, since step 8.2's update builds the maintenance revision from the latest created revision's settings. Co-Authored-By: Claude Opus 5.5 --- docs/project-and-domain-decision.md | 10 +++++---- docs/project-move-runbook.md | 32 ++++++++++++++++++----------- 2 files changed, 26 insertions(+), 16 deletions(-) diff --git a/docs/project-and-domain-decision.md b/docs/project-and-domain-decision.md index 2db1769..8bacb2e 100644 --- a/docs/project-and-domain-decision.md +++ b/docs/project-and-domain-decision.md @@ -28,7 +28,7 @@ version of this memo). | 1. D2, the project | **B: a dedicated project,** holding both environments, each moved in one database copy. | | 2. The project's details | It uses the **same organization and billing account as the current project.** It has **one owner.** Its ID (``) is chosen when it is created and recorded by the operator; after each environment's copy it is also the value of that environment's `TELEMETRY_GCP_PROJECT_ID` variable. | | 3. D3, the address | **C: a global external Application Load Balancer,** with a Google-managed certificate from Certificate Manager. | -| 4. Hostnames | **One load balancer and one hostname per environment,** `` and ``, both under a PolicyEngine-owned domain (``). | +| 4. Hostnames | **One load balancer and one hostname per environment,** `` and ``, both under a PolicyEngine-owned domain (``, the domain both hostnames end with). The operator records the hostnames; from C2 they are also the values of the dashboard's `MICROCOSM_TELEMETRY_COLLECTOR_URL` settings, and from C3 the production hostname is in the emitter's `PRODUCTION_COLLECTOR_URL`. | | 5. DNS | The domain's DNS is **managed outside Google Cloud and edited by hand by Anthony.** Each environment needs two records: an `A` record for its load balancer's static IP, and the `CNAME` for its Certificate Manager DNS authorization. The runbook lists them (step 5.3). | | 6. Order | **Hostname first.** Each environment's load balancer is built in the new project from the start. Until that environment moves, it sends requests across projects to today's service, through a backend service and a serverless network endpoint group left in the current project. Clients switch to the hostname; each environment's move is then one change to its load balancer's URL map. | | 7. The old address | **Maintenance mode after production's copy.** The old production service is deleted once no build uses emitter 0.2.1 and it has had no requests for seven days. | @@ -280,9 +280,11 @@ The plan's constraints: project after the move. - The migration user's default privileges give the runtime user access to every table a migration creates; the migrations grant nothing (#12). Both - environments have them today. The copy recreates the database users, so the - runbook sets them again on each new instance before any migration runs - there. + environments have them today. The copy recreates only two database users, + the migration user and the runtime user (production's unused third user was + dropped, and Cloud SQL makes its admin user on every new instance), so the + runbook sets the default privileges again on each new instance before any + migration runs there. In order: diff --git a/docs/project-move-runbook.md b/docs/project-move-runbook.md index f15fe13..1ec58a3 100644 --- a/docs/project-move-runbook.md +++ b/docs/project-move-runbook.md @@ -75,8 +75,8 @@ OLD_PROJECT_ID='' # TELEMETRY_GCP_PROJECT_ID in both envi OLD_PROJECT_NUMBER=$(gcloud projects describe "$OLD_PROJECT_ID" --format='value(projectNumber)') OLD_DEPLOY_SERVICE_ACCOUNT='' # the TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT secret; found in step 3 NEW_PROJECT_ID='' # chosen in section 4; recorded by the operator -NEW_PROJECT_NUMBER=$(gcloud projects describe "$NEW_PROJECT_ID" --format='value(projectNumber)') # after section 4 creates it -OPERATOR_ACCOUNT='' # the account that runs these steps +NEW_PROJECT_NUMBER=$(gcloud projects describe "$NEW_PROJECT_ID" --format='value(projectNumber)') # empty until section 4 creates the project, which sets it again +OPERATOR_ACCOUNT=$(gcloud config get-value account) # the Google account gcloud runs these steps as ENVIRONMENT_NAME=staging # then production COLLECTOR_HOSTNAME='' # '' for production; recorded by the operator, and in the dashboard's MICROCOSM_TELEMETRY_COLLECTOR_URL after step 5.5 @@ -98,7 +98,7 @@ READ_SECRET_NAME=$(collector_variable TELEMETRY_READ_SECRET_NAME) DATABASE_NAME='' MIGRATION_DATABASE_USER='' RUNTIME_DATABASE_USER='' -ADMIN_DATABASE_USER='' # Cloud SQL's built-in admin user +ADMIN_DATABASE_USER='' # the admin user Cloud SQL makes on every instance; `gcloud sql users list` lists it # Names are scoped to a project, so the new resources can reuse the old ones. NEW_INSTANCE=$OLD_INSTANCE @@ -215,8 +215,9 @@ both from the current project, so neither value is typed: gcloud projects describe "$OLD_PROJECT_ID" --format='value(parent.type)' # must print organization ORGANIZATION_ID=$(gcloud projects describe "$OLD_PROJECT_ID" --format='value(parent.id)') BILLING_ACCOUNT=$(gcloud billing projects describe "$OLD_PROJECT_ID" --format='value(billingAccountName)') -gcloud projects create "$NEW_PROJECT_ID" --name '' --organization "$ORGANIZATION_ID" +gcloud projects create "$NEW_PROJECT_ID" --name '' --organization "$ORGANIZATION_ID" # the console's name for it, chosen by the operator gcloud billing projects link "$NEW_PROJECT_ID" --billing-account "${BILLING_ACCOUNT#billingAccounts/}" +NEW_PROJECT_NUMBER=$(gcloud projects describe "$NEW_PROJECT_ID" --format='value(projectNumber)') # step 4.2 needs it gcloud services enable run.googleapis.com sqladmin.googleapis.com \ secretmanager.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \ iamcredentials.googleapis.com sts.googleapis.com storage.googleapis.com \ @@ -401,7 +402,8 @@ Google Cloud. Four in all, two per environment: them. A proxy would end TLS for requests that carry Hugging Face tokens, and would hide the `CNAME` from Google. - **Names:** if the provider's form adds the domain itself, enter only the - part of each name before `.`. A trailing dot at the + part of each name before `.`, the domain both + hostnames end with. A trailing dot at the end of a printed name marks it as complete; leave it out if the form doesn't accept it. - **Both records at once:** nothing uses the hostname until step 5.5, so the @@ -873,17 +875,23 @@ redeployed production, possibly with migrations, so an earlier revision may not match the database schema. ```bash -rollback_revision=$(gcloud run services describe "$SERVICE" --project "$OLD_PROJECT_ID" \ - --region "$REGION" --format=json \ - | jq -r '.status.traffic[] | select((.percent // 0) > 0) | .revisionName') +old_service_json=$(gcloud run services describe "$SERVICE" --project "$OLD_PROJECT_ID" \ + --region "$REGION" --format=json) +rollback_revision=$(jq -r '.status.traffic[] | select((.percent // 0) > 0) | .revisionName' \ + <<< "$old_service_json") echo "$rollback_revision" +jq -r '.status.latestCreatedRevisionName' <<< "$old_service_json" ``` -It must print one revision name. Write it in the copy's notes too. +Both must print the same one revision name. If they differ, the last +deployment didn't finish (its maintenance or candidate revision is newer +than the serving one): resolve that before going on. Write the revision in +the copy's notes too. ### 8.2 Put the old service in maintenance mode -`services update` makes a new revision from the serving one with only +`services update` makes a new revision from the latest created revision's +settings, which step 8.1 checked are the serving revision's, with only `TELEMETRY_MAINTENANCE_MODE` changed, so it keeps the same image, secrets and database: @@ -1070,7 +1078,7 @@ same value. Step 7.3 has already granted the access this needs. | Setting | Where | New value | When | |---|---|---|---| -| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Preview | staging's new read secret version, `projects//secrets//versions/`, naming the project as section 3's recorded value does | after staging's copy | +| `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Preview | staging's new read secret version, `projects//secrets//versions/`, naming the project as section 3's recorded value does (by ID or number), with the version step 7.3 added (`gcloud secrets versions list` shows it) | after staging's copy | | `MICROCOSM_TELEMETRY_READ_SECRET_VERSION` | Vercel, Production | the same, for production's read secret | after production's copy | | `TELEMETRY_GCP_PROJECT_ID` | calibration-diagnostics GitHub environment `Production` (variable) | `` | with Vercel Production | | `TELEMETRY_READ_SECRET_NAME` | calibration-diagnostics GitHub environment `Production` (variable) | unchanged, because the name was reused | — | @@ -1200,7 +1208,7 @@ unless stated: | `TELEMETRY_GCP_REGION`, `TELEMETRY_ARTIFACT_REGISTRY_NAME`, `TELEMETRY_IMAGE_NAME`, `TELEMETRY_SERVICE`, `TELEMETRY_PRODUCTION_SERVICE`, `TELEMETRY_MIGRATION_JOB`, and the four `TELEMETRY_*_SECRET_NAME` variables | same (variables) | image path, Cloud Run names, `--set-secrets`, `check-secrets-exist` | unchanged, because the new resources reuse the names | | `TELEMETRY_GCP_WIF_PROVIDER` | same (secret) | `google-github-actions/auth`, `workload_identity_provider` | the new provider's resource, in step 8.6 | | `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | same (secret) | `google-github-actions/auth`, `service_account` | the new deployment account, in step 8.6 | -| `TELEMETRY_PUBLIC_ORIGIN` (only once section 16's check exists) | same (variable) | the proposed deployment check | `https://` after step 5.4; unset in step 8.6 and set again in step 8.8 | +| `TELEMETRY_PUBLIC_ORIGIN` (only once section 16's check exists) | same (variable) | the proposed deployment check | `https://` after step 5.4; unset in step 8.6 and set again in step 8.8 | | `COLLECTOR_DEPLOY_ENABLED` | this repo's variable | the deployment jobs' `if:` | unchanged (`true`) | | `roles/artifactregistry.reader` on the new project for the old deployment account and the current project's Cloud Run service agent | Google Cloud IAM (step 4.1) | Cloud Run, when `deploy-production` deploys staging's image into the current project | removed in section 13 | | `DATABASE_URL`, `TELEMETRY_JWT_SECRET`, `TELEMETRY_READ_TOKEN` | Google Secret Manager in the new project | the service, through `--set-secrets` | new database URL; copied signing secret and read token |