diff --git a/frontend/HOSTED_RUNTIME.md b/frontend/HOSTED_RUNTIME.md index d177d505..4655e9d4 100644 --- a/frontend/HOSTED_RUNTIME.md +++ b/frontend/HOSTED_RUNTIME.md @@ -1,28 +1,53 @@ -# Hosted variable runtime - -The Next.js dashboard proxies variable requests to a private Modal service. The service -executes `scripts/microcosm_variable_core.py`, also used by the local command-line tool, -and verifies the installed .py version and the model versions it pins, both release -manifests, the data certification and the H5 bytes before constructing a native -PolicyEngine microsimulation. Weighted results use MicroSeries. The root evaluation -environment and its recorded outputs remain separate. - -The runtime uses Python 3.12 and the hashed `../backend/requirements.lock`. Regenerate -that lock with `uv pip compile backend/requirements.txt --python-version 3.12 ---generate-hashes --output-file backend/requirements.lock` from the repository root. +# Hosted dashboard and variable runtime + +The calibration dashboard is a Next.js application hosted on Vercel. Its variable +lookup, `/api/microcosm_variable`, forwards each request to a private FastAPI service on +Modal (`backend/modal_app.py`, `backend/app.py`). That service runs +`frontend/scripts/microcosm_variable_core.py`, which the local command-line tool also +uses. Before it constructs a native PolicyEngine microsimulation, it verifies the +installed .py version and the model versions that .py pins, both release manifests, +.py's data certification and the H5's SHA-256. Weighted results use MicroSeries. The +root evaluation environment (`pyproject.toml`, `uv.lock`) and its recorded outputs are +separate from this runtime. + +The dashboard server also reads: + +- Hugging Face, for calibration releases and staging runs + ([Hugging Face reads](#hugging-face-reads)); +- the hosted Microcosm telemetry collector, for build progress + ([Reading the telemetry collector](#reading-the-telemetry-collector)); +- private Vercel Blob storage, for the published calibration tree and comparison + artifacts (see [calibration tree artifacts](../docs/calibration-tree-artifacts.md)). + The dashboard's Blob reads pass no credential, so `@vercel/blob` uses the + deployment's Vercel OIDC token with `BLOB_STORE_ID` when both are present, and + `BLOB_READ_WRITE_TOKEN` otherwise. The comparison route + (`frontend/app/api/microcosm/comparison-tree/route.ts`) writes a missing comparison + artifact with `BLOB_READ_WRITE_TOKEN`. + +The Modal runtime uses Python 3.12 and the hashed `backend/requirements.lock`. +Regenerate that lock from the repository root with `uv pip compile +backend/requirements.txt --python-version 3.12 --generate-hashes --output-file +backend/requirements.lock`. The Modal image also installs Debian's current +`ca-certificates` package. When the Next.js server starts, it removes two root +certificates from Node's default list (`frontend/instrumentation.ts`): the compromised +eMudhra root "emSign ECC Root CA - G3", and Sectigo's "AAA Certificate Services", +which cross-signed the affected intermediates. ## Model and data identity -.py (the PyPI package `policyengine`) is the model identity, and -`backend/requirements.txt` declares only `policyengine[us]==VERSION`. **Upgrade by +.py (the PyPI package `policyengine`) is the model identity. `backend/requirements.txt` +pins it as `policyengine[us]==6.7.5` and pins no model package itself. **Upgrade by bumping .py; the dataset must be one that version of .py certifies.** Never pin -`policyengine-us`, `policyengine-core` or `spm-calculator` directly; a test fails if a -bare pin reappears. `scripts/hosted_release.json` records the certifying .py version -in `certifying_package` and the reviewed data identity: repository, immutable commit -`hf_revision`, release ID, H5 filename and SHA-256, data year, and the canonical -digests of both release manifests. +`policyengine-us`, `policyengine-core` or `spm-calculator` directly; +`frontend/tests/test_model_dependency_pins.py` fails if a bare pin reappears. +`frontend/scripts/hosted_release.json` records the certifying .py version in +`certifying_package`, the versions that .py's `us` extra pins in `packages` +(policyengine-us 2.2.1, policyengine-core 3.33.0, spm-calculator 1.0.0), and the +reviewed data identity: repository, immutable commit `hf_revision`, release ID, H5 +filename and SHA-256, data year, and the canonical digests of both release manifests. +The Next.js proxy reads the same file (`frontend/lib/microcosm/production-release.ts`). -`scripts/hosted_release.py` enforces the identity in two places: +`frontend/scripts/hosted_release.py` enforces the identity in two places: - `validate_runtime` requires the installed .py to be the reviewed version, derives the exact `policyengine-us`, `policyengine-core` and `spm-calculator` versions from @@ -36,139 +61,293 @@ digests of both release manifests. The release's own built-with Core version and its build manifest's `runtime` block are not the identity: .py is. For the same reason the build manifest's exported-household -count is checked by `verify_hosted_runtime.py --dataset-file` only when the -.py-certified build records one. +count is checked by `frontend/scripts/verify_hosted_runtime.py --dataset-file` only when +the .py-certified build records one. -The reviewed release is `populace-us-2024-spm-20260915`, the US dataset .py 6.4.1 -certifies. It is a source-enrichment release of +The reviewed release is `populace-us-2024-spm-20260915`, the US dataset that .py 6.7.5 +certifies. .py 6.4.1 certified the same release, and 6.7.5's `us` extra pins the same +three package versions. It is a source-enrichment release of `populace-us-2024-buildp-sparse-rmloss100-cae8640-20260728T011454Z`: the same tables and calibration plus `is_spm_independent_minor_role`, which `policyengine-us` 2.x requires to calculate SPM thresholds. Its parent manifests are the Build P manifests previously reviewed here. -To upgrade: +### Stored-input renames + +The reviewed H5 stores WIC take-up as `would_claim_wic`. policyengine-us 2.2.1 renamed +that input `takes_up_wic_if_eligible` (default `True`), and it sets a stored column as +an input only when it defines a variable of that name, so without a mapping every +WIC-eligible person takes WIC up. After building each simulation, the hosted runtime +(`native_simulation` in `microcosm_variable_core.py`) and the evaluation harness +(`evaluation_harness/adapters/microcosm.py`) call .py's +`policyengine.tax_benefit_models.us.legacy_inputs.apply_legacy_input_renames_to_microsimulation`, +which sets each renamed input from its stored column. Every calculation response +records the renames applied, per geographic scope, in `execution.legacy_input_renames`. +`verify_hosted_runtime.py`, which the Modal image runs while it is built, records the +function it binds and the renames .py reports for the pinned model. + +The hosted runtime calls that function directly instead of .py's +`managed_microsimulation`, which accepts only a dataset path, builds every +economic-assumption year, and cannot take the in-memory state subset. + +### Upgrading .py 1. Bump `policyengine[us]` in `backend/requirements.txt` and regenerate the lock. 2. Read the new .py's `data_releases.us` bundle entry, and set `hosted_release.json`'s data identity and `packages` to what that .py certifies and pins. -3. Refresh the manifest fixtures under `tests/fixtures/`. -4. Regenerate `public/variable-catalog.json` in the hosted environment with - `python frontend/scripts/generate_variable_catalog.py`. It builds the catalog from a - simulation over the reviewed H5, verified by SHA-256 (`--dataset-file` reuses a - local copy), never from the country package's default dataset, and records the +3. Refresh the manifest fixtures under `frontend/tests/fixtures/`. +4. Check the stored-input renames the new .py reports for the pinned model. + `frontend/tests/test_hosted_runtime_verification.py` expects exactly + `would_claim_wic` to `takes_up_wic_if_eligible`. +5. Regenerate `frontend/public/variable-catalog.json` in the hosted-runtime environment + with `python frontend/scripts/generate_variable_catalog.py`. It builds the catalog + from a simulation over the reviewed H5, verified by SHA-256 (`--dataset-file` reuses + a local copy), never from the country package's default dataset, and records the release and package versions it used. -5. Qualify the deployment with the checks below. +6. Qualify the deployment with the [required calculation checks](#required-calculation-checks). ## Automated production deployment -`.github/workflows/deploy.yml` runs after the `CI` workflow succeeds for a push to -`main`. Its trigger accepts only CI runs whose head branch is `main`, and its deploy -job runs only when that CI run was a successful push. The deploy job, not the -workflow, holds the `calibration-diagnostics-production` concurrency group: one deploy -runs, at most one waits, and a newer `main` deploy replaces a waiting one. A run whose -job is skipped, such as one for a pull request from a fork whose branch is named -`main`, must not replace a waiting deploy (#260). The deploy job checks out the exact -commit tested by CI and performs these operations in order: - -1. Deploy a commit-specific Modal application in the `main` Modal environment. -2. Resolve the deployed `web_app` URL and verify its authenticated metadata. -3. Build a production-targeted Vercel deployment with `--skip-domain`, using that Modal - URL and the same source commit. -4. Verify the unaliased frontend metadata, then execute a real 2024 national threshold - calculation followed by a California income-tax calculation. Promotion requires the - reviewed package and data identities, verified H5 bytes, numeric results, and the - expected national-to-state cache transition. If the connection is refused, closes, - resets or times out before a complete response arrives, the check waits until the - backend answers a metadata request (at most 860 seconds after the first request), - then sends the calculation once more; the retried response must meet the same - requirements. A DNS or TLS-certificate failure, an HTTP error, or a response that - fails them is never retried, so a slow calculation still fails: the proxy answers - HTTP 504 after waiting 780 seconds. -5. Promote the Vercel deployment and execute a California calculation through both - `calibration-diagnostics.vercel.app` and the mounted `microcosm.institute` route. - -The workflow names each backend `calibration-diagnostics-<12-character-commit>` so an -older frontend continues to reference its matching backend during rollback. Retain these -applications for the rollback window; remove them separately after they are no longer -referenced by a deployable frontend. +`.github/workflows/deploy.yml` ("Deploy production") is the only production deployment +path. It starts on `workflow_run` for the `CI` workflow, filtered to CI runs whose head +branch is `main`. Its one job runs only when that CI run concluded `success`, was a +`push` and had head branch `main`, so pull-request CI runs never deploy, including one +from a fork whose branch is named `main`. CI runs for every pull request and every push +to `main`, and a newer run for the same pull request or branch cancels an unfinished +one. When two +merges arrive close together, the earlier commit's CI run can therefore be cancelled, +and only the later commit deploys. + +The deploy job, not the workflow, holds the `calibration-diagnostics-production` +concurrency group: one deploy runs, at most one waits, and a newer `main` deploy +replaces a waiting one. A run whose job is skipped never replaces a waiting deploy +(#260). The job runs in the GitHub `Production` environment, which had no required +reviewers or branch policy when read on 2026-10-26, with a 45-minute limit and +permission to request a GitHub OIDC token. It checks out the exact commit CI tested and +performs these steps in order: + +1. Require the Modal, Vercel and Hugging Face credentials and the collector address + (`.github/scripts/deploy-dashboard.sh require-configuration`). A missing one stops + the job before anything is deployed. +2. Authenticate to Google Cloud through GitHub OIDC and Workload Identity Federation, + then confirm that the collector's read secret exists with a metadata-only + `gcloud secrets describe` call. The step never reads the secret's value, and it does + not show that the dashboard's Vercel identity can read the secret. +3. Install Modal 1.5.5 into `.deploy-venv` and Vercel CLI 58.8.0. +4. Deploy the Modal application `calibration-diagnostics-` in the `main` Modal environment. +5. Resolve that application's `web_app` URL, which must be an `https://…modal.run` + address and is masked in the log, and verify its authenticated metadata. +6. Build a production-targeted Vercel deployment from the repository root with + `--skip-domain`, passing the Modal URL, the Modal proxy credentials, the Hugging Face + read token, the collector address and the source commit as `--env` values. +7. Verify the unaliased deployment's metadata through the mounted route + `/calibration/dashboard/api/microcosm_variable`, then send one calculation request + for the national `spm_unit_spm_threshold` and the California `ca_income_tax` in 2024. + Promotion requires the reviewed package and data identities, verified H5 bytes, + finite weighted results, California input for the state result, and both scopes + built fresh, with the national simulation evicted before the state one. If the + connection is refused, closes, resets or times out before a complete response + arrives, the check waits until the backend answers a metadata request (at most 860 + seconds after the first request), then sends the calculation once more; the retried + response must meet the same requirements. A DNS or TLS-certificate failure, an HTTP + error, or a response that fails them is never retried, so a slow calculation still + fails: the proxy answers HTTP 504 after waiting 780 seconds. The metadata checks and + the post-promotion California checks instead make up to six attempts, pausing 5, 10, + 15, 20 and 25 seconds between them, and retry any failure. +8. Install the frontend workspace (`bun install --frozen-lockfile --ignore-scripts`) + and run the calibration release check against the candidate + ([Promotion and rollback](#promotion-and-rollback)). +9. Promote the Vercel deployment with `vercel promote`. +10. Send a California calculation through `calibration-diagnostics.vercel.app` and + through the `microcosm.institute/calibration/dashboard` mount, then run the + calibration release check against `calibration-diagnostics.vercel.app`. + +The workflow names each backend after its commit so an older frontend continues to +reference its matching backend during rollback. Retain these applications for the +rollback window, and remove them separately once no deployable frontend references +them. Nothing removes them automatically: on 2026-10-26 the `main` Modal environment +held 30 deployed `calibration-diagnostics-` applications, plus an older one +named `calibration-diagnostics`. The workflow is fixed to Vercel project `calibration-diagnostics` (`prj_pL7dIJJ3M4hGcKr5pttWaOACVFtu`) in the PolicyEngine team -(`team_xsyTmFLMLGbHH7Qxu70R5G4r`). `vercel.json` disables automatic Git deployment for +(`team_xsyTmFLMLGbHH7Qxu70R5G4r`) through `VERCEL_PROJECT_ID`, `VERCEL_ORG_ID` and +`--scope policy-engine`. `frontend/vercel.json` turns off Vercel's Git deployment for `main` only, so pull-request previews remain enabled. Do not relink the workflow to a different project or remove `--skip-domain`; the frontend must not receive production traffic before its paired backend passes the deployment checks. -Run the Vercel CLI from the repository root. The Vercel project already configures -`frontend` as its root directory, so setting the CLI working directory to `frontend` -would make Vercel look for a nonexistent `frontend/frontend` directory. - -Configure these GitHub Actions secrets before merging the workflow: - -- `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`: deployment credentials for the PolicyEngine - Modal workspace and its `main` environment. -- `MICROCOSM_MODAL_KEY` and `MICROCOSM_MODAL_SECRET`: credentials accepted by the - proxy-authenticated Modal function and passed to the Vercel server runtime. -- `VERCEL_TOKEN`: permission to deploy and promote the fixed PolicyEngine Vercel project. -- `VERCEL_AUTOMATION_BYPASS_SECRET`: permission for the workflow to call the protected, - unaliased Vercel deployment during verification. -- `HF_RELEASE_READ_TOKEN`: a dedicated fine-grained Hugging Face read-only token scoped - to the registered private release repositories and `policyengine/populace-us-staging`. - Keep the publisher's separate `HF_TOKEN` secret unchanged. - -Missing credentials stop the workflow before it creates a Modal application. The Vercel -project's existing production environment variables, including Blob credentials, remain -managed by Vercel; the workflow supplies the backend URL, proxy credentials, release read -token, and source commit for the candidate deployment. -## Preview deployment +Run the Vercel CLI from the repository root. The Vercel project sets `frontend` as its +root directory (read 2026-10-26), so a CLI working directory of `frontend` would make +Vercel look for a nonexistent `frontend/frontend` directory. + +### Deployment settings + +The workflow reads these settings. Every stored secret and variable named here was +present when listed on 2026-10-26; no value was read. + +| Setting | Provider | Where it is read | +| --- | --- | --- | +| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | PolicyEngine organization Actions secrets | The Modal deploy and URL-resolution steps. They let the Modal client deploy to the PolicyEngine workspace's `main` environment. | +| `MICROCOSM_MODAL_KEY`, `MICROCOSM_MODAL_SECRET` | `Production` environment secrets | The backend metadata check, and `vercel deploy --env` for the candidate, whose variable route sends them as the `Modal-Key` and `Modal-Secret` headers. They are a Modal proxy token, which the `requires_proxy_auth` web function requires. | +| `VERCEL_TOKEN` | `Production` environment secret | `deploy-dashboard.sh deploy-frontend` and `vercel promote`. | +| `VERCEL_AUTOMATION_BYPASS_SECRET` | `Production` environment secret | The checks of the unaliased candidate, which send it as `x-vercel-protection-bypass` because Vercel protects generated deployment URLs. | +| `HF_RELEASE_READ_TOKEN` | Repository Actions secret | `vercel deploy --env`, as the candidate's runtime `HF_TOKEN` ([Hugging Face reads](#hugging-face-reads)). | +| `MICROCOSM_TELEMETRY_COLLECTOR_URL` | `Production` environment variable | `vercel deploy --env`, for the candidate's collector client. | +| `TELEMETRY_GCP_WIF_PROVIDER`, `TELEMETRY_GCP_DEPLOY_SERVICE_ACCOUNT` | `Production` environment secrets | `google-github-actions/auth`, used only by the read-secret existence check. The service account is the collector's deploy service account. | +| `TELEMETRY_GCP_PROJECT_ID`, `TELEMETRY_READ_SECRET_NAME` | `Production` environment variables | The read-secret existence check, as `PROJECT_ID` and `READ_SECRET_NAME`. | +| `CALIBRATION_MODAL_APP_NAME` | Output of the backend-name step: `calibration-diagnostics-` | The Modal deploy step, where `backend/modal_app.py` names the application and records the name in the image, and the URL-resolution step. | +| `MODAL_ENVIRONMENT`, `VERCEL_CLI_VERSION`, `VERCEL_ORG_ID`, `VERCEL_PROJECT_ID` | Fixed values in the workflow's top-level `env:` | The Modal deploy and URL-resolution steps (`main`), every Vercel CLI call (`58.8.0`), and the Vercel CLI's choice of team and project. | +| `MICROCOSM_CALCULATION_URL` | Output of the URL-resolution step | `vercel deploy --env`, for the candidate's variable route. | +| `MICROCOSM_BACKEND_SOURCE_COMMIT` | The commit CI tested | `vercel deploy --env`. The variable route compares the backend's packaged commit with `VERCEL_GIT_COMMIT_SHA` when Vercel sets it, and with this value otherwise, and answers 409 when they differ. | + +The candidate also uses settings stored in the Vercel project's Production +environment, which the workflow neither supplies nor changes: +`MICROCOSM_TELEMETRY_READ_SECRET_VERSION` and +`MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER` (the collector), +`POPULACE_UK_STAGING_HF_TOKEN` (UK staging), `BLOB_STORE_ID` and +`BLOB_READ_WRITE_TOKEN` (Blob storage), `HF_WEBHOOK_SECRET` and +`GITHUB_ACTIONS_DISPATCH_TOKEN` (the Hugging Face webhook), +`SLACK_WEBHOOK_POPULACE_US` and `SLACK_WEBHOOK_POPULACE_UK`, and +`CROSS_DATASET_ARTIFACT_BASE_URL` and `CROSS_DATASET_ARTIFACT_BASE_URL_BE`. Each name +was listed on 2026-10-26. The project also stores its own Production `HF_TOKEN` and +`MICROCOSM_TELEMETRY_COLLECTOR_URL`, both of which the workflow also passes with +`--env`; which of the two values a deployment uses was not verified for either. Vercel +provides `VERCEL=1` and the deployment's OIDC token itself. + +## Reading the telemetry collector + +The Microcosm telemetry collector runs on Google Cloud Run. Its source, tests and +deployment are in +[PolicyEngine/microcosm-collector](https://github.com/PolicyEngine/microcosm-collector), +which has deployed it since 2026-10-21. A merge to this repository's `main` deploys only +the dashboard and its Modal backend, never the collector. The dashboard only reads the +collector, from server code (`frontend/lib/server/microcosm/telemetry-collector-client.ts`), +with three server-only settings: + +- `MICROCOSM_TELEMETRY_COLLECTOR_URL`: the collector address. It must use HTTPS, except + on `localhost` or `127.0.0.1`. The workflow passes production deployments the GitHub + `Production` environment variable through `vercel deploy --env`, and the Vercel + project also stores a Production value; which one a deployment uses was not + verified. Previews get it from the Vercel project's Preview settings. +- `MICROCOSM_TELEMETRY_READ_SECRET_VERSION`: the full Google Secret Manager version + resource that holds the collector's read credential, + `projects//secrets//versions/`. It is a Vercel + project setting, present in both the Production and the Preview environment. +- `MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER`: the full Google Workload Identity + provider resource that trusts this Vercel project. It is a Vercel project setting, + present in both environments, and is read only on Vercel. + +Production should name the production collector and its read secret, and Preview the +staging collector and its read secret. Only the names were listed (2026-10-26), so +which collector each environment's values name is not verified here. + +On Vercel (`VERCEL=1`), `frontend/lib/server/secret-manager.ts` exchanges the +deployment's Vercel OIDC token (`@vercel/oidc`) for a Google access token through that +provider, reads the secret version from the Secret Manager API, and keeps the value in +memory for up to one minute. Locally it uses Google Application Default Credentials and +does not read the provider setting. The server sends the credential only to the +collector, as the `X-Telemetry-Read-Token` header; it never reaches the browser and +never passes through GitHub Actions. Unless both the address and the secret version +are set, the build monitor shows only the Hugging Face staging history. If a collector +request fails, the build monitor still shows the Hugging Face runs and reports +`collector` among the runs that could not be read. + +The Vercel project has OIDC turned on with the team issuer (read 2026-10-26). The +Google side, the provider's trust in that issuer and audience and each identity's +access to its own read secret, was provisioned outside this repository and is not +verified here. [docs/build-monitor.md](../docs/build-monitor.md) describes the build +monitor that shows these runs. + +## Hugging Face reads + +- **The Modal backend** downloads the reviewed release's manifests and H5 from the + public dataset repository `policyengine/populace-us` at the commit pinned in + `hosted_release.json`, without a token: the Modal function has no Modal secret + attached. The repository answered an unauthenticated request on 2026-10-26. If it + became private, the backend would need a token, which this code does not provide. +- **The dashboard server** reads calibration releases and staging runs with `HF_TOKEN`, + or `HUGGINGFACE_TOKEN` when `HF_TOKEN` is unset. UK staging is the exception: it reads + only `POPULACE_UK_STAGING_HF_TOKEN`, a Vercel project setting. For production + deployments the workflow sets `HF_TOKEN` from the repository secret + `HF_RELEASE_READ_TOKEN`. That must be a dedicated fine-grained read-only token for the + registered private release repositories (`policyengine/populace-uk-private`, + `policyengine/populace-be-private`) and the private US staging repository + `policyengine/populace-us-staging`. Never reuse the publisher's `HF_TOKEN` Actions + secret, which `.github/workflows/publish-calibration-tree.yml` uses, for the + dashboard. Previews use the Vercel project's Preview `HF_TOKEN`. The tokens' scopes + cannot be read from GitHub or Vercel, so they are not verified here. + +Each registered country's release and staging repositories can be overridden on the +dashboard server by the settings its registration names (`repo_env` and `revision_env` +in `frontend/packages/calibration-tree/src/countries.ts`, such as `POPULACE_HF_REPO` +for the US). None of these overrides was set in the Vercel project on 2026-10-26. -Install the runtime lock into a separate environment, run the hosted Python tests and -`verify_hosted_runtime.py`, then run the frontend tests, typecheck and build. Commit the -reviewed source before building the Modal image: its deployment definition rejects dirty -backend/shared-runtime files and records the actual Git commit and tree digest. +## Preview deployment -Use the PolicyEngine Modal workspace and an explicit task-owned application name: +Vercel's Git integration builds a preview of every pull request. A preview reads its +settings from the Vercel project's Preview environment, which on 2026-10-26 held +`MICROCOSM_CALCULATION_URL`, `MICROCOSM_MODAL_KEY`, `MICROCOSM_MODAL_SECRET`, `HF_TOKEN`, +`POPULACE_UK_STAGING_HF_TOKEN`, `BLOB_STORE_ID`, `BLOB_READ_WRITE_TOKEN`, the three +collector settings and `VERCEL_SUPPORT_LARGE_FUNCTIONS`. A Git preview's +`VERCEL_GIT_COMMIT_SHA` is its own commit, so its variable lookups succeed only when the +configured backend was deployed from that same commit; otherwise the proxy answers 409. +The other dashboard pages do not use the backend. + +To qualify a change to the hosted runtime on a preview, install the runtime lock into a +separate environment, run the hosted Python tests and `verify_hosted_runtime.py`, then +run the frontend tests, typecheck and build. Commit the reviewed source before building +the Modal image: `backend/modal_app.py` refuses uncommitted or untracked files under +`backend/` and `frontend/scripts/`, and records the actual Git commit and a digest of +those directories' Git tree. + +Use the PolicyEngine Modal workspace and an explicit task-owned application name. +`backend/modal_app.py` fails without `CALIBRATION_MODAL_APP_NAME`: ```bash CALIBRATION_MODAL_APP_NAME=calibration-diagnostics-preview-REVIEW_ID \ uv run --no-sync modal deploy backend/modal_app.py --env testing ``` -The service reserves 16 GiB and sets the same hard limit, requests two CPU cores, and -serializes requests (`max_inputs=1`, one container). It scales to zero. Its working disk -holds the normal HF download; there is no Vercel RAM-backed HDF5 workaround. Observe the -actual deployed allocation and response resource measurements rather than assuming the +The service requests two CPU cores and reserves 16 GiB with the same hard limit. It +runs at most one container, which serves one request at a time (`max_containers=1`, +`max_inputs=1`), stops a call after 800 seconds, and scales to zero after 60 idle +seconds. Its working disk holds the normal Hugging Face download. Observe the actual +deployed allocation and response resource measurements rather than assuming the requested allocation is sufficient. Under policyengine-us 1.764.6 the local SPM reference peaked at approximately 12 GB, and -an 8 GiB hosted allocation did not complete the calculation. Under .py 6.4.1 -(policyengine-us 2.2.1, spm-calculator 1.0.0), a local national `spm_unit_spm_threshold` -calculation, then `ca_income_tax`, then the national threshold again, peaked at about -1 GB resident memory (macOS, 2026-10-16). Keep the 16 GiB allocation until a hosted -measurement under the new tuple is recorded. The required checks below must still pass. +an 8 GiB hosted allocation did not complete the calculation. Under .py 6.4.1, whose +`us` extra pins the same policyengine-us 2.2.1, policyengine-core 3.33.0 and +spm-calculator 1.0.0 as 6.7.5, a local national `spm_unit_spm_threshold` calculation, +then `ca_income_tax`, then the national threshold again, peaked at about 1 GB resident +memory (macOS, 2026-10-16). This document records no hosted measurement for that +package set, so keep the 16 GiB allocation until one is recorded. The required checks +below must still pass. The service loads only the reviewed 2024 input. It validates the native dataset year and wraps that `USSingleYearDataset` in a one-entry `USMultiYearDataset` before constructing -the simulation. This compatibility step prevents the pinned country package from -automatically allocating copies for later economic-assumption years: policyengine-us -2.2.1's `Microsimulation` still extends a single-year dataset through every -economic-assumption year and does not let the caller set the end year. Remove the -wrapper only after upgrading to a country-package version that does, and qualify that -upgrade with the same calculation checks. +the simulation. This prevents the pinned country package from automatically allocating +copies for later economic-assumption years: policyengine-us 2.2.1's `Microsimulation` +still extends a single-year dataset through every economic-assumption year and does not +let the caller set the end year. Remove the wrapper only after upgrading to a +country-package version that does, and qualify that upgrade with the same calculation +checks. Only one native simulation stays resident: changing between national and state scopes evicts the preceding simulation before loading the next input tables. Same-scope requests can reuse it; mixed variable lists may rebuild when their scopes change. -Create a dedicated Modal proxy token using `modal workspace proxy-tokens` (Modal 1.5.5 -or later). Store its ID and secret through `agent-secret`; do not print or put them in -source. Where the workspace supports RBAC, allow only the intended Modal environment. -Otherwise the token has workspace-wide proxy scope: keep it server-only and delete the -task token after qualification. Set these **server-only** Vercel preview variables: +Create a dedicated Modal proxy token with `modal workspace proxy-tokens` (Modal 1.5.5 +or later). Keep its ID and secret out of source and logs. Where the workspace supports +RBAC, allow only the intended Modal environment. Otherwise the token has workspace-wide +proxy scope: keep it server-only and delete the task token after qualification. Set +these **server-only** Vercel Preview settings: - `MICROCOSM_CALCULATION_URL`: the actual returned Modal app URL. -- `MICROCOSM_MODAL_KEY` and `MICROCOSM_MODAL_SECRET`: proxy credentials. +- `MICROCOSM_MODAL_KEY` and `MICROCOSM_MODAL_SECRET`: the proxy credentials. -The Next deployment's `VERCEL_GIT_COMMIT_SHA` must match the backend's packaged commit. For local development against a remote backend, set `MICROCOSM_BACKEND_SOURCE_COMMIT` -explicitly. Without a remote URL, local calculations still use the shared Python CLI. +explicitly. Without a remote URL, and outside Vercel, local calculations run +`frontend/scripts/microcosm_variable_value.py` with the local Python. ## Required calculation checks @@ -176,7 +355,7 @@ explicitly. Without a remote URL, local calculations still use the shared Python Next metadata routes return JSON 200 with the expected source, model and configuration. With a nonempty base path, the former root `/api/microcosm_variable` URL first returns a same-origin 307 to the mounted Next route; preserve that redirect in the receipt. - Metadata is not proof that data loaded or a calculation succeeded. It does now refuse a + Metadata is not proof that data loaded or a calculation succeeded. It does refuse a conflicting deployment data selection: the backend validates its resolved `POPULACE_HF_REPO`/`POPULACE_HF_REVISION` against the reviewed release and answers 503, and the proxy compares the response's `environment_configuration` with the reviewed @@ -186,7 +365,7 @@ explicitly. Without a remote URL, local calculations still use the shared Python on the pinned release through the exact Next preview. Require JSON 200, verified H5 SHA-256, exact model/source identity, `execution.simulation_cache_hits.national=false` and a non-null weighted result. Require comparison with a separately executed reference - using the identical source, package tuple and immutable H5. An older Core version's + using the identical source, package versions and immutable H5. An older Core version's results are not a substitute, and an unavailable reference leaves this check incomplete. 3. Preserve the request, response hash, execution ID, source/deployment identities, allocation, peak memory and elapsed time. Exercise a warm request separately and label @@ -200,7 +379,7 @@ explicitly. Without a remote URL, local calculations still use the shared Python flow before promotion; do not hide timeouts with retries or stale outputs. 5. After a national request, execute a real state variable such as `ca_income_tax`. Require recorded national-cache eviction, numeric output with the correct state scope, - and comparison with the same-tuple local reference. Exercise the reverse transition + and comparison with the same-package local reference. Exercise the reverse transition or a mixed-scope request as well; national-only success does not qualify state lookup. ## Promotion and rollback @@ -208,61 +387,69 @@ explicitly. Without a remote URL, local calculations still use the shared Python Obtain independent review and passing CI on the exact candidate. `frontend/vercel.json` prevents Vercel's Git integration from deploying `main`; the backend-first GitHub workflow is the only production deployment path. Keep the project -setting `autoAssignCustomDomains=false` as an additional protection against unintended -domain assignment, and read it back when changing deployment configuration. -Inspect the production-scoped environment entries before the build and require -`POPULACE_HF_REPO` and `POPULACE_HF_REVISION` to be absent for the variable-calculation -service. Its US data selection is compiled from the reviewed release, and a conflicting -runtime override is refused by that service. Dashboard release discovery is independent: -it follows the registered repository's `main` branch through the dashboard manifest. -If either variable is present, record and remove only those two conflicting calculation -overrides, with their prior settings retained for rollback. This specific check also -applies to preview qualification. Do not print or change unrelated environment entries. +setting `autoAssignCustomDomains=false` (it was `false` when read on 2026-10-26) as an +additional protection against unintended domain assignment, and read it back when +changing deployment configuration. + +The backend reads `POPULACE_HF_REPO` and `POPULACE_HF_REVISION` from its own Modal +environment. The Modal function has no Modal secret attached and its image sets neither, +so both are unset and the backend uses the reviewed values; its 503 refusal guards +against a later change that sets them. The variable proxy on Vercel uses the checked-in +identity and never forwards either setting to the backend. On Vercel, `POPULACE_HF_REPO` +changes which repository the dashboard's US release discovery reads, and which +repository's Hugging Face webhook events start a US calibration tree publication +(`frontend/lib/microcosm/hf-webhook-config.ts`). Release discovery follows the +repository's `main` branch through the dashboard manifest. Neither setting was present +in the Vercel project's Production or Preview environment on 2026-10-26. If either +appears, record and remove only that entry, retaining its prior setting for rollback. +Do not print or change unrelated environment entries. The production workflow deploys the matching private backend, stages the exact frontend -with production server-only URL and credentials, verifies metadata and a genuine -calculation, and only then runs `vercel promote`. After promotion, it checks a genuine -calculation through `calibration-diagnostics.vercel.app` and the -`microcosm.institute/calibration/dashboard` mount. A failure before promotion leaves the +with production server-only URL and credentials, verifies metadata, a genuine +calculation and the calibration releases, and only then runs `vercel promote`. After +promotion, it checks a genuine calculation through `calibration-diagnostics.vercel.app` +and the `microcosm.institute/calibration/dashboard` mount, and the calibration releases +through `calibration-diagnostics.vercel.app`. A failure before promotion leaves the existing production aliases unchanged. A failure after promotion must be handled with the rollback procedure below. -The workflow requires the dedicated Actions `HF_RELEASE_READ_TOKEN` secret and passes it -to the candidate as server-only runtime `HF_TOKEN`. Use a fine-grained read-only credential -scoped to every registered private release repository and US staging; never reuse the -publisher's write credential for the dashboard. UK staging retains its -separate `POPULACE_UK_STAGING_HF_TOKEN` deployment configuration. Before -promotion, `frontend/scripts/calibration-deployment-smoke.ts` checks each -registered country's calibration summary and release inventory, and every -supported staging inventory. It requires available calibration targets, a -consistent selected release, and staging `available: true`; an HTTP 200 alone -does not qualify staging. The same check runs against the public dashboard -after promotion. Public US calculations alone do not qualify private release -access. The script imports the `@policyengine/calibration-tree` workspace -package, so the workflow runs `bun install --frozen-lockfile --ignore-scripts` -in `frontend` before the first check. It installs after the Vercel upload, -which sends the repository root, so the uploaded files never include -`node_modules`. Lifecycle scripts are skipped because the job holds Google -Cloud credentials and can request GitHub OIDC tokens. Both checks run -`bun run --no-install`, so Bun never installs a missing package from npm by -itself; without the workspace install, a check fails. - -Calibration qualification retries network failures and HTTP 5xx twice, with short -increasing backoff and a 60-second timeout per attempt. HTTP 4xx, malformed successful -responses, and unavailable or inconsistent data fail immediately. If the post-promotion -check fails, the workflow reports failure while production aliases already point at the -candidate. The release owner must follow the rollback procedure below and run -`vercel rollback PREVIOUS_DEPLOYMENT`; the workflow does not roll back automatically. +`frontend/scripts/calibration-deployment-smoke.ts` is the calibration release check. +For each registered country with the `calibration` capability (the US, the UK and +Belgium), it requires an available calibration summary with a positive target count and +the expected country, and a release inventory whose default and latest release are that +summary's release, with calibration. For each country with the `staging` capability +(the US and the UK), it requires the staging inventory to report `available: true` with +a list of runs; an HTTP 200 alone does not qualify staging. It then requires an +unregistered two-letter country code to be refused as not configured (HTTP 404), never +answered with another country's release. Public US calculations alone do not qualify +private release access. + +The script imports the `@policyengine/calibration-tree` workspace package, so the +workflow runs `bun install --frozen-lockfile --ignore-scripts` in `frontend` before the +first check. It installs after the Vercel upload, which sends the repository root, so the +uploaded files never include `node_modules`. Lifecycle scripts are skipped because the +job holds Google Cloud credentials and can request GitHub OIDC tokens. Both checks run +`bun run --no-install`, so Bun never installs a missing package from npm by itself; +without the workspace install, a check fails. + +The calibration release check retries network failures and HTTP 5xx twice, after one +and then two seconds, with a 60-second timeout per attempt. It never follows a +redirect, so the bypass header cannot reach another address. Any other status than the +expected one (200, or 404 for the unregistered code), a malformed successful response, +and unavailable or inconsistent data fail immediately. If the post-promotion check fails, the workflow reports failure while +production aliases already point at the candidate. The release owner must follow the +rollback procedure below and run `vercel rollback PREVIOUS_DEPLOYMENT`; the workflow +does not roll back automatically. The public `calibration-diagnostics.vercel.app` domain is also the stable Hugging Face webhook origin. Do not point the webhook at a staged deployment URL or the protected `calibration-diagnostics-policy-engine.vercel.app` alias. -Vercel now hosts only Next.js; the previous large-Python-function flag and Performance -memory change are not prerequisites. Remove the obsolete preview-only large-function -exception once the migration is qualified. Do not change other project environment -entries. Production settings and aliases are release-owner actions, never implied by a -successful preview build. +Vercel hosts only Next.js; the Python calculation runs on Modal. The Vercel project's +Preview environment still held `VERCEL_SUPPORT_LARGE_FUNCTIONS` on 2026-10-26, from +when the calculation ran on Vercel. Nothing in this repository reads it, and removing it +is a release-owner action. Production settings and aliases are release-owner actions, +never implied by a successful preview build. Rollback uses `vercel rollback PREVIOUS_DEPLOYMENT` and the backend URL/credentials baked into that deployment. Retain that backend and token through the rollback window. Restore