Segment and cache Stage 12 US simulations

[anth-volk]

Fixes #711
Fixes #792

Same-repository draft PR check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

• Split each Stage 12 US baseline and reform simulation into 20 deterministic segment calls, while keeping the implementation independent of API v1 runner modules.
• Add versioned dataset and baseline cache construction, storage, image loading, manifest identity, and deployment validation.
• Observe baseline and reform concurrently within one shared wait window. Cancel every unfinished simulation, and record it as failed, when one simulation fails, when the window closes, or when the coordinator itself fails after starting a simulation.
• Preserve structured policy-validation errors, including invalid effective-period details, through simulation and report records.
• Document Stage 12 cache deployment, segmented execution, aggregation, and failure handling.
• Rebased onto main after Release Stage 12 database sessions and keep original failures visible #789. Every Stage 12 database operation in this PR opens its connection through Release Stage 12 database sessions and keep original failures visible #789's connection opener, and every report and worker failure record goes through Release Stage 12 database sessions and keep original failures visible #789's persist_failure_record.
• Review fixes: an error reading or writing a failed simulation's own record no longer replaces its failure, each failure is reported to telemetry exactly once, and the Stage 12 precompute app mounts only the read-only pe-uk-private-hf-read-token Hugging Face secret. See "Fixes after review" below.

Fixes after review

A failed simulation's own state error no longer replaces its failure (b6c691d1)

Before this fix, the coordinator read a failed child's record, to choose its failure detail, outside any guard. A reproduction in which one database read of the failed baseline's record fails recorded the report as report_coordination_failed instead of simulation_invocation_failed, recorded the failed baseline itself as simulation_cancelled ("Cancelled after the coordinator failed."), and gave the reform the same wrong reason. A failing write of that record was already caught, but it was logged without the report or simulation identifiers.

• In both the child-wait path and the dispatch-failure path, reading and writing the failed simulation's own record is now non-fatal, like the cancellation writes. The error is logged at ERROR with evaluation_id, simulation_execution_id, the role and the error type, in the message and as record attributes. The error's message is not logged.
• The report keeps the simulation's failure as its code and cause (simulation_invocation_failed or simulation_dispatch_failed, or the typed failure the worker recorded).
• The failed simulation is recorded as failed with its own cause whenever its record can still be written. If the first read fails, the conditional write is still attempted. It is never recorded as simulation_cancelled, and it is not cancelled through Modal.
• Its peers are cancelled with "Cancelled after the simulation failed.".

Each failure is reported once

#789's contract is that each failure is reported once. After the rebase, an unexpected error reached telemetry twice: once through #789's report_original_failure ("Stage 12 execution failed", with the error), and again through this PR's failure_detail_from_exception, which called log_and_redact_exception. That second report was a standard-library ERROR log with the stack trace, runtime.record_exception and a gateway_error_redacted event. It happened in the worker, child-wait, child-dispatch and report-coordination handlers. A segmented simulation's failure was also reported twice: the fan-out logged the segment error, and the worker then reported the wrapping Stage12ExecutionError.

• A new report_failure helper builds the redacted detail without logging it and emits one report_original_failure record. That record carries the same correlation_id as the persisted Simulation failed (correlation_id=...) summary, plus evaluation_id and, for a simulation, simulation_execution_id and the role. Without a runtime, the one standard-library log line includes the correlation ID.
• The worker, the segment fan-out, and the coordinator's child-wait, dispatch and report handlers all use it. The worker does not report a segment failure again, because the fan-out has already reported it.
• failure_detail_from_exception is replaced by redacted_failure_detail, which only describes a failure and never reports it.
• The expected attributes in Release Stage 12 database sessions and keep original failures visible #789's report-ordering tests now include the identifiers and the correlation ID. Each test also checks that the correlation ID matches the persisted summary.

The Stage 12 precompute app mounts only the read-only Hugging Face secret (eef03c3c)

src/modal/stage12_precompute_app.py runs on every Stage 12 deployment. It mounted the retiring policyengine-data-credentials and huggingface-token (required_keys=["HF_TOKEN"]) Modal secrets on its image build and all five functions.

• Its image build (policyengine bundle install ... --country us) and all five functions now mount pe-uk-private-hf-read-token with required_keys=["HUGGING_FACE_TOKEN"]. The secret is declared through the same HF_MODAL_SECRET_NAME and HF_RUNTIME_ENV_NAME constants as app.py and v2_app.py. The functions keep stage12-evaluation-gcp-credentials. The app never read HF_TOKEN itself, and nothing else changes.
• Read scope. Every Hugging Face read the app makes resolves through the US bundle manifest to hf://policyengine/populace-us/.... This covers the bundle install in the image build and any dataset materialization during ensure_datasets or a calculation. PolicyEngine 6.3.4's runtime code names no other Hugging Face repository for the US. policyengine/populace-us is a public, ungated dataset repository. The token's scope covers only the two private UK repositories, and this public repository needs no grant. The same token already downloads these artifacts in the v2 US worker image build. That layer was rebuilt with only this secret in today's 6.3.4 deployment (run 38545891000, staging) and installed populace_us_2024.h5 from policyengine/populace-us. No credential was widened.
• test_deployment_sources_exclude_legacy_hf_secret_names now includes this app. A new declaration test checks the secret on the image build and every function. Both fail on the previous app source.

Environment variables and wiring

Variable	Used by	Provider	Wiring
HUGGING_FACE_TOKEN	Image build (bundle install) and all five functions	GitHub organization secret PE_UK_PRIVATE_HF_READ_TOKEN, copied into Modal secret pe-uk-private-hf-read-token	The prepare job validates the GitHub value, then runs modal secret create pe-uk-private-hf-read-token HUGGING_FACE_TOKEN=... --env=<modal_environment> --force when deploy_existing_stack or deploy_stage12_v2 is true. deploy_stage12_v2 needs: prepare, and its implicit success() condition means the sync has succeeded, in the same Modal environment, before "Build the independent Stage 12 cache" runs. The sync is in the prepare job, not in the same job as the precompute step. The app attaches the secret to the image build and every function.
GOOGLE_APPLICATION_CREDENTIALS_JSON	All five functions (cache bucket access)	Secret Manager secret named by vars.STAGE12_GCP_CREDENTIALS_SECRET_NAME, copied into Modal secret stage12-evaluation-gcp-credentials	Unchanged: stage12-sync-modal-secrets.sh, in the same deploy_stage12_v2 job, runs before the precompute step.
POLICYENGINE_DATA_FOLDER	Functions	Image environment	Unchanged.
STAGE12_CACHE_BUCKET	Local entrypoint on the runner, which passes the bucket to each function	GitHub environment variable vars.STAGE12_CACHE_BUCKET	Unchanged: precompute step env:.
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	modal secret create and modal run	GitHub secrets	Unchanged: each step's env:.

• Both environments. Staging (release_environment: beta, Modal staging) and production (prod, Modal main) call the reusable workflow with deploy_stage12_v2: true and secrets: inherit, from both simulation-deploy.yml and stage12-deploy.yml.
• Modal secret names. A read-only modal secret list showed pe-uk-private-hf-read-token and stage12-evaluation-gcp-credentials in both staging and main, each used on 2026-10-14. Secret values and the GitHub secret were not inspected.
• Not verified directly. Whether PE_UK_PRIVATE_HF_READ_TOKEN is granted to this repository and is valid. Today's successful deployments, which validate and synchronize it, indicate that it is. The precompute app has not run with the new secret, because no deployment was run.

Rebased onto main after #789

The branch is rebased onto 5484278455e7fd48bb8d874f0e26245895d2730c, the merge of #789. The earlier merge commit from main is gone, and each commit of this PR now builds on #789.

Connection opener

• Stage12PersistenceStore.replace_simulation_if_status, which this PR adds, opens its connection through _transaction, like every other store operation. Only the connect step can be retried; its statements and commit are never retried.
• The coordinator and the simulation worker build their stores with runtime_store(runtime), so a refused connection is retried and each retry is logged with the operation name. The Cloud Run entrypoint keeps the non-retrying default.

Failure recording

• Each failure is reported to telemetry once, through Release Stage 12 database sessions and keep original failures visible #789's report_original_failure, before any database write. This covers worker failures, segment failures, child dispatch failures, child failures seen while observing, and report failures. See "Fixes after review" for how the report now carries the correlation ID.
• Report and worker failure records are written through persist_failure_record, which logs a failed write instead of raising it. A failed simulation's own record is updated by the coordinator through _record_child_failure, which logs a failed read or write with the report and simulation identifiers, like the cancellation records.
• This PR's structured failure details are kept. The records now hold the redacted summary with a correlation ID, or the typed public error, instead of the exception type name. Release Stage 12 database sessions and keep original failures visible #789's tests were adapted to these summaries and to the conditional replace_simulation_if_status writes.
• A child dispatch failure, a child failure and a closed wait window each raise Stage12ExecutionError after being reported where they occur. The report handler therefore reports only other errors, so each failure is reported once.
• A closed wait window is now reported too, with failure scope child_wait_timeout, before the cancellation records and the report record are written. A new test makes those writes fail and checks that the report still reached telemetry first.
• Release Stage 12 database sessions and keep original failures visible #789's _mark_child_failed is folded into this PR's _replace_running_simulation, which the dispatch, child-failure and cancellation paths share. It only changes a running record, which this PR requires.
• Cancellation records keep this PR's own handling. A failed cancellation write is logged with evaluation_id and simulation_execution_id, and an unfinished child is then cancelled through Modal. They do not use persist_failure_record, because the coordinator must know whether the record was written.

Known open issue (fixed after review)

The coordinator read a failed child's record outside the failure-recording guard, so an error on that read replaced the child's failure. This is fixed in b6c691d1; see "Fixes after review". #789's test for this case, test_coordinator_reports_a_child_failure_once_when_its_record_is_unwritable, is no longer marked xfail and passes.

Connection retries and the finishing time

#789 retries a refused connection for up to 60 s. The last attempt can start just before 60 s and then wait up to the 10 s connect timeout, so one store operation can take about 70 s when the pooler refuses throughout. After the wait window closes:

• Wait timeout: at most 5 operations (a read and a conditional write for each unfinished child, then the report record), so about 350 s. This path has no production-result wait, so 1,500 s remain.
• Child failure at the last poll: at most 6 operations, so about 420 s, which also fits.
• Direct report succeeds: 1 operation (2 if that write fails), so 70 to 140 s of the 600 s.
• Automatic (shadow) report succeeds: at most 5 operations outside the 900 s production-result wait, so about 350 s. That leaves about 250 s of the 600 s to read the artifacts, aggregate and write. A complete testing-environment US report took 288 s including every calculation, so the budget still holds, but in this worst case the margin falls from 600 s to about 250 s.

If the coordinator ever exceeded 5,400 s, Modal would stop it and the report would stay running without a failure record. The timeouts are unchanged in this rebase.

Replaces #793 and covers #792

This PR replaces #793 and adopts its practices. Issue #792 asks for one shared wait deadline that fits inside the coordinator's timeout, and for every unfinished worker to be cancelled and recorded as failed. Observing baseline and reform concurrently was always part of this PR.

Wait window and timeouts

This PR follows #793's rule: the shared wait equals the workers' Modal timeout, and the coordinator's timeout is that wait plus 900 s for the production-result wait plus 600 s to finish.

• One shared window from coordinator entry. The window opens when coordinate_report starts. It covers claiming, output planning (including loading the country model), retained-artifact reads, dispatch, and the concurrent wait for both simulations. Setup time therefore shortens the simulations' window, not the time the coordinator needs afterwards.
• Window = longest simulation worker timeout = 3,900 s. This is the US worker's timeout, and it covers the UK worker (3,000 s) and each US region-group call (3,600 s). Before this update, the window was 3,000 s, so a US simulation could never use the 3,900 s Modal allowed it.
• Coordinator timeout = 3,900 + 900 + 600 = 5,400 s (was 5,100 s):
  • 3,900 s: shared wait window;
  • 900 s: production-result wait (PRODUCTION_RESULT_WAIT_TIMEOUT_SECONDS);
  • 600 s: read and validate both simulation artifacts, aggregate, write the aggregate and comparison artifacts, and record the outcome or failure.
• Why 600 s is enough. A complete testing-environment US report took 288.2 s, including all 40 region-group calculations, merging, reads, aggregation and writes. A failed report took 26.8 s. So 600 s is ample, and I kept Share one Stage 12 child wait deadline and cancel unfinished children #793's value. See "Connection retries and the finishing time" above for the worst case with Release Stage 12 database sessions and keep original failures visible #789's retries.
• No overrun. The coordinator's last poll no longer sleeps past the deadline, so the window closes on time rather than up to 1 s late, and the 600 s is not eroded.
• Named constants. A new dependency-free module, stage12_timeouts.py, defines the worker timeouts, the window and the finishing time, and derives the coordinator timeout from them. The Modal app declaration, the coordinator and the comparison step all read it.
• Dependents. I searched for anything that relies on the old 5,100 s or 3,000 s values. The Modal app declaration, its declaration test, the coordinator comment and the docs did, and are updated. The entry service, the persistence library, deployment workflows and scripts, and release validation do not depend on them. I also checked the Public API's Stage 12 code in a local clone. It has no dependency on these timeouts, and its status rules let a simulation record move from running to failed and from failed back to running, so failed cancellation records and retries still fit.
• Environment variables. Only the timeout values of run_single_simulation_us, run_single_simulation_uk and coordinate_report change. Their images, worker_secrets (the GCP, Hugging Face and comparison-runtime Modal secrets) and environment are unchanged. The new constants module ships in the existing policyengine_simulation_executor package source that every Stage 12 image already includes. No CI, Docker or Terraform changes.

Cancellation

• Recorded as failed. Every simulation the coordinator cancels is recorded as failed with simulation_cancelled. Its error_summary gives the reason:

  • "Cancelled after the baseline simulation failed." (or reform);
  • "Cancelled after the 3900-second wait window closed.", where the report itself fails with simulation_wait_timeout;
  • "Cancelled after the coordinator failed."

  The earlier incomplete status and the cancelled_after_peer_failure and cancelled_after_coordinator_failure codes are gone. Nothing else needed them: workers treat any status other than running as a cancellation, and the entry service passes statuses and codes through unchanged.

• Only running records change. A simulation that has already succeeded or failed keeps its own outcome.

• Always stopped in Modal. An unsegmented simulation is always cancelled through Modal, even if its record could not be written. A segmented simulation reads its failed record, cancels its own region-group calls and exits. Modal does not cancel spawned calls recursively, so the coordinator cancels a segmented simulation through Modal only if it could not write the record.

• Startup failures too. Any coordinator failure after the first spawn stops every started, unfinished simulation. Examples are a failed dispatch placeholder write, a failed invocation attach (including the simulation whose attach failed), a failed restore and a retained-artifact error. A call restored from an earlier coordinator attempt is stopped too. Before this update, only a failed spawn did this.

• Errors never mask the original failure. Cancellation and state-update errors, including errors reading or writing the failed simulation's own record, are logged, and the report keeps the original failure.

• Traceable logs. When recording or requesting a cancellation fails, the log includes evaluation_id and the simulation's simulation_execution_id, in the message and as record attributes, alongside the role and error type. The error message itself is not logged. Share one Stage 12 child wait deadline and cancel unfinished children #793 used event-name messages with these fields only as attributes. This PR also puts them in the message, because the observability log handler exports only the message.

Carried over from #793

• The timing rule and its declaration test, which reads the real Modal timeouts through fixtures.fake_modal, and the setup-time and exhausted-window tests.
• simulation_cancelled failed records, the running-only guard, and Modal cancellation even when the state write fails.
• Tests: a cancelled and recorded reform, a finished baseline left alone, a restored reform call cancelled on retry, a sibling that already succeeded or failed left unchanged, every unfinished child cancelled on timeout, and cancellation and state errors that do not mask the failure.
• ImmediateCall.cancel now counts cancellations.
• A changelog line. changelog_entry.yaml keeps Release Stage 12 database sessions and keep original failures visible #789's line and this PR's line.
• Not carried over: Share one Stage 12 child wait deadline and cancel unfinished children #793's tests of sequential call.get timeouts, because this PR polls both simulations without blocking.

Validation of the review fixes

• Full executor suite on Python 3.13, with CI's uv sync --extra test environment: 1149 passed, 22 skipped, 3 deselected, 0 xfailed (was 1134 passed and 1 xfailed).
• Stage 12 executor tests (tests/test_stage12_*.py): 213 passed.
• Simulation entry suite: 381 passed. Stage 12 persistence suite: 19 passed.
• New tests, each run against the previous source and failing there:
  • the failed simulation's own record read or write fails once, in the child-wait path (2 cases) and in the dispatch-failure path (2 cases);
  • each unexpected failure is reported exactly once, with a runtime and without one, in the worker, child-wait, dispatch and report paths (8 cases), plus a segment failure (1 case);
  • the precompute app mounts only the read-only secret (1 test), and the guard now includes this app.
• test_coordinator_reports_a_child_failure_once_when_its_record_is_unwritable no longer has its xfail marker and passes. Its expected report attributes now include the identifiers and the correlation ID.
• Pinned Ruff (0.15.22) format --check and check on the 10 changed Python files: clean. Ruff format --check on every projects/*/src and libs/*/src directory: clean. git diff --check: clean.
• Not run: new testing, staging or production deployments.

Merged main after #791

#791 stores a redacted failure cause: the type, the stage, the simulation role and the message. For every unexpected error, this PR stored a generic message with a correlation ID instead. Main is merged in as a merge commit rather than a rebase, because 16 of this PR's 25 commits touch the three files that conflicted. This PR keeps its own failure model and uses #791's cause within it:

• Unexpected errors: redacted_failure_detail stores the redacted cause, followed by the correlation ID, for example OperationalError in stage12_child_state_create: ... (correlation_id=3f9c...). The cause is shortened, never the ID, to fit the 512-character summary. Typed input and SPM failures, failures a worker already recorded, and cancellations are unchanged.
• Stages and roles: the coordinator and the workers open their spans and operations through Record a redacted Stage 12 failure cause instead of only the exception class #791's tracked_stage, so a failure names its stage. The coordinator waits on both simulations at once, so it records each one's role while checking it.
• Telemetry: the original error is still reported once, with its correlation ID. As on main, the redacted cause is also attached to the failing operation, with the original stack.
• Error codes: the report keeps this PR's codes. For example, a failed child gives the report simulation_invocation_failed, where Record a redacted Stage 12 failure cause instead of only the exception class #791 always used report_coordination_failed.
• Tests:
  • Tests that expected the generic message, or no message at all, now expect the redacted cause.
  • Record a redacted Stage 12 failure cause instead of only the exception class #791's six cause tests are ported with this PR's error codes.
  • A new test checks that a long cause is shortened and keeps its correlation ID.
  • The exactly-once telemetry tests now check for one log of the original error and one recorded redacted cause.
• Docs: the README and the Stage 12 migration notes describe the new summary.
• Validation: executor suite: 1230 passed, 22 skipped, 3 deselected. Simulation entry suite: 449 passed. Stage 12 persistence suite: 19 passed. Quality guards, pinned Ruff format --check on every src directory, and git diff --check all pass.

Rebased onto main after #762, #763 and #795

• Move the deploy pipelines' multi-line steps into tested scripts #763 moved the worker validation step into .github/scripts/stage12-validate-v2-workers.sh, which conflicted with this PR's --cache-manifest-sha256 argument. The script now takes the digest as a second required argument and passes it to validate_v2_worker_release. The workflow passes steps.stage12_cache.outputs.manifest_digest, the same pattern Move the deploy pipelines' multi-line steps into tested scripts #763 uses to pass step outputs to its other scripts.
• The digest comes from the Build the independent Stage 12 cache step (stage12-cache-precompute.sh), which runs earlier in the same deploy_stage12_v2 job and writes manifest_digest to GITHUB_OUTPUT. RUNNER_TEMP and GITHUB_OUTPUT come from the runner. MODAL_TOKEN_ID and MODAL_TOKEN_SECRET come from GitHub Actions secrets, unchanged.
• test_deploy_workflow_scripts.py checks that the workflow passes the digest and that the script forwards it. A new test checks that the script fails, without calling uv, when the digest is missing.
• Simulation entry suite: 449 passed. Full executor suite: 1169 passed, 22 skipped, 3 deselected. Quality guards (rules-registry, ci-scripts), pinned Ruff and bash -n pass.

Rebased onto main after #757 and #787

• changelog_entry.yaml keeps Accept null SPM provenance in Stage 12 aggregate artifacts #787's line and this PR's line.
• test_coordinator_records_bounded_timeout_without_an_aggregate combines Accept null SPM provenance in Stage 12 aggregate artifacts #787's valid aggregate stub with this PR's wait_timeout_seconds=0.
• Accept null SPM provenance in Stage 12 aggregate artifacts #787 made the coordinator tests' artifact double write aggregates through the real Stage12ArtifactStore. This PR's four aggregator stubs that must not run now return _aggregate_payload(...), as Accept null SPM provenance in Stage 12 aggregate artifacts #787 did for main's, so an unexpected run shows up as a written aggregate instead of as a rejection hidden by the failure the test expects.
• Full executor suite on Python 3.13: 1152 passed, 22 skipped, 3 deselected. Quality guards (rules-registry, ci-scripts) pass.

Rebased onto the rules registry (#752)

• Rebased onto main after Add a rules registry that makes every enforced and unenforced rule visible #752. Code and the changelog line are unchanged from the previous head (eef03c3); the only differences are main's changes and the registry entry below.
• The rules-registry guard from Add a rules registry that makes every enforced and unenforced rule visible #752 flagged test_stage12_only_deployment_skips_existing_stack_integration_suite, which this PR adds, as a guard test no rule references. It is now one of the nodes of sim.stage12.deployment-scope. The rule's statement now also says that a Stage 12-only deploy never runs the existing stack's complete integration suite. docs/generated/rules.md and the AGENTS.md Rules block were re-rendered.
• python -m scripts.rules guard --base-ref origin/main, scripts/run_quality_guards.py and the rules tool's own tests pass. tests/test_deployment_assets.py: 113 passed.

Validation after rebasing onto #789

• Executor environment synced with CI's uv sync --extra test, plus --locked to confirm the lockfile is current: PolicyEngine 6.3.4 (policyengine-us 2.2.1, policyengine-uk 2.126.12, policyengine-core 3.33.0).
• Full executor suite on Python 3.13: 1134 passed, 22 skipped, 3 deselected, 1 xfailed (the known open issue above).
• Stage 12 executor tests (tests/test_stage12_*.py): 198 passed, 1 xfailed.
• Simulation entry suite: 381 passed. Stage 12 persistence suite: 19 passed.
• Every rebased commit was checked out in turn and its Stage 12 executor tests and persistence suite were run. They pass from Reconcile segmented Stage 12 execution with current typed provenance onwards. The four commits before it fail one worker-validation test, test_us_validation_normalizes_the_frontend_reform_interval, exactly as they did before this rebase: main's reform-interval normaliser predates them, and that commit updates the test.
• Regression check: removing the closed-window report makes the new timeout test fail.
• Pinned Ruff (0.15.22) format --check and check on the 49 changed Python files: clean. Ruff format --check on every projects/*/src and libs/*/src directory, as CI runs it: clean. git diff --check: clean.
• Not run: new testing, staging or production deployments.

Validation of the cancellation and timeout update (before rebasing onto #789)

• Full executor suite on Python 3.13 with uv run --frozen: 809 passed, 23 skipped, 3 deselected. Before that update it was 786 passed.

• Stage 12 executor tests (tests/test_stage12_*.py): 181 passed.

• tests/test_stage12_runtime.py gained 23 tests in that update, covering startup cancellation, cancellation records, Modal fallback, logs, and timing.

• Regression checks. I reverted each behaviour in turn, and each time the matching new tests failed:

  • removing the Modal fallback after a failed state write;
  • letting the last poll sleep past the deadline;
  • overwriting a sibling that already finished;
  • anchoring the deadline at observation instead of coordinator entry.

  The startup-cancellation, cancellation-log and coordinator-entry tests also failed against the head before that update.

• Simulation entry suite: 162 passed. Stage 12 persistence suite: 14 passed. Their cancellation fixtures now use the failed simulation_cancelled record.

• Scoped Pyright on the coordination, comparison and timeouts modules: 0 errors.

Prior validation before the earlier rebases

• Stage 12 simulation executor tests: 130 passed, with 1 warning.
• Stage 12 simulation entry tests: 81 passed.
• Stage 12 persistence tests: 14 passed.
• Ruff lint checks passed for the changed Python surface.
• Repository-wide Ruff formatting found one pre-existing formatting difference in projects/policyengine-simulation-executor/tests/test_simulation_output_geographic.py, which this branch does not modify.
• Deployed and validated all seven Stage 12 functions in the Modal testing environment, including US and UK worker validation and a cache-hit check for the 20-segment, three-year cache.
• Completed a testing-environment US report with 1 coordinator call, 2 simulation calls, and 40 successful segment calls in 288.2 seconds.
• Completed an invalid-reform report in 26.8 seconds: the reform retained invalid_policy_period, the baseline became incomplete with cancelled_after_peer_failure (the behaviour at that commit; a cancelled baseline is now failed with simulation_cancelled), and the report failed with the original validation summary.

Earlier testing-environment deployment note

A fresh UK dataset download was blocked because the configured Hugging Face credential could not access the private UK dataset. The testing deployment reused the previously validated UK and coordinator images from the same commit and rebuilt the US image layer with the validated Stage 12 cache.

Earlier current-main reconciliation

The executor keeps main's PolicyEngine 6.3.4 requirements and frozen lockfile; it does not restore older country packages.

Conflict resolution preserves main's output-planning instrumentation and UK metadata validation while retaining concurrent baseline/reform monitoring and segmented US calculation.

Segment merging accepts the current typed SPMProvenance contract. Every segment must supply the same complete receipt, including configuration, forecast hash, years and runtime versions; incomplete or inconsistent receipts fail. The merged logical simulation keeps one compact receipt, not a legacy spm_config object or a growing receipt list. Five regressions cover preservation, missing receipts and mismatched scenario, runtime versions and years.

The worker-validation unit test expects the complete reform date interval, matching main's normalizer rather than discarding its end date.

The earlier live results above describe an earlier commit on an older PolicyEngine version, not a fresh 6.3.4 qualification. No deployment has been run on 6.3.4.

🤖 Generated with Claude Code