Repository navigation
Beta deploy verification fails on three US simulations, blocking prod deploys since 9 Oct #747
Description
The "Deploy Simulation API" workflow has failed at Complete beta deployment / Verify deployed three-service stack on the last two merges to main, so prod deployment was skipped both times:
| Merge | Run | Date |
|---|---|---|
| #733 (750cd3a) | failure | 2026-10-09 |
| #741 (2ab7156) | failure | 2026-10-12 |
Both runs failed the same three US integration tests. Each simulation job comes back "status": "failed", "error": "Simulation failed (correlation_id=…)", on bundle policyengine 6.2.3, model 2.2.1, data populace-us-2024-spm-20260915 (SPM contract canonical-spm-v1, scenario ce_trend, county geography):
tests/simulation/test_budget_window.py::test_budget_window_multi_year_batch_completes(2026 child job failed, 2027 cancelled; correlation 8f8fb6bc4e5f49bc8549b6f8bf476b9e on 10-12)tests/simulation/test_calculate.py::test_calculate_us_state_region_model(b4d030d4933e44929037ce370ad780fc)tests/simulation/test_calculate.py::test_calculate_specific_model(4ef35a9aebec4237b73d9084d66d032a)
Every other beta job passed: the deploys and unit tests of the Modal executor, Cloud Run entrypoint and Modal gateway, plus Stage 12 v2 and routing. The UK paths aren't in the failing set.
Effect: nothing merged since 9 October is in production, including #741's fix (keep reform stop dates). The correlation ids should identify the worker-side exception in the executor logs. #726 (ACS-local data for US regional simulations) may be related to the region-model failure. I haven't checked that.
Found by the Budget CI W1 session while following #741's deploy.
Activity
MaxGhenis commentedon Oct 12, 2026
Taking this (W4 owns the sim-api deploy).
Cause. I ran the failing request (
country: us,region: state/ut, the refundable CTC reform,time_period: 2026) directly on the stagingpolicyengine-simulation-py6-2-3worker with the Modal SDK. It raises:So the worker fails while building the per-region US dataset, in policyengine 6.2.3's legacy-input rename. That is why the national default-model test passes and the state, region and budget-window tests fail.
Fix. It is not in policyengine-us 2.2.1 or the populace data, and not in sim-api's code. policyengine.py fixed it on 9 Oct in PolicyEngine/policyengine.py 48f3a784 and 4d2541d2: missing ACS WIC decisions are assumed claimed, and missing values anywhere else are still refused. The fix first shipped in policyengine 6.2.5. sim-api main still pins 6.2.3.
The updater's PR to 6.2.7, #746, carries the fix. It also moves:
Its checks are green and an independent review is running. I'll merge it when that agrees, watch the deploy, and close this issue only once beta verification passes and prod serves the new bundle. The US verification tests stay as they are.