Reduce USDA and baseline state CI memory pressure without adding runners #9757

Activity

1. MaxGhenis commented on Oct 2, 2026

   

   MaxGhenis

   on Oct 2, 2026

   Contributor

   More actions

   Root cause of the jump, for the record: policyengine-core 3.32.12, which these jobs started resolving at 04:58Z today (uv lock --upgrade-package policyengine-core). main did not change between the last green runs (02:01Z, core 3.32.11) and the first red ones (09:25Z, core 3.32.12), and the two environments differ only in core.

   3.32.12 gave every holder's in-memory storage an empty set, and every simulation has a holder for each of policyengine-us's 6,194 variables, so each simulation costs about 1.3 MB more. The YAML runner keeps every case's simulation for the whole batch, so the batches grew by 2.1 to 2.6 GB (Linux peak RSS printed by test_batched.py, states-shard-3, three runs each):

   Batch	core 3.32.11	core 3.32.12
   1 (DE, FL, GA, HI)	7.2 GB	9.3 GB
   2 (MI, MN, MO)	7.9 GB	10.5 GB
   3 (NY, OH, OK)	15.0-15.1 GB	at the runner's limit; dies in most runs
   4 (VA, VT, WA)	8.9 GB	11.4 GB

   NY/OH/OK and USDA (14.9 GB) were already at the limit on 3.32.11. On 3.32.11 batch 3 also stalled for 7.5 minutes in ok_count_exemptions.yaml. Its last case has reforms: plus a dotted parameter input, so the runner builds a whole reform system at the end of the batch, when memory is at its peak. The stalls in the logs are swapping, not slow tests.

   • The core fix is Give a storage a set of shared keys only while it shares an array policyengine-core#578 (issue Widen openfisca-tools requirement #577): a storage gets a set only while it shares an array. These jobs pick it up automatically once it is released.
   • Reduce USDA and state CI memory pressure without adding runners #9758's smaller batches are still worth having: 3.32.11 levels leave NY/OH/OK and USDA within about 1 GB of the runner limit.
   • The underlying growth (the runner keeping every case's simulation) is Stop the YAML test runner keeping every case's simulation and reform system policyengine-core#569.

   I'll post local before/after numbers for the NY/OH/OK and USDA batches on #578 when they finish.
2. hua7450

   closed this as completedin #9758on Oct 3, 2026
3. MaxGhenis commented on Oct 3, 2026

   

   MaxGhenis

   on Oct 3, 2026

   Contributor

   More actions

   Follow-up: the core fix is released as policyengine-core 3.32.13 (PolicyEngine/policyengine-core#578, on PyPI 2026-10-03 11:30Z). PE-US CI picks it up through uv lock --upgrade-package policyengine-core.

   A/B on Linux CI: the same main push run (37100870281, commit fd693d6, #9758's batching), with only the two jobs rerun on 3.32.13. Peak RSS per batch, as printed by test_batched.py.

   states-shard-3 (13 per-state batches):

   	Largest batch	Batch peaks	Batch time total
   3.32.12 (attempt 1)	6,315 MB	3,156-6,315 MB	1,323 s
   3.32.13 (attempt 2)	5,472 MB	2,862-5,472 MB	989 s

   Every batch is smaller, by 270-990 MB.

   ssa-usda: SSA revenue 8,809 → 8,703 MB. The 8 USDA batches' largest went 6,382 → 6,257 MB, and each of them is 110-520 MB smaller.

   Both jobs passed on both cores. Since 11:31Z, 27 baseline state and ssa-usda jobs have finished across PE-US PR and push runs, and all passed.

   On local single processes (macOS, peak footprint), the same core change takes the old unsplit batches back to their 3.32.11 size: NY/OH/OK 18,156 → 16,288 MB and USDA 18,884 → 17,116 MB. Details are in #578.