The YAML test job likely runs short of memory and hits its 45-minute limit #2741

The Test / YAML job most likely runs short of memory on its 16 GB arm64 runner. It takes 30 to 45 minutes and often hits its 45-minute limit after every test has passed. #2697 raised the limit from 30 minutes for the same symptom (#2698); it now hits 45.

Evidence

• Of 38 pull request and merge-queue YAML jobs (first attempts) that started between 23:05Z on 28 October and 08:33Z on 29 October, the 19 that passed took 30 to 44 minutes and 7 ran into the 45-minute limit; 12 more ended early. On 10 October the job took about 4 minutes, with 2,946 cases (run 38085920460).
• In run 40830740956 (Value means-tested capital less 10% sale expenses and secured debt (UC reg 49 and legacy equivalents) #1969, rerun job 123339506397), pytest reported 5266 passed ... in 1152.73s. The second-to-last test file finished at 08:30:08, the last one, which runs in a second locally, finished at 08:55:55, and the job ran on without output until the limit cancelled it at 09:08:37.
• On main (2d437eb), on a Mac with memory to spare, policyengine-core test policyengine_uk/tests/policy -c policyengine_uk passed 5,235 cases in 168 s, with memory climbing about 2 GB every 20 s to a peak of 16.1 GiB (process-group RSS), then took 107 s to exit after printing its summary. The CI logs record no memory figures, so the shortage on the runners is an inference from this run and the runner's 16 GB.

Cause

policyengine-core's YAML runner (policyengine_core/tools/test_runner.py, 3.33.12) keeps every case's state for the whole process:

• pytest holds each collected item until the session ends, and each YamlItem keeps simulation and tax_benefit_system after its case has run;
• _tax_benefit_system_cache keeps a full clone of the tax-benefit system for each distinct combination of dotted parameter inputs (gov.x.y: value) and never evicts one. 358 cases in 76 YAML files supply dotted inputs: 588 entries in 157 distinct combinations.

PolicyEngine/policyengine-core#569 fixes both in core, but is still a draft (open since 2 October; last update 15 October), after PolicyEngine/policyengine-core#615 closed as not planned.

Fix in this repository

Until core releases #569, the test conftest's pytest_runtest_teardown can drop each YAML case's simulation and system after it runs, and a file's dotted-input systems after its last case. With that change the same local run peaked at 1.65 GiB and exited 5 s after its summary instead of 107 s, and #2742's CI YAML step took 7 min 25 s.