Stop the YAML test runner keeping every case's simulation and reform system

[MaxGhenis]

Draft: the October 7 CI repair is at 91744d3e6c88b1a56f7e81f1001f8cdeba3c9af8. Runtime validation at this head remains pending. Existing downstream measurements were completed on earlier snapshots, as linked below.

Problem

On 2026-10-02 two policyengine-core test <~1,500 policyengine-us baseline YAML files> -c policyengine_us processes reached 118 GB and 111 GB on a 128 GB Mac and had to be killed. Two of the causes are in tools/test_runner.py:

1. pytest keeps every collected item until the session ends, and each YamlItem kept simulation and tax_benefit_system after its case ran. Every simulation of the run stayed alive.
2. _tax_benefit_system_cache kept every system it built, one per distinct reforms / extensions / dotted-parameter-input combination, and never evicted. Each is a full clone of the baseline. On policyengine-us one measured 1.16 to 1.20 GB and 43 to 47 s to build.

Simulation.__init__ also sets tax_benefit_system.simulation = self, so a cached system kept the last simulation built on it.

The third cause, a copy of the whole parameter tree per instant, is in #570 (lazy ParameterNodeAtInstant); the two PRs are independent.

Change

• YamlItem.teardown drops the case's simulation, its system, and the system's back-reference to that simulation.
• The cache keeps the reform-free system for the baseline's lifetime, plus an LRU of reform_cache_size reform systems (default 2). It is set with policyengine-core test --reform-cache-size N or options["reform_cache_size"]; 0 caches none.

Replaying the cache over policyengine-us's whole tests/policy tree in run order (31,393 cases, 433 distinct reform combinations) gives 434 builds with no bound and 504 at size 2 (453 at 4). Two keeps the per-process peak at about two reform systems, and costs about 16% more builds.

Invariants

• After a case, the runner item drops its simulation/system references and the system's simulation backlink returns to its pre-case value, including failed construction. With external exception owners excluded, no simulation created by the run remains alive; the live-system count does not grow with case count (test_runner_memory.py).
• Traced memory after 300 cases is within 2 MB of memory after the first 100: 0.5 MB with this change, against 4.0 MB (plain cases) and 18.4 MB (every third case a reform) on master.
• Differential test against a reference LRU (test_reform_cache_property.py, Hypothesis): for any request sequence and any size 0 to 3, the cache holds at most size reform systems, returns a cached object exactly when the reference LRU would, always returns a system built for exactly the requested reforms and extensions, and keeps one reform-free system that is never the baseline itself.
• Outcomes are unchanged: caching only decides when a system is rebuilt.

Existing test targets

The following targets were already listed in this PR; they are not claimed as executed at the current head.

• uv run pytest tests/core/tools/test_runner tests/core/test_tax_benefit_system_cache_identity.py tests/core/test_yaml.py
• The country template YAML suite with and without --reform-cache-size 1.

October 7 CI repair

Merged canonical master at 5d68130be487f0b87d47fc75ef300f36ad46f7ca with a history-preserving merge and no conflicts.

• The deepcopy regression had expected a cached tracing wrapper. It now checks the intended plain-node cache, a fresh tracing wrapper for each read, and independent copied nodes/tracers. The parameter-cache implementations and distinct-date tracer/result regressions are unchanged.
• The eight failed-build simulations in CI were retained by pytest-rerunfailures 16.7, which stores failed ExceptionInfo objects on collected items even with no reruns configured. The nested memory-test session disables that dev plugin while the outer session retains its rerun behavior. The probe explicitly retains collected items; failure tests cover both builder paths, an unknown variable, and absent/existing prior simulation backlinks. A successful-build case checks the same ownership invariant.
• This repair changes regression tests and the changelog wording. The product runner's plugin behavior remains unchanged: installing pytest-rerunfailures can still retain failed tracebacks. The test isolation is not a production fix for that third-party behavior.

Validation at the repair head

• Passed: merged and changed Python files formatted before their commits; ruff format --check . (348 files), ruff check ., changed-source AST parsing, and git diff --check.
• Targeted pytest attempt: ~/reviews/us-hub/scripts/heavy_run.sh core569-targeted env UV_CACHE_DIR=/private/tmp/core-569-build-artifacts/uv-cache uv run --no-sync pytest tests/core/test_tracing_parameter_isolation.py tests/core/tools/test_runner/test_runner_memory.py tests/core/test_parameter_tracing_cache.py tests/core/test_parameter_tracing_cache_property.py tests/core/tools/test_runner/test_reform_cache_property.py -q -n 2 -p no:cacheprovider exited 75 before pytest ran. The sandbox denies the wrapper's ps call, preventing its required nice-level check. No bypass was attempted.
• No full suite, country suite, documentation build, microsimulation, or fresh memory benchmark ran in this repair. The bounded assignment permits targeted files only.
• Independent source review approved exact head 91744d3e6c88b1a56f7e81f1001f8cdeba3c9af8 with no unresolved P1/P2 findings in the repair delta. Runtime validation and current green CI remain required; this draft is not marked ready.

Downstream impact

Completed measurements were posted October 3 for core 760a001a, and its combination with #570 4e6bc315, using PE-US 615a0f2d. Those earlier results are not measurements of this repair head. Fresh measurement is pending for 91744d3e6c88b1a56f7e81f1001f8cdeba3c9af8 and its eventual combination with #570.

No country-policy or partner baseline files were changed in this repair; no new partner-impact gate or methodology decision is introduced. Whichever of #569 and #570 lands second must rerun the overlapping parameter-cache and runner-memory tests. Preserve #570's recorded #567 ordering constraint.

🤖 Generated with Claude Code

[MaxGhenis]

Measurements of this PR, alone and combined with #570, on the two policyengine-us CI batches that have been dying on 16 GB runners since 3.32.12 (PolicyEngine/policyengine-us#9757). Each row is one policyengine-core test <paths> -c policyengine_us process on policyengine-us main 615a0f2d. Setup: CPython 3.14.7 on macOS arm64, CI's itemization setting, core imported from a snapshot of each commit. Peak memory is the process's peak physical footprint (/usr/bin/time -l). CPU is user plus system time; the host was shared and loaded, so wall time is not comparable.

Core	NY/OH/OK batch (1,328 cases): peak	CPU	USDA batch (1,373 cases): peak	CPU
3.32.11 (7950c01)	16,282 MB	213 s	17,076 MB	272 s
3.32.12 (b78b0ba)	18,156 MB	302 s	18,935 MB	321 s
#578 on 3.32.12	16,287 MB	243 s	17,055 MB	260 s
#569 (760a001)	9,636 MB	236 s	13,912 MB	237 s
#569 + #570 (760a001 merged with 4e6bc31)	7,033 MB	149 s	6,399 MB	137 s

All runs passed every case.

• On their own, both batches are already above the 16 GB runner limit on 3.32.11 and 3.32.12. In CI, NY/OH/OK is one of four sequential batches and USDA follows the SSA batches.
• Give a storage a set of shared keys only while it shares an array #578 (issue 3.32.12 gives every holder's storage an empty set: about 1.3 MB more per policyengine-us simulation #577) only removes the 3.32.12 regression, an empty set in every holder's storage.
• Stop the YAML test runner keeping every case's simulation and reform system #569 with Resolve parameter nodes at instant on first read #570 is what gives these jobs headroom: under half the 3.32.11 peak on both batches, and less CPU.

Scripts, logs and raw results: ~/reviews/ci-shutdown-core-3-32-12-2026-10-02/ on Max's machine (scripts/measure.sh, results/summary.tsv).