Cut the CPU the Python tests spend in garbage collection

[MaxGhenis]

Fixes #2316

Follows #2269 (test shards), which landed in #2364; this branch has main merged in, so the diff is this PR alone. #2331 (test-shard memory) is stacked on this PR, and the two should land together (see "Memory").

Change

In policyengine_uk/tests/conftest.py, each process that runs tests (each pytest-xdist worker):

• sets the first-generation GC threshold to 10,000 at start-up (pytest_configure), instead of CPython 3.13's 2,000; the other thresholds keep the interpreter's defaults;
• after collection (pytest_collection_finish), runs gc.collect() and then gc.freeze(). That freezes about 1.64–1.66 million objects per worker (gc.get_freeze_count() in every measured worker), of which about 430,000 come from importing policyengine_uk; the rest are test modules, collected items and the tax-benefit systems built at import. The collector never scans them again; frozen objects in reference cycles that later become garbage are not freed until they are unfrozen (objects outside cycles still go with their last reference). Objects the tests create are collected as before;
• undoes both at pytest_unconfigure, so an in-process pytest.main() (an IDE, a REPL) leaves an interpreter that had nothing frozen as it found it (gc.unfreeze() would also unfreeze anything frozen before the session).

A tax-benefit system and its variable modules form reference cycles, so a system that dies after the freeze is never freed. Two test modules built a Simulation at import (test_council_tax_uprating.py's region-to-country map and test_uc_childcare_work_condition_properties.py's minimum care hours); their systems died during collection, so their variable modules were frozen, then released and never freed: about 19,000 objects (11+ MB) per worker, found in review. Both now avoid it (the map is built on first use; the hours come from policyengine_uk.parameters). conftest.py records any variable-module owner added during the session whose system died before the freeze (owners already loaded when the session started, from an earlier in-process run, are exempt), and the new test_no_system_dies_before_the_freeze fails on one (red against the old modules: "2 tax-benefit system(s) were built and dropped while the tests were collected").

release_dead_variable_modules now also counts the frozen tax-benefit systems as live. gc.get_objects() leaves frozen objects out (checked on Python 3.13.15 and 3.14.7). Without this change, the release would take policyengine_uk.tax_benefit_system.system for a dead system and drop its 1,110 variable modules from sys.modules. The new test_frozen_systems_keep_their_modules fails without that line and passes with it.

Measured on CI

Each measurement run adds a workflow to the PR on a branch that will not be merged (gc-measure/2317*; runs 1–3 on cffb3722a, the first commit; run 4 on 50ebd8877, after the review fixes, which differs from the head only by the merge of main). Every setting runs the Python tests as three shards in one workflow run, on separate runners, from one commit, the way #2269's Test (Python K/3) jobs ran them: -n 4 --dist worksteal, coverage with sysmon, Python 3.13 on ubuntu-24.04-arm. Main has since moved to four shards for memory (827b277, .github/workflows/test-suite.yaml); the comparisons below are all on three.

• A plugin records each worker's CPU time (getrusage), its time inside the cycle collector (gc.callbacks, wall-clock perf_counter, so the "in the collector" column is wall time while "Worker CPU" is CPU time) and its peak RSS. Its overhead is negligible next to the differences measured (the earlier review benchmarked the callbacks).
• A sampler records the runner's used memory (MemTotal − MemAvailable) every 2 seconds, and from run 3 on also swap, written to the job log every minute.
• From run 4, each worker also counts the frozen objects that are unreachable when its session ends: the objects the freeze leaks.
• Every setting deselects the same two tests:
  • main's failing test_an_exported_frame_loads_back_and_exports_itself, on which Hypothesis spends minutes shrinking;
  • this PR's test_frozen_systems_keep_their_modules, which asserts the new settings and so fails under the others.

Totals over the three shards, in minutes:

Setting	Run	Worker CPU	in the collector	Test durations (JUnit)	4 × pytest wall	Job time	Slowest shard	Peak runner memory
Python's defaults	38155711978	203.4	83.0	203.3	221.8	56.8	24.6	15.28 GiB
Python's defaults	38158313028	204.2	82.3	203.6	224.5	57.4	23.1	15.28 GiB
freeze only	38155711978	203.4	84.2	202.4	224.0	57.2	24.6	15.32 GiB
threshold 10,000 only	38155711978	190.8	72.7	190.8	207.8	53.1	20.8	15.27 GiB
freeze + threshold 10,000 (this PR)	38155711978	174.1	55.3	173.6	188.2	48.2	19.6	15.28 GiB
freeze + threshold 10,000 (this PR)	38158313028	170.7	54.2	170.2	184.4	47.4	20.3	15.27 GiB
Python's defaults	38168735544 (run 4)	193.6	77.8	194.5	213.3	54.3	22.7	15.28 GiB
freeze + threshold 10,000 (this PR, after review)	38168735544 (run 4)	176.1	56.9	176.1	192.0	48.8	20.1	15.24 GiB
freeze + threshold 25,000	38155711978	runners of shards 1 and 2 shut down at 98–99%

"4 × pytest wall" is the worker-minutes measure #2269 used (four workers, each busy until its shard ends). "Job time" is the three shard jobs' runner time.

• Compute. Over the three paired runs, worker CPU fell 13%, from 200 to 174 minutes (the per-run cuts were 14%, 16% and 9%). Worker-minutes fell from 220 to 188, and time in the collector by almost a third, from 81 to 55. Every shard was faster in every run: 16.1 / 24.6 / 14.7 → 15.2 / 19.6 / 12.2 minutes, 17.9 / 23.1 / 15.2 → 14.3 / 20.3 / 11.5, and 16.6 / 22.7 / 14.0 → 14.9 / 20.1 / 13.1.
• Freezing alone saves nothing. With Python's threshold, freezing raised the number of full collections from 3,359 to 7,697 and left their total time unchanged. It pays only with the higher threshold, which cuts first-generation collections from 1.45 million to 288,000.
• Memory. Shards 1 and 2 already fill the runner (15.3 of its 15.6 GiB; MemTotal 15,947 MiB, plus 3 GiB of swap) under every setting, including Python's defaults, so their peaks cannot show a difference. On shard 3, which doesn't fill it, this PR raised the runner's peak from 14.19 / 14.14 to 14.49 / 14.59 GiB (runs 1 / 2) and the largest worker's from 5.3 to 5.7 GiB. The threshold alone also took that worker to 5.7 GiB; freezing alone did not (5.26 GiB). In runs 1–2 the largest worker peaked at 5.3–5.8 GiB without the higher threshold, 5.7–5.8 with 10,000 and 6.8 with 25,000; in run 4, at 5.87 GiB with Python's defaults and 6.14 with this PR. In run 4, which logged swap, shards 1 and 2 used up to 2,077 / 2,389 MiB of the 3 GiB of swap with Python's defaults and 1,874 / 1,743 MiB with this PR; shard 3 used none either way. So in run 4 this PR used less swap, but neither setting leaves much: the PR's own CI run at 50ebd8877 lost shard 2's runner at 90% ("The runner has received a shutdown signal"; memory exhaustion is inferred from the swap figures, not logged on that run), and with Python's defaults the same shards came as close (2.4 GiB of swap). Stop the test shards' memory growing with every Simulation under sysmon coverage #2331 reports that coverage's sysmon core keeps every compiled code object alive; with its fix on top of this PR the runners peak at about 8.1 GiB with no swap (below), which is why the two should land together.
• A higher threshold isn't safe. With 25,000, the runners of shards 1 and 2 shut down at 98–99% ("The runner has received a shutdown signal"); shard 1 had printed nothing for 3 minutes before. Their memory logs were lost with the runners; later runs also write memory and swap to the job log every minute.
• This PR's own CI. Run 38155535210 at cffb3722a: Python shard jobs took 14.7 / 22.8 / 13.2 minutes, red only on main's then-failing dataframe test. Run 38168650620 at 50ebd8877: shard 2's runner shut down at 90% (memory, as above); no test failed.

Shard count

The task asked whether the compute saved would allow two shards instead of three. Not with this PR alone; yes once #2331 lands. Main has meanwhile moved from three shards to four for memory (827b277).

• This PR on two shards (run 38158313028): both runners shut down at 95–97%; shard 2 had printed nothing for 15 minutes. Run 3 (38164405956) repeated it with memory and swap in the job log. Shard 1's used memory climbed steadily from 9.2 GiB to 15.5 GiB over 12 minutes, swap filled (409 MiB at 18:48, 3,071 MiB, all of it, by 18:53), and the runner shut down at 97% at 18:54:41. Shard 2's runner stopped reporting and the job failed after 65 minutes with "The hosted runner lost communication with the server. Anything in your workflow that ... starves it for CPU/Memory ... can cause this error"; its log was lost with it. So two shards run out of memory with these settings. No run tried two shards with Python's default GC, so this PR's share of that is not separated.
• With Stop the test shards' memory growing with every Simulation under sysmon coverage #2331 on top of this PR (run 38165445604 on Stop the test shards' memory growing with every Simulation under sysmon coverage #2331's measurement branch mem-measure/2331-run2b; peaks, walls, swap and outcomes checked against its artifacts): two shards peaked at 8.11 / 8.09 GiB, pytest wall 1,308 / 1,100 s; three shards 8.24 / 8.00 / 8.19 GiB, 776 / 952 / 637 s; no swap, and no test failed in any job. Two shards then cost the same worker time (4 × wall: 161 against 158 minutes), save one runner pickup and setup (under a minute each here), and make the slowest shard 22 minutes instead of 16.
• With both PRs in, four shards are more than memory needs. Choosing two, three or four is a trade between wall time and ARM runner pickups (on 11 October up to 13 of 15 measurement jobs queued at once), in test-suite.yaml, which W2 owns; it is left as a follow-up for after Stop the test shards' memory growing with every Simulation under sysmon coverage #2331 lands.

Invariant

Same tests, same outcomes: the change alters when memory is reclaimed, not which tests run or what they compute.

• The final Test job's shard check confirms every collected test ran in exactly one shard. In this PR's run, 3,777 were collected, and the shards ran 1,206 + 1,280 + 1,291.
• Differential check: in all three paired measurement runs, every test had the same outcome under every setting that ran them all (runs 1 and 2: 3,775 tests, 3,772 passed and 3 skipped or expected to fail; run 4: 3,776 tests, 3,773 passed and 3 skipped or expected to fail).
• test_frozen_systems_keep_their_modules: the freeze and threshold are in effect, and a frozen live system keeps its variable modules (red without the _FROZEN_SYSTEMS lookup).
• test_no_system_dies_before_the_freeze: nothing frozen is a dead system's modules, which would leak (red against the two old test modules).
• test_systems_built_after_the_freeze_are_still_collected: a system built after the freeze is freed by gc.collect() once dropped. It documents the property; it would also pass without a freeze.
• test_dead_systems_modules_are_released_and_live_ones_kept (existing) still passes.
• Leak, measured in run 4 after the fix: 10 unreachable frozen objects per worker, and 772 in one worker each of shards 1 and 3. The review measured about 18,850 per worker locally before the fix.

Not property-based: the invariant is over the whole suite, so the differential run above is the test for it.

Tests run locally

Python 3.13.9:

• pytest policyengine_uk/tests/test_council_tax_uprating.py policyengine_uk/tests/test_uc_childcare_work_condition_properties.py policyengine_uk/tests/code_health/test_ci_test_shards.py -n 4, after merging main: 62 passed.
• The five tests in test_variable_module_release.py, with full collection and -n 4, after merging main: 5 passed.
• pytest policyengine_uk/tests -k "test_no_system_dies_before_the_freeze or test_frozen_systems_keep" (full collection, with and without -n 2): 2 passed.
• pytest.main() twice in one process: both pass, and the threshold and freeze count are back to (2000, 10, 10) and 0 after each.
• ruff format and ruff check are clean on the changed files. The full suite runs in CI.

Review

• Opus 5.5 (independent) at 057132c1e: REQUEST_CHANGES (~/reviews/uk-hub/reviews/uk2316-gc/review-opus-out.md); every finding addressed in 50ebd8877.
• GPT-6.1 Sol at a8ff60fa0: APPROVE (~/reviews/uk-hub/reviews/uk2316-gc/review2-out.md), with documentation nits. The comment nits are in 37cadb471 (comments and docstrings only; the UK hub checked both files are AST-identical to a8ff60fa0 ignoring them), and the rest are in this description.

axiom: n/a: test infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the UK hub (session happy-turing-d05840 / f9b280dd).

• Head: 37cadb47192adb0110a796cddf5ef2cb2ce6d795. Main is merged in; the diff is this PR's 5 files.
• Review: GPT-6.1 Sol approved a8ff60fa0 (~/reviews/uk-hub/reviews/uk2316-gc/review2-out.md). 37cadb471 changes only comments and docstrings, per Sol's nits; the hub checked both files are AST-identical to a8ff60fa0 once those are ignored. The earlier Opus review's REQUEST_CHANGES at 057132c1e (review-opus-out.md) was addressed in 50ebd8877.
• CI: the head's run is queued (the org is starved). At 50ebd8877, Python shard 2's runner shut down at 90% with no test failure, the same memory pressure Stop the test shards' memory growing with every Simulation under sysmon coverage #2331 fixes.
• Land with Stop the test shards' memory growing with every Simulation under sysmon coverage #2331, which is stacked on this branch and already handed off, so the shards run with the memory fix. The hub plans to fold both into rollup B (Land six reviewed pull requests together (rollup B) #2380).
• Measurement: the four measurement branches gc-measure/2317* are deleted, but their runs stay on GitHub. The downloaded artifacts, the analysis script and each run's tables are in ~/reviews/uk-hub/reviews/uk2316-gc/.
• Follow-ups: re-pick the shard count (main now uses four) once Stop the test shards' memory growing with every Simulation under sysmon coverage #2331 lands; that's a separate chip. There are no decisions for Max.

[MaxGhenis]

UK hub (2026-10-12 11:05Z): this PR's head 37cadb471 is merged unchanged into rollup B, #2380. The rollup now holds #2365, #2366, #2375, #2271, #2317 and #2331, and one CI run tests all six together on current main. Merging #2380 with a merge commit marks this PR merged. The hub cancelled this PR's own queued runs, which the rollup replaces. Please don't push to this branch now: a new commit would not be in the rollup, and this PR would stay open.