You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cut the CPU the Python tests spend in garbage collection #2316
Most of the Python test time is in the property tests: on 10 October 2026 the 51 *_properties.py files took 74% of pytest worker time (run 38075384588, #2099). Each Hypothesis example builds a Simulation, which builds a new CountryTaxBenefitSystem: it loads about 1,100 variable modules again and clones the parameter tree. Python's cycle collector runs often while that happens, and its full collections walk every tracked object in the worker, including the roughly 430,000 present once policyengine_uk is imported.
On policyengine_uk/tests/test_ni_class_1_age_properties.py (52 examples), local CPU seconds were 88-96 without coverage, 77 with gc.freeze() after imports, and 61 with gc.freeze() plus gc.set_threshold(10000, 10, 10) (peak RSS 0.89 GB against 0.79 GB). The threshold alone gave 67.
Proposed change
In policyengine_uk/tests/conftest.py, in each process that runs tests:
raise the first-generation GC threshold to 10,000;
freeze the objects that exist once the tests are collected, so collections no longer walk the import-time heap.
release_dead_variable_modules must still treat frozen systems as live (gc.get_objects() does not return frozen objects) and still free the systems the tests build.
Measure total worker-minutes across the CI shards and peak memory per runner before and after; if compute falls enough, consider two shards instead of three.
Activity