You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The Python test shards run out of memory under sys.monitoring coverage #2330
The Python test shards (#2269, plus #2317) run four pytest-xdist workers on ubuntu-24.04-arm runners (15.6 GiB, 3 GiB swap) with COVERAGE_CORE=sysmon. They are close to running out of memory.
Evidence
Run 38155711978 (branch gc-measure/2317), sampling MemTotal - MemAvailable every 2 s:
Peak used memory was 14.2 to 15.3 GiB on every shard under every GC setting, including Python's defaults.
Python RSS climbs steadily for the whole run. On shard 2 (default GC) it went from 6.9 GiB at 60 s to 14.5 GiB at 1,100 s, then stayed flat near the limit.
Per-worker peak RSS reached 5.8 GiB. The worker with the highest peak in each shard ran the microsimulation/ tests.
With the first-generation threshold at 25,000, two of three shards stalled near the end and were cancelled.
The workflow comment ("together they peak at about 9 GB") no longer holds.
Cause (measured locally, Python 3.13, coverage 7.16.2, policyengine-core 3.33.2)
Coverage's sys.monitoring core keeps every code object it has seen alive until the process exits (SysMonitor.code_objects in coverage/sysmon.py). policyengine-core compiles a system's ~1,110 variable files again for each new system, and every Simulation builds a new system. On test_uc_work_related_groups_properties.py, run serially:
Coverage
Traced heap after 10, 20, 30 systems
Code objects held by coverage
none
160, 163, 171 MB
n/a
ctrace
165, 174, 174 MB
n/a
sysmon
186, 226, 259 MB
38,133, 63,365, 88,610
So under sysmon each Simulation leaves about 2,500 code objects and 3.6 MB of traced heap that are never freed. Main is unaffected today because coverage on Python 3.13 defaults to the C tracer; #2269 switches to sysmon, and Python 3.14 makes it coverage's default.
Activity
Split the Python tests over four runners: three nearly ran out of memory
Compile each variable file once per test process
Compile each variable file once per test process