Skip to content

The Python test shards run out of memory under sys.monitoring coverage #2330

Closed

Description

@MaxGhenis

The Python test shards (#2269, plus #2317) run four pytest-xdist workers on ubuntu-24.04-arm runners (15.6 GiB, 3 GiB swap) with COVERAGE_CORE=sysmon. They are close to running out of memory.

Evidence

Run 38155711978 (branch gc-measure/2317), sampling MemTotal - MemAvailable every 2 s:

  • Peak used memory was 14.2 to 15.3 GiB on every shard under every GC setting, including Python's defaults.
  • Python RSS climbs steadily for the whole run. On shard 2 (default GC) it went from 6.9 GiB at 60 s to 14.5 GiB at 1,100 s, then stayed flat near the limit.
  • Per-worker peak RSS reached 5.8 GiB. The worker with the highest peak in each shard ran the microsimulation/ tests.
  • With the first-generation threshold at 25,000, two of three shards stalled near the end and were cancelled.

The workflow comment ("together they peak at about 9 GB") no longer holds.

Cause (measured locally, Python 3.13, coverage 7.16.2, policyengine-core 3.33.2)

Coverage's sys.monitoring core keeps every code object it has seen alive until the process exits (SysMonitor.code_objects in coverage/sysmon.py). policyengine-core compiles a system's ~1,110 variable files again for each new system, and every Simulation builds a new system. On test_uc_work_related_groups_properties.py, run serially:

Coverage Traced heap after 10, 20, 30 systems Code objects held by coverage
none 160, 163, 171 MB n/a
ctrace 165, 174, 174 MB n/a
sysmon 186, 226, 259 MB 38,133, 63,365, 88,610

So under sysmon each Simulation leaves about 2,500 code objects and 3.6 MB of traced heap that are never freed. Main is unaffected today because coverage on Python 3.13 defaults to the C tracer; #2269 switches to sysmon, and Python 3.14 makes it coverage's default.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions