Cut PR CI from hours to about 20 minutes: Linux arm64, 4 xdist workers, cancel superseded runs

[MaxGhenis]

The PR Test job was the bottleneck for every open PolicyEngine-UK PR. The figures below come from the Actions API: every "Code changes" run created since 2026-10-02T00:00Z whose Test job finished with success or failure. Those 78 jobs on macos-latest waited a median of 139 minutes for a runner and then ran for a median of 242 minutes; p90 end to end was 10.8 hours. Two runs hit GitHub's 6-hour limit: #1992's (run 37009617251) and #2088's (run 37090099823, which shows as cancelled). This PR's own Test job took 22m36s on ubuntu-24.04-arm, and every test node gets the same outcome as on macOS.

What changes, one commit each

1. Cancel superseded runs. pr_code_changes.yaml and pr_docs_changes.yaml get a concurrency group keyed on the PR number, with cancel-in-progress for pull_request events. Pushes to main use code_changes.yaml, which has no concurrency group, so they are never cancelled. Manual dispatches are grouped by ref and not cancelled.
2. Linux arm64 instead of macOS. The Test job moved to macos-latest in 2025 (Some uprating fails on ubuntu but not macos #1092, Move back to correct private pension income index #1207) because uprating differed between Ubuntu and macOS (Ubuntu is returning different results from macos #1209). That is no longer the case (evidence below). ubuntu-24.04-arm keeps the macOS runners' CPU architecture and has 4 physical cores, against 2 on ubuntu-latest. The main workflow switches too, so PR and main CI run on the same platform. The x86 Python-Compat smoke imports are unchanged.
3. Test isolation fixes, needed before running in parallel:
   • test_latest_data_smoke.py set POLICYENGINE_UK_DEFAULT_DATASET to the unpinned dataset and never restored it. Any later Microsimulation() in the same process silently used the unpinned data. The override now lasts only while the fixture builds its simulation.
   • Memory leak. policyengine-core registers every variable module of each new tax-benefit system in sys.modules under the system's id and never removes them (TaxBenefitSystem construction permanently leaks ~5,600 sys.modules entries (~55-60MB) per build — unreclaimable by gc policyengine-core#520). Every Simulation builds its own system, so each leaves about 1,000 modules (roughly 10 MB) behind. The property tests build hundreds. Host memory peaked at 15.8 GB of the Linux runner's 16 GB during the serial pytest run. GitHub's macOS arm64 runners have 7 GB (3 M1 cores), less than half that, which fits the same suite taking anywhere from 74 minutes to over 4 hours there.
   • An autouse fixture now drops the modules of systems that no longer exist after each test. A regression test checks three things: a dead system's modules go, a live system's modules stay, and its simulation still computes. The test also fails if core stops leaking, which is the signal to delete the workaround.
4. pytest-xdist, 4 workers, --dist worksteal. pytest-xdist is a new dev dependency. make test is now test-yaml followed by test-python. CI passes PYTEST_ARGS="-n 4 --dist worksteal"; without that argument, make test still runs serially, as before. Hypothesis settings are untouched, and no test is removed or weakened.
5. timeout-minutes: 60 on both Test jobs. That is more than three times the measured 17m30s.

Before and after (real CI runs)

Run	Runner	Pytest step	Whole job	Peak memory
main 3525263, run 37013961610	macos-latest	4h17m59s	4h58m, after 5h13m queued	(7 GB runner, swapping)
PR #1940, run 37014647328 (a good runner)	macos-latest	1h13m56s	1h20m
hb-ctr-non-dep-exemptions, run 37017928329 (started 15:21Z on 10/2)	macos-latest	4h11m45s	4h55m
main 3c48247, serial, run 37062825983	ubuntu-latest	76m36s	79m	15.7 GB of 16
main 3c48247, serial, run 37062825983	ubuntu-24.04-arm	69m24s	72m	15.8 GB of 16
main 3c48247, 4 workers loadfile, before the module release, run 37062825983	both	runner shut down 14-24 min in		(serial already peaked at 15.8 GB)
this branch, 4 workers loadfile, run 37120279129	ubuntu-latest	18m26s	20m45s	15.2 GB of 16
this branch, 4 workers loadfile, run 37120279129	ubuntu-24.04-arm	17m51s	20m28s	11.5 GB of 16
this branch, 4 workers worksteal, run 37120279129	ubuntu-24.04-arm	14m56s	17m30s	9.2 GB of 16
this PR's own CI, run 37122388608 (head ffdf3e3, current main merged in: 2,374 tests)	ubuntu-24.04-arm	21m56s (pytest's own total 19m39s)	22m36s, after 14m40s queued	not sampled

Peak memory is the runner's used memory from free -m, sampled every 30 s (every 5 s in round 1). It is shown as MiB ÷ 1,000 against a total of 15,947 MiB, so "9.2 GB of 16" means 9,242 of 15,947 MiB (58%), and the serial runs used 99%.

Memory peaks vary from run to run with worksteal scheduling (9.2 GB to 11.5 GB on arm across the two schedulers). If a run ever exhausts memory, the fallback is -n 3. Each runner keeps one job slot whether pytest uses one core or four. Four workers in one job therefore cut runner time per PR about fourfold, where sharding across jobs would only have cut wall time. This matters because the org's Linux runners also queue at busy times.

Evidence that Linux gives the same results

• Parameter tree. I dumped every Parameter object's full value history: 2,706 objects and 43,666 dated values, including the 1,353 baseline copies. The dump is byte-identical on macOS arm64 (local, locked environment) and in CI on ubuntu-latest and ubuntu-24.04-arm (run 37120279129 artifacts), sha256 79abbf29…. Some uprating fails on ubuntu but not macos #1092 traced the original difference to chained uprating and fixed it. Ubuntu is returning different results from macos #1209, which reported a later difference between operating systems, is still open; this dump and the outcome comparison below are the evidence that no difference remains.
• YAML tests. All 1,611 pass on both Linux runners, as on macOS.
• Pytest outcomes, node for node. The macOS main run on 3525263, the Linux serial runs on 3c48247 (the same code plus a version bump) and every xdist run report identical outcomes for all 2,263 nodes: 2,255 passed, 7 xfailed, 1 skipped. The xdist runs add only this branch's two new tests. The comparison parses each log's per-test outcome lines and reports nodes missing from either log and any outcome that differs. The only differences are the two new tests. (In the experiment logs a memory sampler sometimes printed onto the end of a result line; the parser strips that.) This PR's own run reports 2,365 passed, 8 xfailed and 1 skipped, because main has gained tests since.

Invariants

• The module release only removes sys.modules entries whose owning tax-benefit system no longer exists. It finds live systems with a gc scan, and only once more than eight systems own modules, so the scan is rare. A live system's variables, and anything that introspects them, are untouched. The regression test checks both sides.
• Under xdist every collected test runs exactly once. Each xdist run reports the same node set as the serial run (2,265 = 2,263 + 2).
• A serial make test without PYTEST_ARGS runs exactly the two commands it ran before.

For open PRs

Each open PR picks this up only after it merges main (or rebases). A PR whose CI starts before that still queues for macOS. Re-running an old run does not help, because a re-run keeps the workflow file it started with.

Not in this PR

• The real fix for the leak belongs in policyengine-core (Bump openfisca-tools #520): evict each module from sys.modules after exec_module. Once that ships, the regression test's precondition fails and the conftest workaround can be deleted.
• Every Simulation() re-imports all ~1,000 variable files to build a fresh system, about 1 s each. That is most of the property tests' cost and is worth a separate look.
• The Codecov upload has been failing with "Token required because branch is protected". It is non-blocking and unchanged here.

axiom: n/a: CI and test infrastructure only; no policy logic changes.

🤖 Generated with Claude Code

[MaxGhenis]

Merged at the reviewed head ffdf3e3 with a merge commit, on these gates:

• gh pr checks exited 0 on run 37122388608: Test 22m36s on ubuntu-24.04-arm.
• The PR was MERGEABLE, not a draft, and had no review requesting changes.
• An independent Opus 5.5 review (Subfleet job 20261003-081849-pe-uk-2115-review-b) returned APPROVE for ffdf3e3. It raised no P1s. I addressed its two P2s and its PR-text P3s in the description before merging: the PR's own CI row is filled in, the Ubuntu is returning different results from macos #1209 wording is corrected, the 78-job statistics now name their source, and the memory headroom note gives -n 3 as the fallback.
• I left two code nits from the review for a follow-up PR, to avoid holding this one for another CI cycle:
  • The conftest comment cites host memory, but says one process.
  • The regression test's precondition should skip rather than fail once policyengine-core#520 is fixed upstream.

Open PRs pick this up by merging main. A re-run of an old run keeps the macOS workflow it started with.