Stop Rest after cancellation, run Contrib after Baseline at 11 jobs, and share YAML runners by memory

[hua7450]

What and why

PR CI runs about 25 jobs at once and about 11.3 runner-hours per run, which competes for the organization's runners when many PRs arrive together. Two failures also recur as tests grow:

• A Rest job survives cancellation. It keeps running after the run is cancelled.
• YAML batches reach the 16 GB runner limit. They end in "The runner has received a shutdown signal", and today the fix is to re-split the batch layout by hand.

This PR changes CI only: no model, parameter, test-case or partner-contract changes, and no test is removed or skipped. Nothing in the new layout depends on hand-tuned groupings, so test growth needs no rearranging.

1. The second Rest shard stops when a run is cancelled

Full Suite - Rest (Python + variables) had a job-level if: ${{ always() }}. A job-level always() is still true after a manual cancel or a cancel-in-progress supersede, so the job kept running, or even started, after the cancel. Two stale runs needed GitHub's force-cancel API: 38625576761 and 38627128097.

The job now uses !cancelled(), so it still runs after a shard-1 failure and its aggregate check still reports. In release CI it also skips when shard 1 was skipped because the release lock failed. The two shards stay sequential.

2. Baseline first, then Contrib, 6 groups at a time

• Order: Contrib now needs Baseline. Most PRs change baseline policy, so their Baseline result arrives first.
• Lanes: each suite keeps one job per group (same checks, commands and 60-minute timeouts) with strategy.max-parallel: 6. GitHub starts the next group on a fresh runner as one finishes, so the lanes balance from real durations, with nothing to re-pair as groups grow.
• Runners at once: the Pull request workflow runs at most 11 jobs at once, down from about 25. It still uses 27 runners in total.
• When Baseline fails: Contrib is skipped instead of spending about 4 runner-hours. "Re-run failed jobs" then reruns only the failed Baseline group and runs Contrib for the first time, so no finished work repeats (fork test 38687143315). A Contrib that ran anyway would have all its groups rerun.
• Release CI (push.yaml) uses the same order, strategies, env and steps; a guard test checks that they match.

Contrib still runs on every PR. From April to October, 54 failed PR runs failed in Contrib while Baseline passed: baseline changes that broke contrib reform tests (for example Kentucky's family size credit change broke contrib/states/ky/hb13).

3. Two memory-aware YAML subprocesses per runner

A 16 GB, 4-CPU runner running one YAML subprocess is about 25% busy. The Baseline and Contrib jobs now set POLICYENGINE_BATCH_WORKERS: 2, and test_batched.py runs a second batch subprocess only while the runner's memory allows:

• Admission: another subprocess starts only while MemAvailable is at least 6,000 MB, and at most one starts every 20 seconds while the last one loads the model.
• Hand-off: a subprocess hands its unstarted files to a fresh one at a file boundary once the runner has less than 4,000 MB available; the remainder waits for admission.
• Eviction: if the runner falls below 2,000 MB mid-file, the newest of two subprocesses is stopped and its files rerun alone once the other finishes. Nothing else starts in between, and a part that already reported keeps its result.
• Splitting: a batch is split, whole files only, when there are fewer batches than workers.
• Elsewhere: where memory cannot be read (off Linux) one subprocess runs at a time; a plain --workers N (the partners target) behaves as before; local make test is unchanged.

Fork experiment, 20 groups × 5 variants, three runs (38687143268, 38689387981, 38710504296); runner-minutes for the 20 groups per run:

Variant	Mean	vs today	Lowest free memory	Runner shutdowns	Cases
One subprocess (today)	508	—	2.6–8.7 GB	0	34,794
Two, fixed 5 GB budget	327	−36%	0.08–0.3 GB	2 (crfb group)	crashed job: at least 50 of its 68 never ran
Two, memory-aware (this PR)	345	−32%	1.9 GB or more	0	34,794
Three / four, memory-aware (runs 1–2)	358 / 402	−30% / −21%	1.8 GB or more	0	34,794
Two + reform-combo admission / 6 GB budget / both (run 3)	340 / 345 / 348	within noise	1.9 GB or more	0	34,794

Three or four subprocesses are no faster because memory, not CPU, runs out first. Every memory-aware job passed exactly the same cases as one subprocess.

4. YAML batches size themselves to a memory budget

Memory accumulates across the YAML files one subprocess runs; on the last PR run single batches peaked at about 15 GB on the 16 GB runner (household 15.2 GB, gov/hhs 15.1 GB, New York 14.3 GB). test_batched.py's subprocess wrapper adds a small pytest plugin to every YAML batch (policyengine-core calls pytest.main(..., plugins=[...]) through the module, so no Core change):

• after each test file the subprocess reads its own peak memory (VmHWM on Linux; on macOS the lifetime peak physical footprint, which counts compressed memory);
• past 8,000 MB it writes the files it has not started to a remainder file and exits, and run_batch continues them in a fresh subprocess; the batch passes only if every part passes;
• it stops only between files, so a file always runs whole; an inherited --junitxml report is merged across parts;
• POLICYENGINE_BATCH_MEMORY_BUDGET_MB overrides the budget, and 0 runs one file per subprocess.

Verification

• Rehearsal of this wiring on the fork (38726097373): the real baseline-tests.yaml and contrib-tests.yaml, Contrib after Baseline. All 20 groups passed; Baseline finished at 34 minutes and Contrib at 68, with 345 Baseline + Contrib runner-minutes (515 on this PR's previous CI run) and at most 6 of their jobs at once.
• Cancellation: a draft conversion and a manual cancel on real CI stopped running jobs within about 90 seconds; waiting lanes and Rest shard 2 never started (runs 38667326165 and 38671743189 attempt 1). Fork sleep-job tests covered cancel-before-start, cancel-while-running, failure, skipped roots and called workflows; an always() control kept running in every cancel case.
• Memory budget against the full US model: household at a forced 3,000 MB budget ran as 20 subprocesses and passed exactly the same 644 cases as without the guard; previous PR CI run: no subprocess above 12.3 GB, against six batches at 12.3–15.2 GB before.
• Tests: 321 code-health tests pass locally. New guards cover admission, the stagger, the memory reader, eviction and its exclusive rerun, a part stopped after it reported, splitting, plain --workers, the floor hand-off (Linux), the 11-job cap, Contrib after Baseline, and PR/release parity. Five deliberate mutants of the scheduler (no rerun reservation, no admission, no stagger, a lost reservation, an empty split) each fail them.
• Independent review: adversarial review rounds over the experiment and the production diff; confirmed findings are fixed.

Designs tried first

1. Grouping 2–3 named groups as steps on 5 + 4 runners (CI run 38656284185): fixed pairs to rebalance, and a runner shutdown would lose every group on it.
2. 5 Baseline + 4 Contrib lanes side by side (this PR's previous head, run 38671743189 attempt 2): 14 jobs at once, Baseline at 65 minutes.
3. Skipping Contrib for baseline-only PRs: rejected; see the 54 Contrib-only failures above.

Test plan

• For policy changes: not applicable (CI only).
• make format
• Changelog fragment in changelog.d/
• Fork rehearsal of this wiring passes all 20 groups.
• This PR's CI passes with at most 11 jobs at once, Baseline before Contrib.

Hub microsimulation impact

None: only CI workflows, batch routing, code-health guards and the changelog change.

axiom: n/a: CI job routing

🤖 Generated with Claude Code