Cancel superseded full-suite shards on main (decision needed: skips superseded PyPI versions)

[MaxGhenis]

Decision needed

Should policyengine-us stop publishing superseded versions to PyPI, in exchange for about 630 runner-min/day saved during merge bursts?

• Cost: version gaps. When a newer Update PolicyEngine US commit reaches a full-suite shard while an older one is still testing, the older shard is cancelled. That older version's Publish is skipped, and only the newest version is published.

  • From Oct 6 00:00Z to Oct 8 16:28Z, 12 of 28 versions would not have reached PyPI. Each would still have a CHANGELOG.md entry with no matching release.
  • One version is already skipped today: run 37719726248's Rest job hit its 60-minute limit. Skip superseded Push versioning instead of failing, and bound every Push job with a timeout #10016 raises that limit.

• Cost: publishing waits for a green suite. If the newest suite fails, nothing is published until a fix lands. Today, the older passing version would still ship.

• Benefit. The change saves ~630 runner-min/day: 1,692 runner-min over the 2.687-day window. Excluding one duplicate push event, it saves ~420/day. The saving is concentrated in merge bursts, when two 23-job suites otherwise run at once on the org's 60 shared runner slots. Averaged over the day, 630 min is only 0.44 of a slot.

• Alternative: test-then-bump.

  1. Run the full suite on the merge commit, with the same per-shard cancellation.
  2. Bump the version and publish only for the newest commit that passed.

  This keeps every published version tested, but versions would still be skipped for superseded merges. The bump commit itself (version, lock root, changelog) would be built and checked by release_build.py but not fully re-tested. It is a larger change and needs its own review.

• Owner: Max, or the policyengine-us release owners.

• Options:

  • Accept: mark this ready and merge it after Skip superseded Push versioning instead of failing, and bound every Push job with a timeout #10016.
  • Reject: close this PR. Skip superseded Push versioning instead of failing, and bound every Push job with a timeout #10016 stands on its own.
  • Ask for test-then-bump instead.

Summary

Stacked on #10016 (base ci-capacity/push-versioning-timeouts). The change touches only .github/workflows/push.yaml, plus a changelog fragment.

Five Push test jobs each get a job-level concurrency group, one per matrix shard:

• ${{ github.workflow }}-${{ github.ref }}-baseline-${{ matrix.group }};
• …-contrib-${{ matrix.group }};
• …-household-api-partners;
• …-rest;
• …-microsimulation.

All five set cancel-in-progress: ${{ github.run_attempt == '1' }}. These are Baseline (11 shards), Contrib (9 shards), HouseholdAPIPartners, Rest and Microsimulation, which run only on the bot's version-bump commit. The baseline- and contrib- prefixes keep Baseline states-shard-1 and Contrib states-shard-1 in different groups.

• Never grouped, so never cancelled: ReleaseLock, ReleaseLockTests, Lint, versioning and Publish.
• What happens to an older run whose shards are cancelled. Its Publish shows skipped, because a needed job did not succeed. This was observed in run 37719726248, where Rest was cancelled at its time limit. The newer bump commit contains every change in the older one.
• Re-runs (run_attempt > 1) do not cancel. A re-run of an old run's shard waits behind the newer run's shard, so it can never cancel the newest release's tests.

Why job-level, not workflow-level

Workflow-level concurrency may use only the github, inputs and vars contexts, so it cannot be per shard. With cancel-in-progress it would also cancel the whole older run, including a merge push's versioning job or a Publish in flight. Job-level groups may also use needs, strategy and matrix (GitHub docs, workflow syntax for jobs.<job_id>.concurrency).

Semantics this change relies on, from the docs:

• A group holds at most one running and one pending job.
• A newly queued job cancels the pending one, and with cancel-in-progress the running one too.
• Group names are case-insensitive.

Data and method

• Source. GitHub REST API: every policyengine-us workflow run created from 2026-10-06T00:00Z to 2026-10-08T16:28:45Z (2.687 days), with all of each run's jobs (filter=all).
• Runner-minutes. completed_at − started_at, counted only for jobs that recorded a runner.
• Push totals. 104 Push runs used 15,551 runner-min. 28 of them ran the full suite, all for policyengine[bot] version-bump commits, at a mean of 545 runner-min each.

Savings model.

1. For each full-suite run i, take the next full-suite run j.
2. i counts as superseded if j was created before i's last test job completed.
3. For each test job k of i, compare it with j's job of the same name:
   • if k started at or after j's job was created, all of k's runner time is saved;
   • otherwise, only the time after that creation is saved.

Expected runner-min/day saved

• 13 of 28 full-suite runs were superseded, and 12 of them had a nonzero saving. Those 12 are the 12 unpublished versions above.
• Saved: 1,692 runner-min, and 1,692 / 2.687 days ≈ 630 runner-min/day. That is 10.9% of Push runner-minutes (15,551), or about 1% of org usage in the window.
• 562 of the 1,692 came from one duplicate push event: runs 37453034547 and 37453035390 were created 1 s apart for the same commit, cause unknown. Without it, the saving is 1,130 min, or ≈420/day.

Invariants

• Only the 23 test-job instances are grouped, and each has its own group. Across the instances, the groups are 23 distinct case-insensitive names.
• The release and guard jobs are never cancelled. ReleaseLock, ReleaseLockTests, Lint, versioning and Publish have no group, and there is no workflow-level concurrency.
• Publish's needs is unchanged, so nothing reaches PyPI without every shard passing on the exact commit it builds.
• A re-run cannot cancel a newer run's running shard, because cancel-in-progress is false when run_attempt > 1.

Risks and how they are bounded

• Version gaps on PyPI. This is the decision above.

• Two semantics are inferred, not stated in the docs.

  • That a matrix job's group is evaluated per matrix instance.
  • That a job with needs enters its group when it is created, after its needs finish.

  Check this on the first two overlapping bot pushes after merge: the older run's cancelled jobs should be exactly the shards that the newer run had reached. If the semantics differ, the effect is less saving, not a broken release, because Publish still needs every shard to succeed.

• Groups are ordered by when a job enters them, not by commit order. If an older run's test jobs entered after a newer run's, the older run would cancel the newer one. The older version would then publish, and the newest would wait for the next bump. All of a run's test jobs need only ReleaseLock, so they enter together, and a mixed outcome where neither run publishes cannot come from entry order. In the window, 0 of the 378 pairs of full-suite runs had a newer run's test jobs created before an older run's.

• A failed newest suite blocks publishing until a fix lands. Today, the older passing version would ship. This is part of the decision.

Runner budget

CLAUDE.md's "Test design and CI cost" section governs runner-budget changes. This PR adds no jobs, matrix entries, runners or concurrent processes. It only cancels already-superseded test jobs on main.

CI on this PR

pr.yaml triggers only on PRs into main, and this is a draft stacked on #10016's branch, so no CI runs here yet. After #10016 merges, GitHub retargets this PR to main. Marking it ready then runs the full PR suite.

Verification

Run on this branch: #10016's commits, rebased onto the latest origin/main (e44d7c3), plus this commit. The run used a fresh worktree with uv sync --locked --extra dev on Python 3.14. This branch's push.yaml is byte-identical to the originally reviewed three-commit tip.

Command	Result
actionlint .github/workflows/push.yaml (actionlint 1.7.12)	Clean, exit 0
RELEASE_LOCK_REAL_UV=1 python -m unittest discover -s .github/tests -p 'test_release_*.py' (includes test_release_build.py, which parses the Publish job)	43 tests OK
uv run pytest policyengine_us/tests/code_health/test_rest_python_groups.py policyengine_us/tests/code_health/test_make_test_matches_ci.py policyengine_us/tests/code_health/test_pr_runner_budget.py	21 passed
uv run ruff format --check . (ruff 0.15.5)	6723 files already formatted

A property check on the parsed push.yaml also passes:

• Each of the 5 test jobs has a group, and cancel-in-progress is ${{ github.run_attempt == '1' }}.
• Expanding the matrices gives 23 test-job instances with 23 distinct case-insensitive group names.
• ReleaseLock, ReleaseLockTests, Lint, versioning and Publish have no group, and there is no workflow-level concurrency.
• Publish still needs the full suite.
• Every job has a timeout, and Skip superseded Push versioning instead of failing, and bound every Push job with a timeout #10016's versioning gate and its stub-gh cases still pass.

As a negative control, the "no groups" check from #10016 fails on this file, as it should.

axiom: n/a: CI infrastructure change

🤖 Generated with Claude Code

[MaxGhenis]

Closing this draft. It cancels a superseded full-suite shard on main when a newer version-bump commit reaches the same shard, which means the superseded PyPI versions are never published. That release-policy question was never decided, and the problem it addresses (bursts of merges to main, each running a full suite) is mostly removed by the integration-branch pilot in #10018, where small fixes reach main once per promotion.

Context: the PolicyEngine org was at GitHub's 60-concurrent-job limit on 2026-10-06..08 (state and measurements: ~/reviews/pe-actions-concurrency-2026-10-08/STATE.md). The companion #10016 (skip superseded versioning instead of failing, job timeouts) merged on 2026-10-09.

If merge bursts on main still saturate runners after #10018 is adopted, reopen this; the branch ci-capacity/push-shard-concurrency is kept.