Skip superseded Push versioning instead of failing, and bound every Push job with a timeout

[MaxGhenis]

Summary

Two changes to .github/workflows/push.yaml, plus a changelog fragment:

1. Versioning exits cleanly when a newer push to main has superseded its commit. Today that job fails. It caused 47 of the 51 red Push runs on main between Oct 6 and Oct 8.
2. Every Push job gets a timeout. Five jobs had none and so inherited GitHub's 360-minute default. The push Rest job's limit rises from 60 to 90 minutes, matching the PR Rest job after Raise the Full Suite - Rest timeout from 60 to 90 minutes #9785.

This is part of an org-wide GitHub Actions capacity investigation. The org's 60 standard-runner slots were saturated on Oct 6 and Oct 7, with demand of about 84k runner-min/day against 86.4k of capacity.

Data and method

• Source. GitHub REST API: every policyengine-us workflow run created from 2026-10-06T00:00Z to 2026-10-08T16:28:45Z (2.687 days), with all of each run's jobs (filter=all).
• Runner-minutes. completed_at − started_at, counted only for jobs that recorded a runner_name.
• Push totals. 104 Push runs used 15,551 runner-min, and 51 of them concluded failure.
• Failure causes. Read from job conclusions and from job logs: versioning job 113391313632 and Publish job 112262572664.
• When main moved. Every push to main creates a Push run, so a Push run's created_at marks a push to main.

1. Versioning skips instead of failing

What happens today

On a PR merge, Push runs Release lock guard tests and then Update versioning. Versioning checks out github.sha, bumps the version, runs towncrier, and pushes an Update PolicyEngine US commit (EndBug/add-and-commit, fetch: false).

Both jobs wait for a runner. On merge pushes in the window, with the wait measured per job as started_at − created_at:

Job	n	Median wait	p90 wait	Max wait
Release lock guard tests	75	43.1 min	113.8 min	220.4 min
Update versioning	74	29.4 min	169.3 min	212.6 min

When several PRs merge inside that window, every versioning job except the newest commits on a stale parent. Its push is then rejected. The log of job 113391313632 shows ! refs/heads/main:refs/heads/main [rejected] (non-fast-forward) at the add-and-commit step.

Measured

Cause of the 51 failed Push runs	Runs
Versioning push rejected because main had moved	47
PyPI connection reset in Publish, on a duplicate run for the same commit (the twin run published)	1
Test jobs failed on a version-bump commit	2
Release lock guard tests failed after 44.9 min	1

• In all 47 failed versioning jobs, a newer push to main had been created before the job started.
• In all 27 successful versioning jobs, none had.

Why nothing is lost

towncrier builds all of changelog.d, and bump_version.py infers the bump from every fragment. So the newest merge's bump consumes every pending fragment. For example, 7c857cf removed the three fragments from the burst of three merges at 13:02Z on Oct 8. The 47 failures were false alarms, and they hide the 4 real failures.

The fix

A new first step asks GitHub's compare API how main relates to this commit: gh api repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...main --jq .status, using the job's GITHUB_TOKEN. partner-api-impact-notice.yaml already uses gh api on ubuntu-latest.

• Status ahead (this commit is an ancestor of main, and main has more). The step writes a ::notice titled "Versioning superseded". Every later step (all 9) is gated on steps.head.outputs.current == 'true', so they skip. The job succeeds in seconds and skips the full-history checkout.
• Any other status (identical, behind, diverged, empty) or a failed lookup. The job runs exactly as before; a failed lookup also writes a ::warning.

The first version compared main's head SHA with github.sha for equality. An independent review showed that a stale read returning an older head would then skip the newest merge's job too, leaving no bump at all. Skipping only on ahead closes that: the newest merge's job skips only if GitHub reports a newer descendant of its commit on main, so a stale read cannot make it skip falsely (assuming main only moves forward).

The job has no concurrency group, so concurrency never cancels it.

2. Timeouts

The table below uses Push job durations from the window. "n" counts jobs that completed with success or failure.

Job	Before	After	Observed
ReleaseLock	360 (default)	10	n=29, max 0.4 min
ReleaseLockTests	360 (default)	10	n=104 with a runner, max 1.4 min. Run 37650158537 failed after 44.9 min with no runner recorded
Lint	360 (default)	10	n=28, max 0.7 min
versioning	360 (default)	20	Successful runs max 4.5 min, any run max 12.0 min
Publish	360 (default)	20	n=25, max 1.3 min
Rest	60	90	n=27, p50 51.2 min, max 58.8 min. Run 37719726248 hit 60.3 min and was cancelled, which skipped that version's Publish
Baseline, Contrib, Microsimulation, HouseholdAPIPartners	60	unchanged	Successful runs max 35.3, 45.9, 32.4 and 12.4 min

Expected runner-min/day saved

• Versioning skip: about 61 runner-min/day.
  • The 47 failed versioning jobs used 175 runner-min (median 3.65 min each). Nearly all of that was the fetch-depth: 0 checkout, which took 3.6 of 3.7 min in job 113391313632.
  • A skipped job runs only the head check, which takes seconds. So the saving is about 47 × 3.5 ≈ 165 min, and 165 / 2.687 days ≈ 61/day.
• Timeouts: about 0 measured in this window.
  • The one long ReleaseLockTests failure (run 37650158537, 44.9 min) recorded no runner, so it counted 0 runner-min.
  • The value of the timeouts is a cap: a hung job now holds a slot for at most 10 to 20 min instead of 360.
• Rest 60 → 90 can add minutes. A Rest job that would have hit 60 min can now run up to 30 min longer. In the window that happened once, so the cost is at most 30 / 2.687 ≈ 11/day. In exchange, that version publishes instead of being skipped.
• Net: about 50 to 61 runner-min/day. That is small against ~84k/day of demand. The main gains are that a red Push run on main means something again, and that hung jobs release their runner early.

Invariants

• A job skips only when main already contains its commit and more. Then its own push would be non-fast-forward: the job commits on top of github.sha and pushes to main without fetching. A later job (main's newest merge) is responsible for the bump. Any doubt (a failed, empty or stale answer) runs the job as today.
• A failed lookup changes nothing. The job behaves exactly as it does today.
• No changelog fragment is dropped. A skipped job leaves its fragments in changelog.d. The newer push's versioning job, or the next merge's, consumes them.
• Every Push job has a finite timeout. The new limits are 1.67× the observed maximum for versioning (20 against 12 for any run) and 7× or more for the others. The four suites left at 60 have 1.3× (Contrib, 45.9) to 4.8× (HouseholdAPIPartners) headroom.

Risks and how they are bounded

• A stale read. For the newest merge, a compare against an older main returns behind or identical, so the job runs and behaves as today.
• A merge between the check and the push. If one lands in the remaining few minutes, the push is rejected exactly as today.
• A timeout that is too tight on a slow day.
  • versioning (20 min against a 12-min max): it fails red, and the next merge's versioning picks up the fragments.
  • Publish (20 min against a 1.3-min max): a re-run is safe because the upload uses skip-existing: true.
• Permissions. The check makes one read-only compare call with the default GITHUB_TOKEN; it needs contents: read, which the workflow inherits (it declares no permissions). It uses no new secret.
• The newest merge's versioning job fails after older ones skipped. No bump lands until it is re-run or another PR merges. Before this PR the same happened, with the older jobs red as well.

Runner budget

CLAUDE.md's "Test design and CI cost" section governs runner-budget changes. This PR adds no jobs, matrix entries, runners or concurrent processes. It adds one step (one API call) to an existing job. It lowers or caps job durations, except the push Rest limit, which now matches the PR Rest job.

Verification

Run on this branch after it was rebased onto the latest origin/main (e44d7c3), in a fresh worktree with uv sync --locked --extra dev on Python 3.14:

Command	Result
actionlint .github/workflows/push.yaml (actionlint 1.7.12)	Clean, exit 0
RELEASE_LOCK_REAL_UV=1 python -m unittest discover -s .github/tests -p 'test_release_*.py' (the PR validation step; it includes test_release_build.py, which parses the Publish job)	43 tests OK
uv run pytest policyengine_us/tests/code_health/test_rest_python_groups.py policyengine_us/tests/code_health/test_make_test_matches_ci.py policyengine_us/tests/code_health/test_pr_runner_budget.py (the code-health tests that parse the workflows)	21 passed
uv run ruff format --check . (ruff 0.15.5)	6723 files already formatted

A property check on the parsed push.yaml also passes:

• The job set is unchanged (10 jobs), and every job has the timeout in the table.
• The head check is versioning's first step, and all 9 later steps are gated on it.
• No job, and not the workflow, has a concurrency group.
• Publish still needs the full suite.
• The head-check script was run under bash -e with a stub gh for three cases (first version; superseded by the committed test below).
• As a negative control, the same check fails on origin/main's push.yaml.

Second round (commit 59d4fbd): a new committed test, policyengine_us/tests/code_health/test_push_versioning_superseded.py, runs the step's script with a stub gh for every compare status (ahead, identical, behind, diverged, empty, null) and a failed lookup, checks that the compare goes from this commit to main, and checks that every later step is gated. 9 passed. Mutation check: loosening the condition to "anything but identical" fails 5 of the 9. actionlint .github/workflows/push.yaml is clean on base and head. The branch also merges current origin/main (f3456c7).

actionlint on the whole repo reports only existing shellcheck style notes in pr.yaml and weekly-uv-lock.yaml, which this PR does not touch.

After merge, check: the next burst of merges should show green versioning jobs with a "Versioning superseded" notice. A burst usually gets one Update PolicyEngine US commit; it gets two when one merge's bump lands before the next merge.

Known wording limit (left as is to keep the reviewed head): the skip notice says the newest merge's versioning job bumps the version with this commit's fragments. When main is ahead only because that bump already landed, the fragments were already consumed.

Related

#10017, a draft stacked on this branch, holds the per-shard concurrency change from the same investigation. It needs a release-policy decision, because superseded versions would not be published to PyPI. This PR does not depend on it.

axiom: n/a: CI infrastructure change

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the US + core hub (from session local_cfdb0a5b, the PolicyEngine Actions concurrency chip).

• Head: 59d4fbd54d8fbb448fd0bc3289983eaec5fe2cec
• Reviews: an independent Codex reviewer (Subfleet) requested changes at 32905c36d5: a stale read of main's head could skip the newest merge's versioning job, leaving no bump. Fixed at 59d4fbd54d, which skips only when the compare API reports main ahead of the commit, plus a committed test over every compare status and a failed lookup (mutation-checked). Round 2 at this head: approve, no blocking findings. Files: ~/reviews/pe-actions-concurrency-2026-10-08/review-us-10016-out.md and review-us-10016-round2-out.md. The description was corrected per the round-2 notes; the head is unchanged since that review.
• Passed / pending: CI is running at this head (queued behind the org backlog). mergeable was still computing at hand-off. main has no required status checks; one approving review is required.
• What it changes: Push only. versioning skips instead of failing when main already contains its commit and more; every Push job gets a timeout; push Rest goes from 60 to 90 minutes to match the PR job. Merging it publishes a normal patch release with its changelog fragment.
• Depends on: nothing. Decision: none needed.
• Follow-up owed: none. One wording limit is noted in the description: the skip notice can overstate where the fragments go when main is ahead only by an already-landed bump.

[MaxGhenis]

US + core hub merge audit, us#10016 at 59d4fbd54d8fbb448fd0bc3289983eaec5fe2cec (squash)

• Change: the Push workflow only.
  • Versioning skips, instead of failing, when main already contains its commit plus more. It skips only when the compare API reports main ahead of the commit.
  • Every Push job gets a timeout.
  • Push Rest goes from 60 to 90 minutes, to match the PR job.
• Reviews:
  • An independent Codex reviewer requested changes at 32905c36d5: a stale read of main's head could skip the newest merge's bump.
  • The fix, with a committed test over every compare status and a failed lookup (mutation-checked), is approved by round 2 at this head.
• Hand-off: from the Actions concurrency session (local_cfdb0a5b). No decision is needed.
• Gates checked live at merge:
  • gh pr checks exits 0;
  • the latest Pull request run at the head concluded success;
  • MERGEABLE;
  • not a draft;
  • no CHANGES_REQUESTED review;
  • the semantic-overlap guard;
  • head pinned.
• Impact: CI workflow only; no model code changes. Merging publishes a normal patch release with its changelog fragment.
• --admin: used only because a required approving review is the sole block.