Skip to content

Cancel superseded CI runs and add job timeouts - #76

Merged
MaxGhenis merged 2 commits into
mainfrom
ci-capacity/ci-hygiene
Oct 9, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
ci-capacity/ci-hygiene

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What changes

  • pr.yaml: add the concurrency group ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} with cancel-in-progress: ${{ github.event_name == 'pull_request' }}. A new push to a PR cancels that PR's older run. Each push to main gets a group of its own (its run id), so main runs are never cancelled and never left pending behind one another.
  • A 10-minute timeout on each job (Build, Tests, Lint) instead of GitHub's 6-hour default. The slowest of these jobs took 2.0 minutes in the window below.

Why

PolicyEngine is on GitHub Team, which runs at most 60 standard GitHub-hosted jobs at once across all the org's repos (limits). That pool was saturated from 2026-10-06 to 10-08, mostly by policyengine-us PR CI. At 2026-10-08 16:28Z all 60 slots were busy and 227 jobs were queued.

The jobs here are short, but each run waits in that queue. In the window below:

  • the PR workflow ran 92 times (78 on pull requests, 14 on main);
  • its jobs waited a median of 33 minutes for a runner (p90 58);
  • 44 of the 92 runs (37 PR, 7 main) were superseded by a newer commit on the same branch before they finished. Their remaining job time was 67 runner-minutes on PRs and 15 on main.

How this was measured

Data: every workflow run created in the PolicyEngine org from 2026-10-06 00:00Z to a snapshot at 2026-10-08 16:28:45Z (2.687 days), with all of each run's jobs, fetched from the GitHub REST API.

  • Runner-minutes: for each job that got a runner, completed_at (or the snapshot time, if still running) minus started_at.
  • Queue wait: started_at minus created_at for the same jobs.
  • Cancel-in-progress counterfactual: runs are grouped by workflow file, event and head branch. For each run that has a newer run in its group, the job time it used after the newer run was created is what a cancel-in-progress group would have cancelled: the sum over its jobs of max(0, end - max(start, newer run's created_at)).

Expected saving

  • Cancel-in-progress on pull requests: 67 runner-min ÷ 2.687 days ≈ 25 runner-min/day. It also takes about 14 superseded PR runs a day out of the queue (37 ÷ 2.687). (The first version also cancelled superseded main runs, another 15 runner-min in the window. That was dropped so every main commit keeps its own result.)
  • Timeouts: 0 today, since no job hung. They cap a hang at 10 minutes instead of 360.

Risks and how they are bounded

  • Superseded PR runs show as cancelled. main runs are never cancelled. This is the repo's only workflow, and it only builds, tests and lints. Nothing deploys from it, so a cancelled run skips no deploy. The newest commit's run still runs every job, so required checks keep reporting on the PR head.
  • The group is keyed on the PR number (or the run id for main), so different PRs never cancel each other.
  • A 10-minute timeout is 5× the slowest observed job. Raising it is a one-line change.

Verification

Run on the branch after git fetch origin and git rebase origin/main (already on the tip, ee50e7d9):

  • actionlint 1.7.12 on origin/main's workflows and on the branch's: 0 findings on both.
  • No test in this repo reads .github/workflows, and no application code changed. This PR's own CI runs the edited workflow.
  • Re-ran the counterfactual on the run data: 37 + 7 superseded runs, 67 + 15 min.

🤖 Generated with Claude Code

The PR workflow ran 92 times from 2026-10-06 to 10-08 with no
concurrency group; 44 of those runs were superseded by a newer commit on
the same branch before they finished, and their jobs waited a median 33
minutes for a shared org runner. The workflow only builds, tests and
lints, so cancel the older run, and give each job a 10-minute timeout
(the slowest took 2 minutes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policyengine-slides Ready Ready Preview Oct 8, 2026 11:41pm UTC

Request Review

The concurrency group cancelled superseded runs on main as well as on
pull requests. A cancelled run leaves that main commit with no result,
and anything that reads main's status sees "cancelled". Pull request
runs still cancel when a newer commit arrives. Each push to main now
gets a group of its own (run_id), so it is neither cancelled nor left
pending behind another push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Landing on the merge gates. Audit at head 2c8d0ea6ad84040854391cd0a3344a6523482936:

  • gh pr checks exits 0 (5 passed, 0 skipped by design); mergeable is MERGEABLE; not a draft; no changes-requested review.
  • Independent review: a Codex reviewer on a Subfleet lane (run 20261008-194225-pe-ci-six-prs), which did not write the change, approved this head with no blocking findings. Verdict: ~/reviews/pe-actions-concurrency-2026-10-08/review-six-out.md. Its notes on stale description text were fixed in the description.
  • The landing session (local_cfdb0a5b) read the full diff at this head. The change touches only CI workflow configuration.
  • Context: the PolicyEngine org is at GitHub's limit of 60 concurrent standard-runner jobs for the Team plan, so superseded and hung jobs delay every repo's checks.

@MaxGhenis
MaxGhenis merged commit 3f0d5b4 into main Oct 9, 2026
5 checks passed

This branch was successfully deployed

1 active deployment
Preview – policyengine-slides — 2c8d0ea6 Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant