You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: the lanes App installation token runs out under many concurrent PRs #15497
On 2026-11-11/12 the CI installation token hit its hourly limit four times (per the hub session's count), failing every in-flight PR check with API rate limit exceeded for installation. Example: run 43143452524 (mark-lanes could not mark ci-lanes pending) and the CodeQL jobs in run 43143445098 on #15468. Recovery today is a manual gh pr update-branch per PR after the reset, and the kill switch CLAUDE_LANES_REPO_DISABLED stays on until then.
Expected
A burst of ~15 open PRs (the stamp-sweep batches plus their splits) completes CI without exhausting the token, or a run that hits the limit re-runs itself after the reset instead of failing ci-status.
Evidence to gather first
Calls per PR run by workflow (the lanes already log rate-limit headers, #8062), and which jobs spend most.
Related
Found during the Fable 5.2 adoption effort (Q11 stamp sweep, #15468).
Recurred 2026-11-12 03:32-03:34 UTC (fifth exhaustion this night): the installation core pool hit used=5000 remaining=0 (reset 03:53 UTC) and failed review-status on #15469 (run 43148031786, run 43148031744) and ci-status's file read (run 43148031104). Load at the time: about 12 open PRs from one effort, four flipped ready within a few minutes.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: when many PRs run CI at once (about 12–15 from one effort), the lanes App installation token reaches its 5000/hour core limit. Checks such as mark-lanes, review-status, ci-status and CodeQL then fail, and recovery is a manual branch update per PR after the reset. The goal is either a burst that completes without exhausting the token, or a run that retries itself after the reset.
The first step is the evidence you listed: API calls per PR run by workflow and the jobs that spend the most. The rate-limit header logging from #8062 should supply that data.
I found no open duplicate and no merged pull request that already fixes this.
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
The token ran out again on 2026-11-16, twice in under an hour: the brake set at 19:29Z, and the start gate tripped at 20:11Z with 6% left, 39 minutes into the window that started at 19:32Z. About 18 PRs were open.
Recorded core calls per workflow job, from the hub's per-job call log (gh-job-calls.jsonl, bucketed by the job's finish time):
What this shows: no single workflow drains the token. The spend is volume across the open PRs. CI Status ran about 65 times in 40 minutes for about 18 PRs, and 58 of those runs were re-runs dispatched by pr-dispatch-feedback. The one outlier per job is reconcile-lost-runs, at 162 calls in a single job.
Proposed first cuts, in order:
Fewer CI Status re-runs: skip or coalesce rerun ci status when nothing CI Status reads has changed since its last run on that head.
A per-run call cap in reconcile-lost-runs, or paging it across runs.
Both are UNVERIFIED as savings until a window is measured after the change. About 1,300 of the window's calls were not yet in the log at 20:12Z (jobs still running, or workflows not logged), so the ranking may still move.
Correction to my earlier comment on this issue (#15497 (comment)): proposed cut 1 is withdrawn. My count of CI Status re-runs was wrong.
In the 19:32-20:12Z window, 62 rerun ci status jobs ran, but only 8 of them re-ran CI Status. Each of the others is about 1.8 calls of checking. 6 of those 8 re-runs turned "Require both review lanes on this head" from red to green, so they were needed. The dispatcher's find step already re-runs only after the last review lane on a head finishes (pr-dispatch-feedback.yml:922-1116).
No CI Status run reads anything twice. One job reads the PR's files once and reuses pr-files.json (pr-require-checks.yml:5293, reused at :6918 and :7390), and pr_read caches the PR, its labels and its events (:5107-5149). The per-job notices show pulls/{n}/files=1 and issues/{n}/labels=1.
Considered and rejected: a per-PR cancel-in-progress group for CI Status runs triggered by label and edit events. It saves under 1% (7 of 34 such runs were superseded in 100 minutes), and it could cancel the run holding the current verdict.
Where the calls actually go: first-attempt CI Status runs, one per push, label change or edit, on each of about 18 open PRs. 87 full runs plus 24 label- and edit-triggered first attempts ran in 100 minutes. The spend grows with how many PRs are open and how often each is touched, not with any waste in one workflow.
Each exhaustion also ejected clean PRs from the merge queue (16598 and 16619 at 20:17Z), so a trip costs more than failed lanes.
Remaining cut: reconcile-lost-runs, about 162 calls in one job, running twice an hour, so about 6% of the hourly limit. Beyond that, the limit itself can only be raised by moving these reads to a separate token pool. Choosing between that and pacing how many PRs are open at once is a design decision, not a fix.
Follow-up on the remaining cut, reconcile-lost-runs: no safe REST cut, and its share was overstated.
Full passes alternate with paced ticks that cost 1-2 calls, so the job uses about 162 core calls an hour (about 3% of the limit), not 6%. Full passes in the call log cost 37 to 249 calls.
In run 43941232026 (164 calls), 100 calls were attempts/{n}/jobs reads, about 87 of them the fault-held check (reconcile.mjs:835-850, called from :1456). It reads each new successful pr-verify or pr-review-main run once and then settles it, so no run is read twice.
Ruled out:
a lane-hold-* artifact prefilter: its upload is continue-on-error, so a missing marker would skip a lost run;
ETag/304: each run is read once, so nothing repeats;
lowering FAULT_HELD_READS: that only defers the reads;
a since-last-clean-pass lookback: already in place.
One option is left: batch the fault-held reads through GraphQL nodes(ids:) on the runs' check-run node ids. That's about 85 fewer core calls per full pass, at about 5 GraphQL calls (estimate). It needs proof that GraphQL steps match REST for the hold check. With the saving under 2% of the hourly limit, it's low priority.
Conclusion: no single workflow is wasting the token. The spend tracks how many PRs are open and how often each is pushed, labeled or edited. Raising the ceiling is a design choice: pace concurrent PRs, or move part of the CI reads to a separate token pool.
Owner decision, 2026-11-16 about 5:1x PM EST, typed in the hub session (hub question Q197, "Go with the recommendation"): no structural change to raise the GITHUB_TOKEN ceiling for now. No Enterprise Cloud upgrade, no read-only App, and no open-PR cap. The brake stays, and runs the token floor breaks recover automatically after the reset (#16642, PR #16650). The GraphQL route was measured at about 6% at most and is not pursued (PR #16644, closed unmerged). This issue's Expected line ('or a run that hits the limit re-runs itself after the reset instead of failing ci-status') is delivered by PR #16650; this issue closes when that merges.
Closed: PR #16650 merged at 22:15:33Z. Runs the token floor breaks now recover automatically after the reset, the Expected line of this issue. Under the owner's Q197 decision (comment above), no structural change raises the ceiling for now; the brake stays.
Activity
kyle-sexton commentedon Nov 12, 2026
Recurred 2026-11-12 03:32-03:34 UTC (fifth exhaustion this night): the installation core pool hit used=5000 remaining=0 (reset 03:53 UTC) and failed review-status on #15469 (run 43148031786, run 43148031744) and ci-status's file read (run 43148031104). Load at the time: about 12 open PRs from one effort, four flipped ready within a few minutes.
github-actions commentedon Nov 12, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).
github-actions commentedon Nov 12, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: when many PRs run CI at once (about 12–15 from one effort), the lanes App installation token reaches its 5000/hour core limit. Checks such as mark-lanes, review-status, ci-status and CodeQL then fail, and recovery is a manual branch update per PR after the reset. The goal is either a burst that completes without exhausting the token, or a run that retries itself after the reset.
The first step is the evidence you listed: API calls per PR run by workflow and the jobs that spend the most. The rate-limit header logging from #8062 should supply that data.
I found no open duplicate and no merged pull request that already fixes this.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 12, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 16, 2026
The token ran out again on 2026-11-16, twice in under an hour: the brake set at 19:29Z, and the start gate tripped at 20:11Z with 6% left, 39 minutes into the window that started at 19:32Z. About 18 PRs were open.
Recorded core calls per workflow job, from the hub's per-job call log (
gh-job-calls.jsonl, bucketed by the job's finish time):Top consumers, 19:32-20:12:
What this shows: no single workflow drains the token. The spend is volume across the open PRs. CI Status ran about 65 times in 40 minutes for about 18 PRs, and 58 of those runs were re-runs dispatched by
pr-dispatch-feedback. The one outlier per job is reconcile-lost-runs, at 162 calls in a single job.Proposed first cuts, in order:
rerun ci statuswhen nothing CI Status reads has changed since its last run on that head.Both are UNVERIFIED as savings until a window is measured after the change. About 1,300 of the window's calls were not yet in the log at 20:12Z (jobs still running, or workflows not logged), so the ranking may still move.
kyle-sexton commentedon Nov 16, 2026
Correction to my earlier comment on this issue (#15497 (comment)): proposed cut 1 is withdrawn. My count of CI Status re-runs was wrong.
rerun ci statusjobs ran, but only 8 of them re-ran CI Status. Each of the others is about 1.8 calls of checking. 6 of those 8 re-runs turned "Require both review lanes on this head" from red to green, so they were needed. The dispatcher's find step already re-runs only after the last review lane on a head finishes (pr-dispatch-feedback.yml:922-1116).pr-files.json(pr-require-checks.yml:5293, reused at:6918and:7390), andpr_readcaches the PR, its labels and its events (:5107-5149). The per-job notices showpulls/{n}/files=1andissues/{n}/labels=1.Where the calls actually go: first-attempt CI Status runs, one per push, label change or edit, on each of about 18 open PRs. 87 full runs plus 24 label- and edit-triggered first attempts ran in 100 minutes. The spend grows with how many PRs are open and how often each is touched, not with any waste in one workflow.
Each exhaustion also ejected clean PRs from the merge queue (16598 and 16619 at 20:17Z), so a trip costs more than failed lanes.
Remaining cut: reconcile-lost-runs, about 162 calls in one job, running twice an hour, so about 6% of the hourly limit. Beyond that, the limit itself can only be raised by moving these reads to a separate token pool. Choosing between that and pacing how many PRs are open at once is a design decision, not a fix.
kyle-sexton commentedon Nov 16, 2026
Follow-up on the remaining cut, reconcile-lost-runs: no safe REST cut, and its share was overstated.
attempts/{n}/jobsreads, about 87 of them the fault-held check (reconcile.mjs:835-850, called from:1456). It reads each new successful pr-verify or pr-review-main run once and then settles it, so no run is read twice.lane-hold-*artifact prefilter: its upload iscontinue-on-error, so a missing marker would skip a lost run;FAULT_HELD_READS: that only defers the reads;nodes(ids:)on the runs' check-run node ids. That's about 85 fewer core calls per full pass, at about 5 GraphQL calls (estimate). It needs proof that GraphQLstepsmatch REST for the hold check. With the saving under 2% of the hourly limit, it's low priority.Conclusion: no single workflow is wasting the token. The spend tracks how many PRs are open and how often each is pushed, labeled or edited. Raising the ceiling is a design choice: pace concurrent PRs, or move part of the CI reads to a separate token pool.
kyle-sexton commentedon Nov 16, 2026
Owner decision, 2026-11-16 about 5:1x PM EST, typed in the hub session (hub question Q197, "Go with the recommendation"): no structural change to raise the GITHUB_TOKEN ceiling for now. No Enterprise Cloud upgrade, no read-only App, and no open-PR cap. The brake stays, and runs the token floor breaks recover automatically after the reset (#16642, PR #16650). The GraphQL route was measured at about 6% at most and is not pursued (PR #16644, closed unmerged). This issue's Expected line ('or a run that hits the limit re-runs itself after the reset instead of failing ci-status') is delivered by PR #16650; this issue closes when that merges.
kyle-sexton commentedon Nov 16, 2026
Closed: PR #16650 merged at 22:15:33Z. Runs the token floor breaks now recover automatically after the reset, the Expected line of this issue. Under the owner's Q197 decision (comment above), no structural change raises the ceiling for now; the brake stays.