Skip to content

ci: the lanes App installation token runs out under many concurrent PRs #15497

Closed

Description

@kyle-sexton

Problem

On 2026-11-11/12 the CI installation token hit its hourly limit four times (per the hub session's count), failing every in-flight PR check with API rate limit exceeded for installation. Example: run 43143452524 (mark-lanes could not mark ci-lanes pending) and the CodeQL jobs in run 43143445098 on #15468. Recovery today is a manual gh pr update-branch per PR after the reset, and the kill switch CLAUDE_LANES_REPO_DISABLED stays on until then.

Expected

A burst of ~15 open PRs (the stamp-sweep batches plus their splits) completes CI without exhausting the token, or a run that hits the limit re-runs itself after the reset instead of failing ci-status.

Evidence to gather first

Calls per PR run by workflow (the lanes already log rate-limit headers, #8062), and which jobs spend most.

Related

Found during the Fable 5.2 adoption effort (Q11 stamp sweep, #15468).

Activity

  1. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Recurred 2026-11-12 03:32-03:34 UTC (fifth exhaustion this night): the installation core pool hit used=5000 remaining=0 (reset 03:53 UTC) and failed review-status on #15469 (run 43148031786, run 43148031744) and ci-status's file read (run 43148031104). Load at the time: about 12 open PRs from one effort, four flipped ready within a few minutes.

  2. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).

  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: when many PRs run CI at once (about 12–15 from one effort), the lanes App installation token reaches its 5000/hour core limit. Checks such as mark-lanes, review-status, ci-status and CodeQL then fail, and recovery is a manual branch update per PR after the reset. The goal is either a burst that completes without exhausting the token, or a run that retries itself after the reset.

    The first step is the evidence you listed: API calls per PR run by workflow and the jobs that spend the most. The rate-limit header logging from #8062 should supply that data.

    I found no open duplicate and no merged pull request that already fixes this.

    Automated triage. A maintainer reviews it.

  4. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    The token ran out again on 2026-11-16, twice in under an hour: the brake set at 19:29Z, and the start gate tripped at 20:11Z with 6% left, 39 minutes into the window that started at 19:32Z. About 18 PRs were open.

    Recorded core calls per workflow job, from the hub's per-job call log (gh-job-calls.jsonl, bucketed by the job's finish time):

    Window (UTC) Core calls recorded Jobs
    18:32-19:32 4,459 1,022
    19:32-20:12 (partial) 3,360 832

    Top consumers, 19:32-20:12:

    Workflow: job Calls Jobs Per job
    pr-require-checks: ci status 528 65 8.1
    pr-verify: check intent 278 41 6.8
    pr-review-hosted + pr-review-security-hosted: review subscription 334 39 8.6
    maintenance-reconcile-lost-runs: reconcile 162 1 162
    pr-dispatch-feedback: rerun ci status 148 58 2.6
    pr-merge: merge 148 2 74

    What this shows: no single workflow drains the token. The spend is volume across the open PRs. CI Status ran about 65 times in 40 minutes for about 18 PRs, and 58 of those runs were re-runs dispatched by pr-dispatch-feedback. The one outlier per job is reconcile-lost-runs, at 162 calls in a single job.

    Proposed first cuts, in order:

    1. Fewer CI Status re-runs: skip or coalesce rerun ci status when nothing CI Status reads has changed since its last run on that head.
    2. A per-run call cap in reconcile-lost-runs, or paging it across runs.

    Both are UNVERIFIED as savings until a window is measured after the change. About 1,300 of the window's calls were not yet in the log at 20:12Z (jobs still running, or workflows not logged), so the ranking may still move.

  6. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Correction to my earlier comment on this issue (#15497 (comment)): proposed cut 1 is withdrawn. My count of CI Status re-runs was wrong.

    • In the 19:32-20:12Z window, 62 rerun ci status jobs ran, but only 8 of them re-ran CI Status. Each of the others is about 1.8 calls of checking. 6 of those 8 re-runs turned "Require both review lanes on this head" from red to green, so they were needed. The dispatcher's find step already re-runs only after the last review lane on a head finishes (pr-dispatch-feedback.yml:922-1116).
    • No CI Status run reads anything twice. One job reads the PR's files once and reuses pr-files.json (pr-require-checks.yml:5293, reused at :6918 and :7390), and pr_read caches the PR, its labels and its events (:5107-5149). The per-job notices show pulls/{n}/files=1 and issues/{n}/labels=1.
    • Considered and rejected: a per-PR cancel-in-progress group for CI Status runs triggered by label and edit events. It saves under 1% (7 of 34 such runs were superseded in 100 minutes), and it could cancel the run holding the current verdict.

    Where the calls actually go: first-attempt CI Status runs, one per push, label change or edit, on each of about 18 open PRs. 87 full runs plus 24 label- and edit-triggered first attempts ran in 100 minutes. The spend grows with how many PRs are open and how often each is touched, not with any waste in one workflow.

    Each exhaustion also ejected clean PRs from the merge queue (16598 and 16619 at 20:17Z), so a trip costs more than failed lanes.

    Remaining cut: reconcile-lost-runs, about 162 calls in one job, running twice an hour, so about 6% of the hourly limit. Beyond that, the limit itself can only be raised by moving these reads to a separate token pool. Choosing between that and pacing how many PRs are open at once is a design decision, not a fix.

  7. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Follow-up on the remaining cut, reconcile-lost-runs: no safe REST cut, and its share was overstated.

    • Full passes alternate with paced ticks that cost 1-2 calls, so the job uses about 162 core calls an hour (about 3% of the limit), not 6%. Full passes in the call log cost 37 to 249 calls.
    • In run 43941232026 (164 calls), 100 calls were attempts/{n}/jobs reads, about 87 of them the fault-held check (reconcile.mjs:835-850, called from :1456). It reads each new successful pr-verify or pr-review-main run once and then settles it, so no run is read twice.
    • Ruled out:
      • a lane-hold-* artifact prefilter: its upload is continue-on-error, so a missing marker would skip a lost run;
      • ETag/304: each run is read once, so nothing repeats;
      • lowering FAULT_HELD_READS: that only defers the reads;
      • a since-last-clean-pass lookback: already in place.
    • One option is left: batch the fault-held reads through GraphQL nodes(ids:) on the runs' check-run node ids. That's about 85 fewer core calls per full pass, at about 5 GraphQL calls (estimate). It needs proof that GraphQL steps match REST for the hold check. With the saving under 2% of the hourly limit, it's low priority.

    Conclusion: no single workflow is wasting the token. The spend tracks how many PRs are open and how often each is pushed, labeled or edited. Raising the ceiling is a design choice: pace concurrent PRs, or move part of the CI reads to a separate token pool.

  8. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Owner decision, 2026-11-16 about 5:1x PM EST, typed in the hub session (hub question Q197, "Go with the recommendation"): no structural change to raise the GITHUB_TOKEN ceiling for now. No Enterprise Cloud upgrade, no read-only App, and no open-PR cap. The brake stays, and runs the token floor breaks recover automatically after the reset (#16642, PR #16650). The GraphQL route was measured at about 6% at most and is not pursued (PR #16644, closed unmerged). This issue's Expected line ('or a run that hits the limit re-runs itself after the reset instead of failing ci-status') is delivered by PR #16650; this issue closes when that merges.

  9. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Closed: PR #16650 merged at 22:15:33Z. Runs the token floor breaks now recover automatically after the reset, the Expected line of this issue. Under the owner's Q197 decision (comment above), no structural change raises the ceiling for now; the brake stays.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    high

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions