Skip to content

The workflow token reaches its rate-limit floor each hour and its steps fail instead of waiting #16642

Open

Description

@kyle-sexton

Problem

On 2026-11-16 the workflow token (token=github-token in the api-budget notices) reached its core rate-limit floor in two consecutive hourly windows. Each time, steps that needed one call failed their run instead of waiting, and a person had to re-run them.

Window 1, reset 20:32:29Z:

  • Refine on PR 16631, 20:12Z: "left GitHub's core API 65 of 5000 requests, at or under its stop-floor share 0.02 … this step fails"; the pr-refine dispatch "was not sent".
  • Mark Lanes on PR 16631 head 0237d839 failed at 20:22Z (run 43949103070).
  • Both review lanes on that head did not run: CI Status reported "'PR Review / Status' did not run … a kill switch was on when it started" for runs 43949103692 and 43949103789.
  • 13 of the 15 PR Verify runs created between 20:22Z and 20:28Z concluded failure; the one opened (43950072774) failed at "Report the kill switch".
  • Recovery took three manual re-runs after the reset (gh run rerun 43949103070 --failed, and all jobs of the two lane runs).

Window 2, reset 21:32:30Z:

  • Refine on PR 16641, 21:24Z: limit=5000 remaining=88 used=4912 resource=core, "no run dispatched".

UNVERIFIED: what spends about 4,900 core requests an hour on this token. Nothing here measured the callers.

Done when

  • The callers that spent the token's core budget in these two windows are named, with their request counts, from the api-budget notices or the run logs.
  • Either the hourly spend is brought under the floor, or a step that finds the token at its floor waits for x-ratelimit-reset (or is re-run by the reconciler) instead of failing a run that then needs a person.
  • A pull request pushed while the token is at its floor gets its review lanes and Mark Lanes without a manual re-run.

Activity

  1. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Callers that spent the token, by workflow and job, from the hub's per-job call log (gh-job-calls.jsonl, bucketed by each job's finish time). The log does not attribute every call: in the 20:32Z window it recorded 4,082 core calls against 4,781 used.

    Window (reset) Top consumers (core calls, jobs)
    19:32Z CI Status 528 (65), PR Verify Check Intent 278 (41), the two review subscriptions 334 (39), reconcile-lost-runs 162 (1), rerun-ci-status 148 (58, of which 8 re-ran CI Status), pr-merge 148 (2)
    20:32Z pr-merge 876 (10: Merge 495 in 4, Enable Auto Merge 208, Dispatch post merge runs 173), CI Status 513 (53), Check Intent 284 (38), review subscriptions 307 (38), reconcile-lost-runs 251 (2)

    What the hub found:

    In flight on #15497: moving the heaviest repeated reads in CI Status, PR Verify and the review lanes to the GITHUB_TOKEN's separate GraphQL pool. That's estimated to raise the ceiling from about 21 to about 26 open PRs. Analysis: #15497 (comment) and the comments after it. This issue's ask (wait at the floor instead of failing) is a separate fix to the same problem, so link the two rather than merge them.

  2. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    First live test of PR #16650, after the 22:17Z brake (cleared 22:33Z): reconcile run https://github.com/melodic-software/claude-code-plugins/actions/runs/43971789832 (22:37Z).

    • Done-when item 2 is shown live: "chore(release): release 20 plugins #16652: re-ran the failed jobs of run 43968483728 (.github/workflows/pr-require-checks.yml) attempt 1, which failed only where the GITHUB_TOKEN hit its rate-limit floor". It also re-ran 2 failed pr-verify lane runs and 2 pr-dispatch-feedback runs whose verdict the kill switch had held.
    • Item 3 is not met at scale: the same pass logged 45 lane runs "deferred, all 3 re-run places taken" (pr-verify.yml, some from about 19:40Z). At 3 re-runs per pass and 2 passes an hour, a backlog that size takes hours to drain. A PR whose pr-verify is in that queue stays unmerged meanwhile. For example, docs(enforcement-ladder): add the architecture and retro Selection rules rows in ladder 3.2.0 #16602 left the merge queue at 22:15Z because pr-verify had not judged its head.
      Remaining: drain the post-brake backlog faster, keeping the stagger. Two options: a larger cap for the first pass after a brake clear, or re-running only the newest lost run per PR, since older heads are superseded. Basis: the reconcile log lines above.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions