Skip to content

fix(ci): re-run checks a rate-limit floor failed, ahead of lane runs - #16650

Merged
kyle-sexton merged 1 commit into
mainfrom
fix/16642-recover-after-token-floor
Nov 16, 2026
Merged

kyle-sexton merged 1 commit into
mainfrom
fix/16642-recover-after-token-floor

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #16642

Summary

After the GITHUB_TOKEN reached its core floor on 2026-11-16, the lost-run reconciler left PRs red whose checks failed on that floor. This PR adds a floor kind. A checks run that failed only on rate-guard refusals now has its failed jobs re-run once, ahead of lane runs, and the pacer can no longer skip the tick after a pass that left recovery work waiting.

Why the reconciler missed them, read from the 20:36Z pass, the 21:12Z pass and the 21:37Z pass:

Fix

  • floor kind. It covers a pr-require-checks run at attempt 1, inside the 24-hour lookback, whose failed jobs carry a rate guard's refusal and nothing else besides that refusal's "could not …" lines and the exit note (floorNoted). The failed jobs are Mark Lanes, or CI Status when it failed alone.
    • The failed jobs are re-run once (rerun-failed-jobs), with a record, as a flake is. The difference is placement: floor sets go in the first band, before lane runs.
    • The refusal texts are matched as their writers emit them: rate-guard.mjs RateLimitStop and its refusal log (:507, :771, :776), and pr-require-checks.yml's api_budget refusals (:697, :710). A test pins each writer.
    • A real error beside a refusal (the human-gated error, for example) leaves the run alone. So does a server error's Retry-After hold.
  • Pacing. A pass that defers any place that holds more than a flake re-run no longer writes clean. The pacer's mustRun then runs the next tick. The shared pace.mjs is unchanged.
  • Review lanes. These recover over two passes through the existing path. The floor re-run of the checks run lets CI Status name the stopped review. On the next pass the review kind re-runs it, with a record, and rerun-ci-status follows. Re-running the review and the checks run together could start an unrecorded third checks attempt, which is why the two steps are kept apart.
  • Fresh event (item 3). Not needed. GitHub's re-run docs say "The workflow will also use the same GITHUB_SHA (commit SHA) and GITHUB_REF (git ref) of the original event" (docs). A floor failure is not a code defect. The re-run is on the same head, and act checks again that the PR is open and still at that head before it writes (reconcile.mjs act). So reusing the original resolution is correct.
  • Stagger. The cap of 3 places per pass is kept. Recovery runs at about 3 PR sets per 30-minute tick, with writes at least a second apart.

Verification

  • node --test over .github/scripts/maintenance-reconcile-lost-runs/*.test.mjs and .github/actions/log-api-budget/pace.test.mjs: 243 pass, 0 fail.
  • New tests use the recorded annotations of check runs 133584509558, 133589388248 and 133592162315 (fixtures/annotations-floor.json). They cover:
    • CI Status failed alone at the floor;
    • Mark Lanes at the floor placed before lane runs, against a real Mark Lanes failure that stays a flake;
    • real failures that are not re-run;
    • attempt 2 and past the lookback;
    • closed, draft and moved heads, both at listing and at act;
    • the two-pass recovery of a review the kill switch stopped;
    • clean on a deferral;
    • the writer pins.
  • Disabling each change in turn failed the matching tests: the floor branches failed 4 tests, and the clean rule failed 1.
  • scripts/check-affected.sh --base origin/main --no-fetch: 13 passed, 6 skipped, over 4 changed files.

Remaining

  • Live confirmation: the next floor trip should show floor re-runs in the reconciler's summary.
  • Brake clear: no file in this repository sets or clears CLAUDE_LANES_REPO_DISABLED; the hub does. A tick the kill switch stops never writes clean, so the first tick after the brake clears is never paced. If the hub wants recovery to start right after BRAKE-CLEAR, it can dispatch maintenance-reconcile-lost-runs.yml.
  • Throughput: the backlog of lost pr-verify lane runs still drains at 3 per tick.

Related

🤖 Generated with Claude Code

After the GITHUB_TOKEN reached its core floor on 2026-11-16, the lost-run
reconciler did not recover the PRs whose checks failed on it:

- A pr-require-checks run where Mark Lanes failed on a rate-guard refusal
  was a flake set, placed after about 45 lost pr-verify lane runs under the
  cap of 3, so the 20:36Z pass deferred #16629, #16633 and #16634.
- A run where CI Status alone failed on a refusal (#16615) matched no
  kind: firstFailure sets an aggregate-only failure aside.
- The 20:36Z pass marked clean, so the pacer skipped the 21:12Z tick.

Add a floor kind: a checks run at attempt 1 whose failed jobs carry only a
rate guard's refusal, its "could not" lines and the exit note has its
failed jobs re-run once, recorded, in the first band before lane runs. A
pass that defers anything but a flake-only set no longer writes clean, so
the next tick is never paced while recovery work waits.

Refs: #16642

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 21:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T21:55:43.327781Z f127bd2 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed f127bd2cded3e0af479f37b8422dac1aba4b2632: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

This PR adds a floor kind to the lost-run reconciler. A pr-require-checks run whose failed jobs carry only rate-guard refusals, plus the "could not …" line beside each refusal and the exit note, has its failed jobs re-run once with a record. Those re-runs are placed ahead of lane runs. The PR also stops writing clean when a pass defers any place that holds more than flake re-runs.

I read all four changed files in full: the fixture, reconcile.mjs, reconcile.test.mjs and the workflow. I checked them against the code they call:

  • Placement. floor sets are not flake-only, so they fall into the first band in place, before lane runs (reconcile.mjs#L1629-L1638).
  • Records and act.
    • withRecord adds a record to every pr-require-checks item at attempt 1, so floor items get one (L701-L705).
    • FAILED_JOBS_WHY and lostNow both handle floor.
  • Re-run limits. The CI Status-only branch goes through reconcilerRetries, which limits it to attempt 1 inside the lookback. A failure in the aggregate job alone still never reaches the flake path through firstFailure.
  • Refusal patterns.
    • FLOOR_NOTE matches the refusal texts in rate-guard.mjs (:507, :776) and pr-require-checks.yml (:697, :710).
    • It does not match the server-error Retry-After lines (:695, :708) or the "stop could not be recorded" line (:689).
    • The new test pins each of those writers.
  • Precedence. A review-stop note or kill-switch note still takes precedence over floor in the CI Status-only branch. A pending-review note next to a refusal makes floorNoted false, so that path is unchanged.

I did not run the tests. The PR's 243 passing tests and the ablation results are the author's claims and unverified here.

Spend (not a hold): a pass that defers lane runs or floor sets no longer writes clean, so the pacer skips fewer reconciler ticks during a backlog. That costs more GITHUB_TOKEN core calls, plus one annotations read per failed lane job of each flake-candidate checks run, and standard-runner Actions minutes. Nothing here starts a billed resource.

Closing issue: the PR context says only Refs: #16642 and includes no closing-issue section, so I checked no acceptance criteria.

Verifier dropped 0 of 0 candidates.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed f127bd2cded3e0af479f37b8422dac1aba4b2632: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16650, re-run checks that a rate-limit floor failed

I found no security issues. I read all 4 changed files in full in the diff: the fixture, reconcile.mjs, reconcile.test.mjs and maintenance-reconcile-lost-runs.yml. maintenance-reconcile-lost-runs.yml is on the trusted-path list, so I read its whole change and what it gates.

What I checked

  • The new ci-status-only floor branch (reconcile.mjs#L1226-L1227). It applies to every PR's author, not only the lanes App's. It does nothing except call rerun-failed-jobs once, at attempt 1, on the same head. Before it writes, act checks again that the PR is open and still at that head.
    • pr-require-checks.yml has no triggering_actor or github.actor branch, so a re-run by the reconciler gains no trust.
    • The hold steps read their config and code from the default branch's tip or the base, never from the head (pr-require-checks.yml#L5514-L5529).
    • So a re-run checks every hold again from scratch. A failure that is not the floor's (human-gated, trust-gate, untrusted-provenance) fails again. No merge hold is bypassed.
  • Kill switch. The floor branch comes after the switchNoted branches. floorNoted rejects any failure annotation that is not a refusal, a could not companion line or the exit note, so a kill-switch note never matches it. A kill-switch failure at attempt 2 still needs a person. The kill-switch checks themselves are unchanged.
  • Who can write the annotations. The Mark Lanes and CI Status annotations come from steps run from base or tip config, not from PR code. The ^could not companion match is broad. At worst it adds one more re-run that checks the gates again, which has no security impact.
  • Pacing. clean = failed === 0 && deferred === 0 only makes the next tick run more often. No kill switch or gate is weakened.
  • Permissions. No permissions:, token request, trigger or trusted-actor change is in the diff.

Spend (not a hold): the change can add re-runs of pr-require-checks jobs and keep the pacer from skipping a tick after a deferred pass. That costs more GITHUB_TOKEN core calls and CI minutes, with no model use.

The PR text contains no embedded instructions to the review. This was a full review, not blocking only; no earlier head had been reviewed.

critical-findings: 0

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 81d5671 Nov 16, 2026
83 checks passed
@kyle-sexton
kyle-sexton deleted the fix/16642-recover-after-token-floor branch November 16, 2026 22:15
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant