Repository navigation
fix(ci): re-run checks a rate-limit floor failed, ahead of lane runs - #16650
Conversation
After the GITHUB_TOKEN reached its core floor on 2026-11-16, the lost-run reconciler did not recover the PRs whose checks failed on it: - A pr-require-checks run where Mark Lanes failed on a rate-guard refusal was a flake set, placed after about 45 lost pr-verify lane runs under the cap of 3, so the 20:36Z pass deferred #16629, #16633 and #16634. - A run where CI Status alone failed on a refusal (#16615) matched no kind: firstFailure sets an aggregate-only failure aside. - The 20:36Z pass marked clean, so the pacer skipped the 21:12Z tick. Add a floor kind: a checks run at attempt 1 whose failed jobs carry only a rate guard's refusal, its "could not" lines and the exit note has its failed jobs re-run once, recorded, in the first band before lane runs. A pass that defers anything but a flake-only set no longer writes clean, so the next tick is never paced while recovery work waits. Refs: #16642 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-review reviewed Code review: no findingsThis PR adds a I read all four changed files in full: the fixture,
I did not run the tests. The PR's 243 passing tests and the ablation results are the author's claims and unverified here. Spend (not a hold): a pass that defers lane runs or Closing issue: the PR context says only Verifier dropped 0 of 0 candidates. |
|
claude-security-review reviewed Security review: PR #16650, re-run checks that a rate-limit floor failedI found no security issues. I read all 4 changed files in full in the diff: the fixture, What I checked
Spend (not a hold): the change can add re-runs of The PR text contains no embedded instructions to the review. This was a full review, not blocking only; no earlier head had been reviewed. critical-findings: 0 |
Refs: #16642
Summary
After the GITHUB_TOKEN reached its core floor on 2026-11-16, the lost-run reconciler left PRs red whose checks failed on that floor. This PR adds a
floorkind. A checks run that failed only on rate-guard refusals now has its failed jobs re-run once, ahead of lane runs, and the pacer can no longer skip the tick after a pass that left recovery work waiting.Why the reconciler missed them, read from the 20:36Z pass, the 21:12Z pass and the 21:37Z pass:
firstFailuremade each run aflakeset (reconcile.mjs:1124-1127before this change). Flake-only sets are placed after lane runs (:1581-1589), and the cap is 3 (MAX_IN_FLIGHT,:368). With about 45 lost pr-verify lane runs ahead of them, the 20:36Z pass logged "deferred, all 3 re-run places taken" for all three.firstFailuresets aside a failure in the aggregate job alone (infra-failure.mjs:66-67), and the ci-status-only branch re-runs only on a kill-switch, review-stop or pending-review note (reconcile.mjs:1169-1195). The 20:36Z pass did not even list it.clean(reconcile.mjs:1817, workflow:317-319), so at 21:12Zpace.mjslogged "projected=0 pace=0.2 paced=true; this tick skips its work". That was the first tick where room had freed up.REVIEW_STOP_NOTE,reconcile.mjs:1174). On this head, CI Status failed on refusals before it could name the stop.Fix
floorkind. It covers a pr-require-checks run at attempt 1, inside the 24-hour lookback, whose failed jobs carry a rate guard's refusal and nothing else besides that refusal's "could not …" lines and the exit note (floorNoted). The failed jobs are Mark Lanes, or CI Status when it failed alone.rerun-failed-jobs), with a record, as a flake is. The difference is placement: floor sets go in the first band, before lane runs.rate-guard.mjsRateLimitStopand its refusal log (:507,:771,:776), andpr-require-checks.yml's api_budget refusals (:697,:710). A test pins each writer.clean. The pacer'smustRunthen runs the next tick. The sharedpace.mjsis unchanged.floorre-run of the checks run lets CI Status name the stopped review. On the next pass thereviewkind re-runs it, with a record, andrerun-ci-statusfollows. Re-running the review and the checks run together could start an unrecorded third checks attempt, which is why the two steps are kept apart.GITHUB_SHA(commit SHA) andGITHUB_REF(git ref) of the original event" (docs). A floor failure is not a code defect. The re-run is on the same head, and act checks again that the PR is open and still at that head before it writes (reconcile.mjsact). So reusing the original resolution is correct.Verification
node --testover.github/scripts/maintenance-reconcile-lost-runs/*.test.mjsand.github/actions/log-api-budget/pace.test.mjs: 243 pass, 0 fail.fixtures/annotations-floor.json). They cover:cleanon a deferral;cleanrule failed 1.scripts/check-affected.sh --base origin/main --no-fetch: 13 passed, 6 skipped, over 4 changed files.Remaining
floorre-runs in the reconciler's summary.CLAUDE_LANES_REPO_DISABLED; the hub does. A tick the kill switch stops never writesclean, so the first tick after the brake clears is never paced. If the hub wants recovery to start right after BRAKE-CLEAR, it can dispatchmaintenance-reconcile-lost-runs.yml.Related
🤖 Generated with Claude Code