From f127bd2cded3e0af479f37b8422dac1aba4b2632 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 16:51:03 -0500 Subject: [PATCH] fix(ci): re-run checks a rate-limit floor failed, ahead of lane runs After the GITHUB_TOKEN reached its core floor on 2026-11-16, the lost-run reconciler did not recover the PRs whose checks failed on it: - A pr-require-checks run where Mark Lanes failed on a rate-guard refusal was a flake set, placed after about 45 lost pr-verify lane runs under the cap of 3, so the 20:36Z pass deferred #16629, #16633 and #16634. - A run where CI Status alone failed on a refusal (#16615) matched no kind: firstFailure sets an aggregate-only failure aside. - The 20:36Z pass marked clean, so the pacer skipped the 21:12Z tick. Add a floor kind: a checks run at attempt 1 whose failed jobs carry only a rate guard's refusal, its "could not" lines and the exit note has its failed jobs re-run once, recorded, in the first band before lane runs. A pass that defers anything but a flake-only set no longer writes clean, so the next tick is never paced while recovery work waits. Refs: #16642 Co-Authored-By: Claude Opus 5.5 --- .../fixtures/annotations-floor.json | 57 ++++++++ .../reconcile.mjs | 67 ++++++++-- .../reconcile.test.mjs | 122 +++++++++++++++++- .../maintenance-reconcile-lost-runs.yml | 14 +- 4 files changed, 246 insertions(+), 14 deletions(-) create mode 100644 .github/scripts/maintenance-reconcile-lost-runs/fixtures/annotations-floor.json diff --git a/.github/scripts/maintenance-reconcile-lost-runs/fixtures/annotations-floor.json b/.github/scripts/maintenance-reconcile-lost-runs/fixtures/annotations-floor.json new file mode 100644 index 00000000000..5eec7d5198d --- /dev/null +++ b/.github/scripts/maintenance-reconcile-lost-runs/fixtures/annotations-floor.json @@ -0,0 +1,57 @@ +{ + "ci-status-133584509558": [ + {"annotation_level":"notice","message":"github-token repos/{repo}/pulls/{n}/files=1 rest=0"}, + {"annotation_level":"notice","message":"github-token=1 not-modified=0 graphql=0 core-unread=1"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"failure","message":"could not read the files of #16615; re-run this run."}, + {"annotation_level":"failure","message":"'gh api repos/melodic-software/claude-code-plugins/pulls/16615/files?per_page=100 --paginate --jq .[] | [.filename, (.previous_filename // \"\")] | @tsv' was not sent: GitHub's core API rate limit holds until 2026-11-16T21:19:18Z"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"notice","message":"github-token=3 not-modified=0 graphql=1"}, + {"annotation_level":"failure","message":"could not list this workflow's runs on 654485e5d346c0fde9a0d50eb598a27583854b6b, so a full run in flight cannot be ruled out; re-run this run."}, + {"annotation_level":"warning","message":"repos/melodic-software/claude-code-plugins/actions/workflows/374767837/runs is rate limited (HTTP 403). Cannot tell whether a full run is in flight on this SHA, so this run fails closed; re-run it."}, + {"annotation_level":"warning","message":"GET repos/melodic-software/claude-code-plugins/actions/workflows/374767837/runs?head_sha=654485e5d346c0fde9a0d50eb598a27583854b6b&per_page=100 is rate limited and needs a 767s wait, more than the 60s left of rate-limit-wait-seconds (60); not waiting. The limit resets at 1794861149 (2026-11-16T20:32:29Z)."}, + {"annotation_level":"warning","message":"GET repos/melodic-software/claude-code-plugins/actions/workflows/374767837/runs?head_sha=654485e5d346c0fde9a0d50eb598a27583854b6b&per_page=100 returned HTTP 403 (rate-limit): API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID 46AE:991D2:6E399F:15746A4:6AFB655E and timestamp 2026-11-16 20:19:42 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service); x-ratelimit-remaining=0 x-ratelimit-reset=1794861149 retry-after=absent x-accepted-github-permissions=absent"}, + {"annotation_level":"warning","message":"could not identify this run's gate job and running step in repos/melodic-software/claude-code-plugins/actions/jobs/133584509558: repos/melodic-software/claude-code-plugins/actions/jobs/133584509558 could not be read (HTTP 403) after 1 reads. The yield listing and the writer check fail closed on it."}, + {"annotation_level":"warning","message":"repos/melodic-software/claude-code-plugins/actions/jobs/133584509558 is rate limited (HTTP 403). The yield listing and the writer check fail closed on it."}, + {"annotation_level":"warning","message":"GET repos/melodic-software/claude-code-plugins/actions/jobs/133584509558 is rate limited and needs a 767s wait, more than the 60s left of rate-limit-wait-seconds (60); not waiting. The limit resets at 1794861149 (2026-11-16T20:32:29Z)."}, + {"annotation_level":"warning","message":"GET repos/melodic-software/claude-code-plugins/actions/jobs/133584509558 returned HTTP 403 (rate-limit): API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID ABDA:29F441:1E4F626:5EEC6E3:6AFB655E and timestamp 2026-11-16 20:19:42 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service); x-ratelimit-remaining=0 x-ratelimit-reset=1794861149 retry-after=absent x-accepted-github-permissions=absent"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"failure","message":"could not read the labels of #16615; re-run this run."}, + {"annotation_level":"failure","message":"'gh api repos/melodic-software/claude-code-plugins/issues/16615/labels?per_page=100 --paginate' was not sent: GitHub's core API rate limit holds until 2026-11-16T21:19:18Z"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"failure","message":"could not read the files of #16615; re-run this run."}, + {"annotation_level":"failure","message":"'gh api repos/melodic-software/claude-code-plugins/pulls/16615/files?per_page=100 --paginate' hit GitHub's API rate limit: no call to it before 2026-11-16T21:19:18Z (fallback); the call returns 1"}, + {"annotation_level":"notice","message":"github-token=1 not-modified=0 graphql=1"} + ], + "mark-lanes-133589388248": [ + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"notice","message":"github-token repos/{repo}/statuses/{sha}=1 rest=0"}, + {"annotation_level":"notice","message":"github-token=1 not-modified=0 graphql=0 core-by-reset=1794861149:1 core-unread=0"}, + {"annotation_level":"notice","message":"token=github-token workflow=.github/workflows/pr-require-checks.yml job=mark-lanes run=43949883313 attempt=1 limit=5000 remaining=0 used=5000 reset=1794861149 resource=core cost=na"}, + {"annotation_level":"failure","message":"could not mark ci-lanes pending on bbb34397b1a702d5ffd9eb6a1787306ac1523af6; a contract-only run could carry an older verdict forward while this run is in flight."}, + {"annotation_level":"failure","message":"'gh api --method POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 -f state=pending -f context=ci-lanes -f description=full run 43949883313 in flight -f target_url=https://github.com/melodic-software/claude-code-plugins/actions/runs/43949883313' was not sent: GitHub's core API rate limit holds until 2026-11-16T20:32:29Z"}, + {"annotation_level":"failure","message":"'gh api --method POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 -f state=pending -f context=ci-lanes -f description=full run 43949883313 in flight -f target_url=https://github.com/melodic-software/claude-code-plugins/actions/runs/43949883313' was not sent: GitHub's core API rate limit holds until 2026-11-16T20:32:29Z"}, + {"annotation_level":"failure","message":"'gh api --method POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 -f state=pending -f context=ci-lanes -f description=full run 43949883313 in flight -f target_url=https://github.com/melodic-software/claude-code-plugins/actions/runs/43949883313' hit GitHub's core API rate limit: no call to it before 2026-11-16T20:32:29Z (x-ratelimit-reset); the call returns 1"} + ], + "ci-status-133592162315": [ + {"annotation_level":"notice","message":"github-token repos/{repo}/pulls/{n}/files=1 rest=0"}, + {"annotation_level":"notice","message":"github-token=1 not-modified=0 graphql=0 core-unread=1"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"failure","message":"could not read the files of #16633; re-run this run."}, + {"annotation_level":"failure","message":"'gh api repos/melodic-software/claude-code-plugins/pulls/16633/files?per_page=100 --paginate --jq .[] | [.filename, (.previous_filename // \"\")] | @tsv' was not sent: GitHub's core API rate limit holds until 2026-11-16T21:32:01Z"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"notice","message":"github-token=2 not-modified=0 graphql=0"}, + {"annotation_level":"failure","message":"could not record ci-lanes on bbb34397b1a702d5ffd9eb6a1787306ac1523af6 (HTTP 403): rate limited past rate-limit-wait-seconds; re-run this job after the limit resets"}, + {"annotation_level":"warning","message":"POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 is still rate limited after the wait; not retrying."}, + {"annotation_level":"warning","message":"POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 returned HTTP 403 (rate-limit): API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID 520C:1A3B03:86C681:1A3C9C9:6AFB685D and timestamp 2026-11-16 20:32:29 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service); x-ratelimit-remaining=0 x-ratelimit-reset=1794861149 retry-after=absent x-accepted-github-permissions=absent"}, + {"annotation_level":"warning","message":"POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 is rate limited; waiting 3s, then retrying once."}, + {"annotation_level":"warning","message":"POST repos/melodic-software/claude-code-plugins/statuses/bbb34397b1a702d5ffd9eb6a1787306ac1523af6 returned HTTP 403 (rate-limit): API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID EF10:2F29F0:2038B37:658C658:6AFB685A and timestamp 2026-11-16 20:32:26 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service); x-ratelimit-remaining=0 x-ratelimit-reset=1794861149 retry-after=absent x-accepted-github-permissions=absent"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"failure","message":"could not read the labels of #16633; re-run this run."}, + {"annotation_level":"failure","message":"'gh api repos/melodic-software/claude-code-plugins/issues/16633/labels?per_page=100 --paginate' was not sent: GitHub's core API rate limit holds until 2026-11-16T21:32:01Z"}, + {"annotation_level":"failure","message":"Process completed with exit code 1."}, + {"annotation_level":"failure","message":"could not read the files of #16633; re-run this run."}, + {"annotation_level":"failure","message":"'gh api repos/melodic-software/claude-code-plugins/pulls/16633/files?per_page=100 --paginate' hit GitHub's API rate limit: no call to it before 2026-11-16T21:32:01Z (fallback); the call returns 1"}, + {"annotation_level":"notice","message":"github-token=1 not-modified=0 graphql=1"} + ] +} diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs index 11597d91765..886139a9058 100755 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs @@ -22,6 +22,14 @@ // flaky test the diff did not cause can pass. pr-unblock's fix-ci does not // wait for this re-run unless it is already queued or going, so it may run // beside a fix-ci run (CI minutes, no model). +// - Floor: a pr-require-checks run, at attempt 1 inside the lookback, whose +// failed jobs (Mark Lanes, or CI Status alone) failed only where the +// GITHUB_TOKEN hit its rate-limit floor (floorNoted: a rate guard's refusal, +// its "could not" lines and the exit note, nothing else). Its failed jobs +// are re-run once, recorded, as a flake is, but placed before lane runs: the +// floor is gone once the pool resets, and nothing else re-runs a ci-status +// that failed alone. Its CI Status then reports a review a kill switch +// stopped, which the next pass re-runs (Review). // - Lane: a pr-verify or pr-review-main run whose first attempt ended as above // or failure (a judgment that broke, a review every credential leg stopped // on, or a pr-verify run the kill switch stopped, which fails), and whose @@ -172,7 +180,8 @@ // closed PR, or a second run for one head, takes no place before a live one, // and the newer runs each kill-switch trip adds no longer push the oldest // live ones back pass after pass. One pass uploads at most MAX_IN_FLIGHT -// records, so what would pass that waits too. +// records, so what would pass that waits too. A pass that defers anything but +// a flake-only set writes no `clean`, so the pacer runs the next tick. // Writes are at least WRITE_GAP_MS apart. The first rate-limit error stops the // run. // @@ -606,6 +615,23 @@ export const pendingReviewNoted = (annotations) => { const exits = failures.filter((m) => EXIT_NOTE.test(m)).length; return pending > 0 && exits <= 1 && pending + exits === failures.length; }; +// The refusals a rate guard writes when a call is not sent, or not answered, +// because the GITHUB_TOKEN's pool is at its floor (rate-guard.mjs RateLimitStop +// and its refusal log; pr-require-checks.yml's api_budget refusals; +// reconcile.test.mjs pins each writer). A server error's Retry-After hold is +// not one. +export const FLOOR_NOTE = + /(?:was not sent: GitHub's .+? rate limit holds until |hit GitHub's .+? rate limit(?: \(status [0-9]+\))?: no call to it before |at or under its stop-floor share [0-9.]+ \(status [0-9]+\): no call to it before )/; +// The line a failing step writes beside such a refusal, naming what it could +// not do ("could not read the files of #N; re-run this run."). +const FLOOR_COMPANION = /^could not /; +// True when a job failed only where the token hit its floor: at least one +// refusal, and every other failure annotation is a refusal's companion line or +// the exit note. A real failure beside a refusal (a gate's own error) is not. +export const floorNoted = (annotations) => { + const failures = (Array.isArray(annotations) ? annotations : []).filter((a) => a?.annotation_level === "failure").map((a) => String(a.message ?? "")); + return failures.some((m) => FLOOR_NOTE.test(m)) && failures.every((m) => FLOOR_NOTE.test(m) || FLOOR_COMPANION.test(m) || EXIT_NOTE.test(m)); +}; // How long after the last review completes before a Pending re-run: at least // rerun-ci-status's `timeout-minutes: 15` (reconcile.test.mjs pins it), so its // own re-run has had the time to land. Under its `queue: max` concurrency that @@ -628,7 +654,7 @@ const lostNow = (item, r, since) => { if (r?.head_sha !== item.head) { return false; } - if (item.kind === "infra" || item.kind === "flake" || item.kind === "switch" || item.kind === "pending" || item.kind === "upstream") { + if (item.kind === "infra" || item.kind === "flake" || item.kind === "floor" || item.kind === "switch" || item.kind === "pending" || item.kind === "upstream") { return r.status === "completed" && r.conclusion === "failure" && r.run_attempt === item.attempt; } if (item.kind === "review") { @@ -743,6 +769,22 @@ const withAnnotations = (call, repo, jobs, wanted = () => true) => ), ); +// True when every failed lane job of a run failed only at the token's floor +// (floorNoted), reading the annotations a job does not already carry. A carried +// job read carries none, so they are read here. +const floorFailed = async (call, repo, jobs) => { + const lanes = jobs.filter((job) => FAILED.has(job?.conclusion) && !AGGREGATE_JOBS.has(job.name)); + if (lanes.length === 0 || !lanes.every((job) => job.conclusion === "failure" && Number.isSafeInteger(job.id))) { + return false; + } + for (const job of lanes) { + if (!floorNoted(job.annotations ?? (await call("GET", `/repos/${repo}/check-runs/${job.id}/annotations?per_page=100`)))) { + return false; + } + } + return true; +}; + const NOT_STARTED = "before any of its jobs started"; // The schedule's interval: a listing of no jobs read this long after its run // last changed is final, as one read sooner may not yet show them. @@ -1123,7 +1165,8 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp } if (!infraOnly(jobs)) { if (firstFailure(r, jobs, now)) { - set.push({ kind: "flake", pr: pull.number, id: r.id, path: r.path, conclusion: "failure", attempt: 1, head: pull.head.sha }); + const floor = RECORDED_UPSTREAMS.has(r.path) && (await floorFailed(call, repo, jobs)); + set.push({ kind: floor ? "floor" : "flake", pr: pull.number, id: r.id, path: r.path, conclusion: floor ? "failure (rate-limit floor)" : "failure", attempt: 1, head: pull.head.sha }); } continue; } @@ -1180,6 +1223,8 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp set.push({ kind: "switch", pr: pull.number, id: checks.id, path: checks.path, conclusion: "failure (kill switch)", attempt: 1, head: pull.head.sha }); } else if (checks.run_attempt >= 2 && switchNoted) { say(`#${pull.number}: ci-status failed on the kill switch at attempt ${checks.run_attempt}; a re-run needs a person on the trusted-actor list`); + } else if (reconcilerRetries(checks, now) && floorNoted(notes)) { + set.push({ kind: "floor", pr: pull.number, id: checks.id, path: checks.path, conclusion: "failure (rate-limit floor)", attempt: 1, head: pull.head.sha }); } else if (isFailedChecks(checks) && notes && pendingReviewNoted(notes)) { // Every PR, not only the lanes App's: a lost rerun-ci-status strands // a person's head the same way. ci-status's own pick of each review: @@ -1567,6 +1612,9 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp let room = Math.max(0, MAX_IN_FLIGHT - inFlight); let uploads = MAX_IN_FLIGHT; const items = []; + // Places deferred that hold more than flake re-runs: recovery work the + // next scheduled tick must not skip (main's `clean`, which the pacer reads). + let deferred = 0; const place = (what, taken) => { const records = taken.filter(recorded).length; if (room > 0 && records <= uploads) { @@ -1574,6 +1622,7 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp room -= 1; uploads -= records; } else { + deferred += taken.some((item) => item.kind !== "flake") ? 1 : 0; say(`${what}: deferred, ${room > 0 ? "no record upload left this pass" : `all ${MAX_IN_FLIGHT} re-run places taken`}`); } }; @@ -1588,7 +1637,7 @@ async function plan({ repo, call, say, since, now, readStatus, judgeSpec, laneAp place(`#${set[0].pr}`, set); } items.push(...intake, ...verdicts); - return { items, failed, settle, heldSince, jobs: [...keepJobs.values()], passes: [...new Map(keepPasses.map((e) => [passesKey(e.pr, e.head), e])).values()] }; + return { items, failed, deferred, settle, heldSince, jobs: [...keepJobs.values()], passes: [...new Map(keepPasses.map((e) => [passesKey(e.pr, e.head), e])).values()] }; } // The item kinds that name no PR: act labels them by kind and reads no PR. @@ -1599,6 +1648,7 @@ const FAILED_JOBS_WHY = new Map([ ["verdict", ""], ["infra", ", which failed on infra only"], ["flake", ", its one re-run before fix-ci takes it"], + ["floor", ", which failed only where the GITHUB_TOKEN hit its rate-limit floor"], ["switch", ": ci-status failed while the lanes kill switch was on"], ["pending", ": ci-status failed on a review that has since finished"], ["upstream", ": the upstream-status gate stopped its review, and the gate lets it run now"], @@ -1729,8 +1779,9 @@ export async function main({ env = process.env, github, fetchStatus: readStatus } let result = { items: [], failed: 0 }; - // A full plan that read every listing; the kill switch, an Actions outage - // or a failed listing leaves it false (the pacer's marker reads it). + // A full plan that read every listing and deferred no recovery work; the + // kill switch, an Actions outage, a failed listing or a deferred place + // that is not a flake re-run leaves it false (the pacer's marker reads it). let clean = false; const write = () => { writeFileSync(env.PLAN_PATH, JSON.stringify(result)); @@ -1795,7 +1846,7 @@ export async function main({ env = process.env, github, fetchStatus: readStatus console.log(`::warning title=reconcile::no settled runs carried, so every run is read: ${escapeCommand(error?.message ?? error)}`); } } - const { settle, jobs, passes, heldSince, ...planned } = await plan({ + const { settle, jobs, passes, heldSince, deferred, ...planned } = await plan({ repo, call, say, @@ -1814,7 +1865,7 @@ export async function main({ env = process.env, github, fetchStatus: readStatus if (result.items.some(recorded) && !env.RECORD_PATH) { throw new Error("RECORD_PATH is not set"); } - clean = result.failed === 0; + clean = result.failed === 0 && deferred === 0; write(); if (env.SETTLED_PATH) { writeFileSync(env.SETTLED_PATH, JSON.stringify({ version: SETTLED_VERSION, repository: repo, runs: [...new Map(settle.map((r) => [r.id, r])).values()], jobs, passes, held_since: new Date(heldSince).toISOString() })); diff --git a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs index 4519da130e0..a51848809b7 100644 --- a/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs +++ b/.github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs @@ -18,8 +18,8 @@ import path from "node:path"; import { test } from "node:test"; import { createGitHub, GitHubError } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; -import { createRateLimitLatch } from "../../actions/log-api-budget/rate-guard.mjs"; -import { CI_WORKFLOWS, GITHUB_STATUS_URL, JUDGE_PREDECESSOR, JUDGE_STEP, KILL_SWITCH_NOTE, main, NPM_STATUS_URL, MISSING_REVIEW_NOTE, PENDING_BACKOFF_MINUTES, PENDING_REVIEW_NOTE, prefilterSpec, REVIEW_STEP, REVIEW_STOP_NOTE, REVIEW_STOP_STEP, REVIEW_SWITCHES, SETTLED_VERSION, TOO_OLD_DAYS, UPSTREAM_STOP_STEP, VERDICT_STOP_STEP, WRITE_STEP } from "./reconcile.mjs"; +import { createRateLimitLatch, RateLimitStop } from "../../actions/log-api-budget/rate-guard.mjs"; +import { CI_WORKFLOWS, FLOOR_NOTE, floorNoted, GITHUB_STATUS_URL, JUDGE_PREDECESSOR, JUDGE_STEP, KILL_SWITCH_NOTE, main, NPM_STATUS_URL, MISSING_REVIEW_NOTE, PENDING_BACKOFF_MINUTES, PENDING_REVIEW_NOTE, prefilterSpec, REVIEW_STEP, REVIEW_STOP_NOTE, REVIEW_STOP_STEP, REVIEW_SWITCHES, SETTLED_VERSION, TOO_OLD_DAYS, UPSTREAM_STOP_STEP, VERDICT_STOP_STEP, WRITE_STEP } from "./reconcile.mjs"; import { switchHold } from "../pr-merge/merge-holds.mjs"; import { tempDir } from "../../../lib/test-tmp.mjs"; @@ -3076,3 +3076,121 @@ test("the pending backoff is at least rerun-ci-status's time limit", () => { assert.ok(limit > 0, "rerun-ci-status sets timeout-minutes"); assert.ok(PENDING_BACKOFF_MINUTES >= limit); }); + +// Runs that failed only where the GITHUB_TOKEN hit its rate-limit floor on +// 2026-11-16 (#16642). The annotations are recorded: CI Status of #16615 (check +// run 133584509558, failed alone), and Mark Lanes and CI Status of #16633 +// (133589388248, 133592162315). +const FLOOR = fixture("annotations-floor.json"); +const floorStatus = (notes = FLOOR["ci-status-133584509558"]) => [ + { id: 601, name: "Lint Repo", status: "completed", conclusion: "success", steps: [] }, + { id: 602, name: "CI Status", status: "completed", conclusion: "failure", steps: [], notes }, +]; +const markLanes = (notes = FLOOR["mark-lanes-133589388248"]) => [ + { + id: 611, + name: "Mark Lanes", + status: "completed", + conclusion: "failure", + steps: [ + { name: "Set up job", number: 1, status: "completed", conclusion: "success" }, + { name: "Mark the lanes in flight", number: 2, status: "completed", conclusion: "failure" }, + { name: "Complete job", number: 3, status: "completed", conclusion: "success" }, + ], + notes, + }, + { id: 612, name: "CI Status", status: "completed", conclusion: "failure", steps: [], notes: FLOOR["ci-status-133592162315"] }, +]; +// ci-status's human-gated error (pr-require-checks.yml), a failure no re-run clears. +const GATED_LINE = "#1 carries 'human-gated'. Removing the label approves this pull request; ci-status re-runs on the unlabel."; +const failure = (message) => ({ annotation_level: "failure", message }); + +test("a CI Status that failed alone at the rate-limit floor has its failed jobs re-run once, recorded", async () => { + const { code, failedJobReruns, reruns, outputs } = await reconcile({ pulls: [pr(1)], runs: [failed(101, sha(1), { jobs: floorStatus() })] }); + assert.equal(code, 0); + assert.deepEqual(failedJobReruns, [101]); + assert.deepEqual(reruns, []); + assert.deepEqual(records(outputs), [`reconcile-rerun-101-2-${sha(1)}`, "", ""]); +}); + +test("a Mark Lanes refusal at the floor is re-run before lane runs; a real Mark Lanes failure waits behind them as a flake", async () => { + const lanes = [201, 202, 203].map((id, i) => verifyRun(id, { created: `2026-10-19T19:1${i}:00Z` })); + const floor = await reconcile({ pulls: [pr(1)], runs: [failed(101, sha(1), { jobs: markLanes() }), ...lanes] }); + assert.deepEqual(floor.failedJobReruns, [101]); + assert.deepEqual(floor.reruns, [201, 202]); + assert.deepEqual(records(floor.outputs), [`reconcile-rerun-101-2-${sha(1)}`, `reconcile-rerun-201-2-${MAIN_TIP}`, `reconcile-rerun-202-2-${MAIN_TIP}`]); + const real = await reconcile({ pulls: [pr(1)], runs: [failed(101, sha(1), { jobs: markLanes([failure("Process completed with exit code 1."), failure("ci-lanes is not a status this run may write.")]) }), ...lanes] }); + assert.deepEqual(real.failedJobReruns, []); + assert.deepEqual(real.reruns, [201, 202, 203]); +}); + +test("a CI Status failure that is not only the floor's is not re-run", async (t) => { + t.mock.method(console, "log", () => {}); + const cases = [ + ["a real error beside a refusal", floorStatus([...FLOOR["ci-status-133584509558"], failure(GATED_LINE)])], + ["a real error alone", floorStatus([failure("Process completed with exit code 1."), failure(GATED_LINE)])], + ["a server error's Retry-After hold", floorStatus([failure("'gh api x' was not sent: GitHub answered HTTP 502 with a Retry-After; no call to it before 2026-11-16T21:19:18Z")])], + ]; + for (const [what, jobs] of cases) { + const { code, calls } = await reconcile({ pulls: [pr(1)], runs: [failed(101, sha(1), { jobs })] }); + assert.equal(code, 0, what); + assert.deepEqual(calls.filter((c) => c.startsWith("POST ")), [], what); + } +}); + +test("a floor failure is re-run once: not at attempt 2, nor past the lookback", async () => { + for (const opts of [{ attempt: 2 }, { created: "2026-10-18T19:00:00Z" }]) { + const { code, calls } = await reconcile({ pulls: [pr(1)], runs: [failed(101, sha(1), { jobs: floorStatus(), ...opts })] }); + assert.equal(code, 0); + assert.deepEqual(calls.filter((c) => c.startsWith("POST ")), []); + } +}); + +test("a floor failure on a closed or draft PR, or on a head the PR has moved from, is left alone", async () => { + const runs = [failed(101, sha(1), { jobs: floorStatus() })]; + for (const pulls of [[], [pr(1, { draft: true })], [{ ...pr(1), head: { ...pr(1).head, sha: sha(2) } }]]) { + const { code, calls } = await reconcile({ pulls, runs }); + assert.equal(code, 0); + assert.deepEqual(calls.filter((c) => c.startsWith("POST ")), []); + } + // Closed or pushed between the listing and the write. + for (const changed of [{ state: "closed" }, { head: { ...pr(1).head, sha: sha(2) } }]) { + const moved = (github) => async (method, apiPath, body) => (method === "GET" && apiPath.endsWith("/pulls/1") ? { ...pr(1), ...changed } : github(method, apiPath, body)); + const { code, outputs, failedJobReruns } = await reconcile({ pulls: [pr(1)], runs, github: moved }); + assert.equal(code, 0); + assert.equal(outputs["record-1"], `reconcile-rerun-101-2-${sha(1)}`, "the plan took it"); + assert.deepEqual(failedJobReruns, []); + } +}); + +test("a review a kill switch stopped on a floor-failed head is re-run the pass after its checks run", async () => { + const first = await reconcile({ pulls: [pr(1)], runs: [failed(101, sha(1), { jobs: floorStatus() }), stoppedRun(201, sha(1))] }); + assert.deepEqual(first.failedJobReruns, [101], "the checks run first, so its CI Status can name the stopped review"); + assert.deepEqual(first.reruns, []); + const second = await reconcile({ pulls: [pr(1)], runs: [redChecks(101, sha(1), { attempt: 2 }), stoppedRun(201, sha(1))] }); + assert.deepEqual(second.reruns, [201]); +}); + +test("a pass that defers floor work writes no clean, so the pacer runs the next tick; deferred flakes still write it", async () => { + const pulls = [1, 2, 3, 4].map((n) => pr(n)); + const floors = await reconcile({ pulls, runs: [1, 2, 3, 4].map((n) => failed(100 + n, sha(n), { jobs: floorStatus() })) }); + assert.deepEqual(floors.failedJobReruns, [101, 102, 103]); + assert.equal(floors.outputs.clean, ""); + const flakes = await reconcile({ pulls, runs: [1, 2, 3, 4].map((n) => failed(100 + n, sha(n), { jobs: MIXED_JOBS, updated: "2026-10-19T19:00:00Z" })) }); + assert.deepEqual(flakes.failedJobReruns, [101, 102, 103]); + assert.equal(flakes.outputs.clean, "true"); +}); + +test("each rate guard still words its floor refusal as the reconciler matches it", () => { + assert.ok(floorNoted([failure(new RateLimitStop("'gh api repos/o/r/pulls/1/files'", "core", 1794864751).message)])); + const guard = readFileSync(new URL("../../actions/log-api-budget/rate-guard.mjs", import.meta.url), "utf8"); + for (const what of ["at or under its stop-floor share ${stopFloor} (status ${response.status})", "hit GitHub's ${poolLabel(verdict.pool)} rate limit (status ${response.status})"]) { + assert.ok(guard.includes(what), what); + } + assert.ok(guard.includes("${method} ${url} ${what}: no call to it before ${resetLabel(verdict.reset)}")); + assert.match("GET u left GitHub's core API 65 of 5000 requests, at or under its stop-floor share 0.02 (status 200): no call to it before 2026-11-16T20:32:29Z (x-ratelimit-reset); this step fails", FLOOR_NOTE); + assert.match("GET u hit GitHub's core API rate limit (status 403): no call to it before 2026-11-16T20:32:29Z (x-ratelimit-reset); this step fails", FLOOR_NOTE); + const checks = readFileSync(new URL("../../workflows/pr-require-checks.yml", import.meta.url), "utf8"); + assert.ok(checks.includes(`'$2' was not sent: GitHub's $(api_budget_label "$1") rate limit holds until $(api_budget_when "$until")`)); + assert.ok(checks.includes(`'$4' hit GitHub's $(api_budget_label "$1") rate limit: no call to it before $(api_budget_when "$2") ($3)`)); +}); diff --git a/.github/workflows/maintenance-reconcile-lost-runs.yml b/.github/workflows/maintenance-reconcile-lost-runs.yml index dfc00d61152..901c723dfee 100644 --- a/.github/workflows/maintenance-reconcile-lost-runs.yml +++ b/.github/workflows/maintenance-reconcile-lost-runs.yml @@ -26,7 +26,10 @@ name: Maintenance Reconcile Lost Runs # any other failure has its failed jobs re-run once, at attempt 1 only, so a # flaky test the diff did not cause can pass. pr-unblock's fix-ci waits only on # a re-run already queued or going, or on an infra-only first failure's re-run -# while it is due (infra-failure.mjs RERUN_DUE_MINUTES). +# while it is due (infra-failure.mjs RERUN_DUE_MINUTES). A pr-require-checks +# run whose failed jobs (Mark Lanes, or CI Status alone) failed only where the +# GITHUB_TOKEN hit its rate-limit floor, by the rate guard's own refusal text, +# has its failed jobs re-run once the same way, placed before lane runs. # # A pr-require-checks run that passed on its first attempt is re-run once, # recorded, when the head's pr-verify pass predates a change to its spec (an @@ -155,7 +158,9 @@ jobs: # at reset, projected at this window's burn rate, is under limits.api # github-token pace (pr-pipeline.yaml over rate-limits.json; 0 is off), # and only when the previous scheduled run marked a clean pass, so the timer - # lanes slow to half cadence before the breaker stops every lane. Any + # lanes slow to half cadence before the breaker stops every lane. A pass + # that deferred recovery work for want of a place marks none, so the tick + # after it always runs. Any # other event, or a read that fails, runs as before # (.github/actions/log-api-budget/pace.mjs). - name: Pace the tick on the GITHUB_TOKEN burn rate @@ -310,8 +315,9 @@ jobs: if-no-files-found: ignore retention-days: 1 - # A green pass whose plan read every listing (reconcile.mjs writes - # clean; a failed listing, the kill switch or an Actions outage does + # A green pass whose plan read every listing and deferred no recovery + # work (reconcile.mjs writes clean; a failed listing, the kill switch, + # an Actions outage or a deferred place other than a flake re-run does # not). The pacer paces the next scheduled tick only after this step # (.github/actions/log-api-budget/pace.mjs). - name: Mark this tick's work done