Skip to content

fix(ci): skip lost lane runs that follow no open head's checks - #16658

Merged
kyle-sexton merged 2 commits into
mainfrom
fix/16642-drain-lost-run-backlog
Nov 16, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
fix/16642-drain-lost-run-backlog

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Refs: #16642

Summary

The first lost-run reconcile pass after the 22:33Z brake cleared (run 43971789832) logged 45 items as "deferred, all 3 re-run places taken": 44 pr-verify lane runs and the CI set of #16624.

41 of the 44 lane runs had nothing live left to judge, because their PRs had merged or their heads had moved. The reconciler's "a newer run judges the head" check covers only runs cancelled before any job started. Failed and deferred runs never get it.

The 3 live runs were not stuck behind the 41: laneOrder already sorted matched runs first. The 41 still cost something:

  • While they were deferred, no pass wrote clean, so the pacer ran every tick.
  • Each of them would have been re-run as places freed up: about 14 passes of wasted re-runs, around 290 core calls.

This change removes that waste. Latency for live runs is unchanged.

Fix

.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs, laneOrder: a lost lane run that no open head could own is left out of the queue. Such a run takes no place, does not count as deferred, and is named in the log as not re-run. A head could own a run created from 60 s (MATCH_BEFORE_MS) before its newest checks completion to 2 hours (OWNED_AFTER_MS) after it.

  • Why the lower bound is safe: a lane run is created after the checks completion it follows, so a run created earlier follows none of that head's current checks.
  • Why the upper bound is 2 hours (judgment): it is 8× the 15-minute match window (MATCH_AFTER_MS). A live run whose workflow_run queue was backed up past that window is still kept, and a test covers it. Without an upper bound, one open head left untouched for hours would keep every later stale run queued. feat(app-control): bring map refresh in line with the maintain-verification-skill upkeep pass #16481 (do-not-merge, newest checks completion 03:59Z) is open now.
  • When nothing is left out:
    • an open head's run listing failed;
    • no open head has a completed checks run.
  • Fault-held runs (fault-held) are never left out.
  • Only a checks run that starts the lane counts (LANE_TRIGGERS). For each lane, "newest checks completion" means the newest one whose conclusion starts that lane: pr-verify starts on success or failure, pr-review-main on success only. A later failed checks run therefore never drops a lost main-run review. A test checks these conclusions against each workflow's trigger condition.
  • What does not change: MAX_IN_FLIGHT stays 3, which keeps the stagger. After the drop, the backlog drains in two passes, so (b) one run per head and (c) a larger cap after a brake clear were not needed.

Verification

The 44 deferred lane runs at 22:37Z, by category. Method:

  1. Listed the pr-require-checks.yml runs and matched each lane run by time to a checks completion, using the window laneOrder uses.
  2. Looked up each head's PR.
  3. Checked for pr-verify-<pr>-<head>-* verdict artifacts.
Category Runs Created This change
PR merged since 37 19:32Z to 20:34Z left out
Head moved (#16634 ×3, #16652 ×1) 4 20:15Z to 20:33Z left out
Live: current head, no verdict at pass time 3 22:19Z to 22:22Z re-run
#16624's CI set (not a lane run) 1 n/a unchanged
  • Replaying laneOrder as of 22:37Z over the 9 PRs open then:
  • chore(release): release 20 plugins #16652: its live run was judged at 22:41Z by the pr-verify run its floor re-run started (artifact from run 43972110749).
  • docs(enforcement-ladder): add the architecture and retro Selection rules rows in ladder 3.2.0 #16602: it left the merge queue at 22:15Z, but it is not in this backlog. Its head was judged at 22:10:41Z (pass-notes, run 43967067284).
  • Drain with this change, replayed on the 22:37Z pass:
  • Tests: node --test over the reconciler's four test files: 235 pass, 0 fail.
    • New tests:
      • Runs from before an open head's checks, or from before its checks ran again, are left out, the pass writes clean, and a fault-held run is kept.
      • The edges of the ownership bounds: 30 s before and 1 min short of 2 h after are kept; 61 s before and 2 h 1 s after are left out.
      • An open head untouched for hours keeps only the runs near its own checks.
      • Nothing is left out when an open head's listing fails.
      • With 4 live runs and 2 superseded ones, 3 are placed, 1 is deferred and clean is withheld.
    • Three existing tests changed:
      • The ordering test's 16:00Z run is now left out.
      • The match-window test adds a second open head whose completion owns its runs.
      • The floor test's lane runs are created after their head's checks.
  • Mutants: each one fails at least one test.
    • Without the 60 s offset: 1 test fails.
    • Without the 2-hour bound: 2 fail.
    • Without the "some open head has a completion" guard: 22 fail.
    • Without the head-read guard: 1 fails.
    • Without the fault-held exemption: 1 fails.
  • Affected checks: scripts/check-affected.sh --base origin/main --no-fetch passed.

Remaining

Done-when item 3 still needs live evidence. On the next brake clear, the first reconcile pass should name stale lane runs as not re-run, defer only live work, and write clean once that work is placed.

Related

🤖 Generated with Claude Code

After a brake cleared, the lost-run reconciler deferred 44 pr-verify runs
at 3 re-runs a pass. 41 of them were created before every open pull
request's newest checks run completed: their PRs had merged or their
heads had moved, so a re-run would judge nothing live. They kept the pass
from writing clean and would each have spent a re-run once places freed.

A lost lane run created, for every open head, more than 60 s before its
newest checks completion or more than 2 hours after it now takes no
place and is named in the log. A lane run starts after the checks
completion it follows, so such a run follows no open head's current
checks. The 2-hour bound keeps a live run whose queue was backed up far
past the 15-minute match window, while stopping one long-untouched open
head from keeping every later stale run queued. Nothing is left out when
an open head's runs could not be listed or no open head has a completed
checks run, and fault-held runs are never left out.

Refs: #16642

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 23:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T23:05:23.036199Z 9317539 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93175395f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 5ff0f6072f8dcd54bf29b7eab938a7b83b9ffcf2: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16658 (incremental, blocking only), no security issues found

This was an incremental review covering the changes since 93175395. This lane already reviewed an earlier head, so it reported only blocking (CRITICAL or IMPORTANT) findings. I read every hunk of both files in scope: .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs and reconcile.test.mjs.

What changed: a new LANE_TRIGGERS map names the checks-run conclusions that start each recorded lane: pr-verify starts on success or failure, and pr-review-main only on success. A new laneStarts(own) helper records, for each lane, the time of the head's newest completed checks run that starts that lane. laneOrder now leaves out a lost lane run only when some open head has a start time for that run's lane and the run's created_at falls outside every such head's window. That window was 60 s before to 2 h after the newest checks completion; it is now measured from the newest completion that starts the lane. The new tests check that a failed checks run leaves out a lost verify run and keeps a lost main-run review. They also check that LANE_TRIGGERS matches each workflow's github.event.workflow_run.conclusion == '…' tests.

What I checked:

  • Direction of change: this narrows what gets left out. Before, a run was left out based on any completed checks run. Now only one that starts its lane counts, so fewer lost runs are left out and more are re-run. The old guard drop && openChecks.length > 0 becomes drop plus a per-run starts.length > 0 check. That is the same rule or a stricter one: no matching start means nothing is left out. Fault-held runs (kind !== "lane") stay exempt.
  • Merge holds and trust binding: leaving a run out only skips a re-run. No check conclusion, verdict or hold is written. The re-run path, and its binding to the PR through check-trusted-trigger and resolve-pr, is unchanged.
  • Untrusted input: the decision reads only GitHub API run fields (path, event, status, conclusion, updated_at, created_at) for runs whose head_repository.full_name is this repository. No text written by a PR author reaches it.
  • Trust controls: neither file is on the trusted-path list. No token scope, trusted-actor list, kill switch, permissions: block or manifest changes. No instruction file changes.

No embedded instructions were found in the PR context. The prior summary finding (0 findings) has nothing to restate.

Spend: more stale-but-live main-run review re-runs than in the previous head. This spends Actions runs and API calls from prepaid or subscription use only, with no new charge.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 5ff0f6072f8dcd54bf29b7eab938a7b83b9ffcf2: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, blocking only): no findings

Scope. This review covers only what changed since 93175395, in reconcile.mjs and reconcile.test.mjs. I read the whole incremental diff for both files. I also read the unchanged code the new lines depend on in reconcile.mjs:

  • laneOrder, including its match-pair loop.
  • The per-PR listing loop in plan.
  • The everyHeadRead guard.

This lane already reviewed an earlier head, so this review reports blocking findings only. The PR refs #16642 but closes no issue, so there are no closing-issue criteria to check.

What changed. The drop rule in laneOrder no longer compares a lost lane run against a head's newest checks completion of any conclusion. It now uses the newest completion whose conclusion starts that run's lane, through LANE_TRIGGERS (reconcile.mjs#L366-L369 and L795-L801). This fixes the case in the settled thread: a later failed checks run no longer drops a lost pr-review-main review.

What I checked:

  • No open head has a completion that starts the lane. starts.length > 0 replaces the old openChecks.length > 0 guard for each lane, so the run is kept. That is the conservative choice.
  • A lane in RECORDED_LANES but missing from LANE_TRIGGERS. That lane gets no starts, so its runs are never dropped. The new drift test pins the two lanes against each workflow's workflow_run.conclusion == '…' conditions.
  • Ordering is unchanged. Match pairing still uses done, the newest completion of any conclusion. The new starts field only feeds the drop rule.
  • everyHeadRead is unaffected. Each openChecks entry is still pushed only when done is defined, so the head-read guard keeps working.
  • New test. It encodes the reproduction from the settled thread: success at 17:00, failure at 19:00. The pr-review-main run from 17:00:04 is kept and re-run. The pr-verify run from 17:00:30 is dropped.

I did not run the tests. The "235 pass" result is the author's claim and is unverified.

Spend. This change keeps the earlier reduction in lane re-runs and starts no new charge, so no hold.

Verifier dropped 0 of 0 candidates.

pr-review-main starts only on a passed PR Require Checks run, while
pr-verify starts on a passed or failed one. The superseded test read a
head's newest checks completion of any conclusion, so a later failed
checks run on an unchanged head dropped a lost main-run review that
nothing would replace. Each lane now compares against the head's newest
checks completion whose conclusion starts it (LANE_TRIGGERS), and a test
pins those conclusions to each workflow's trigger condition.

Refs: #16642

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 86b6f0f Nov 16, 2026
77 checks passed
@kyle-sexton
kyle-sexton deleted the fix/16642-drain-lost-run-backlog branch November 16, 2026 23:29
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

Killed 6, survived 1, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 86%; mutation score 86%.

File Killed Survived Timeout No coverage Invalid Not run
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs 6 1 0 0 0 0

Surviving mutants

Location Operator Disposition Cause
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:752 other productive input-gap

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant