fix(ci): clear the zizmor red on cla.yml and repair the stalled-PR cron (BACKLOG #1457) - #938
Open
wshallwshall wants to merge 2 commits into
Open
fix(ci): clear the zizmor red on cla.yml and repair the stalled-PR cron (BACKLOG #1457)#938wshallwshall wants to merge 2 commits into
wshallwshall wants to merge 2 commits into
Conversation
…te the uses (BACKLOG #1457) zizmor 1.30.0 arrived in ci/locks/ci-scanners.lock via PR 891 and added the self-repository audit. It fires one low on .github/workflows/cla.yml:102, where the vendored action is referenced as `uses: ./.github/actions/cla-assistant-lite`, and the job exits 12 on any finding. The gate ran on the bump PR, failed with this exact finding, and merged anyway because zizmor is not a required context. The finding is true and the syntax it recommends is real: GitHub shipped same-repository `uses: $/...` on 2026-07-30, needing runner 2.336.0 or newer, and hosted runners are on 2.337.0. Verified against GitHub's changelog rather than taken on the lint's word. It is recorded as an accepted residual instead of adopted, for four reasons that are all about which file this is: 1. cla.yml runs on pull_request_target, merge_group and issue_comment, and all three execute the workflow from the default branch. No pull request can test a change to it; its green certifies the version being replaced. `cla` is a required context, so a wrong edit blocks every pull request and the repair is itself gated by the check it broke. 2. This exact line already did that. PR 621 vendored the action and rewrote this uses without adding a checkout; 7 of 21 open pull requests were blocked at once and clearing it needed an administrator. 3. scripts/quality/workflow_local_action_check.py matches only values starting with `./`, so adopting `$/` makes the reference invisible to the guard built to prevent reason 2. A correct adoption has to teach that checker the new form in the same change. 4. `$/` needs no working copy, so it makes the actions/checkout step above the uses obsolete for its stated purpose, and that step carries the load-bearing reasoning about checking out the base and never the head. The entry is line-anchored, matching the two suppressions already in this file, so a future local uses added to cla.yml re-fires rather than being swallowed. What would flip the decision is recorded beside it: a way to exercise cla.yml before it reaches main. Verified with the pinned scanner, one variable, both arms run with an explicit --config so the same file is read each time: zizmor 1.30.0 with online audits enabled exits 12 on HEAD's config and 0 with this entry, ignored 30 to 31. BACKLOG #1457 is filed by PR 903, which is open and not yet on main, so this citation does not resolve in the ledger yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…visible The daily stalled-PR cron has failed three mornings running -- 2026-09-03, 09-04 and 09-05, runs 33753049186, 33870995359 and 33962763074 -- each with `gh pr list failed (1): unexpected end of JSON input`. Nothing anywhere said so. CAUSE, reproduced locally rather than inferred. The check asked for statusCheckRollup alongside every other field across every open pull request in one `gh pr list`. At 67 open pull requests that query returns HTTP 502/504 in about 11 seconds; the same query without the rollup returns all 67 in about 3. The rollup is the cost: it pulls every check run of every pull request, about 29 each here, so the node count grows with the open set and crossed GitHub's GraphQL timeout somewhere between 2026-09-02 (green) and 2026-09-03 (red). FIX. List cheaply, then fetch the rollup one pull request at a time and only for the ones that could still be a stall. No single query now carries more than one pull request's rollup, so the node count that crossed the timeout cannot reassemble however far the repository grows. One shared predicate, could_be_stalled, gates both the fetch and the scan: written out twice, a pull request skipped by one and read by the other would be announced as a stall on a rollup nobody fetched, because _counts(None) is (0, 0) and that is indistinguishable from fully green. AND THE SILENCE, which is the real defect. A cron whose failure nobody sees is a gate that cannot fire, and this one reports on OTHER pull requests, so a report that stopped arriving looks exactly like a day with nothing to report. nightly-notice.yml now watches `Stalled PRs` on the same seam it already uses for CI, Security and DAST -- the workflow has a schedule trigger, so the notice job's `workflow_run.event == 'schedule'` gate can match it. Four smaller things the split itself made necessary: * A truncation guard. A returned count equal to --limit is a truncation signal, not a population, so the check now refuses to report on a capped list rather than under-reporting silently. The limit moves from 100 to 300; the open count is 68. * A retry. One query became one plus one per BEHIND pull request, so a flaky 502 went from a one-in-one chance to one-in-N, and every failure now opens an issue. Retried serially, never concurrently: parallel requests on a single token invite a secondary-rate-limit 403, which is this outage again. * Per-call timeouts and a job timeout-minutes. One call at 180s used to bound the job at about three minutes; that ceiling went with the split, leaving GitHub's 360-minute default on a job nobody watches run. * Fail closed on an unreadable rollup response, for the same reason as the unusable-number branch beside it. An earlier draft of this change stored None there, which would have reported the pull request as a stall. Verified: ruff check and format, mypy strict (268 files), 261 tests across the workflow-adjacent suites, zizmor 1.30.0 exit 0 with online audits, and the shipped script against the live repository -- 68 open pull requests scanned, exit 0, 28 seconds, where the previous code returned HTTP 504 in 11. Every new test was proved to discriminate with a hash-verified mutation harness: nine mutants planted, nine killed, including reverting the query split, dropping the truncation guard, unwatching the workflow and falling open on an unreadable rollup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two CI reds, both re-verified before acting, both fixed. The diff is
.github/plus one script and its tests; it does not touchdocs/BACKLOG.md.1. zizmor is red on main, and nothing will clear it on its own
zizmor1.30.0 reachedci/locks/ci-scanners.lockthrough PR 891 and added theself-repositoryaudit. It fires one low on.github/workflows/cla.yml:102and the job exits 12. The gate ran on the bump PR, failed with this exact finding, and merged anyway because zizmor is not a required context. That is not a bypass; an advisory check went red and nobody was obliged to stop.The finding is true and the syntax it recommends is real. GitHub shipped same-repository
uses: $/...on 2026-07-30; it resolves to the workflow's own repository at the running commit with no checkout and needs runner 2.336.0 or newer, and hosted runners are on 2.337.0. Verified against GitHub's own changelog, because PR 903 correctly flagged that the lint's help text had not been checked.I did not adopt it, and the reason is which file it is. Four reasons, each recorded beside the suppression:
cla.ymlruns onpull_request_target,merge_groupandissue_comment, and all three execute the workflow from the default branch. No pull request can test a change to it -- its green certifies the version being replaced.clais a required context, so a wrong edit blocks every pull request and the repair is itself gated by the check it broke.uses:without adding a checkout; 7 of 21 open pull requests were blocked at once and clearing it needed an administrator.scripts/quality/workflow_local_action_check.pyexists because of it.$/would silently retire that checker. Its_LOCAL_USESpattern matches only values starting with./, so a$/reference is invisible to the guard built to prevent reason 2. A correct adoption must teach it the new form in the same change.$/needs no working copy, which makes theactions/checkoutstep above theuses:obsolete for its stated purpose -- and that step carries the load-bearing reasoning about taking the base and never the head.So it is a line-anchored accepted residual in
.github/zizmor.yml, matching the two entries already there, with what would flip the decision written next to it. The rule is not weakened anywhere else:self-repositorystays live on every other file and every other line, and a future localuses:added tocla.ymlre-fires rather than being swallowed.Exit codes, and an instrument error worth reporting
help[self-repository],cla.yml:102:15)31 ignored, 45 suppressed, was 30 ignored)Both arms zizmor 1.30.0 with online audits, one variable changed.
My first "after" reading was wrong and said exit 12. zizmor discovered its config from the primary checkout rather than from this worktree, because the worktree lives inside that directory tree, so it never read the file I had edited. Both arms above pass
--configexplicitly. This does not affect CI, where the checkout is the repository root and discovery finds the right file. Recording it because a worktree nested under the primary will do this to the next person too.2. The daily stalled-PR report has failed three times and nothing watches it
stalled-prs.ymlfailed on 2026-09-03, 09-04 and 09-05 (runs 33753049186, 33870995359, 33962763074), each withgh pr list failed (1): unexpected end of JSON input.The cause, reproduced rather than guessed
The check asked for
statusCheckRollupalongside every other field across every open pull request in onegh pr list. Measured locally against this repository:statusCheckRollup, 67 open PRsThe rollup is the cost: it pulls every check run of every pull request, about 29 each here, so the node count grows with the open set and crossed GitHub's GraphQL timeout between 2026-09-02 (green) and 2026-09-03 (red). The
unexpected end of JSON inputseen in CI isghfailing to parse the empty body a timed-out query returns.The fix
List cheaply, then fetch the rollup one pull request at a time and only for the ones that could still be a stall. No single query now carries more than one pull request's rollup, so the node count cannot reassemble however far the repository grows.
One shared predicate,
could_be_stalled, gates both the fetch and the scan. Written out twice, a pull request skipped by one and read by the other would be announced as a stall on a rollup nobody fetched --_counts(None)is(0, 0), which is indistinguishable from fully green. I asserted the opposite in a first draft and checking it caught me; there is now a test pinning it.Then the silence, which is the real defect
A cron whose failure nobody sees is a gate that cannot fire, and this one reports on other pull requests, so a report that stopped arriving looks exactly like a day with nothing to report.
nightly-notice.ymlnow watchesStalled PRs, on the seam it already uses for CI, Security and DAST. That workflow has aschedule:trigger, so the notice job'sworkflow_run.event == 'schedule'gate can match it, andtests/test_nightly_notice.pyalready pins that precondition for every watched name. BACKLOG #1384 independently reasoned that this exact shape -- one name in the watch list plus its pinned test row -- is the right fix, for a different unwatched workflow.Four things the split itself made necessary
--limitis a truncation signal, not a population, so the check refuses to report on a capped list rather than under-reporting silently. The limit moves from 100 to 300; the open count is 68.timeout-minutes: 15. One call at 180s used to bound the job at about three minutes. That ceiling went with the split, leaving GitHub's 360-minute default on a job nobody watches run.Nonethere, which would have reported the pull request as a stall.Checks
ruff checkandruff format --check: passmypy messagefoundry(strict): 268 source files, no issuespytest: 261 passed acrosstest_stalled_prs,test_nightly_notice,test_zizmor_config_hygiene,test_required_contexts,test_required_workflow_state,test_failure_signal,test_unread_signal,test_unread_prs,test_workflow_local_action_check,test_workflow_shell_syntax,test_security_posture,test_ci_venv_pinning,test_gate_ci_mirror_parity,test_quality_advisory_invariants,test_dep1_lock_resync_lockstepzizmor1.30.0 with online audits: 12 before, 0 afterscripts/quality/workflow_local_action_check.py: pass, 28 workflows, 1 local referenceEvery new test was proved to discriminate. A hash-verified mutation harness planted nine mutants and killed nine: reverting the query split, fetching a rollup for every pull request, dropping the truncation guard, unwatching the workflow, falling open on an unreadable rollup, removing the retry, retrying forever, lengthening the view timeout, and ignoring the injected runner. A mutant that failed to apply was scored invalid rather than surviving.
/simplifywas run; its four review agents produced the retry, the per-call timeouts, the job timeout, the fail-closed rollup branch, the_ghand_rowshelpers and the late-bound runner.Legs a hosted runner must read
This branch changes three workflow files, and no local run can report on them. Read
zizmor,Required workflow state, and theCI gateroll-up after they land.Decisions this brief left open
$/, for the four reasons above. Reasons 3 and 4 are the ones I would not have found without readingworkflow_local_action_check.py, and together they make adoption a two-part change with a live required-context outage as its downside.stalled-prs-croninstead. Allocating an item means adding a row todocs/BACKLOG.md, which two sibling Builders are editing right now, and this diff is otherwise entirely outside the ledger.BACKLOG #1457is cited but does not resolve yet. PR 903 files it and was open andDIRTYat the time of writing, so the number exists and is allocated but is not onmain. The two are independent and can merge in either order; nothing here depends on 903, and this PR is the fix rather than a second filing.Not done, and deliberately
_run_ghis the sixth copy of the sameghsubprocess wrapper in this tree, alongsidereport_ci_red.py,check_unread_prs.py,check_required_workflow_state.py,check_required_contexts_drift.pyandscripts/security/vuln_metrics.py. Hoisting a shared runner intoscripts/ci/_pr_checks.pyis a cross-script refactor well outside these two findings, and this change adds no copy -- the same block was already inline. Worth its own item.docs/CI.md:220saysnightly-notice.ymlwatches onlyci.yml. Already stale before this change, and staler now. Left alone to keep the diff tight..github/required-contexts.txtand the pinned count intests/test_required_contexts.pyare untouched, and bothtest_required_contexts.pyandtest_required_workflow_state.pypass. Saying so rather than claiming a set-equal read I did not get.Generated with Claude Code (https://claude.com/claude-code)