Skip to content

fix(ci): clear the zizmor red on cla.yml and repair the stalled-PR cron (BACKLOG #1457) - #938

Open
wshallwshall wants to merge 2 commits into
mainfrom
builder-ci-reds
Open

fix(ci): clear the zizmor red on cla.yml and repair the stalled-PR cron (BACKLOG #1457)#938
wshallwshall wants to merge 2 commits into
mainfrom
builder-ci-reds

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Two CI reds, both re-verified before acting, both fixed. The diff is .github/ plus one script and its tests; it does not touch docs/BACKLOG.md.

1. zizmor is red on main, and nothing will clear it on its own

zizmor 1.30.0 reached ci/locks/ci-scanners.lock through PR 891 and added the self-repository audit. It fires one low on .github/workflows/cla.yml:102 and the job exits 12. The gate ran on the bump PR, failed with this exact finding, and merged anyway because zizmor is not a required context. That is not a bypass; an advisory check went red and nobody was obliged to stop.

The finding is true and the syntax it recommends is real. GitHub shipped same-repository uses: $/... on 2026-07-30; it resolves to the workflow's own repository at the running commit with no checkout and needs runner 2.336.0 or newer, and hosted runners are on 2.337.0. Verified against GitHub's own changelog, because PR 903 correctly flagged that the lint's help text had not been checked.

I did not adopt it, and the reason is which file it is. Four reasons, each recorded beside the suppression:

  1. cla.yml runs on pull_request_target, merge_group and issue_comment, and all three execute the workflow from the default branch. No pull request can test a change to it -- its green certifies the version being replaced. cla is a required context, so a wrong edit blocks every pull request and the repair is itself gated by the check it broke.
  2. This exact line already did that. PR 621 vendored the action and rewrote this uses: without adding a checkout; 7 of 21 open pull requests were blocked at once and clearing it needed an administrator. scripts/quality/workflow_local_action_check.py exists because of it.
  3. Adopting $/ would silently retire that checker. Its _LOCAL_USES pattern matches only values starting with ./, so a $/ reference is invisible to the guard built to prevent reason 2. A correct adoption must teach it the new form in the same change.
  4. It is not a one-line change. $/ needs no working copy, which makes the actions/checkout step above the uses: obsolete for its stated purpose -- and that step carries the load-bearing reasoning about taking the base and never the head.

So it is a line-anchored accepted residual in .github/zizmor.yml, matching the two entries already there, with what would flip the decision written next to it. The rule is not weakened anywhere else: self-repository stays live on every other file and every other line, and a future local uses: added to cla.yml re-fires rather than being swallowed.

Exit codes, and an instrument error worth reporting

arm exit
before, HEAD's config 12 (1 low, help[self-repository], cla.yml:102:15)
after, this branch 0 (31 ignored, 45 suppressed, was 30 ignored)

Both arms zizmor 1.30.0 with online audits, one variable changed.

My first "after" reading was wrong and said exit 12. zizmor discovered its config from the primary checkout rather than from this worktree, because the worktree lives inside that directory tree, so it never read the file I had edited. Both arms above pass --config explicitly. This does not affect CI, where the checkout is the repository root and discovery finds the right file. Recording it because a worktree nested under the primary will do this to the next person too.

2. The daily stalled-PR report has failed three times and nothing watches it

stalled-prs.yml failed on 2026-09-03, 09-04 and 09-05 (runs 33753049186, 33870995359, 33962763074), each with gh pr list failed (1): unexpected end of JSON input.

The cause, reproduced rather than guessed

The check asked for statusCheckRollup alongside every other field across every open pull request in one gh pr list. Measured locally against this repository:

query result
with statusCheckRollup, 67 open PRs HTTP 502/504, about 11s, zero bytes
identical query without it exit 0, about 3s, all 67 rows

The rollup is the cost: it pulls every check run of every pull request, about 29 each here, so the node count grows with the open set and crossed GitHub's GraphQL timeout between 2026-09-02 (green) and 2026-09-03 (red). The unexpected end of JSON input seen in CI is gh failing to parse the empty body a timed-out query returns.

The fix

List cheaply, then fetch the rollup one pull request at a time and only for the ones that could still be a stall. No single query now carries more than one pull request's rollup, so the node count cannot reassemble however far the repository grows.

One shared predicate, could_be_stalled, gates both the fetch and the scan. Written out twice, a pull request skipped by one and read by the other would be announced as a stall on a rollup nobody fetched -- _counts(None) is (0, 0), which is indistinguishable from fully green. I asserted the opposite in a first draft and checking it caught me; there is now a test pinning it.

Then the silence, which is the real defect

A cron whose failure nobody sees is a gate that cannot fire, and this one reports on other pull requests, so a report that stopped arriving looks exactly like a day with nothing to report.

nightly-notice.yml now watches Stalled PRs, on the seam it already uses for CI, Security and DAST. That workflow has a schedule: trigger, so the notice job's workflow_run.event == 'schedule' gate can match it, and tests/test_nightly_notice.py already pins that precondition for every watched name. BACKLOG #1384 independently reasoned that this exact shape -- one name in the watch list plus its pinned test row -- is the right fix, for a different unwatched workflow.

Four things the split itself made necessary

  • A truncation guard. A returned count equal to --limit is a truncation signal, not a population, so the check refuses to report on a capped list rather than under-reporting silently. The limit moves from 100 to 300; the open count is 68.
  • A retry. One query became one plus one per BEHIND pull request, so a flaky 502 went from one-in-one to one-in-N, and every failure now opens an issue. Serial, never concurrent: parallel requests on a single token invite a secondary-rate-limit 403, which is this outage again arriving slower.
  • Per-call timeouts and timeout-minutes: 15. One call at 180s used to bound the job at about three minutes. That ceiling went with the split, leaving GitHub's 360-minute default on a job nobody watches run.
  • Fail closed on an unreadable rollup response. An earlier draft stored None there, which would have reported the pull request as a stall.

Checks

  • ruff check and ruff format --check: pass
  • mypy messagefoundry (strict): 268 source files, no issues
  • pytest: 261 passed across test_stalled_prs, test_nightly_notice, test_zizmor_config_hygiene, test_required_contexts, test_required_workflow_state, test_failure_signal, test_unread_signal, test_unread_prs, test_workflow_local_action_check, test_workflow_shell_syntax, test_security_posture, test_ci_venv_pinning, test_gate_ci_mirror_parity, test_quality_advisory_invariants, test_dep1_lock_resync_lockstep
  • zizmor 1.30.0 with online audits: 12 before, 0 after
  • scripts/quality/workflow_local_action_check.py: pass, 28 workflows, 1 local reference
  • pre-commit on both commits, actionlint included: pass
  • End-to-end against the live repository: 68 open pull requests scanned, exit 0, 28 seconds, where the previous code returned HTTP 504 in 11

Every new test was proved to discriminate. A hash-verified mutation harness planted nine mutants and killed nine: reverting the query split, fetching a rollup for every pull request, dropping the truncation guard, unwatching the workflow, falling open on an unreadable rollup, removing the retry, retrying forever, lengthening the view timeout, and ignoring the injected runner. A mutant that failed to apply was scored invalid rather than surviving.

/simplify was run; its four review agents produced the retry, the per-call timeouts, the job timeout, the fail-closed rollup branch, the _gh and _rows helpers and the late-bound runner.

Legs a hosted runner must read

This branch changes three workflow files, and no local run can report on them. Read zizmor, Required workflow state, and the CI gate roll-up after they land.

Decisions this brief left open

  1. The zizmor remedy. Suppress rather than adopt $/, for the four reasons above. Reasons 3 and 4 are the ones I would not have found without reading workflow_local_action_check.py, and together they make adoption a two-part change with a live required-context outage as its downside.
  2. No new backlog item for the stalled-PR half. Took the free-text claim stalled-prs-cron instead. Allocating an item means adding a row to docs/BACKLOG.md, which two sibling Builders are editing right now, and this diff is otherwise entirely outside the ledger.
  3. BACKLOG #1457 is cited but does not resolve yet. PR 903 files it and was open and DIRTY at the time of writing, so the number exists and is allocated but is not on main. The two are independent and can merge in either order; nothing here depends on 903, and this PR is the fix rather than a second filing.
  4. Retry and job timeout included, thread pool rejected. Concurrency would take the run from 30s to about 5s once a day, at the cost of secondary-rate-limit exposure on a fail-closed check and worse error attribution. GitHub's own guidance for a single token is serial requests.

Not done, and deliberately

  • _run_gh is the sixth copy of the same gh subprocess wrapper in this tree, alongside report_ci_red.py, check_unread_prs.py, check_required_workflow_state.py, check_required_contexts_drift.py and scripts/security/vuln_metrics.py. Hoisting a shared runner into scripts/ci/_pr_checks.py is a cross-script refactor well outside these two findings, and this change adds no copy -- the same block was already inline. Worth its own item.
  • docs/CI.md:220 says nightly-notice.yml watches only ci.yml. Already stale before this change, and staler now. Left alone to keep the diff tight.
  • I could not read live branch protection. The GitHub API rate limit, shared across the fleet and pushed over by my own rollup calls, refused it twice. This change adds and removes no required context; .github/required-contexts.txt and the pinned count in tests/test_required_contexts.py are untouched, and both test_required_contexts.py and test_required_workflow_state.py pass. Saying so rather than claiming a set-equal read I did not get.

Generated with Claude Code (https://claude.com/claude-code)

wshallwshall and others added 2 commits September 5, 2026 19:40
…te the uses (BACKLOG #1457)

zizmor 1.30.0 arrived in ci/locks/ci-scanners.lock via PR 891 and added the
self-repository audit. It fires one low on .github/workflows/cla.yml:102, where
the vendored action is referenced as `uses: ./.github/actions/cla-assistant-lite`,
and the job exits 12 on any finding. The gate ran on the bump PR, failed with
this exact finding, and merged anyway because zizmor is not a required context.

The finding is true and the syntax it recommends is real: GitHub shipped
same-repository `uses: $/...` on 2026-07-30, needing runner 2.336.0 or newer,
and hosted runners are on 2.337.0. Verified against GitHub's changelog rather
than taken on the lint's word.

It is recorded as an accepted residual instead of adopted, for four reasons that
are all about which file this is:

1. cla.yml runs on pull_request_target, merge_group and issue_comment, and all
   three execute the workflow from the default branch. No pull request can test
   a change to it; its green certifies the version being replaced. `cla` is a
   required context, so a wrong edit blocks every pull request and the repair is
   itself gated by the check it broke.
2. This exact line already did that. PR 621 vendored the action and rewrote this
   uses without adding a checkout; 7 of 21 open pull requests were blocked at
   once and clearing it needed an administrator.
3. scripts/quality/workflow_local_action_check.py matches only values starting
   with `./`, so adopting `$/` makes the reference invisible to the guard built
   to prevent reason 2. A correct adoption has to teach that checker the new
   form in the same change.
4. `$/` needs no working copy, so it makes the actions/checkout step above the
   uses obsolete for its stated purpose, and that step carries the load-bearing
   reasoning about checking out the base and never the head.

The entry is line-anchored, matching the two suppressions already in this file,
so a future local uses added to cla.yml re-fires rather than being swallowed.
What would flip the decision is recorded beside it: a way to exercise cla.yml
before it reaches main.

Verified with the pinned scanner, one variable, both arms run with an explicit
--config so the same file is read each time: zizmor 1.30.0 with online audits
enabled exits 12 on HEAD's config and 0 with this entry, ignored 30 to 31.

BACKLOG #1457 is filed by PR 903, which is open and not yet on main, so this
citation does not resolve in the ledger yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…visible

The daily stalled-PR cron has failed three mornings running -- 2026-09-03,
09-04 and 09-05, runs 33753049186, 33870995359 and 33962763074 -- each with
`gh pr list failed (1): unexpected end of JSON input`. Nothing anywhere said so.

CAUSE, reproduced locally rather than inferred. The check asked for
statusCheckRollup alongside every other field across every open pull request in
one `gh pr list`. At 67 open pull requests that query returns HTTP 502/504 in
about 11 seconds; the same query without the rollup returns all 67 in about 3.
The rollup is the cost: it pulls every check run of every pull request, about 29
each here, so the node count grows with the open set and crossed GitHub's
GraphQL timeout somewhere between 2026-09-02 (green) and 2026-09-03 (red).

FIX. List cheaply, then fetch the rollup one pull request at a time and only for
the ones that could still be a stall. No single query now carries more than one
pull request's rollup, so the node count that crossed the timeout cannot
reassemble however far the repository grows. One shared predicate,
could_be_stalled, gates both the fetch and the scan: written out twice, a pull
request skipped by one and read by the other would be announced as a stall on a
rollup nobody fetched, because _counts(None) is (0, 0) and that is
indistinguishable from fully green.

AND THE SILENCE, which is the real defect. A cron whose failure nobody sees is a
gate that cannot fire, and this one reports on OTHER pull requests, so a report
that stopped arriving looks exactly like a day with nothing to report.
nightly-notice.yml now watches `Stalled PRs` on the same seam it already uses for
CI, Security and DAST -- the workflow has a schedule trigger, so the notice job's
`workflow_run.event == 'schedule'` gate can match it.

Four smaller things the split itself made necessary:

* A truncation guard. A returned count equal to --limit is a truncation signal,
  not a population, so the check now refuses to report on a capped list rather
  than under-reporting silently. The limit moves from 100 to 300; the open count
  is 68.
* A retry. One query became one plus one per BEHIND pull request, so a flaky 502
  went from a one-in-one chance to one-in-N, and every failure now opens an
  issue. Retried serially, never concurrently: parallel requests on a single
  token invite a secondary-rate-limit 403, which is this outage again.
* Per-call timeouts and a job timeout-minutes. One call at 180s used to bound the
  job at about three minutes; that ceiling went with the split, leaving GitHub's
  360-minute default on a job nobody watches run.
* Fail closed on an unreadable rollup response, for the same reason as the
  unusable-number branch beside it. An earlier draft of this change stored None
  there, which would have reported the pull request as a stall.

Verified: ruff check and format, mypy strict (268 files), 261 tests across the
workflow-adjacent suites, zizmor 1.30.0 exit 0 with online audits, and the
shipped script against the live repository -- 68 open pull requests scanned,
exit 0, 28 seconds, where the previous code returned HTTP 504 in 11.

Every new test was proved to discriminate with a hash-verified mutation harness:
nine mutants planted, nine killed, including reverting the query split, dropping
the truncation guard, unwatching the workflow and falling open on an unreadable
rollup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the ci-red A required check went red. Attribute it before retrying. label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-red A required check went red. Attribute it before retrying.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant