fix: run zizmor's online audits and name the mode it chose - #256
Merged
Conversation
Issue #239. Spec and ADR 0036 only; no code yet. The ADR number is provisional: the orchestrator assigned 0035 to a sibling and had not answered the request for one when this was written.
Adversarial review of ADR 0036 reproduced that a token is not connectivity: gh auth token reads the keyring without network, and zizmor handed a token it cannot use exits 1 with 'fatal: no audit was performed' rather than degrading. The fallback would therefore have reddened just verify and the pre-push hook for every developer whenever the API was unreachable. Mode is now selected from GH_TOKEN, GITHUB_TOKEN and ZIZMOR_GITHUB_TOKEN alone, so online mode is an opt-in and the script never reads a token's value. Also corrects an overstated verified: ground, records both directions of the CI residual, and states that the mode line reports the token found rather than that every online audit reached the API.
Second adversarial pass reproduced that ZIZMOR_OFFLINE and ZIZMOR_NO_ONLINE_AUDITS each override a present token: the run completes offline at exit 0 and prints no WARN at all. A gate inferring its mode from the token alone would have announced 'online mode' over an unaudited run - worse than today, since the label is the whole product of this change. Mode selection now tests all five variables, the two mode controls outranking the tokens, and the offline line names which one chose it. Also names the four online audit IDs from zizmor's docs, reports GH_HOST on the online line, fixes two verified: grounds that carried no command or source, adds the rejected bullet for scoping the token to a dedicated CI step, and cuts the tripled measurements.
Third adversarial pass reproduced three defects in the previous revision, all re-run here: - ZIZMOR_OFFLINE=false selects online, so a presence test would have forced --offline and reported the operator's own variable as the cause of a mode they asked against. - An empty GH_TOKEN is a clap usage error (exit 2, 'GitHub token cannot be empty') even with --offline on argv, so 'empty is not a token' was false and the gate would have announced a mode the run never entered. - The audit list came from the docs page and matched neither it nor the binary. Measured instead: zizmor 1.29.0 skips impostor-commit, known-vulnerable-actions, ref-confusion, stale-action-refs and ref-version-mismatch; typosquat-uses runs offline. The mode variables are now read by value, an empty token variable is removed from the child environment, a non-boolean mode value exits 2 before any mode is announced, and Consequences records that online audits make the required check depend on upstream state. Also cuts the record and drops an asserted premise.
Fourth adversarial pass, no high findings. Four corrections and one addition: - The online path is the only one that can fail, and actions-check runs in the pre-push hook, so a failed online audit blocks git push with no stated remedy. The script now prints one after a non-zero online status. - 'the exposure is to first-party reviewed code' was false: just verify runs an unpinned npm-installed CLI, as the spec's own boundary inventory records. What bounds the exposure is the permission and the public repository. - '[possible values: true, false]' comes from clap's rejection path, not from --help; the attribution moves to the measurement that produced it. - Adds the missing alternative - run the online audits off the required check - and rejects it: a finding that lands after merge is a pin already on main. - Cuts two rejection bullets that restated arguments made above them.
Resumed on explicit operator approval (ADR 0029). - The online-failure hint now keys on zizmor's tool-failure status alone, not on any non-zero exit. Measured on 1.29.0: 0 clean, 1 tool failure, 2 usage error, 14 findings. Offering the hint at 14 would have advised switching off the audit that just caught something. - A malformed mode value now warns and falls through instead of exiting 2. An explicit --offline shadows ZIZMOR_OFFLINE, so exiting would have refused where zizmor runs happily and would have regressed ZIZMOR_OFFLINE=1, green today. - Records the version-drift residual as measured-on-1.29.0, since CI installs zizmor unpinned. - Grounds the gh-auth rejection with a dead-proxy command and marks the contents: read argument as reasoning pending the CI run that settles it. - Splits the CI-fail rejection so it answers silent loss, not only deliberate removal.
actions-check invoked zizmor --offline unconditionally, so the pin-provenance audits never ran anywhere: impostor-commit, known-vulnerable-actions, ref-confusion, stale-action-refs and ref-version-mismatch, measured as skipped via zizmor --offline -vv on 1.29.0. scripts/run-zizmor.sh now resolves the mode from all five variables that decide it -- the two clap booleans by value, then zizmor's three token names in its own order -- prints the mode and the condition that chose it before scanning, and re-raises zizmor's status unchanged. The offline path keeps --offline, so every audit that passes today passes identically, and states that provenance was not audited. CI's Verify step gains GH_TOKEN, which is what puts the online audits on the merge path; permissions: contents: read and persist-credentials: false are deliberately unchanged. Verified against the real zizmor on this host: tokenless with gh off PATH exits 0 reporting the offline condition, and with a token exits 0 having scheduled all five provenance audits across both workflows. just verify green, 20 suites.
Branch review and the security pass returned two low findings each; all four are fixed here. - The online-failure hint said 'set ZIZMOR_OFFLINE=true'. CI is where a token is always set and so where a tool failure is most likely, and on a runner the only way to act on that is to edit verify.yml or the Justfile - disabling the five provenance audits for good, on a red required check where the pressure to go green is highest. It now leads with the diagnosis and scopes the remedy to a local run. - The malformed-value warning promised to ignore the value. Nothing shadows --no-online-audits, so zizmor still exits 2 there after the gate announced a mode; the wording no longer claims an outcome the gate cannot deliver. - ADR 0036's rg scan cited 'exactly one hit ... on this branch' and returns 64, because this change's own files match the pattern. Corrected to a path list that reproduces as written. - The threat model's B2 gave the egress destination as api.github.com unconditionally; GH_HOST selects it. B2, its control list and the accepted residuals now say so. just verify green, exit 0.
…ames
Second branch-review pass proved two coverage gaps by mutation, both re-run
here.
- Every online case asserted only the absence of --offline, so a mutant adding
--no-online-audits, --min-severity high, --no-exit-codes or -o to the online
invocation passed the suite: unaudited AND labelled audited, which the design
calls worse than the status quo. The online argv is now pinned exactly, and all
four of those mutants redden.
- The empty-token removal was asserted for GH_TOKEN alone; deleting it for the
other two passed. Both are separately fatal to zizmor (exit 2, 'GitHub token
cannot be empty'), so all three now have a case, plus one setting all three.
- GH_HOST was the one variable read as ${VAR:-}, so an exported-but-empty value
went unreported on the line whose job is to say where the run was pointed. It
now uses ${VAR+set} like the rest, with a case.
- Corrects the GH_HOST rationale: the fatal's causal chain does name the host, so
the clause's value is stating it before the scan. Records that the audit which
failed was artipacked - not one of the five - so a token pulls more than those
five onto the network.
just verify green, exit 0.
Third branch-review pass, one low finding. The remedy line said 'an API or token fault' on every status 1, but zizmor exits 1 for causes that are neither - measured here, 'invalid input: <path>' for a missing directory exits 1, and so does an unloadable zizmor.yml, where ZIZMOR_OFFLINE=true fixes nothing. That is the cause-not-carried-by-the-observation defect ADR 0025 decision 2 forbids, in the one line this change adds to the red path. The line now reports what status 1 does carry - the run failed rather than reporting findings - points at zizmor's own error, and offers the remedy conditionally. The ((status == 1)) gate and the online-mode guard are unchanged; both are correct and the suite kills mutants of each. Also corrects measurement 4: a malformed workflow exits 3, so the status list is 'at least these' rather than an enumeration, and only the 1-versus-14 split is load-bearing. just verify green, exit 0.
Fourth branch-review pass. Pass 3 reworded the online-failure hint but left four assert_lacks needles looking for 'could not run', which the script no longer prints - so all four passed unconditionally and two mutants that break the hint survived: keying it on any non-zero status (which would advise disabling the audit on a real finding, exit 14) and dropping the online-mode guard. Both now die, as does a third that mislabelled the offline condition when a mode variable and a missing token compete. That is the verify-tests-bite rule failing exactly where it was applied: the mutation run happened before the rewording and was not repeated after it. Also adds the missing discriminator cases for a mode variable with no token, and records two ADR corrections - a non-boolean ZIZMOR_OFFLINE is inert offline but fatal online, and a mode-variable rename is the silent failure, not the loud one.
…dits-239 # Conflicts: # .claude-plugin/plugin.json
…dits-239 # Conflicts: # .claude-plugin/plugin.json
Fifth branch-review pass. CI is the only place the five provenance audits run - a workstation is offline by default and says so - and the runner takes the online branch solely because github.token resolves non-empty. If it ever resolved empty, run-zizmor.sh would report the no-token condition and exit 0, leaving the required check green with no provenance audited anywhere: this change's own subject, one level up, mitigated only by a log line. The Verify step now asserts the token is non-empty before just ci. It re-types no gate command. ADR 0036's rejection of a ZIZMOR_REQUIRE_ONLINE flag is rewritten: the concern was right, the flag is the wrong shape for it, and the old ground - that it would redden an unrelated PR - did not survive, since the record already accepts exactly that when a tag moves or an advisory lands. Also corrects the threat model's default egress host to github.com per zizmor --help, and states that a true ZIZMOR_NO_ONLINE_AUDITS is executed as the stronger --offline, which the mode line discloses. just verify green, actionlint clean.
Sixth branch-review pass plus the security rescan. Committed to protect the
work; the full just verify on this exact tree was interrupted and has NOT run.
- The CI guard I added in the previous commit named a cause its observation did
not carry: -z cannot tell an unset GH_TOKEN (the env: wiring is gone, fix is in
verify.yml) from a set-but-empty one (github.token resolved empty, fix is in
repository settings). Two arms now, using the ${VAR+set} idiom run-zizmor.sh
already uses three lines away. Same ADR 0025 defect I had just fixed in the
script, reintroduced in the workflow.
- run-zizmor.sh forwarded "$@" unchecked, so a mode flag passed as an input
reached zizmor on argv and won over the announced mode - measured, --offline
produced 'online mode' followed by all five provenance audits skipped. Flags
are now refused at exit 2 before any mode is announced, with a suite case.
- Reconciles both records with what ships: the guard moves into ADR Decision 3 so
its own cross-reference resolves, the spec's quoted YAML carries it, and the
spec's B1 control list gains it. Narrows the ZIZMOR_REQUIRE_ONLINE rejection,
which overstated coverage - the guard closes the empty-token route only.
- Corrects the threat model's default egress host to github.com per zizmor --help.
Version set to 2.9.17 per the campaign ledger reassignment.
…dits-239 # Conflicts: # .claude-plugin/plugin.json
…dits-239 # Conflicts: # .claude-plugin/plugin.json
…dits-239 # Conflicts: # .claude-plugin/plugin.json
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #239
just actions-checkranzizmor --offline, which disables every online auditrule. Offline mode confirms the shape of an action pin — that a full
40-character SHA is written rather than a mutable tag. The audits it disables
are the ones that confirm the pin's provenance: whether that SHA is reachable
in the repository the
uses:names, and whether the pinned revision is knownbad. A run that skipped those and a run that passed them looked identical from
the outside.
Nothing is wrong today. All five pins were resolved by hand at review time and
an online run with a token returned no findings. The gap is prospective: the
next bump, whether hand-edited or opened by the dependabot config now landing
alongside this, would have received only the offline subset.
What changes
Mode selection moves into
scripts/run-zizmor.sh, which reads all fivevariables in zizmor's own vocabulary — the two mode booleans first, then
GH_TOKEN,GITHUB_TOKENandZIZMOR_GITHUB_TOKEN— announces the mode itselected and the condition that selected it before the scan starts, and
runs zizmor in the strongest mode the environment supports.
CI's Verify step gains
GH_TOKEN: ${{ github.token }}. The job'spermissions: contents: readis not widened and checkout'spersist-credentials: falseis not relaxed. The security-posture change isthat the token's value becomes visible to the
just cistep — not that anypermission grows.
A workstation with no token degrades to the offline subset, stays green, and
says so, naming the condition.
just verifycontinues to work with no networkand no credentials, as every other gate in this repository does.
Why a script rather than an inline recipe
Anatomy rule 1 says a supporting file is the exception and must be argued for
in the pull request that adds it, so here is the argument.
A
Justfilerecipe body is invisible toscripts/list-shell-sources.sh. Logicplaced there would be unseen by
shellcheckandshfmt, and could carry notest suite at all. The mode ladder is exactly the kind of branching that needs
both: five variables, two of them clap booleans where the value is the
instruction —
ZIZMOR_OFFLINE=falseis a request for online mode — and amalformed value that must be reported and ignored rather than silently
selecting something.
scripts/run-zizmor-test.shships beside it.This is rule 2's bar, met on its own terms: not "a script is tidier", but that
the repository's own guardrails cannot see the alternative.
Governed by ADR 0025
docs/adr/0025-a-skip-reports-the-condition-not-the-cause.mdrules that a skipreports the condition it observed. Degrading silently to the offline subset
would have restored the original gap without saying so, which is precisely what
that record forbids. ADR 0036 records the decision and the rejected
alternatives, including why
--offlinewas chosen over--no-online-audits.Measured, not assumed
zizmor's exit codes were measured on the installed binary (1.29.0, arm64
macOS), not read from documentation:
That distinction is load-bearing. An earlier draft keyed its remedy hint on any
non-zero exit, which would have advised setting
ZIZMOR_OFFLINE=trueinresponse to a genuine security finding. It compares against the tool-failure
status instead.
The CLI contract above is a property of 1.29.0, and CI installs zizmor
unpinned. That residual is recorded in ADR 0036 rather than stated as though it
were version-independent.
Verification
just verifyrun bare, exit 0. The managed pre-push hook re-ran the wholesuite in its isolated worktree on push, green.
Not yet verified: that
permissions: contents: readis sufficient for theonline audits. The argument is structural — the audits read other repositories'
refs and the advisory database, which no permission on this repository grants
— and structural reasoning is not a green run. This pull request's own CI is
the first execution of the online path; read it before trusting the claim. If it
proves insufficient, that is a finding to report, not a licence to widen
permissions:.