fix(artifacts): catch escaped and unnamed run credentials, and withhold a rejected report (#1217)

[mrthankyou]

Closes #1217.

Summary

The artifact secret gate matched configured credentials byte for byte against the published bytes. JSON escapes a quote, a backslash, a tab and a line break, so a credential holding one never matched report.json, and a verified report could publish it. Auditing the gate turned up more gaps than the issue names. This PR closes the ones inside the gate, and changes what happens when the gate rejects the final report.

1. The gate reads decoded JSON (the issue)

• A JSON publication is also scanned as its decoded strings, keys included. The strings are joined by NUL, which no pattern treats as a word gap, so a recovery phrase split by a line break inside one field is found, while words from adjacent fields never form one phrase (the issue's second acceptance criterion).
• report.md needs no separate pass: it is a projection of report.json and is always checked beside it.
• The fix: a reworded finding no longer discards the final report #1204 rejected-report path drops its own copy of this logic and relies on the gate.

Every credential format in the existing gate test, embedded in a report.json field, was re-run through the old and new gate. Sixteen of eighteen were already caught; two leaked on unstable:

Format inside a JSON field	unstable	this PR
Recovery phrase split by a line break	leak	caught
Quoted Bearer header ("Authorization: Bearer …")	leak	caught
Configured value with \ " tab or line break	leak	caught

2. The gate knows every credential an agent can see

All gate calls used sensitiveEnvironmentValues(process.env) with no explicit names, so only variables named like *_API_KEY or *_TOKEN were matched exactly. A configured api_key_env such as AZURE_OPENAI_KEY (which the Codex adapter hands the agent as CODEX_API_KEY), or an allowlisted variable whose value is a credential, could be published verbatim.

• New publicationForbiddenSecretValues (artifacts) adds the controller's ULTRAFUZZ_PROVIDER_CREDENTIAL_ENV_NAMES and ULTRAFUZZ_SENSITIVE_AGENT_ENV_NAMES. The verifier uses it.
• New runPublicationForbiddenSecretValues (runtime) derives the same names from the run's smithers/resolved-config.json and the allowlist, for checks outside the controller: synchronization and the rejected-report reader. sensitiveAllowlistedEnvironmentVariableNames moves there from start-run.ts unchanged.
• Pre-existing false positive fixed: ULTRAFUZZ_PROVIDER_CREDENTIAL_ENV_NAMES matches the credential-name pattern, so its value — a list of names like ANTHROPIC_API_KEY — was itself treated as a secret, and any output mentioning that variable name failed the gate. The two name lists are now excluded from the values.
• The controller's re-check on read (verified-output.ts) stays patterns-only on purpose: it runs on every later read, and matching the reader's current environment could make an already-verified run unreadable after the environment changes.

3. Encodings agents produce by accident

A configured value is also found percent-encoded (an RPC URL), with Markdown backslash escapes (\_ in agent-written .md), or broken across lines. Deliberate encodings (base64, hex, case changes) are out of reach of any scan and are documented as such.

4. Modal public checks

The Modal public bundle, public eval diagnostics and recovery lifecycle checks had the same escaped-JSON hole. They now reuse the gate's check through the new publicationContainsSecrets. Their inputs are already redacted upstream, so this is defense in depth.

5. A rejected final report is withheld, not a failed campaign

Failing the gate used to fail the report node and therefore the whole run, at its last step. Now:

• The report's verify task (only) is generated with continueOnFail, so the engine ends succeeded-with-failures.
• When the verifier's NodeFailed error carries code: ARTIFACT_SECRET_GATE_REJECTED (set only by the controller's gate, never by agent output), the report node's failure records a new optional reason_code and the node is non-blocking: the run ends succeeded when everything else did.
• status reports the report unavailable with the new reason report-withheld-secret, synchronization adds a TERMINAL_REPORT_WITHHELD warning naming the recovery command, and ultrafuzz report exits non-zero as for any unavailable report. The unchecked fallback is withheld too.
• ultrafuzz resume <run-id> --retry-failed regenerates the report. For a failed verifier it resets the producing agent (retryProducerForFailedVerifier), so the retry writes a fresh report rather than re-checking the withheld one, and it reads failed tasks from failedChildKeys for a success-terminal run.
• Any other report verification failure still fails the run, and so does a withheld report under --require-complete.

Schema changes: optional reason_code on node failure provenance (run-state.schema.json), and report-withheld-secret in report-publication-status.schema.json. A new failure category was avoided because it would change the public eval diagnostics schema.

Not in this PR

• Full-run report bundles ship raw agent transcripts and rejected artifacts without a secret scan #1289: full-run report bundles ship raw agent transcripts (engine-logs/) and gate-rejected files from artifacts/ without any scan. This is the largest remaining leak path, but it is a separate subsystem with a product decision attached.
• Formats no pattern recognizes (a bare Etherscan key, a QuickNode URL, a Codex rt_ refresh token) are caught only when configured; section 2 makes the configured ones count. Unlabelled 64-hex stays allowed because it matches every transaction hash.

Testing

All suites run in a Linux container (macOS-only Modal failures exist on unstable too; filed as #1290).

Suite	Result
@ultrafuzz/artifacts	336 / 336
@ultrafuzz/security	pass
@ultrafuzz/cli	197 / 197
@ultrafuzz/modal	all pass
runtime: publication-secret-values, unverified-report, final-report-markdown	86 / 86
runtime: runtime.test.ts sync, report, redaction and #1217 tests	128 / 128
runtime: smithers-report-retry.integration	2 / 2
runtime: generated-workflow-verifier	123 / 124 — the one failure (pinned source proof) fails identically on unstable

New tests:

• the gate rejects each escaped credential, every known format inside JSON, and the percent / Markdown / line-break encodings, and does not flag ordinary escaped prose or a mentioned variable name;
• the name lists reach the exact-value list, including from a run's resolved config;
• sync: a secret-gate rejection withholds the report and the run succeeds; another verifier failure still fails the run; --require-complete still fails; a simulated --retry-failed publishes the verified report.

The new tests fail on unstable's gate. Strict lint, Prettier and schema snapshots are clean.

Not verified live: resume --retry-failed on a withheld report is checked in code and in simulated sync tests, not in a real campaign. To check it live, launch any run with a credential-named variable whose value is the run ID (for example E2E_RUN_TOKEN=<run-id>, which every report contains), confirm the run ends succeeded with report-withheld-secret, then unset it and run ultrafuzz resume <run-id> --retry-failed.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; both previous findings are fixed and no new blocking issues were found.

Summary

The PR strengthens secret checks and withholds rejected final reports without failing otherwise successful runs.

• Publication checks catch credentials hidden in escaped or encoded output.
• A report rejected for secrets is withheld without failing an otherwise successful run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Report draft] --> B{Secret check}
  B -->|Pass| C[Publish verified report]
  B -->|Reject| D[Save rejection reason]
  D --> E[Withhold report and unchecked fallback]
  E --> F[Retry report agent]
  F --> A

Loading

_{Reviews (2) · Last reviewed commit: "fix(artifacts): keep a gate-withheld rep..." · Reviewed by Greptile}