feat(runtime): allow best-effort success with a secret-withheld report

CHANGELOG.md

@@ -34,6 +34,7 @@
 - **[runtime] [docs]** Preserve the submodule hydration and verification entry points used by persisted workflows when resuming after the dependency-cache upgrade. Older pinned runs rebuild a missing cache from their recorded execution snapshot, including after `resume --refresh-controller`, and continue to reject dependency edits.
 - **[runtime] [docs]** Task worktrees of an ordinary, unpinned run now get the launch checkout's initialized submodules, such as Foundry's `lib/forge-std` and `lib/openzeppelin-contracts` (#1251). Before, only runs launched from the invariant-pinned branch hydrated submodules, so in a normal `ultrafuzz run` every task saw empty `lib/` directories and the first `forge test` cloned dependencies over the network. `run` now records the checked-out submodule files and their hashes at launch and copies them into a commit-keyed dependency cache in the repository's Git directory (`.git/ultrafuzz/submodule-cache/`), with task-local Git configuration that keeps submodules from fetching. Tasks hydrate from that cache, not from the sealed execution snapshot, which no longer carries dependency files for any run, pinned runs included (#921). A missing or damaged cache entry is rebuilt from the submodules' Git objects at the recorded commits; when neither is available, an ordinary task runs without hydration and its verification marker records why in `dependency_hydration_unavailable`. When an agent changes those files, the task no longer fails: Ultrafuzz restores the sealed files, records the changed paths in the task's verification marker, restates them in `report.json` as `run_metadata.dependency_changes`, and adds a fixed notice to the `report.md` Run summary. Pinned runs still fail the task on any dependency edit. Unlike a pinned source, an ordinary checkout may keep its shared `.git/modules` and local `submodule.*` configuration; local `url.*` rewrites are still refused. A nested submodule that was never initialized is copied as an empty directory. When the checkout cannot be sealed, `run` still launches and reports a `SUBMODULE_HYDRATION_UNAVAILABLE` warning with the reason. The first sealed run sets `extensions.worktreeConfig = true` in the target repository's Git config. Also fixes the sealed-file check, which rejected a valid tree when a file name sorted before a sibling directory with the same prefix, such as OpenZeppelin's `fv-requirements.txt` and `fv/`. That bug could also fail pinned runs.
 - **[runtime] [artifacts] [prompts] [docs]** Invariant suites that cannot fail are now visible (#687). Recon runs in assertion mode, where only a failed `assert` is a finding and any other revert, including a failed `require`, a custom error or an arithmetic panic, is discarded. The invariant prompts claimed every reached revert was "the signal the campaign records"; they now say how assertion mode works, require `assert` for every invariant check, and describe the selector-aware liveness oracle: the typed call in `try`, the catalog's documented selectors accepted in `catch`, and `assert(false)` for any other failure. The campaign's verifier warns, without failing the node, about generated test code Recon cannot fail: constant-true assertions, a `require` that checks an outcome after a target call, a `catch` that asserts nothing (rethrowing does not count), and an assertion that computes its expected value with code imported from production sources. The finding dedupe node carries these warnings to the final report, and its warnings companion lists them first under "Invariant checks that may not be able to fail", with their count. They are heuristics that mark a check for review, not proof that its property went untested.
+- **[runtime] [docs]** Under the default best-effort completion policy, a final report rejected by the artifact secret gate no longer fails an otherwise successful campaign. The report remains unavailable, its task stays failed, and `ultrafuzz resume <run-id> --retry-failed` regenerates it. Other report verification failures and a withheld report under `--require-complete` still fail the campaign.
 - **[runtime] [docs]** A subscription-mode `ClaudeAgent` now runs on a headless host: `CLAUDE_CODE_OAUTH_TOKEN`, the long-lived login from `claude setup-token`, reaches Claude agents the way an API key reaches its own adapter, and every other model child has it blanked (#1266). Before, Ultrafuzz stripped it from every agent, even when allowlisted, and each Claude task failed the engine's preflight with "Claude Code is not logged in". `ultrafuzz validate` now warns with `AGENT_ENV_ALLOWLIST_CREDENTIAL_BLANKED` when `ULTRAFUZZ_AGENT_ENV_ALLOWLIST` names a credential that no selected agent uses, since every agent would receive it blank.
 - **[runtime] [artifacts] [docs]** The artifact secret gate also scans the decoded strings of a JSON publication, so a run credential that holds a quote, a backslash, a tab or a line break, which JSON escapes, no longer passes it in `report.json` or any other verified JSON output (#1217). The strings are joined by NUL, so a recovery phrase split by a line break inside one field is found, while words from adjacent fields never form one. A configured value is also found percent-encoded, with Markdown backslash escapes, or broken across lines. The verifier, synchronization and the rejected-report check now match the run's configured `api_key_env` values and allowlisted variables recognized as credentials even when their names do not look like one, such as `AZURE_OPENAI_KEY`; before, only names like `*_API_KEY` or `*_TOKEN` were matched exactly. The verifier no longer treats the value of `ULTRAFUZZ_PROVIDER_CREDENTIAL_ENV_NAMES`, a list of variable names, as a credential, so an output that mentions `ANTHROPIC_API_KEY` publishes. The Modal public bundle, public eval diagnostics and recovery lifecycle checks use the same scan. A final report the gate rejects stays unavailable even after its credential is unset or rotated: the report task records `reason_code: ARTIFACT_SECRET_GATE_REJECTED`, `status` reports `report-withheld-secret`, and `ultrafuzz resume <run-id> --retry-failed` generates the report again. The run retains its configured completion policy.
 - **[runtime]** Production copies of `proxy-addr`, `vue`, `source-map-js` and `prosemirror-view` move past the Critical advisory GHSA-jqcg-44mw-7w3h (`proxy-addr` trusted every client as a proxy when a trust subnet was an IPv4-mapped IPv6 address with a short prefix) and the High advisories GHSA-g2v6-rqmx-r4w6 (XSS in `@vue/server-renderer`), GHSA-68fv-2mgg-jv7q (event-loop denial of service in `source-map-js`) and GHSA-c8x8-7fp4-3x9w (XSS in `prosemirror-view` paste handling). All four reach Ultrafuzz only through the workflow engine: Express under `@modelcontextprotocol/sdk`, and the Milkdown editor in `@smthrs/ui`. In `pnpm-lock.yaml`, `proxy-addr` moves from 2.0.7 to 2.0.8, `vue` and its `@vue/*` packages from 3.5.40 to 3.5.42, `source-map-js` from 1.2.1 to 1.2.2 and `prosemirror-view` from 1.42.2 to 1.42.3, all within their parents' ranges. Like any install that changes the engine's dependency tree, `pnpm install` of this change moves the engine to a new directory: a run still in progress in that checkout can fail until `ultrafuzz resume` continues it, and long-lived processes such as the dashboard need a restart. The engine that `run` installs from npm does not read that lockfile; it resolves every open range as of a fixed instant, which moves from `2026-08-18T06:00:00Z` to `2026-10-01T00:00:00Z`. Newly launched runs therefore also get the fixed releases (`vue` 3.5.43 and `prosemirror-view` 1.42.6 there), and `fast-uri` 3.1.8 and `undici` 7.30.0, which clear seven High advisories in that install that earlier lockfile fixes did not reach (GHSA-5jgf-p345-68v8, GHSA-f65p-4m7j-42xc, GHSA-fph4-wmhf-6fwf, GHSA-jqff-g426-hqxp and GHSA-qw65-cvwx-89v3 in `fast-uri`, GHSA-rfgv-xxqx-mfg5 and GHSA-w293-vg96-wgc3 in `undici`). The new instant also moves 94 other packages of the roughly 700 in that install to their newest in-range release as of that date, including `react` and `react-dom` from 19.2.8 to 19.3.0 and `ai` from 7.0.66 to 7.0.126.

docs/reference/artifacts-reports.md

@@ -698,8 +698,11 @@ at verification is the exception described below.
 
 A report the artifact secret gate rejects, because it contains a run credential
 or a recognized secret, remains unavailable (#1217). The report task's failure
-records `reason_code: ARTIFACT_SECRET_GATE_REJECTED`; the run retains its configured
-completion policy. `ultrafuzz status` reports the report unavailable with reason
+records `reason_code: ARTIFACT_SECRET_GATE_REJECTED`. Under the default best-effort
+completion policy, withholding this report does not fail an otherwise successful
+run. Other report verification failures still fail the run, and
+`--require-complete` treats a withheld report as unsuccessful too.
+`ultrafuzz status` reports the report unavailable with reason
 `report-withheld-secret`, synchronization adds a `TERMINAL_REPORT_WITHHELD`
 warning, and `ultrafuzz report` fails as it does for any unavailable report. The
 unchecked fallback below is withheld too, even after the credential it matched

docs/reference/cli.md

@@ -578,7 +578,11 @@ counts:
 - `report`: `status` (`available`, `unavailable`, `pending`, or `unknown`),
   `reason`, `completion` (`complete`, `partial`, or `unknown`), `verification`
   (`verified`, `not-checked`, or `unknown`), `json_path`, and `markdown_path`.
-  A failed run may have an available PARTIAL report. If the report agent did not
+  A failed run may have an available PARTIAL report. A successful best-effort run
+  may have an unavailable report with reason `report-withheld-secret` when the
+  artifact secret gate rejects it; `resume <run-id> --retry-failed` regenerates
+  the report. Strict completion treats that rejection as a failed run.
+  If the report agent did not
   produce usable output, the report is unavailable; status does not create a
   replacement. Missing or unreadable publication records leave availability
   unknown. Unavailable and unknown reports expose no stale report paths.

docs/reference/configuration.md

@@ -542,6 +542,12 @@ attempt eligible agent-written reporting. A failed report agent leaves report
 availability unavailable; no replacement report is synthesized. Custom
 topologies keep their declared group failure policies.
 
+Under best-effort completion, a final report rejected by the artifact secret
+gate does not fail an otherwise successful run. The report remains unavailable
+and its task stays failed; `ultrafuzz resume <run-id> --retry-failed` regenerates
+it. Strict completion still fails the run, as do other report verification
+failures under either policy.
+
 `ultrafuzz report --require-verified` is independent: it requires verification of
 an available agent-written report, not complete execution coverage.
 

docs/security.md

@@ -61,7 +61,9 @@ itself:
   included automatically in the exact-value scan.
   A rejected final-report draft stays unavailable even after the credential
   that matched it is unset or rotated; generate a fresh report with
-  `ultrafuzz resume <run-id> --retry-failed`.
+  `ultrafuzz resume <run-id> --retry-failed`. With default best-effort completion,
+  a secret-withheld report does not fail an otherwise successful run;
+  `--require-complete` still fails the run when its report is withheld.
   Exact-value substring matching ignores credentials shorter than eight
   characters to avoid rejecting unrelated content on collision-prone values.
 - Materialization requires explicit selected copies and confirmation. Patch

packages/runtime/src/templates/smithers/workflows/workflow.tsx

@@ -1310,6 +1310,10 @@ function verifiedCanonicalPropertyCatalog(
   return { path: catalog.relativePath, value: parsed.value };
 }
 
+function producesFinalReport(task: (typeof taskSpecs)[number]): boolean {
+  return task.outputs.some((output) => output.contract === "ultrafuzz/report@3");
+}
+
 function declaredFinalReportOutputPair(task: (typeof taskSpecs)[number]):
   | {
       report: (typeof taskSpecs)[number]["outputs"][number];
@@ -9294,7 +9298,10 @@ export default smithers((ctx) => {
                 needs={{ agent: task.id }}
                 deps={{ agent: outputs.agentProcess }}
                 depsOptional
-                continueOnFail={task.continueOnFail}
+                // Let the runtime apply completion policy to a report the
+                // secret gate rejects (#1217). Strict completion and other
+                // report verification failures still fail the campaign.
+                continueOnFail={task.continueOnFail || producesFinalReport(task)}
                 retries={0}
                 metadata={{
                   category: "artifact-contract",

packages/runtime/src/workflow-sync.ts

@@ -793,7 +793,10 @@ export async function synchronizeLinkedWorkflowRun(
   const evidenceComplete = syncResult.syncedNodes >= loaded.tasks.length;
   const nonBlockingNodeIds = requiresCompleteRun(evidence)
     ? new Set<string>()
-    : nonBlockingRuntimeNodeIds(loaded.graph);
+    : new Set([
+        ...nonBlockingRuntimeNodeIds(loaded.graph),
+        ...withheldReportStateIds(loaded.graph, loaded.tasks, readRunState(layout).nodes)
+      ]);
   if (workflowSucceeded(inspect) && syncResult.syncedNodes < loaded.tasks.length) {
     diagnostics.push({
       code: "WORKFLOW_TASK_EVIDENCE_MISSING",
@@ -4945,8 +4948,8 @@ function producesFinalReport(node: PlannedGraphNode): boolean {
 }
 
 /**
- * Report tasks whose output the secret gate rejected. The rejected draft stays
- * unavailable until `resume --retry-failed` generates it again (#1217).
+ * Report tasks whose output the secret gate rejected. Their reports remain
+ * unavailable; `resume --retry-failed` generates them again (#1217).
  */
 function withheldReportNodeIds(nodes: Readonly<Record<string, NodeState>>): ReadonlySet<string> {
   return new Set(
@@ -4960,6 +4963,45 @@ function withheldReportNodeIds(nodes: Readonly<Record<string, NodeState>>): Read
   );
 }
 
+/** Keep report failures visible while exempting aggregates explained entirely by withheld reports. */
+function withheldReportStateIds(
+  graph: PlannedGraph,
+  tasks: readonly StoredWorkflowTask[],
+  nodes: Readonly<Record<string, NodeState>>
+): ReadonlySet<string> {
+  const ids = new Set(withheldReportNodeIds(nodes));
+  const rollUp = (stateId: string, children: readonly string[]): void => {
+    if (nodes[stateId]?.status !== "failed") return;
+    const unsuccessful = children.filter((id) => {
+      const status = nodes[id]?.status;
+      return status !== "succeeded" && status !== "reused-from-prior-run";
+    });
+    if (unsuccessful.length > 0 && unsuccessful.every((id) => ids.has(id))) ids.add(stateId);
+  };
+  const tasksByConcreteNode = new Map<string, StoredWorkflowTask[]>();
+  for (const task of tasks) {
+    const siblings = tasksByConcreteNode.get(task.concreteNodeId) ?? [];
+    siblings.push(task);
+    tasksByConcreteNode.set(task.concreteNodeId, siblings);
+  }
+  const statesByConcreteNode = new Map<string, string[]>();
+  for (const [concreteNodeId, siblings] of tasksByConcreteNode) {
+    const attemptIds = siblings.map((task) => task.attemptId);
+    const aggregateId = siblings[0]?.metadata.node.storageId ?? concreteNodeId;
+    rollUp(aggregateId, attemptIds);
+    statesByConcreteNode.set(concreteNodeId, nodes[aggregateId] === undefined ? attemptIds : [aggregateId]);
+  }
+  for (const node of graph.nodes) {
+    if (node.dynamic?.status !== "expanded") continue;
+    const generatedIds = node.dynamic.generated_node_ids ?? [];
+    // Stale state cannot stand in for a generated task missing from the current manifest.
+    if (generatedIds.some((id) => !statesByConcreteNode.has(id))) continue;
+    const children = generatedIds.flatMap((id) => statesByConcreteNode.get(id) ?? []);
+    rollUp(node.id, children);
+  }
+  return ids;
+}
+
 function nonBlockingRuntimeNodeIds(graph: PlannedGraph): ReadonlySet<string> {
   const ids = new Set<string>();
   for (const node of graph.nodes) {

packages/runtime/test/generated-workflow-verifier.test.ts

@@ -5836,7 +5836,10 @@ test("generated Smithers workflow quarantines optional tasks and reads only veri
   assert.doesNotMatch(baseAgent, /executionSnapshotRoot|path\.dirname|controls/u);
   assert.doesNotMatch(source, /addDir:\s*\[task\.artifactDir, \.\.\.task\.dependencyArtifactDirs\]/u);
   assert.match(workflow, /continueOnFail=\{task\.continueOnFail\}/u);
-  assert.equal(workflow.match(/continueOnFail=\{task\.continueOnFail\}/gu)?.length, 3);
+  assert.equal(workflow.match(/continueOnFail=\{task\.continueOnFail\}/gu)?.length, 2);
+  // The report verifier alone also tolerates failure, so the runtime can
+  // withhold a report the secret gate rejects instead of failing the run (#1217).
+  assert.equal(workflow.match(/continueOnFail=\{task\.continueOnFail \|\| producesFinalReport\(task\)\}/gu)?.length, 1);
 });
 
 test("generated optional admission rejects a present malformed marker before publishing agent access", () => {