diff --git a/CHANGELOG.md b/CHANGELOG.md index fe58f82fc..8ced8182f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ - **[runtime] [docs]** Preserve the submodule hydration and verification entry points used by persisted workflows when resuming after the dependency-cache upgrade. Older pinned runs rebuild a missing cache from their recorded execution snapshot, including after `resume --refresh-controller`, and continue to reject dependency edits. - **[runtime] [docs]** Task worktrees of an ordinary, unpinned run now get the launch checkout's initialized submodules, such as Foundry's `lib/forge-std` and `lib/openzeppelin-contracts` (#1251). Before, only runs launched from the invariant-pinned branch hydrated submodules, so in a normal `ultrafuzz run` every task saw empty `lib/` directories and the first `forge test` cloned dependencies over the network. `run` now records the checked-out submodule files and their hashes at launch and copies them into a commit-keyed dependency cache in the repository's Git directory (`.git/ultrafuzz/submodule-cache/`), with task-local Git configuration that keeps submodules from fetching. Tasks hydrate from that cache, not from the sealed execution snapshot, which no longer carries dependency files for any run, pinned runs included (#921). A missing or damaged cache entry is rebuilt from the submodules' Git objects at the recorded commits; when neither is available, an ordinary task runs without hydration and its verification marker records why in `dependency_hydration_unavailable`. When an agent changes those files, the task no longer fails: Ultrafuzz restores the sealed files, records the changed paths in the task's verification marker, restates them in `report.json` as `run_metadata.dependency_changes`, and adds a fixed notice to the `report.md` Run summary. Pinned runs still fail the task on any dependency edit. Unlike a pinned source, an ordinary checkout may keep its shared `.git/modules` and local `submodule.*` configuration; local `url.*` rewrites are still refused. A nested submodule that was never initialized is copied as an empty directory. When the checkout cannot be sealed, `run` still launches and reports a `SUBMODULE_HYDRATION_UNAVAILABLE` warning with the reason. The first sealed run sets `extensions.worktreeConfig = true` in the target repository's Git config. Also fixes the sealed-file check, which rejected a valid tree when a file name sorted before a sibling directory with the same prefix, such as OpenZeppelin's `fv-requirements.txt` and `fv/`. That bug could also fail pinned runs. - **[runtime] [artifacts] [prompts] [docs]** Invariant suites that cannot fail are now visible (#687). Recon runs in assertion mode, where only a failed `assert` is a finding and any other revert, including a failed `require`, a custom error or an arithmetic panic, is discarded. The invariant prompts claimed every reached revert was "the signal the campaign records"; they now say how assertion mode works, require `assert` for every invariant check, and describe the selector-aware liveness oracle: the typed call in `try`, the catalog's documented selectors accepted in `catch`, and `assert(false)` for any other failure. The campaign's verifier warns, without failing the node, about generated test code Recon cannot fail: constant-true assertions, a `require` that checks an outcome after a target call, a `catch` that asserts nothing (rethrowing does not count), and an assertion that computes its expected value with code imported from production sources. The finding dedupe node carries these warnings to the final report, and its warnings companion lists them first under "Invariant checks that may not be able to fail", with their count. They are heuristics that mark a check for review, not proof that its property went untested. +- **[runtime] [docs]** Under the default best-effort completion policy, a final report rejected by the artifact secret gate no longer fails an otherwise successful campaign. The report remains unavailable, its task stays failed, and `ultrafuzz resume --retry-failed` regenerates it. Other report verification failures and a withheld report under `--require-complete` still fail the campaign. - **[runtime] [docs]** A subscription-mode `ClaudeAgent` now runs on a headless host: `CLAUDE_CODE_OAUTH_TOKEN`, the long-lived login from `claude setup-token`, reaches Claude agents the way an API key reaches its own adapter, and every other model child has it blanked (#1266). Before, Ultrafuzz stripped it from every agent, even when allowlisted, and each Claude task failed the engine's preflight with "Claude Code is not logged in". `ultrafuzz validate` now warns with `AGENT_ENV_ALLOWLIST_CREDENTIAL_BLANKED` when `ULTRAFUZZ_AGENT_ENV_ALLOWLIST` names a credential that no selected agent uses, since every agent would receive it blank. - **[runtime] [artifacts] [docs]** The artifact secret gate also scans the decoded strings of a JSON publication, so a run credential that holds a quote, a backslash, a tab or a line break, which JSON escapes, no longer passes it in `report.json` or any other verified JSON output (#1217). The strings are joined by NUL, so a recovery phrase split by a line break inside one field is found, while words from adjacent fields never form one. A configured value is also found percent-encoded, with Markdown backslash escapes, or broken across lines. The verifier, synchronization and the rejected-report check now match the run's configured `api_key_env` values and allowlisted variables recognized as credentials even when their names do not look like one, such as `AZURE_OPENAI_KEY`; before, only names like `*_API_KEY` or `*_TOKEN` were matched exactly. The verifier no longer treats the value of `ULTRAFUZZ_PROVIDER_CREDENTIAL_ENV_NAMES`, a list of variable names, as a credential, so an output that mentions `ANTHROPIC_API_KEY` publishes. The Modal public bundle, public eval diagnostics and recovery lifecycle checks use the same scan. A final report the gate rejects stays unavailable even after its credential is unset or rotated: the report task records `reason_code: ARTIFACT_SECRET_GATE_REJECTED`, `status` reports `report-withheld-secret`, and `ultrafuzz resume --retry-failed` generates the report again. The run retains its configured completion policy. - **[runtime]** Production copies of `proxy-addr`, `vue`, `source-map-js` and `prosemirror-view` move past the Critical advisory GHSA-jqcg-44mw-7w3h (`proxy-addr` trusted every client as a proxy when a trust subnet was an IPv4-mapped IPv6 address with a short prefix) and the High advisories GHSA-g2v6-rqmx-r4w6 (XSS in `@vue/server-renderer`), GHSA-68fv-2mgg-jv7q (event-loop denial of service in `source-map-js`) and GHSA-c8x8-7fp4-3x9w (XSS in `prosemirror-view` paste handling). All four reach Ultrafuzz only through the workflow engine: Express under `@modelcontextprotocol/sdk`, and the Milkdown editor in `@smthrs/ui`. In `pnpm-lock.yaml`, `proxy-addr` moves from 2.0.7 to 2.0.8, `vue` and its `@vue/*` packages from 3.5.40 to 3.5.42, `source-map-js` from 1.2.1 to 1.2.2 and `prosemirror-view` from 1.42.2 to 1.42.3, all within their parents' ranges. Like any install that changes the engine's dependency tree, `pnpm install` of this change moves the engine to a new directory: a run still in progress in that checkout can fail until `ultrafuzz resume` continues it, and long-lived processes such as the dashboard need a restart. The engine that `run` installs from npm does not read that lockfile; it resolves every open range as of a fixed instant, which moves from `2026-08-18T06:00:00Z` to `2026-10-01T00:00:00Z`. Newly launched runs therefore also get the fixed releases (`vue` 3.5.43 and `prosemirror-view` 1.42.6 there), and `fast-uri` 3.1.8 and `undici` 7.30.0, which clear seven High advisories in that install that earlier lockfile fixes did not reach (GHSA-5jgf-p345-68v8, GHSA-f65p-4m7j-42xc, GHSA-fph4-wmhf-6fwf, GHSA-jqff-g426-hqxp and GHSA-qw65-cvwx-89v3 in `fast-uri`, GHSA-rfgv-xxqx-mfg5 and GHSA-w293-vg96-wgc3 in `undici`). The new instant also moves 94 other packages of the roughly 700 in that install to their newest in-range release as of that date, including `react` and `react-dom` from 19.2.8 to 19.3.0 and `ai` from 7.0.66 to 7.0.126. diff --git a/docs/reference/artifacts-reports.md b/docs/reference/artifacts-reports.md index fc86057f2..da906f063 100644 --- a/docs/reference/artifacts-reports.md +++ b/docs/reference/artifacts-reports.md @@ -698,8 +698,11 @@ at verification is the exception described below. A report the artifact secret gate rejects, because it contains a run credential or a recognized secret, remains unavailable (#1217). The report task's failure -records `reason_code: ARTIFACT_SECRET_GATE_REJECTED`; the run retains its configured -completion policy. `ultrafuzz status` reports the report unavailable with reason +records `reason_code: ARTIFACT_SECRET_GATE_REJECTED`. Under the default best-effort +completion policy, withholding this report does not fail an otherwise successful +run. Other report verification failures still fail the run, and +`--require-complete` treats a withheld report as unsuccessful too. +`ultrafuzz status` reports the report unavailable with reason `report-withheld-secret`, synchronization adds a `TERMINAL_REPORT_WITHHELD` warning, and `ultrafuzz report` fails as it does for any unavailable report. The unchecked fallback below is withheld too, even after the credential it matched diff --git a/docs/reference/cli.md b/docs/reference/cli.md index aa9b15a24..1dbc666b8 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -578,7 +578,11 @@ counts: - `report`: `status` (`available`, `unavailable`, `pending`, or `unknown`), `reason`, `completion` (`complete`, `partial`, or `unknown`), `verification` (`verified`, `not-checked`, or `unknown`), `json_path`, and `markdown_path`. - A failed run may have an available PARTIAL report. If the report agent did not + A failed run may have an available PARTIAL report. A successful best-effort run + may have an unavailable report with reason `report-withheld-secret` when the + artifact secret gate rejects it; `resume --retry-failed` regenerates + the report. Strict completion treats that rejection as a failed run. + If the report agent did not produce usable output, the report is unavailable; status does not create a replacement. Missing or unreadable publication records leave availability unknown. Unavailable and unknown reports expose no stale report paths. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index df38c7377..9808d0604 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -542,6 +542,12 @@ attempt eligible agent-written reporting. A failed report agent leaves report availability unavailable; no replacement report is synthesized. Custom topologies keep their declared group failure policies. +Under best-effort completion, a final report rejected by the artifact secret +gate does not fail an otherwise successful run. The report remains unavailable +and its task stays failed; `ultrafuzz resume --retry-failed` regenerates +it. Strict completion still fails the run, as do other report verification +failures under either policy. + `ultrafuzz report --require-verified` is independent: it requires verification of an available agent-written report, not complete execution coverage. diff --git a/docs/security.md b/docs/security.md index 4ad0fff6f..cee7e7d49 100644 --- a/docs/security.md +++ b/docs/security.md @@ -61,7 +61,9 @@ itself: included automatically in the exact-value scan. A rejected final-report draft stays unavailable even after the credential that matched it is unset or rotated; generate a fresh report with - `ultrafuzz resume --retry-failed`. + `ultrafuzz resume --retry-failed`. With default best-effort completion, + a secret-withheld report does not fail an otherwise successful run; + `--require-complete` still fails the run when its report is withheld. Exact-value substring matching ignores credentials shorter than eight characters to avoid rejecting unrelated content on collision-prone values. - Materialization requires explicit selected copies and confirmation. Patch diff --git a/packages/runtime/src/templates/smithers/workflows/workflow.tsx b/packages/runtime/src/templates/smithers/workflows/workflow.tsx index de6901edc..d51f870f8 100644 --- a/packages/runtime/src/templates/smithers/workflows/workflow.tsx +++ b/packages/runtime/src/templates/smithers/workflows/workflow.tsx @@ -1310,6 +1310,10 @@ function verifiedCanonicalPropertyCatalog( return { path: catalog.relativePath, value: parsed.value }; } +function producesFinalReport(task: (typeof taskSpecs)[number]): boolean { + return task.outputs.some((output) => output.contract === "ultrafuzz/report@3"); +} + function declaredFinalReportOutputPair(task: (typeof taskSpecs)[number]): | { report: (typeof taskSpecs)[number]["outputs"][number]; @@ -9294,7 +9298,10 @@ export default smithers((ctx) => { needs={{ agent: task.id }} deps={{ agent: outputs.agentProcess }} depsOptional - continueOnFail={task.continueOnFail} + // Let the runtime apply completion policy to a report the + // secret gate rejects (#1217). Strict completion and other + // report verification failures still fail the campaign. + continueOnFail={task.continueOnFail || producesFinalReport(task)} retries={0} metadata={{ category: "artifact-contract", diff --git a/packages/runtime/src/workflow-sync.ts b/packages/runtime/src/workflow-sync.ts index fd8d7c23e..6195285f4 100644 --- a/packages/runtime/src/workflow-sync.ts +++ b/packages/runtime/src/workflow-sync.ts @@ -793,7 +793,10 @@ export async function synchronizeLinkedWorkflowRun( const evidenceComplete = syncResult.syncedNodes >= loaded.tasks.length; const nonBlockingNodeIds = requiresCompleteRun(evidence) ? new Set() - : nonBlockingRuntimeNodeIds(loaded.graph); + : new Set([ + ...nonBlockingRuntimeNodeIds(loaded.graph), + ...withheldReportStateIds(loaded.graph, loaded.tasks, readRunState(layout).nodes) + ]); if (workflowSucceeded(inspect) && syncResult.syncedNodes < loaded.tasks.length) { diagnostics.push({ code: "WORKFLOW_TASK_EVIDENCE_MISSING", @@ -4945,8 +4948,8 @@ function producesFinalReport(node: PlannedGraphNode): boolean { } /** - * Report tasks whose output the secret gate rejected. The rejected draft stays - * unavailable until `resume --retry-failed` generates it again (#1217). + * Report tasks whose output the secret gate rejected. Their reports remain + * unavailable; `resume --retry-failed` generates them again (#1217). */ function withheldReportNodeIds(nodes: Readonly>): ReadonlySet { return new Set( @@ -4960,6 +4963,45 @@ function withheldReportNodeIds(nodes: Readonly>): Read ); } +/** Keep report failures visible while exempting aggregates explained entirely by withheld reports. */ +function withheldReportStateIds( + graph: PlannedGraph, + tasks: readonly StoredWorkflowTask[], + nodes: Readonly> +): ReadonlySet { + const ids = new Set(withheldReportNodeIds(nodes)); + const rollUp = (stateId: string, children: readonly string[]): void => { + if (nodes[stateId]?.status !== "failed") return; + const unsuccessful = children.filter((id) => { + const status = nodes[id]?.status; + return status !== "succeeded" && status !== "reused-from-prior-run"; + }); + if (unsuccessful.length > 0 && unsuccessful.every((id) => ids.has(id))) ids.add(stateId); + }; + const tasksByConcreteNode = new Map(); + for (const task of tasks) { + const siblings = tasksByConcreteNode.get(task.concreteNodeId) ?? []; + siblings.push(task); + tasksByConcreteNode.set(task.concreteNodeId, siblings); + } + const statesByConcreteNode = new Map(); + for (const [concreteNodeId, siblings] of tasksByConcreteNode) { + const attemptIds = siblings.map((task) => task.attemptId); + const aggregateId = siblings[0]?.metadata.node.storageId ?? concreteNodeId; + rollUp(aggregateId, attemptIds); + statesByConcreteNode.set(concreteNodeId, nodes[aggregateId] === undefined ? attemptIds : [aggregateId]); + } + for (const node of graph.nodes) { + if (node.dynamic?.status !== "expanded") continue; + const generatedIds = node.dynamic.generated_node_ids ?? []; + // Stale state cannot stand in for a generated task missing from the current manifest. + if (generatedIds.some((id) => !statesByConcreteNode.has(id))) continue; + const children = generatedIds.flatMap((id) => statesByConcreteNode.get(id) ?? []); + rollUp(node.id, children); + } + return ids; +} + function nonBlockingRuntimeNodeIds(graph: PlannedGraph): ReadonlySet { const ids = new Set(); for (const node of graph.nodes) { diff --git a/packages/runtime/test/generated-workflow-verifier.test.ts b/packages/runtime/test/generated-workflow-verifier.test.ts index 1aad7b4cd..c4ca0d705 100644 --- a/packages/runtime/test/generated-workflow-verifier.test.ts +++ b/packages/runtime/test/generated-workflow-verifier.test.ts @@ -5836,7 +5836,10 @@ test("generated Smithers workflow quarantines optional tasks and reads only veri assert.doesNotMatch(baseAgent, /executionSnapshotRoot|path\.dirname|controls/u); assert.doesNotMatch(source, /addDir:\s*\[task\.artifactDir, \.\.\.task\.dependencyArtifactDirs\]/u); assert.match(workflow, /continueOnFail=\{task\.continueOnFail\}/u); - assert.equal(workflow.match(/continueOnFail=\{task\.continueOnFail\}/gu)?.length, 3); + assert.equal(workflow.match(/continueOnFail=\{task\.continueOnFail\}/gu)?.length, 2); + // The report verifier alone also tolerates failure, so the runtime can + // withhold a report the secret gate rejects instead of failing the run (#1217). + assert.equal(workflow.match(/continueOnFail=\{task\.continueOnFail \|\| producesFinalReport\(task\)\}/gu)?.length, 1); }); test("generated optional admission rejects a present malformed marker before publishing agent access", () => { diff --git a/packages/runtime/test/runtime.test.ts b/packages/runtime/test/runtime.test.ts index 545c233f5..3ec1331b2 100644 --- a/packages/runtime/test/runtime.test.ts +++ b/packages/runtime/test/runtime.test.ts @@ -2653,7 +2653,7 @@ function writeRequiredArtifactSet(runRoot: string, nodeId: string, required: str writeCurrentArtifactVerificationMarker(runRoot, nodeId); } -function writeEmptyFinalReportArtifactSet(runRoot: string, runId: string) { +function writeEmptyFinalReportArtifactSet(runRoot: string, runId: string, attemptId = "final-report") { const finalReport = projectCanonicalFinalReport({ schema_version: "ultrafuzz.report.v3", run_metadata: { @@ -2681,11 +2681,11 @@ function writeEmptyFinalReportArtifactSet(runRoot: string, runId: string) { reason: "property-implementation-track-not-declared" } }); - const reportDir = path.join(runRoot, "artifacts", "final-report"); + const reportDir = path.join(runRoot, "artifacts", attemptId); fs.mkdirSync(reportDir, { recursive: true }); fs.writeFileSync(path.join(reportDir, "report.json"), `${JSON.stringify(finalReport.report, null, 2)}\n`); fs.writeFileSync(path.join(reportDir, "report.md"), finalReport.markdown); - writeCurrentArtifactVerificationMarker(runRoot, "final-report"); + writeCurrentArtifactVerificationMarker(runRoot, attemptId); return finalReport; } @@ -11215,7 +11215,12 @@ test("compileSmithersWorkflow marks specialist attempts and their artifact hando { attemptId: "direct-strategy", verifierId: "verify:direct-strategy", optional: false }, { attemptId: "optional-specialist", verifierId: "verify:optional-specialist", optional: true } ]); - assert.equal(workflowSource.match(/continueOnFail=\{task\.continueOnFail\}/gu)?.length, 3); + assert.equal(workflowSource.match(/continueOnFail=\{task\.continueOnFail\}/gu)?.length, 2); + // The report verifier also tolerates failure, so a secret-gate rejection withholds the report (#1217). + assert.equal( + workflowSource.match(/continueOnFail=\{task\.continueOnFail \|\| producesFinalReport\(task\)\}/gu)?.length, + 1 + ); }); test("current-controller rendering reapplies continue policy and optional dependencies", async () => { @@ -19902,15 +19907,15 @@ test("syncRun publishes a report after a resumed report agent succeeds", async ( }); for (const variant of ["secret-gate", "other-verifier-failure", "require-complete"] as const) { - test(`syncRun records secret-gate rejection without changing completion policy (#1217, ${variant})`, async () => { + test(`syncRun withholds a report the secret gate rejects instead of failing the run (#1217, ${variant})`, async () => { const { readReportPublicationStatus } = await import("../src/report-publication-status.js"); const project = tempProject(); initProject({ projectRoot: project, force: true }); writeSingleFinalReportTopology(project); const runId = `sync-withheld-report-${variant}`; const workflowRunId = `ultrafuzz-${runId}`; - // Even a runner that tolerated the wrapper failure must not override - // this required report node's failed completion status. + // The verifier task tolerates failure, so the runner ends the run + // succeeded-with-failures whatever rejected the report. const verifierError = variant === "other-verifier-failure" ? { name: "Error", message: "artifact-contract failure: report parity failed" } @@ -19975,12 +19980,12 @@ for (const variant of ["secret-gate", "other-verifier-failure", "require-complet ), JSON.stringify(sync.diagnostics) ); - assert.equal(sync.value?.status, "failed", JSON.stringify(sync.diagnostics)); - // Both default and strict completion retain the report node's failure. + // Strict completion still treats the missing report as an unsuccessful run. if (variant === "require-complete") { assert.equal(sync.value?.status, "failed"); return; } + assert.equal(sync.value?.status, "succeeded", JSON.stringify(sync.diagnostics)); // `resume --retry-failed` reruns the report task; its accepted output is published. const recoveredEnv = fakeLifecycleSmithersEnv(project, { @@ -20012,6 +20017,85 @@ for (const variant of ["secret-gate", "other-verifier-failure", "require-complet }); } +for (const variant of ["all-withheld", "successful-sibling", "ordinary-failure", "require-complete"] as const) { + test(`syncRun preserves withheld-report completion policy across model fanout (${variant})`, async () => { + const project = tempProject(); + initProject({ projectRoot: project, force: true }); + writeSingleFinalReportTopology(project); + const topologyPath = path.join(project, ".ultrafuzz", "topology.yml"); + fs.writeFileSync( + topologyPath, + fs + .readFileSync(topologyPath, "utf8") + .replace( + " prompt: setup/project-discovery.md\n", + " prompt: setup/project-discovery.md\n model_profiles: [default, claude]\n" + ) + ); + const runId = `withheld-fanout-${variant}`; + const workflowRunId = `ultrafuzz-${runId}`; + const env = fakeLifecycleSmithersEnv(project, { + inspect: workflowInspect({ workflowRunId, status: "running", state: "running", steps: [] }) + }); + const run = await startRun({ + projectRoot: project, + runId, + env, + ...(variant === "require-complete" + ? { runtimeOverrides: { run: { completionPolicy: "require-complete" as const } } } + : {}) + }); + assert.ok(run.ok && run.value, JSON.stringify(run.diagnostics)); + const runRoot = run.value.run_root; + const { tasks } = parseSmithersTaskManifestBytes(fs.readFileSync(path.join(runRoot, "smithers", "tasks.json"))); + assert.equal(tasks.length, 2); + const [firstTask] = tasks; + assert.ok(firstTask); + const steps: Parameters[0]["steps"] = []; + const events: Parameters[1] = []; + for (const [index, task] of tasks.entries()) { + const succeeded = index === 1 && variant === "successful-sibling"; + const ordinaryFailure = index === 1 && variant === "ordinary-failure"; + steps.push( + { id: task.smithersNodeId, state: "finished", attempt: 1 }, + { id: task.verifierSmithersNodeId, state: succeeded ? "finished" : "failed", attempt: 1 } + ); + events.push( + { type: "NodeStarted", nodeId: task.smithersNodeId, attempt: 1 }, + { type: "NodeFinished", nodeId: task.smithersNodeId, attempt: 1 }, + succeeded + ? { type: "NodeFinished", nodeId: task.verifierSmithersNodeId, attempt: 1 } + : { + type: "NodeFailed", + nodeId: task.verifierSmithersNodeId, + attempt: 1, + error: ordinaryFailure + ? { message: "artifact-contract failure: report parity failed" } + : { message: "artifact publication contains sensitive data", code: "ARTIFACT_SECRET_GATE_REJECTED" } + } + ); + if (succeeded) writeEmptyFinalReportArtifactSet(runRoot, runId, task.attemptId); + } + events.push({ type: "RunFinished" }); + const terminalEnv = fakeLifecycleSmithersEnv(project, { + inspect: workflowInspect({ workflowRunId, status: "finished", state: "succeeded-with-failures", steps }), + events: workflowEvents(workflowRunId, events) + }); + const expectedStatus = variant === "all-withheld" || variant === "successful-sibling" ? "succeeded" : "failed"; + for (let pass = 0; pass < 2; pass += 1) { + const synced = await syncRun({ projectRoot: project, runId, env: terminalEnv }); + assert.equal(synced.ok, true, JSON.stringify(synced.diagnostics)); + assert.equal(synced.value?.status, expectedStatus, JSON.stringify(synced.diagnostics)); + // A tolerated failure stays visible both on its attempt and its aggregate. + const state = readRunState(layoutForRunRoot(runRoot)); + assert.equal(state.nodes[firstTask.attemptId]?.status, "failed"); + assert.equal(state.nodes["final-report"]?.status, "failed"); + } + if (variant === "successful-sibling") assert.ok(loadReportSnapshot(runRoot)); + else assert.throws(() => loadReportSnapshot(runRoot), /Report unavailable/u); + }); +} + test("syncRun follows the runner past a retry-failed recovery a pre-#961 build left prepared", async () => { const project = tempProject(); initProject({ projectRoot: project, force: true }); diff --git a/packages/runtime/test/withheld-report-state.test.ts b/packages/runtime/test/withheld-report-state.test.ts new file mode 100644 index 000000000..a6f1c3f9d --- /dev/null +++ b/packages/runtime/test/withheld-report-state.test.ts @@ -0,0 +1,173 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; +import ts from "typescript"; + +import { ARTIFACT_SECRET_GATE_ERROR_CODE, isRecord, type NodeState, type NodeStatus } from "@ultrafuzz/artifacts"; + +type Task = { + concreteNodeId: string; + attemptId: string; + metadata: { node: { storageId?: string } }; +}; +type Graph = { nodes: Array<{ id: string; dynamic?: { status: string; generated_node_ids: string[] } }> }; +type Nodes = Record>; + +/** Exercise the private production selection without loading the workflow controller's I/O. */ +function loadWithheldReportStateIds(): (graph: Graph, tasks: readonly Task[], nodes: Nodes) => ReadonlySet { + const directory = path.dirname(fileURLToPath(import.meta.url)); + const runtimeRoot = fs.existsSync(path.resolve(directory, "../src/workflow-sync.ts")) + ? path.resolve(directory, "..") + : path.resolve(directory, "../.."); + const filename = path.join(runtimeRoot, "src/workflow-sync.ts"); + const source = ts.createSourceFile(filename, fs.readFileSync(filename, "utf8"), ts.ScriptTarget.Latest, true); + const names = new Set(["withheldReportNodeIds", "withheldReportStateIds", "recordField"]); + const declarations = source.statements.flatMap((statement) => + ts.isFunctionDeclaration(statement) && statement.name !== undefined && names.has(statement.name.text) + ? [statement.getText(source)] + : [] + ); + assert.equal(declarations.length, names.size); + const emitted = ts.transpileModule(declarations.join("\n"), { + compilerOptions: { module: ts.ModuleKind.None, target: ts.ScriptTarget.ES2022 } + }).outputText; + return new Function("ARTIFACT_SECRET_GATE_ERROR_CODE", "isRecord", `${emitted}; return withheldReportStateIds;`)( + ARTIFACT_SECRET_GATE_ERROR_CODE, + isRecord + ) as ReturnType; +} + +const withheldReportStateIds = loadWithheldReportStateIds(); + +function withheldNode(attemptId: string): Nodes[string] { + return { + status: "failed", + provenance: { + failure: { + category: "artifact-contract", + causal_task_id: attemptId, + causal_failure_category: "artifact-contract", + dependent_task_ids: [], + reason_code: ARTIFACT_SECRET_GATE_ERROR_CODE + } + } + }; +} + +function fixture(): { graph: Graph; tasks: Task[]; nodes: Nodes } { + return { + graph: { + nodes: [ + { id: "reports", dynamic: { status: "expanded", generated_node_ids: ["report-a", "report-b"] } }, + { id: "report-a" }, + { id: "report-b" } + ] + }, + tasks: [ + { concreteNodeId: "report-a", attemptId: "a-claude", metadata: { node: { storageId: "storage-a" } } }, + { concreteNodeId: "report-a", attemptId: "a-codex", metadata: { node: { storageId: "storage-a" } } }, + { concreteNodeId: "report-b", attemptId: "b-codex", metadata: { node: { storageId: "storage-b" } } } + ], + nodes: { + "a-claude": withheldNode("a-claude"), + "a-codex": { status: "succeeded" }, + "b-codex": { status: "reused-from-prior-run" }, + "storage-a": { status: "failed" }, + "storage-b": { status: "reused-from-prior-run" }, + reports: { status: "failed" } + } + }; +} + +function selected(input: ReturnType): string[] { + return [...withheldReportStateIds(input.graph, input.tasks, input.nodes)].sort(); +} + +test("withheld report attempts roll up through model fanout and an expanded dynamic group", () => { + const input = fixture(); + assert.deepEqual(selected(input), ["a-claude", "reports", "storage-a"]); + // The group is visited before the concrete graph nodes, and task order is not significant. + input.tasks.reverse(); + assert.deepEqual(selected(input), ["a-claude", "reports", "storage-a"]); + // A generated node without a separate aggregate uses its attempts directly. + delete input.nodes["storage-b"]; + assert.deepEqual(selected(input), ["a-claude", "reports", "storage-a"]); +}); + +test("unfinished or independently failed siblings keep both report aggregates blocking", () => { + const blocking: NodeStatus[] = [ + "pending", + "ready", + "runnable", + "running", + "skipped", + "failed", + "timed-out", + "invalidated" + ]; + for (const status of blocking) { + const input = fixture(); + input.nodes["a-codex"] = { status }; + assert.deepEqual(selected(input), ["a-claude"], status); + } + const missing = fixture(); + delete missing.nodes["a-codex"]; + assert.deepEqual(selected(missing), ["a-claude"]); +}); + +test("a dynamic group cannot omit missing generated tasks or attempt states", () => { + const missingTask = fixture(); + missingTask.tasks = missingTask.tasks.filter((task) => task.concreteNodeId !== "report-b"); + assert.deepEqual(selected(missingTask), ["a-claude", "storage-a"]); + missingTask.nodes["report-b"] = withheldNode("report-b"); + assert.deepEqual(selected(missingTask), ["a-claude", "report-b", "storage-a"]); + missingTask.nodes["report-b"] = { status: "succeeded" }; + assert.deepEqual(selected(missingTask), ["a-claude", "storage-a"]); + + const missingState = fixture(); + delete missingState.nodes["storage-b"]; + delete missingState.nodes["b-codex"]; + assert.deepEqual(selected(missingState), ["a-claude", "storage-a"]); +}); + +test("only failed aggregates explained by at least one withheld child are exempt", () => { + const notFailed: NodeStatus[] = [ + "pending", + "ready", + "runnable", + "running", + "skipped", + "timed-out", + "invalidated", + "succeeded", + "reused-from-prior-run" + ]; + for (const status of notFailed) { + const input = fixture(); + input.nodes["storage-a"] = { status }; + assert.deepEqual(selected(input), ["a-claude"], `concrete aggregate ${status}`); + + const group = fixture(); + group.nodes.reports = { status }; + assert.deepEqual(selected(group), ["a-claude", "storage-a"], `dynamic aggregate ${status}`); + } + const allSucceeded = fixture(); + allSucceeded.nodes["a-claude"] = { status: "succeeded" }; + allSucceeded.nodes["unrelated-withheld"] = withheldNode("unrelated-withheld"); + assert.deepEqual(selected(allSucceeded), ["unrelated-withheld"]); +}); + +test("unexpanded and empty dynamic groups remain blocking", () => { + const unexpanded = fixture(); + const pendingGroup = unexpanded.graph.nodes[0]?.dynamic; + assert.ok(pendingGroup !== undefined); + pendingGroup.status = "pending"; + assert.deepEqual(selected(unexpanded), ["a-claude", "storage-a"]); + const empty = fixture(); + const emptyGroup = empty.graph.nodes[0]?.dynamic; + assert.ok(emptyGroup !== undefined); + emptyGroup.generated_node_ids = []; + assert.deepEqual(selected(empty), ["a-claude", "storage-a"]); +});