CL-8978: prevent recoverable subagent failures from stalling parent (#1171)

* test(subagent): prove recoverable failure yields continuable wait marker

* feat(subagent): mark recoverable failures continuable in wait output

* fix(agent): except continuable same-brief from identical-brief refuse

Wait output for a recoverable child failure tells the parent to spawn
one successor with the same brief. The standing orchestration rule still
refused any identical brief, so Skywalker would refuse the spawn the
marker asks for.

Author: TheGreatAxios

docs/ARCHITECTURE.md

@@ -139,7 +139,7 @@ Two directors, selected by role:
   Auto mode is toggled by CLI flags (`--auto` / `--no-auto`); there is currently no in-session key to toggle it (default on; constrained envelope — workspace writes and unconstrained shell auto-allow; installs, recursive rm, force/uncontained worktree changes, sensitive-path and opaque-wrapper shell still ask; contained non-force `git worktree add`/`remove`/`prune` and `list` auto-allow; shell file-mutation denied). It is not a separate edit/plan mode.
 
 - **SubAgentDirector** (delegated work, `src/subagent/index.ts`) — Drives a dispatched worker until a turn arrives with no tool calls, then replies with the final assistant text and ends the run. A tool-less turn **after tools** completes only with the four-heading envelope (Summary, Findings, Blockers, Paths). Assistant text that prints explicit `<tool_call>` markup is treated as attempted tool use, not narration: one **verbatim-tool-call** nudge asks the worker to re-issue a real `tool_call` and does not count toward the tool-less spiral. A missing envelope otherwise nudges once (**incomplete-report**) and a second tool-less turn still without the envelope salvages as **incomplete-report-stop** (once; later tool-less turns wait). Explore/read-only workers that used tools then replied with findings remain normal completes; `requireEvidence` (off by default, set per director) additionally requires at least one read before a tool-less spawn-only reply can complete. `requirePlanSubstance` (counsel or `intent=plan`, not `modelRole === "plan"`) additionally requires Findings to contain files/paths, acceptance criteria, non-goals, risks, and ordered steps with a non-placeholder line each — four headings with stub Findings are incomplete-report, not an attachable plan. After real tool work, wrap-up Findings that are not placeholder or outline-only complete instead of being salvaged as a stub plan. Reads done through `run_shell` count as evidence too — `src/subagent/shell-evidence.ts` classifies shell reads (`cat`, `grep`, `sed` without `-i`, …) over the same subject expansion the auto-shell policy uses — but there is no corresponding shell-write evidence or file-write requirement: a run that never touches a file still completes normally once it replies with the envelope. There is no turn budget. Operator/parent cancel after any progress returns a **cancelled** salvage report (partial findings + tool activity) instead of a bare cancel string; cancel before progress still surfaces as cancelled-by-operator. There is no repetition/no-progress/never-acted/never-edited hard stop and no fingerprint-based re-dispatch block — a genuinely stuck worker runs until it completes, stalls, hits an opt-in wall-clock deadline, or is cancelled.
-  `spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
+  `spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
 
 #### Model-family policy (`src/agent/model-family-policy.ts`)
 

src/agent/prompts.test.ts

@@ -223,7 +223,7 @@ Scope and conventions:
 Orchestration:
 - One focused task per spawned worker. Fan-out width follows independent lanes (one lane per PR/path/ownership). Break multi-step or parallel work into those dispatches with distinct lenses; prefer \`spawn_agent\` (fire several in one turn when jobs are independent), then reply with who is running and end the turn — workers keep running while you are idle. Mailbox mail arrives as inbound when a worker finishes; read it and do not poll. \`list_agents\` shows the fleet without blocking; after a parked ask is surfaced, answer with \`send_input\` and do not poll \`list_agents\`.
 - Pass the typed spawn contract and keep it tight: \`intent\`, \`success_criteria\` (done-when; required for implement/review and their default directors), \`do_not\` (scope fence), and \`report_focus\`. Free-form \`prompt\` without \`success_criteria\` fail-closes for implement/review and their default directors.
-- After workers return, classify fail / incomplete-report vs parent-initiated interrupt vs operator-cancel vs clean complete. Fail-path (\`status: failed\` or salvage \`incomplete-report\`): diagnose from the report or error and MAY spawn one successor with a changed brief. Parent-initiated interrupt (\`interrupt_agent\` / \`send_input\` with \`interrupt:true\` unblocks wait with \`stop_reason: interrupted\`): the worker is often still running and often has no report — \`resume_agent\`, or idle for its mailbox mail; do not \`spawn_agent\` a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancel (\`stop_reason\` cancelled): wait for the operator; do not auto-retry. Identical brief: refuse. Merge Summary/Findings into a coherent answer for the operator; do not paste raw fleet-agent dumps.
+- After workers return, classify fail / incomplete-report vs parent-initiated interrupt vs operator-cancel vs clean complete. Fail-path (\`status: failed\` or salvage \`incomplete-report\`): diagnose from the report or error and MAY spawn one successor with a changed brief. Parent-initiated interrupt (\`interrupt_agent\` / \`send_input\` with \`interrupt:true\` unblocks wait with \`stop_reason: interrupted\`): the worker is often still running and often has no report — \`resume_agent\`, or idle for its mailbox mail; do not \`spawn_agent\` a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancel (\`stop_reason\` cancelled): wait for the operator; do not auto-retry. Identical brief: refuse. Recoverable/continuable child failure (\`continuable: true\`) MAY spawn one successor with the same brief; identical brief is still refused otherwise. Merge Summary/Findings into a coherent answer for the operator; do not paste raw fleet-agent dumps.
 - Use manage_tasks for your own coordination checklist; spawning workers is \`spawn_agent\`, not manage_tasks.
 - If context is compacted automatically, do not stop tasks early due to token fear; persist progress via manage_tasks and worker reports.`);
   });

src/agent/prompts.ts

@@ -217,7 +217,7 @@ const GUIDELINE_SUB_BLOCKS: Record<
         (ctx.waitAgentsMounted
           ? " or re-wait"
           : ", or idle for its mailbox mail") +
-        "; do not `spawn_agent` a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancel (`stop_reason` cancelled): wait for the operator; do not auto-retry. Identical brief: refuse. Merge Summary/Findings into a coherent answer for the operator; do not paste raw fleet-agent dumps.",
+        "; do not `spawn_agent` a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancel (`stop_reason` cancelled): wait for the operator; do not auto-retry. Identical brief: refuse. Recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; identical brief is still refused otherwise. Merge Summary/Findings into a coherent answer for the operator; do not paste raw fleet-agent dumps.",
       "- Use manage_tasks for your own coordination checklist; spawning workers is `spawn_agent`, not manage_tasks.",
       "- If context is compacted automatically, do not stop tasks early due to token fear; persist progress via manage_tasks and worker reports.",
     ];

src/inference-error-message.ts

@@ -68,6 +68,31 @@ export function classifyInferenceErrorCategory(
     : error.category;
 }
 
+/**
+ * Whether a normalized provider-failure category is transient enough that a
+ * parent may spawn one successor with the same brief (CL-8978). Allowlist:
+ * retryable/timeout — including 429 overload, which normalizes to retryable.
+ * Fatal categories win explicitly: credential, quota, and context-overflow
+ * failures must never read as continuable.
+ */
+const FATAL_PROVIDER_FAILURE_CATEGORIES: ReadonlySet<string> = new Set([
+  "credential_failure",
+  "quota_exhausted",
+  "context_overflow",
+]);
+
+const RECOVERABLE_PROVIDER_FAILURE_CATEGORIES: ReadonlySet<string> = new Set([
+  "retryable",
+  "timeout",
+]);
+
+export function isRecoverableProviderFailureCategory(
+  category: string,
+): boolean {
+  if (FATAL_PROVIDER_FAILURE_CATEGORIES.has(category)) return false;
+  return RECOVERABLE_PROVIDER_FAILURE_CATEGORIES.has(category);
+}
+
 function codexUsageLimitLine(error: InferenceErrorLike): string | undefined {
   // Match normalizeCodexUsageLimitError: never brand a known non-Codex source.
   if (

src/prompts.test.ts

@@ -194,6 +194,9 @@ test("primary chat prompt classifies fail-path successor vs interrupt resume vs
   expect(guidelines).toContain("wait for the operator");
   expect(guidelines).toContain("do not auto-retry");
   expect(guidelines).toContain("Identical brief: refuse");
+  expect(guidelines).toContain("continuable");
+  expect(guidelines).toContain("MAY spawn one successor with the same brief");
+  expect(guidelines).toContain("identical brief is still refused otherwise");
   expect(guidelines).toContain("resume_agent");
   expect(guidelines).toContain("still-live worker");
   expect(guidelines).not.toContain("interrupted-incomplete");

src/subagent/agent-fleet.ts

@@ -107,7 +107,10 @@ import {
 } from "./authority.js";
 
 import { formatSubAgentSpawnAuthFailureMessage } from "./inference-auth-failure.js";
-import { isResolvedProviderFailureError } from "../inference-error-message.js";
+import {
+  isRecoverableProviderFailureCategory,
+  isResolvedProviderFailureError,
+} from "../inference-error-message.js";
 import { errorMessage } from "../agent/error-message.js";
 import { isSubAgentCancelError } from "./dispose.js";
 import {
@@ -125,6 +128,8 @@ interface FleetRecord {
   error?: string;
   stopReason?: string;
   providerFailure?: true;
+  /** CL-8978: transient provider failure — the parent may spawn one successor. */
+  recoverableFailure?: true;
   /** Set once a wait_agents caller has been handed this result. */
   collected?: boolean;
   /** Set once a waiter or occupancy take handed report/error. */
@@ -158,6 +163,8 @@ interface FleetOverlay {
   tombstoned?: boolean;
   hint?: string;
   providerFailure?: true;
+  /** CL-8978: transient provider failure — the parent may spawn one successor. */
+  recoverableFailure?: true;
 }
 
 const RECOVERY_HINT =
@@ -265,6 +272,17 @@ class FleetMailbox {
     existing.providerFailure = true;
   }
 
+  /**
+   * CL-8978: stamp a transient (retryable/timeout/overload) provider failure
+   * alongside sessions.fail. Survives session eviction like providerFailure —
+   * snapshot projects it even once the payload is tombstoned.
+   */
+  markRecoverable(id: string): void {
+    const existing = this.records.get(id);
+    if (existing === undefined) return;
+    existing.recoverableFailure = true;
+  }
+
   markQueued(id: string): void {
     const existing = this.records.get(id);
     if (existing === undefined || existing.collected === true) return;
@@ -462,6 +480,9 @@ class FleetMailbox {
         : {}),
       ...(stopReason !== undefined ? { stopReason } : {}),
       ...(overlay.providerFailure === true ? { providerFailure: true } : {}),
+      ...(overlay.recoverableFailure === true
+        ? { recoverableFailure: true }
+        : {}),
       ...(ask !== undefined
         ? { question: ask.question, questionId: ask.questionId }
         : {}),
@@ -600,7 +621,10 @@ export const waitAgentsToolDefinition: ToolDefinition = {
     `slot), "running", and "awaiting_director". interrupt_agent unblocks this wait immediately with ` +
     `status "interrupted" (a parent-initiated pause — resume_agent, do not spawn_agent a successor against the still-live worker). ` +
     `close_agent also unblocks with status "interrupted" but is permanent. Terminal JSON includes stop_reason when the session recorded one ` +
-    `(interrupted, cancelled, incomplete-report, and similar). awaiting_director is not terminal: re-wait while still pending re-delivers the same question. ` +
+    `(interrupted, cancelled, incomplete-report, and similar). A "failed" entry with "continuable": true is a recoverable transient ` +
+    `provider failure (retryable/timeout/overload) — terminal, not a timeout and not a stall: do not re-wait it, and you may spawn at most ` +
+    `one successor with the same brief. "failed" without the marker (auth, quota, context-overflow, or other errors) is not continuable — ` +
+    `do not respawn it. awaiting_director is not terminal: re-wait while still pending re-delivers the same question. ` +
     `Answer with send_input (soft). Do not call this in a tight zero-progress loop: a timeout means the targets are still ` +
     `queued, running, or awaiting a director answer, not "try again right away" — do other work, reply to the operator, or change the brief. Calling again with the ` +
     `same targets is a real timed wait, not a spin, but wastes turns if nothing has changed. ` +
@@ -1530,6 +1554,17 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
               if (isProviderFailure || providerFailureObserved) {
                 deps.fleetRecords.markProviderFailure(session.id);
               }
+              // CL-8978: a classified transient failure stays wait-terminal
+              // failed, but carries a continuable marker so the parent can
+              // spawn one successor instead of stalling on the failure.
+              // Fatal categories (credential/quota/context-overflow) and
+              // unclassified throws never mark — no auto-retry is added here.
+              if (
+                isResolvedProviderFailureError(err) &&
+                isRecoverableProviderFailureCategory(err.category)
+              ) {
+                deps.fleetRecords.markRecoverable(session.id);
+              }
               deps.sessions.fail(session.id, failReason);
             })
             .finally(() => {

src/subagent/fleet-dry-drive.ts

@@ -34,6 +34,8 @@ export interface FleetDryMailboxRecord {
   readonly description?: string;
   readonly hint?: string;
   readonly providerFailure?: true;
+  /** CL-8978: transient provider failure — the parent may spawn one successor. */
+  readonly recoverableFailure?: true;
   readonly stopReason?: string;
 }
 
@@ -58,9 +60,25 @@ export interface CollectedWorkerReport {
   error?: string;
   hint?: string;
   provider_failure?: true;
+  /**
+   * CL-8978: failed entries from a transient provider failure carry this
+   * marker plus single-successor guidance in continue_with. Capped affordance:
+   * at most one respawn with the same brief, never a retry loop.
+   */
+  continuable?: true;
+  continue_with?: string;
   stop_reason?: string;
 }
 
+/**
+ * Single-successor guidance for a failed+continuable entry. The marker is
+ * advisory only — no runtime auto-retry backs it.
+ */
+export const RECOVERABLE_FAILURE_CONTINUE_GUIDANCE =
+  "This worker failed with a transient provider error (retryable/timeout/overload) " +
+  "and is terminal — do not re-wait it. You may spawn at most one successor with " +
+  "the same brief; do not retry in a loop.";
+
 export function shouldDriveOpenTasks(input: {
   previousRunning?: number | undefined;
   running?: number | undefined;
@@ -169,6 +187,12 @@ export function projectMailboxRecord(
     ...(error !== undefined ? { error } : {}),
     ...(taken.hint !== undefined ? { hint: taken.hint } : {}),
     ...(taken.providerFailure === true ? { provider_failure: true } : {}),
+    ...(taken.status === "failed" && taken.recoverableFailure === true
+      ? {
+          continuable: true as const,
+          continue_with: RECOVERABLE_FAILURE_CONTINUE_GUIDANCE,
+        }
+      : {}),
     ...(taken.stopReason !== undefined
       ? { stop_reason: taken.stopReason }
       : {}),