fix(auth): recover from provider credential failures (#1188)

* fix(inference): recover OAuth credentials on the active source

* fix(auth): preserve actionable recovery diagnostics

* fix(inference): preserve explicit provenance at refresh seams

* fix(tui): resume uncommitted turns after auth failure

* fix(auth): fence credential rotation and scrub diagnostics

* fix(inference): classify normalized auth failures before retry

* refactor(auth): remove obsolete credential rotator

* test(auth): cover opaque worker credential redaction

* fix(auth): return the committed OAuth refresh winner

* refactor(auth): require complete credential registrations

* fix(inference): freeze retry identity and credential history

* fix(tui): preserve exact recovery selections

* test(auth): assert authoritative store update results

* fix(tui): make model option identities unambiguous

* fix(auth): install and sanitize refresh winners exactly

* test(tui): use opaque model option identities

* fix(auth): preserve sanitized endpoint error types

* fix(tui): decode setup model option identities

* fix(auth): seal refresh failure boundaries

* docs(auth): document credential recovery behavior

Author: TheGreatAxios

docs/ARCHITECTURE.md

@@ -91,6 +91,16 @@ In TUI chat mode there is no completion gate — the session stays open across t
 - The default user-global settings path, the per-repo `.corbits/settings.json`, and the project/global grant store (`.corbits/permissions.json`) are on the static secret-guard denylist for path-keyed tools, so the agent cannot `read_file` credentials there or persist standing auto-approvals. An arbitrary path selected with `--config <path>` is not added to that denylist at runtime. In normal and auto modes, shell commands that reference a statically protected path require explicit operator approval; yolo/skip-permissions allows those shell references after catastrophic authorization checks, while path-keyed access to statically protected paths remains hard-denied.
 - Credential-surface ownership: each auth store module enumerates its own files (`*_AUTH_FILENAME` / `MCP_AUTH_DIRNAME`), the data-only registry in `src/auth/credential-surface.ts` turns them into denylist patterns, `secret-guard-plugin.ts` owns matching (lexical plus realpath), and `@mention` resolution consumes the resolved check — never the registry directly. A new `*-auth.json` token store is denied only once its store module exports its filename and the registry lists it; the coverage test scans store sources for `*-auth.json` literals (registered dirnames get an includes-check instead) and fails the build until both exist.
 
+### Inference credential recovery
+
+Every inference source registers both live credential material and explicit provenance: OAuth credentials identify their provider and named profile, while API-key and keyless sources are classified separately. The harness freezes the complete source at call start, so classification, the one permitted recovery attempt, and any terminal diagnostic retain the exact provider/profile identity that actually failed even if the session changes provider concurrently.
+
+On the first credential failure for an uncommitted call, the retry policy may refresh an OAuth credential from that same named profile and retry once immediately at the harness boundary. It does not substitute another profile or provider. Credential failures that survive that same-source attempt are terminal to the reactor's silent source-failover path. API-key, keyless, missing-provenance, and later credential failures do not enter OAuth refresh recovery.
+
+OAuth stores serialize refresh writes and treat the value observed under the store lock as authoritative. A process that loses a compare-and-swap race adopts the concurrent winner, including its complete token set, instead of publishing or continuing with stale loser material. Credential material and refresh tokens are sanitized recursively before reactor events, run records, worker reports, terminal diagnostics, logs, or other diagnostic sinks can receive them.
+
+The interactive TUI may offer an explicit alternate-provider/model selector only after the same-profile retry also ends in a terminal credential failure. That choice is generation-scoped and consumed once. It may continue the original operator message only when the failed attempt emitted no committing inference event; after any commitment it switches the live provider without replaying the message. `/connect` replaces or adds credentials and `/model` switches the live source explicitly. Exec and fleet workers use the same one-shot same-profile recovery but never open an auth or alternate-provider prompt; an exhausted failure terminates that attempt with a sanitized recovery diagnostic.
+
 ### TUI Runner (`src/tui/runner/`)
 
 - Builds a chat-mode agent using the `ChatDirector`

docs/IMPLEMENTATION.md

@@ -382,6 +382,22 @@ Providers and credentials are read exclusively from settings files: the global `
 
 **Models-first connect.** There is no standalone `/login` command. `/model` opens on a flat **models-only** list (Recent, Favorites, then connected provider/model rows) built by `buildModelsFirstList` (`src/tui/model-picker.ts`); type-to-filter owns printable keys. Selecting a row runs `applyLiveModelSwitch` (`src/session/live-model-switch.ts`) so inference sources, permission-gate identity, grant persistence identity, and advertised tool schemas cut over together. **Alt+A** or `/connect` opens Connect via `addProviderSelectorChoices` (`src/tui/provider-setup.ts`), which lists every first-class kind including Custom — never bare `c` / Ctrl+A, and never in-list “connect →” rows. First-class API-key rows use a named-instance + auth-only form (instance name, key; catalog base URL is display-only); Custom keeps the full manual form. **Alt+F** toggles favorites; recent/favorite pairs live in global settings (`recentModels` / `favoriteModels`). **Alt+D** sets the default via `setDefaultModel` (global `defaultProvider` + that provider's `defaultModel`) plus `persistConnectedSelection` without switching the live session. First-class providers ship from `packages/first-class-providers` (corbits-agnostic defs) and `packages/opencode-go` (Go catalog, auth validate, multi-protocol endpoints, usage). OAuth providers open the existing browser login modal with a named account step; API-key providers share the same multi-instance naming and pre-seed models on save so selection works without restart. Both OAuth and API-key (including Custom) connects share `persistConnectedSelection` in `provider-setup-submit.ts` so project-local provider/model selection is written alongside global credentials. OpenCode Go forces `OPENCODE_GO_BASE_URL` when `opencodeGo` is set so subscription traffic is not billed as Zen PAYG.
 
+### Inference authentication recovery
+
+`src/config/source-credentials.ts` stores each source's live `CredentialMaterial` beside `SourceCredentialProvenance`: `{ kind: "oauth", provider, profile }`, `{ kind: "api-key" }`, or `{ kind: "keyless" }`. Codex and xAI source builders register the exact named OAuth profile plus any credential-derived identity header in the same material record. The vendored harness resolves that record at send time and shallow-freezes the complete call-start source; retries therefore keep the original source identity rather than consulting mutable session selection.
+
+`createCorbitsRetryPolicy` normalizes the attempt error before counting credential failures. On ordinal 1 only, an OAuth source calls `refreshSourceCredentialByProvenance` for that exact provider/profile and retries immediately. A refresh error is replaced with a profile-specific `/connect` diagnostic. Non-OAuth credentials, unknown provenance, later credential failures, and a failed post-refresh retry abort. The vendored harness treats the resulting `credential_failure` as terminal instead of feeding it to automatic source failover.
+
+Codex and xAI refresh sessions use the auth store's locked compare-and-swap update. `updateTokens` returns the authoritative profile observed under the lock; callers replace the complete mutable token object with that winner. If a refresh request loses to a concurrent process, the valid stored winner is adopted rather than overwritten, including removal of optional fields absent from the winner. The in-memory source credential cell also rotates only if the record used to start refresh is still current.
+
+`sanitizeDiagnosticText` strips terminal controls, replaces exact configured credentials, and scrubs secret-shaped text; `sanitizeDiagnosticValue` applies the same rules recursively. Refresh failures sanitize the refresh token through the error's message, stack, detail, and cause chain before it leaves the auth boundary. TUI, exec, and worker event paths sanitize before run sinks, persistent error records, terminal/UI notices, logs, and parent-facing worker output.
+
+TUI recovery state lives in `src/tui/runner/credential-recovery.ts`. It arms only for an operator-originated send that observed both the automatic credential retry and a terminal credential failure, and only when another configured provider/model can be assembled. A monotonically increasing generation makes stale accept/cancel actions inert, and acceptance consumes the generation once. The selector excludes the failed provider. It switches the live provider/model and sends a content-less, generation-correlated director continuation only when no committing inference event occurred; after text, tool, or other committed output it switches without replay. Provider/model option IDs are opaque internal identities used to preserve arbitrary provider and model strings; only their labels are UI contract.
+
+Exec and fleet workers install the same retry policy and sanitization but no recovery selector or credential prompt, including TTY exec. Their exhausted credential failure is terminal and reports how to repair the profile for a later run. In the TUI, `/connect` reauthorizes or adds a profile and refreshes the live catalog, while `/model` switches to a connected provider/model; either is available after a terminal failure.
+
+This release changes no settings, OAuth-store, or session persistence format and runs no migration. New sessions and resumed sessions resolve provider selection and credentials from the stores as they exist when the process starts or rebuilds; persisted transcripts do not pin old credential material. A Corbits process already running during upgrade still has the old in-memory harness and credential cells and must be restarted to acquire this recovery behavior.
+
 **OpenCode Go multi-protocol.** Selectable ids come from live `GET /zen/go/v1/models` (process-cached; packaged seed when cold or the fetch fails). Byte and model-count caps fail closed as malformed — never a truncated prefix. Protocol routing stays on the local map; unknown live ids use Chat Completions. `buildGoSource` / `resolveGoEndpoint` pick the adapter and base URL per model (not a single provider-wide OpenAI route). When Go is the active provider, subscription usage is fetched for the status bar and omitted on auth/network failure.
 
 ### CLI Verbs and Flags

docs/TUI.md

@@ -572,6 +572,12 @@ pair as the default (global `defaultProvider` + that provider's `defaultModel`
   active, bare `j`/`k` type into the filter rather than moving the highlight —
   use arrow keys (or the filtered list's navigation) to move.
 
+After an operator-originated send fails authentication, Corbits first attempts one silent OAuth refresh and retry against the same named profile. If that retry also ends in a terminal credential failure, the shell waits until it is idle and may open a dedicated model/provider picker containing only assemblable models from providers other than the failed provider. It does not appear for API-key failures, the first failure before recovery is attempted, non-credential terminal errors, sends not originated by the operator, or when no alternate is available. Escape consumes the recovery offer without changing provider or replaying input.
+
+Each recovery offer belongs to the failed send's generation. Enter accepts a displayed provider/model identity once; stale, malformed, duplicate, interrupted, cleared, or superseded acceptance is inert. The selected provider/model becomes live. If the failed attempt emitted no assistant text, tool activity, or other committing inference event, Corbits continues the preserved original operator message once on the selected provider. If anything committed, it **never replays the message**: selection only switches the live provider for the next operator action. The internal option IDs that preserve arbitrary provider/model names are opaque implementation details and are never shown as user-facing syntax.
+
+The terminal failure remains visible and names `/connect` as the path to reauthorize a profile. `/connect` refreshes the provider catalog after successful authorization; `/model` remains available to switch explicitly to any connected provider/model. Neither command retroactively replays committed work.
+
 `/mcp` uses the same longest-first overlay-hint footer as the model picker:
 **Alt+A** add (omitted while local MCP settings shadow global), **Alt+D**
 disable, **Alt+R** remove — never bare letters. A remove confirm drops those

docs/VENDORING.md

@@ -108,11 +108,22 @@ verbatim (including the two upstream deletions,
 a ledger entry), and the second re-applies each `PATCHES.md` entry with an
 as-is / adapt / subsumed triage recorded in the ledgers. No entry was
 subsumed upstream. Upstream replaced inline provider `apiKey` plumbing
-with a `credentialId` + credential-cell model; no ledger entry touches
-auth so the vendored trees needed no migration, but first-party callers
-were migrated to the new model (each built source registers its secret
-in `src/config/source-credentials.ts`, handed to the vendored trees as
-their resolver).
+with a `credentialId` + credential-cell model. At sync time no existing
+ledger entry touched auth, so the pristine trees needed no auth-patch
+migration; first-party callers were migrated to the new model (each built
+source registers its secret in `src/config/source-credentials.ts`, handed
+to the vendored trees as their resolver).
+
+CL-9347 subsequently adds the coupled `harness-ts-auth-recovery` entry in
+`vendor/intx-inference/PATCHES.md` and `runtime-ts-auth-recovery-context` in
+`vendor/intx-types/PATCHES.md`; it does not change the upstream pin or
+retrieval metadata. Together they atomically resolve bearer material and
+credential-derived identity headers, freeze the exact call-start source,
+expose per-call credential-failure history to the retry policy, preserve a
+classified refresh diagnostic on abort, and make an exhausted credential
+failure terminal to automatic source failover. Re-syncs must carry or replace
+both entries together so retry identity cannot drift from the credential
+material used by the attempt.
 
 `vendor/intx-workflow-host/workflow-definition-loader.ts` is new in this
 sync: a second partial-tree path alongside `adapters/`, carrying

src/agent/director.test.ts

@@ -135,6 +135,55 @@ describe("toolSetDigest", () => {
   });
 });
 
+describe("ChatDirector credential recovery continuation", () => {
+  const continuation = (generation: number): ReactorInboundEvent =>
+    ({
+      type: "message.received",
+      message: {
+        ref: { uid: 0, mailbox: "system" },
+        headers: {
+          from: "user@local",
+          to: ["agent@local"],
+          date: "2026-09-26T00:00:00.000Z",
+          messageId: `credential-recovery-${generation}@local`,
+          interchangeType: "system.credential.refresh",
+          interchangeCorrelationId: String(generation),
+        },
+        flags: [],
+        content: "",
+        signatureStatus: "missing",
+      },
+    }) as ReactorInboundEvent;
+
+  test("consumes one matching armed continuation and rejects stale or repeated delivery", async () => {
+    const director = createChatDirector("system", [], {});
+    const capabilities = makeCapabilities();
+
+    expect(
+      actionsArray(
+        await director.decide(continuation(4), mockState, capabilities),
+      ),
+    ).toEqual([{ type: "reply", content: "" }]);
+
+    director.armCredentialRecoveryContinuation(5);
+    expect(
+      actionsArray(
+        await director.decide(continuation(4), mockState, capabilities),
+      ),
+    ).toEqual([{ type: "reply", content: "" }]);
+    expect(
+      actionsArray(
+        await director.decide(continuation(5), mockState, capabilities),
+      ).map((action) => action.type),
+    ).toEqual(["infer"]);
+    expect(
+      actionsArray(
+        await director.decide(continuation(5), mockState, capabilities),
+      ),
+    ).toEqual([{ type: "reply", content: "" }]);
+  });
+});
+
 describe("ChatDirector tool-only loop protection", () => {
   const providerlessPolicy = { providerName: "test-provider" };
 

src/agent/director.ts

@@ -457,6 +457,8 @@ function applyManageTasksToolCall(
 // (inference., tool., reactor., fork.).
 export const CHAT_TASKS_CHANGED_EVENT = "custom.chat.tasks.changed";
 export const CHAT_TOOLS_ACTIVATE_EVENT = "custom.chat.tools.activate";
+export const CREDENTIAL_RECOVERY_INTERCHANGE_TYPE =
+  "system.credential.refresh" as const;
 export const ChatTasksChangedDataSchema = type({
   tasks: TaskSchema.array(),
 });
@@ -591,6 +593,7 @@ class ChatDirectorImpl extends DefaultDirector {
   // preserves the queue for the next successful turn instead of desyncing
   // the host from already-mutated director state.
   private coordinatorRethrowNoted = false;
+  private credentialRecoveryGeneration: number | undefined;
 
   constructor(
     systemPrompt: string,
@@ -775,6 +778,16 @@ class ChatDirectorImpl extends DefaultDirector {
     this.clearDenials = clear;
   }
 
+  armCredentialRecoveryContinuation(generation: number): void {
+    this.credentialRecoveryGeneration = generation;
+  }
+
+  cancelCredentialRecoveryContinuation(generation: number): void {
+    if (this.credentialRecoveryGeneration === generation) {
+      this.credentialRecoveryGeneration = undefined;
+    }
+  }
+
   updateToolDefinitions(toolDefinitions: ToolDefinition[]): void {
     const before = toolSetDigest(this._toolDefinitions);
     const after = toolSetDigest(toolDefinitions);
@@ -991,6 +1004,23 @@ class ChatDirectorImpl extends DefaultDirector {
     const recovery = this.compaction.interceptOverflow(event, capabilities);
     if (recovery !== null) return recovery;
 
+    if (
+      event.type === "message.received" &&
+      event.message.ref?.mailbox === "system" &&
+      event.message.headers?.interchangeType ===
+        CREDENTIAL_RECOVERY_INTERCHANGE_TYPE
+    ) {
+      const generation = Number(event.message.headers.interchangeCorrelationId);
+      if (
+        Number.isSafeInteger(generation) &&
+        generation === this.credentialRecoveryGeneration
+      ) {
+        this.credentialRecoveryGeneration = undefined;
+        return capabilities.infer();
+      }
+      return capabilities.wait();
+    }
+
     // A forged or replayed compaction continuation arrives as an empty
     // message.received with no outstanding compact state (the legit resume
     // is consumed above). Answering it with infer would burn a billable
@@ -1437,6 +1467,8 @@ export interface ChatDirector extends ReactorDirector {
   setWorkflowCoordinator(coordinator: WorkflowCoordinator | undefined): void;
   setAllowIdleWithFleet(value: boolean): void;
   setClearDenials(clear: (() => void) | undefined): void;
+  armCredentialRecoveryContinuation(generation: number): void;
+  cancelCredentialRecoveryContinuation(generation: number): void;
   getTasks(): Task[];
   restoreTasks(tasks: Task[]): void;
   getContextEstimate(): { tokens: number; isEstimate: boolean };

src/agent/prompts.ts

@@ -447,6 +447,15 @@ export function buildSkillsSection(skills: readonly SkillSummary[]): string {
   ].join("\n");
 }
 
+export function buildCorbitsRecoveryCommands(): string {
+  return [
+    "Corbits recovery commands:",
+    "- For provider authentication, login, reauthentication, or credential failures, recommend /connect and name the provider/profile.",
+    "- To switch the active provider or model, recommend /model.",
+    "- For OAuth profiles, never recommend Codex CLI login or API-key setup, and never request or expose secrets.",
+  ].join("\n");
+}
+
 export function buildChatSystemPrompt(
   extensions?: string[],
   env?: EnvironmentInfo,
@@ -474,6 +483,7 @@ export function buildChatSystemPrompt(
       coreToolNamesForSessionMode(sessionMode, toolAvailability),
       { advertiseArchive: true },
     ),
+    buildCorbitsRecoveryCommands(),
   ];
   if (skills.length > 0) sections.push(buildSkillsSection(skills));
   sections.push(contextSection(env));