fix(upstream): apply fresh-connection recovery on the sidecar and loop retry legs

devlog/_plan/260830_lane_o_reset_recovery_parity/000_units.md

@@ -0,0 +1,87 @@
+# Lane O — connection-reset recovery parity on the sidecar and loop legs
+
+Unit for the work-phase that followed Lane N. Its trigger was a hook restating
+issue #2885 fallout as "the web-search sidecar ignores `upstreamHttpVersion` and
+skips the fresh-connection retry treatment". Half of that was already shipped;
+the other half turned out to be wider than the sidecar.
+
+## What was already true
+
+PR #2908 (`22f2df614`) landed the transport-pin half. `src/web-search/loop.ts:326`
+resolves `deps.incomingMeta.providerFetch`, both send legs use it, and
+`src/server/responses/core.ts:5006` rebuilds it at send time so a 429 rotation
+cannot pin a stale credential. `src/web-search/executor.ts:77` wraps the sidecar
+leg in `withUpstreamHttpVersion(forwardProvider)`. Nothing in that description is
+outstanding, and no part of this unit re-does it.
+
+## The gap that was real
+
+`applyUpstreamRecoveryInit` (`src/lib/upstream-retry.ts:295`) exists for one
+reason: Bun has ignored the hop-by-hop `Connection: close` header
+(oven-sh/bun#20492), so leaving a half-closed pooled socket needs the
+transport-level `keepalive: false` extension as well. Setting the header alone
+lets the retry land back on the same dead socket.
+
+The main lanes call it — `src/server/chat-native.ts:207`,
+`src/server/responses/compact.ts:715`, and six sites in
+`src/server/responses/core.ts` (3831, 3902, 4103, 4163, 5521, 6038). The
+web-search loop and the images loop take the `retryRecovery` argument
+`fetchWithResetRetry` hands them, spend it on `deps.onAttemptSend` telemetry, and
+then build a plain init. Every sidecar executor passes a zero-argument thunk: it
+still retries, but it cannot ask for fresh-connection recovery.
+
+Be precise about the consequence. `fetchWithResetRetry` retries a reset up to
+three times, and on these legs each replay stays *eligible* to reuse the pooled
+socket the reset came from — not guaranteed to, since the pool may hand out
+another. That is enough to make recovery a matter of luck, and the retry then
+reports as exhausted rather than as the pool problem it is. It matches the
+failure shape #2885 reported without explaining it, and this unit does not claim
+to close that issue.
+
+`src/adapters/kiro-retry.ts` already hand-rolls the same two fields (header at
+168, `keepalive` at 173) and is out of scope; an independent audit confirmed it
+correct.
+
+## Diff
+
+Thread the recovery init through the legs that already receive the recovery kind,
+and give the sidecar thunks the argument they were missing:
+
+- `src/web-search/loop.ts` and `src/images/loop.ts` — pass the existing
+  `retryRecovery` through `applyUpstreamRecoveryInit`, preserving the
+  `accept-encoding: identity` handling and the provider-scoped executor.
+- the sidecar executors — accept the recovery argument and route their init the
+  same way, composed so a protocol pin and the recovery fields cannot displace
+  each other.
+
+## Composition constraint
+
+`withUpstreamHttpVersion` spreads `{...(init ?? {}), protocol}` and is typed to
+return `RequestInit | undefined`; `applyUpstreamRecoveryInit` spreads
+`{...init, headers}` and adds `keepalive`. The order is not free. Recovery goes
+**inside**:
+
+```ts
+withUpstreamHttpVersion(url, applyUpstreamRecoveryInit(baseInit, recovery), provider)
+```
+
+so the recovery helper always receives a defined init and the version helper
+spreads the result, keeping headers, `keepalive`, body, signal, and redirect
+alongside `protocol`. The reverse nesting needs a `?? baseInit` fallback to type-check
+at all and would otherwise dereference the `undefined` branch. An independent
+audit probed the composed object under Bun and observed `protocol`,
+`keepalive: false`, `connection: close`, the body, and `redirect: "manual"`
+surviving together. The regression asserts a pinned provider still sees its
+`protocol` on the replay.
+
+What is not verified: no wire capture was taken. Under an HTTP/2 pin,
+`Connection` is a prohibited hop-by-hop header and Bun may normalize it away
+while still honoring `keepalive: false`. The same canonical helper already runs
+on provider paths that support an HTTP/2 pin, so this is a documented unknown
+rather than a reason to exclude a site.
+
+## Evidence standard
+
+A green suite proves nothing here. Each assertion is driven red by reverting its
+own site to the plain init, and any assertion that stays green under that
+mutation is deleted rather than kept.

src/images/loop.ts

@@ -21,7 +21,7 @@ import type { AttemptRecoveryKind } from "../usage/log";
 import { bridgeToResponsesSSE } from "../bridge";
 import { clearableDeadline, idleDeadline } from "../lib/abort";
 import { readBoundedResponseBody } from "../lib/bounded-body";
-import { fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry";
 import { rateLimitRetryDelayMs } from "../providers/key-failover";
 import {
   isTranslatorBudgetExceededError,
@@ -521,12 +521,15 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise<Respons
                 deps.onAttemptSend?.(retryRecovery ?? recovery);
                 const h = new Headers(request.headers);
                 if (!h.has("accept-encoding")) h.set("accept-encoding", "identity");
-                return fetchImpl(request.url, {
+                // Same reset-recovery parity as the web-search loop: the replay needs
+                // `keepalive: false` to abandon the pooled socket, because Bun has ignored the
+                // hop-by-hop header alone (oven-sh/bun#20492).
+                return fetchImpl(request.url, applyUpstreamRecoveryInit({
                   method: request.method,
                   headers: h,
                   body: request.body,
                   signal: headerDeadline.signal,
-                });
+                }, retryRecovery));
               },
               { abortSignal: headerDeadline.signal, label: "image-bridge-loop" },
             );

src/vision/anthropic-describe.ts

@@ -2,7 +2,7 @@ import type { OcxProviderConfig } from "../types";
 import { CLAUDE_CODE_HEADERS, claudeCodeSessionId } from "../adapters/client-fingerprint";
 import { signalWithTimeout, cancelBodyOnAbort } from "../lib/abort";
 import { sidecarEnter } from "../lib/sidecar-tracker";
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import { getValidAccessToken, publicOAuthAuthenticationErrorMessage } from "../oauth";
 import { ANTHROPIC_OAUTH_BETA, CLAUDE_CODE_SYSTEM_INSTRUCTION } from "../oauth/anthropic";
 import type { DescribeOutcome, VisionSettings } from "./describe";
@@ -155,12 +155,12 @@ export async function describeImageAnthropic(
   const startedAt = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(`${base}/v1/messages`, {
+      recovery => fetch(`${base}/v1/messages`, applyUpstreamRecoveryInit({
         method: "POST",
         headers,
         body: JSON.stringify(body),
         signal: linkedSignal.signal,
-      }),
+      }, recovery)),
       { abortSignal: linkedSignal.signal, label: "vision-sidecar-anthropic" },
     );
     if (!res.ok) {

src/vision/describe.ts

@@ -4,7 +4,7 @@ import { FORWARD_HEADERS } from "../adapters/openai-responses";
 import { signalWithTimeout, cancelBodyOnAbort } from "../lib/abort";
 import { redactSecretString } from "../lib/redact";
 import { sidecarEnter } from "../lib/sidecar-tracker";
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import { parseSidecarSSE } from "../web-search/parse";
 import type { SidecarOutcomeRecorder } from "../web-search/executor";
 
@@ -90,7 +90,9 @@ export async function describeImage(
   const t0 = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(`${forwardProvider.baseUrl}/responses`, {
+      // The replay needs `keepalive: false` to abandon the half-closed pooled socket; Bun has
+      // ignored a bare `Connection: close` (oven-sh/bun#20492).
+      recovery => fetch(`${forwardProvider.baseUrl}/responses`, applyUpstreamRecoveryInit({
         method: "POST",
         headers,
         body: JSON.stringify(body),
@@ -99,7 +101,7 @@ export async function describeImage(
         // across origins but forwards nonstandard headers such as `chatgpt-account-id`,
         // `session_id`, and `x-codex-turn-metadata` to the redirect target.
         redirect: "manual",
-      }),
+      }, recovery)),
       { abortSignal: linkedSignal.signal, label: "vision-sidecar" },
     );
     const detachBodyGuard = cancelBodyOnAbort(res.body, linkedSignal.signal);

src/web-search/anthropic-executor.ts

@@ -4,7 +4,7 @@ import { ANTHROPIC_OAUTH_BETA, CLAUDE_CODE_SYSTEM_INSTRUCTION } from "../oauth/a
 import { CLAUDE_CODE_HEADERS, claudeCodeSessionId } from "../adapters/client-fingerprint";
 import { signalWithTimeout, cancelBodyOnAbort } from "../lib/abort";
 import { sidecarEnter } from "../lib/sidecar-tracker";
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import type { WebSearchSource } from "./parse";
 import { BASE_INSTRUCTION, IMAGE_INSTRUCTION, type SidecarOutcome, type SidecarSettings } from "./executor";
 
@@ -162,7 +162,14 @@ export async function runAnthropicWebSearch(
   const t0 = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(url, { method: "POST", headers, body: JSON.stringify(body), signal: linkedSignal.signal }),
+      // The replay needs `keepalive: false` to leave the half-closed pooled socket; Bun has
+      // ignored a bare `Connection: close` (oven-sh/bun#20492).
+      recovery => fetch(url, applyUpstreamRecoveryInit({
+        method: "POST",
+        headers,
+        body: JSON.stringify(body),
+        signal: linkedSignal.signal,
+      }, recovery)),
       { abortSignal: linkedSignal.signal, label: "web-search-sidecar-anthropic" },
     );
     // Guard before any branch reads the body: the failure branch's `res.text()` ran ahead of

src/web-search/exa-executor.ts

@@ -8,7 +8,7 @@
  * redirect: "manual" because Bun forwards custom headers across redirects.
  * Never throws; every error string passes redactSecretString.
  */
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import { cancelBodyOnAbort, signalWithTimeout } from "../lib/abort";
 import { readBoundedResponseBytes } from "../lib/bounded-body";
 import { sidecarEnter } from "../lib/sidecar-tracker";
@@ -39,13 +39,13 @@ export async function runExaWebSearch(
   const t0 = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(EXA_SEARCH_URL, {
+      recovery => fetch(EXA_SEARCH_URL, applyUpstreamRecoveryInit({
         method: "POST",
         headers: { "Content-Type": "application/json", "x-api-key": apiKey },
         body: JSON.stringify({ query, numResults: EXA_NUM_RESULTS, contents: { text: { maxCharacters: EXA_SNIPPET_CHARS } } }),
         signal: linkedSignal.signal,
         redirect: "manual",
-      }),
+      }, recovery)),
       { abortSignal: linkedSignal.signal, label: "exa-web-search-sidecar" },
     );
     const detachBodyGuard = cancelBodyOnAbort(res.body, linkedSignal.signal);

src/web-search/executor.ts

@@ -3,7 +3,7 @@ import { FORWARD_HEADERS } from "../adapters/openai-responses";
 import { signalWithTimeout, cancelBodyOnAbort } from "../lib/abort";
 import { redactSecretString } from "../lib/redact";
 import { sidecarEnter } from "../lib/sidecar-tracker";
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import { withUpstreamHttpVersion } from "../lib/upstream-http-version";
 import { parseSidecarSSE, type WebSearchResult } from "./parse";
 import type { CodexUpstreamOutcome } from "../codex/routing";
@@ -74,7 +74,11 @@ export async function runWebSearch(
   const t0 = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(url, withUpstreamHttpVersion(url, {
+      // Recovery nests INSIDE the version helper: applyUpstreamRecoveryInit then always receives a
+      // defined init, and withUpstreamHttpVersion spreads the result, so `protocol` and the
+      // recovery fields (`connection: close` + Bun's transport-level `keepalive: false`) survive
+      // together. The reverse order needs a `?? init` fallback to type-check at all.
+      recovery => fetch(url, withUpstreamHttpVersion(url, applyUpstreamRecoveryInit({
         method: "POST",
         headers,
         body: JSON.stringify(body),
@@ -83,7 +87,7 @@ export async function runWebSearch(
         // across origins but forwards nonstandard headers such as `chatgpt-account-id`,
         // `session_id`, and `x-codex-turn-metadata` to the redirect target.
         redirect: "manual",
-      }, forwardProvider)),
+      }, recovery), forwardProvider)),
       { abortSignal: linkedSignal.signal, label: "web-search-sidecar" },
     );
     // Attach the body guard before ANY branch reads it. The success path guarded itself below,

src/web-search/gemini-executor.ts

@@ -10,7 +10,7 @@
  */
 import type { OcxProviderConfig } from "../types";
 import { getValidAccessTokenSnapshot, publicOAuthAuthenticationErrorMessage } from "../oauth";
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import { cancelBodyOnAbort, signalWithTimeout } from "../lib/abort";
 import { readBoundedResponseBytes } from "../lib/bounded-body";
 import { sidecarEnter } from "../lib/sidecar-tracker";
@@ -69,7 +69,7 @@ export async function runGeminiWebSearch(
   const t0 = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(`${base}/v1internal:generateContent`, {
+      recovery => fetch(`${base}/v1internal:generateContent`, applyUpstreamRecoveryInit({
         method: "POST",
         headers: {
           "Content-Type": "application/json",
@@ -79,7 +79,7 @@ export async function runGeminiWebSearch(
         body: JSON.stringify(envelope),
         signal: linkedSignal.signal,
         redirect: "manual",
-      }),
+      }, recovery)),
       { abortSignal: linkedSignal.signal, label: "gemini-web-search-sidecar" },
     );
     const detachBodyGuard = cancelBodyOnAbort(res.body, linkedSignal.signal);

src/web-search/loop.ts

@@ -13,7 +13,7 @@ import type { WebSearchBackendId } from "./index";
 import { clearableDeadline } from "../lib/abort";
 import { redactSecretString } from "../lib/redact";
 import { readBoundedResponseBody } from "../lib/bounded-body";
-import { fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry";
 import { rateLimitRetryDelayMs } from "../providers/key-failover";
 import {
   isTranslatorBudgetExceededError,
@@ -460,12 +460,17 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise<Respons
                 deps.onAttemptSend?.(retryRecovery ?? recovery);
                 const h = new Headers(request.headers);
                 if (!h.has("accept-encoding")) h.set("accept-encoding", "identity");
-                return routedProviderFetch(request.url, {
+                // A connection-reset replay must leave the half-closed pooled socket, not just
+                // ask politely: Bun has ignored a bare `Connection: close` (oven-sh/bun#20492),
+                // so the transport-level `keepalive: false` this helper adds is what actually
+                // opens a new connection. Spending `retryRecovery` on telemetry alone left every
+                // replay on this leg eligible for the same dead socket the reset came from.
+                return routedProviderFetch(request.url, applyUpstreamRecoveryInit({
                   method: request.method,
                   headers: h,
                   body: request.body,
                   signal: headerDeadline.signal,
-                });
+                }, retryRecovery));
               },
               { abortSignal: headerDeadline.signal, label: "web-search-loop" },
             );

src/web-search/xai-executor.ts

@@ -12,7 +12,7 @@
  */
 import type { OcxProviderConfig } from "../types";
 import { getValidAccessToken, publicOAuthAuthenticationErrorMessage } from "../oauth";
-import { fetchWithResetRetry } from "../lib/upstream-retry";
+import { applyUpstreamRecoveryInit, fetchWithResetRetry } from "../lib/upstream-retry";
 import { cancelBodyOnAbort, signalWithTimeout } from "../lib/abort";
 import { sidecarEnter } from "../lib/sidecar-tracker";
 import { redactSecretString } from "../lib/redact";
@@ -102,14 +102,14 @@ export async function runXaiWebSearch(
   const t0 = Date.now();
   try {
     const res = await fetchWithResetRetry(
-      () => fetch(url, {
+      recovery => fetch(url, applyUpstreamRecoveryInit({
         method: "POST",
         headers: { "Content-Type": "application/json", "Authorization": `Bearer ${token}` },
         body: JSON.stringify(body),
         signal: linkedSignal.signal,
         // Credential-bearing: never follow a redirect off the pinned origin.
         redirect: "manual",
-      }),
+      }, recovery)),
       { abortSignal: linkedSignal.signal, label: "xai-web-search-sidecar" },
     );
     const detachBodyGuard = cancelBodyOnAbort(res.body, linkedSignal.signal);

tests/exa-web-search.test.ts

@@ -242,7 +242,8 @@ describe("runExaWebSearch key hygiene (canary)", () => {
       expect(captured).toHaveLength(1);
       expect(captured[0]!.url).toBe("https://api.exa.ai/search");
       expect(captured[0]!.init.redirect).toBe("manual");
-      expect((captured[0]!.init.headers as Record<string, string>)["x-api-key"]).toBe("key-1");
+      // Representation-independent: the init may carry a plain record or a Headers instance.
+      expect(new Headers(captured[0]!.init.headers).get("x-api-key")).toBe("key-1");
     } finally {
       globalThis.fetch = realFetch;
     }

tests/gemini-web-search.test.ts

@@ -129,9 +129,12 @@ describe("runGeminiWebSearch request shape (review P1)", () => {
       expect(new URL(req.url).origin).toBe("https://daily-cloudcode-pa.googleapis.com");
       expect(req.url).toContain("/v1internal:generateContent");
       expect(req.init.redirect).toBe("manual");
-      const headers = req.init.headers as Record<string, string>;
-      expect(headers["Authorization"]).toBe("Bearer gem-token-abc");
-      expect(headers["User-Agent"]).toContain("antigravity");
+      // Read through Headers so the credential assertion holds whether the init carries a plain
+      // record or a Headers instance: the reset-recovery helper normalizes headers on the send
+      // path, and this canary is about WHICH bearer goes out, not how the init spells it.
+      const headers = new Headers(req.init.headers);
+      expect(headers.get("Authorization")).toBe("Bearer gem-token-abc");
+      expect(headers.get("User-Agent")).toContain("antigravity");
       const body = JSON.parse(String(req.init.body));
       expect(body.project).toBe("proj-9");
       expect(body.userAgent).toBe("antigravity");
@@ -176,7 +179,7 @@ describe("runGeminiWebSearch request shape (review P1)", () => {
     try {
       const out = await runGeminiWebSearch("q", "google-antigravity", cca, { model: "gemini-3.7-flash", reasoning: "low", timeoutMs: 5000 });
       expect(out.text).toBe("ok");
-      expect((request!.headers as Record<string, string>)["Authorization"]).toBe("Bearer token-a");
+      expect(new Headers(request!.headers).get("Authorization")).toBe("Bearer token-a");
       expect(JSON.parse(String(request!.body)).project).toBe("project-a");
       expect(accountSets["google-antigravity"]!.activeAccountId).toBe("account-b");
     } finally {