fix(cluster): let an ACP write survive a shim channel renewal

[joerideturck]

Summary

A bug fix for a cluster agent that stops answering for good after one unlucky credential renewal, observed on our deployment on 2026-09-17: every turn — new sessions and replies alike — failed with ⚠️ Agent failed to respond: shim channel renewed for hours while the sandbox pod sat there healthy, until someone deleted the pod by hand.

What happens

The shim credential renews at half its TTL, and ShimSession.attach aborts the requests in flight on the socket it replaces. That is documented and correct: the reply was lost, ask again (ShimChannelLostError is typed for exactly that). The ACP stdin write is the one caller that could not act on it. Its request rides inside the WritableStream.write of createRemoteRuntime, and a WritableStream that rejects a write stays errored — every later write() rejects with the same stored error object. The ShimSession survives the renewal (that is what it is for), the runtime never exits, so AcpHost stays warm on a dead stdin and every subsequent turn dies on its first byte.

The tell in the daemon log: repeated relay im dispatch failed …: ShimChannelLostError: shim channel renewed with an identical ShimChannel.abort ← ShimSession.attach stack, at times with no matching shim: bound agent … line. Same error, replayed.

Fix

Asking again is only safe if the pod can tell a re-send from a new write — ND-JSON applied twice is a corrupted frame, which is why §6 of cluster-spawn-and-shim.md terminates a tunnel stream in the same situation. Terminating the ACP stream is not acceptable there (it ends the runtime, and the write in flight is usually the turn), so this stream buys the missing property instead:

• shim/acp-stream.ts — chunks carry an optional per-stream seq.
• shim/acp-runner.ts — a seq is applied at most once, recorded after the write so a failed write is still owed. Chunk writes run in arrival order: the client serves requests concurrently, and a chunk that reached the pod just before the socket closed can still be inside its stdin write when the re-send lands on the new socket.
• shim/client.ts — the open reply announces resumableWrites: true. Announced rather than assumed: against a shim that predates it the daemon must not ask twice.
• k8s/remote-runtime.ts — writes are numbered; a write rejected with ShimChannelLostError is re-sent unchanged (seq included) once the session is attached. When it cannot be re-sent — no announcement, no re-attach within the grace window, or the re-send fails too — the runtime is ended deliberately, which reapTerminalHost respawns on the next message. One failed turn is an acceptable outcome; an agent that answers nothing until a human intervenes is not.
• shim/session.ts — waitForAttach(timeoutMs). Ordinarily already satisfied when the rejection arrives (attach binds the replacement before aborting the old channel), so this is the ordering guarantee rather than a wait anyone expects to spend time in.

Backward compatible in both directions: an older shim ignores seq and does not announce, so the daemon falls back to the terminal path; an older daemon never sends seq, so the new shim's dedupe is inert.

docs/designs/cluster-spawn-and-shim.md §6 gains a subsection stating why the ACP stream re-sends where the tunnels terminate, so the next reader does not "fix" one into the other.

Testing

packages/daemon/test/shim-acp-write-resume.test.ts (new, 7 tests):

• AcpRunner with cat as the runtime — the evidence is what reached stdin, not what the shim answered: a re-sent chunk is applied once; a re-send arriving while its first attempt is still writing is still applied once; a re-send whose first attempt failed is applied.
• ShimSession.waitForAttach on the real session: settles at once when attached, on the next attach otherwise, rejects on the grace window and once the launch is lost.
• createRemoteRuntime: a write the renewal failed goes out again unchanged (seq 1, 2, 2); against a shim that did not announce the dedupe the runtime exits instead of the stream erroring forever.

Five of the seven fail on an unmodified checkout. shim-pod-env, shim-provider-env, shim-dsh-preset, shim-handshake, cluster-metrics pass alongside (113 total). tsc --noEmit, eslint and Prettier clean on the touched files. shim-cancellation.test.ts fails 5 cases on this macOS host identically on an unmodified checkout (needs tooling the host lacks).

The production incident was cleared by deleting the sandbox pod (generation 26 → 27, session resumed via session/load); this goes onto our deployment next.

🤖 Generated with Claude Code

[agentconnect-md-test]

The sequence-based retry and shim capability handshake address the ambiguous-write case cleanly, but the new terminal fallback can leave the actual ACP child running in the sandbox. Because the synthetic exit clears AcpHost’s spawned-runtime handle before host reaping calls stop(), a subsequent message can launch a second adapter beside the orphaned one. Please make the failure path stop/close the remote runner before publishing terminal exit, or otherwise preserve a handle that teardown can use.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

[P1] Stop the sandbox runner before publishing terminal exit

failRuntime() only detaches the event listener and calls finish(). That fires SpawnedRuntime.onExit; AcpHost then clears its spawned handle before reapTerminalHost() reaches host.stop(), so teardown no longer sends this stream an op: 'close'. When the retry fails after a renewal (for example, a second renewal, timeout, or shim error), the AcpRunner and its child can therefore remain alive in ShimClient.acpStreams, and the next message spawns another adapter in the same pod. Please close/stop the remote stream before firing the terminal notification, or keep the spawned handle available until host teardown has invoked stop().

[joerideturck]

Right — the exit finish() publishes clears AcpHost's spawned handle, so stop() early-returns and this stream never gets its close. Fixed in 5c727b4: the failure path now shares teardown's close (closeStream, same stopped latch so a later stop() does not ask twice) and awaits it before the exit goes out. The fallback test now asserts the order ['close:s1', 'exit'] and that a following stop() sends nothing further; it fails on the previous head.

[agentconnect-md-test]

The ACP write-resume path now safely deduplicates ambiguous writes across shim renewal, preserves backpressure and compatibility gating, and terminates the remote child before publishing a synthetic runtime exit. The follow-up fixes the prior orphaned-runner finding by sharing the close path and verifying close-before-exit ordering. I found no remaining blocking issues in this revision.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[spacedragon]

Thank you.

Merging.