Skip to content

Shared per-workspace broker is torn down by any session's SessionEnd, killing other sessions' in-flight jobs #671

Description

@bwish71

Summary

The companion broker is keyed on the workspace, but its teardown is triggered by an individual session ending, with no check for other sessions' in-flight work. When several Claude Code sessions run in the same checkout — normal on a shared repo — any session closing kills whatever job the others are running. The victim sees codex app-server connection closed., and for the stop-review gate this surfaces as its session-stop being blocked by a failure it did not cause.

Observed as an intermittent stop-gate failure: 3 of 7 runs died over one day, with no correlation to run duration — a 6-minute run survived while ~5-minute runs died. The variable was never the run; it was whether another session happened to close during it.

Environment

  • plugin 1.0.6, Node v24, Linux
  • 7 Claude Code sessions live in one git checkout (verified: every one of their /proc/<pid>/cwd resolves to the same repo root)
  • one broker serving them all: app-server-broker.mjs serve --endpoint unix:/tmp/cxc-XXXXXX/broker.sock --cwd <repo>, reparented to PID 1

Bug 1 (primary) — unguarded workspace-wide teardown

scripts/session-lifecycle-hook.mjshandleSessionEnd() resolves the broker via loadBrokerSession(cwd), which reads broker.json from the workspace-keyed state dir (lib/state.mjs resolveStateDir). It then unconditionally calls sendBrokerShutdown(endpoint) and teardownBrokerSession({... killProcess: terminateProcessTree}), which kills the pid, unlinks socket/pid/log files, and rmdirs the session dir.

lib/broker-lifecycle.mjs teardownBrokerSession has no refcount, no owning-session check, and no in-flight-job check. cleanupSessionJobs() correctly scopes itself to the ending session's own job rows — but the broker kill beside it is workspace-wide.

Victim path: BrokerCodexAppServerClient's socket fires closehandleExit(this.exitError) with exitError === null → every pending request rejects with new Error("codex app-server connection closed.") (lib/app-server.mjs). No reconnect, no retry.

Expected: a session ending does not disturb work owned by another session.
Actual: it terminates it, surfacing as an opaque connection-closed.

Bug 2 — the broker honours broker/shutdown mid-turn

Independently of Bug 1: scripts/app-server-broker.mjs answers broker/shutdown and immediately runs shutdown(server), which socket.end()s every connected client, closes the upstream app-server, and exits. It consults neither activeRequestSocket nor activeStreamSocket.

So the graceful RPC alone is enough to kill a live turn — the SIGTERM in Bug 1 is not even required. A complete fix needs both a guard at the teardown call site and a refusal inside the broker.

Bug 3 — the shared broker is single-flight, so concurrent sessions collide

scripts/app-server-broker.mjs rejects a request with BROKER_BUSY_RPC_CODE (-32001) "Shared Codex broker is busy." whenever activeRequestSocket/activeStreamSocket is held by a different socket. A review turn is turn/start, which is in STREAMING_METHODS, so it holds activeStreamSocket for the whole multi-minute turn.

With N sessions sharing one broker, any two overlapping turns mean one fails, and unlike Bug 1 this scales with turn duration. Arguably a design limit of sharing one app-server rather than an oversight, but it makes the shared broker unsuitable as-is for multi-session workspaces.

Bug 4 (minor, two parts)

  • Orphaned job rows. When Bug 1 kills a job, the victim's row keeps status: "running" forever — only the ending session's rows are reaped, so nothing reconciles the victim's. The next session's stop hook then reports a phantom running job.
  • Diagnostics are destroyed by the next state write. saveState unlinks the log file of any job dropped from state.json, and jobs are dropped both by the SessionEnd filter and by pruneJobs() at MAX_JOBS = 50. A failure's log is routinely gone before anyone can investigate — which is why this report leans on code reading rather than logs.

Reproduction

  1. Open two Claude Code sessions in the same git checkout.
  2. In session A, start a long-running Codex job (/codex:review, or let the stop gate run).
  3. While it runs, exit session B.
  4. Session A's job fails with codex app-server connection closed. and its state.json row is left at status: "running".

Deterministic variant, no Codex turns needed: seed the workspace state dir with a broker.json plus a foreign status: "running" job row, run session-lifecycle-hook.mjs SessionEnd with a different session_id, and observe broker.json cleared and the broker pid killed.

Suggested fix

  1. handleSessionEnd — before shutting the broker down, check whether any job belonging to a different session is queued/running; if so, reap only your own rows and leave the broker alone. Bound the check by staleness (e.g. the gate's own 15-minute timeout) so an orphaned row from a crashed session cannot wedge teardown permanently. An idle broker is already reclaimed by ensureBrokerSession's liveness check on next use, so leaking it is safe.
  2. app-server-broker.mjs broker/shutdown — refuse while another client's request or stream is active, replying with the existing BROKER_BUSY_RPC_CODE instead of shutting down.

Longer term, reference-counting the broker across sessions, or giving each session its own app-server, would also remove Bug 3.

I've applied both guards locally and verified them with a mutation test in an isolated workspace: it passes with the guard, fails on exactly the guard's assertion when the predicate is neutralised to false, and the stale-row and no-foreign-job controls tear down normally in both states. Happy to open a PR if that's useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions