Skip to content

CL-6451: one room participant, one live run — command paths reuse resident agents; stale turns fail - #181

Merged
TheGreatAxios merged 4 commits into
mainfrom
cl-6451-single-run
Aug 21, 2026
Merged

TheGreatAxios merged 4 commits into
mainfrom
cl-6451-single-run

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Root cause

Two independent defects, one wedge:

  1. A mention could mint a sibling run. A participant's mention handle derives from its definition's display name (Myra -> @myra), while the workflow-command registrar names commands after the definition's wire name (assistant). dispatchWorkbenchCommand's known-handle guard compared the handle against the command name, so @assistant sailed past it into dispatchAtCommand -> startWorkflowCommand -> launchAndJoinAgent, minting run B for an agent the room already held as run A. Later messages fanned out to both, run A never parked, and the dispatch died at the supervisor's waitForRunTerminalOrPark 300s backstop. The same class hit /name re-invocations and any handle drift (dedupe suffixes, slugging).
  2. A failed dispatch left the turn row running forever. dispatchTurn closes its row only when sendMail throws; a dispatch that fails later in the workflow-host supervisor (the backstop) emits no message.run.ended, so nothing ever closed the row — stuck typing indicator, no failed-turn strip.

Fix

  • dispatchWorkbenchCommand now resolves an @name command against the definitions the room's agents were launched from (findResidentAgentForDefinition, comparing by definition asset so re-deployed rows still match). A resident match returns a routing decision instead of a command dispatch: the message posts normally and sendWorkbenchMessage forces the resident participant into the recipient set, so the turn rides the ordinary pipeline — turn row, one-in-flight queueing, reply correlation — into the run the room already has.
  • startWorkflowCommand applies the same residency rule to slash/command dispatch: a resident definition gets the args delivered as mail to the existing run, never a second launch.
  • The explicit invite affordance deliberately still always launches — a second instance of one definition per room stays possible (that is what handle de-dup echo/echo-2 exists for, and what the CL-6329 live proof pins), guarded by a new test.
  • Both AgentTurnStore implementations now fail any row still running past AGENT_TURN_STALE_MS (per-occurrence timeout + grace) on their next read or write — an interrupted dispatch surfaces as a visible failed turn instead of zombieing, and a later reply can never be attributed to a dead row.

Reproduced at scale (longevity campaign, live forensics)

The longevity campaign wedged on exactly this defect while the fix was being built. Read-only forensics from its scratch DB (longevity_scratch), captured mid-wedge:

  • One room, 17 agent participants, 17 minted runs — 4 intended agents plus 13 duplicate daily-heartbeat participants (daily-heartbeat through daily-heartbeat-13), one minted per scheduled command invocation (18:00, 18:01, 18:04, 18:27, 18:47, 19:00, 19:06, 19:44, 20:00, 20:07, 20:32, 21:00) — the startWorkflowCommand mint point this PR removes.
  • 909 running turn rows, 849 older than 10 minutes, 0 ever marked failed — the zombie-row defect this PR's stale-turn expiry closes. All 905 wedged rows sit on the one working agent (sales-analyst).
  • Progressive collapse in lockstep with duplication: sales-analyst completed 77 turns in hour one, then 7, then 2, then 0 after ~20:09, while ~300 new turns/hour kept opening. Hub and sidecar both saturated (~90% CPU in lock waits + replay churn) with zero inference starts.
  • The duplicates' opening mails also bypass the turn projection entirely (no turn rows at all for the 13 siblings). With this PR the invocation reaches the resident run instead of minting; the @name form rides the full turn pipeline (turn row, queueing, reply correlation).

Verification

  • Red/green at the routes/service/store layers: mention-of-resident routes into the existing run and opens turn__0/turn__1 on the same occurrence sequence; command reuse by id and by asset; stale-turn expiry in both stores (drizzle suite run against a scratch Postgres).
  • Live proof on one real stack (scripts/e2e/cl-6451-single-run-proof.ts, scratch DB + real Ollama): invite assistant (handle myra), send @assistant — no command interception, one agent participant, reply as turn__0 of run A; a second @assistant message answers as turn__1 of the same run and recalls a fact stated in turn 1, proving the bootstrap exchange lives in the same stepId-keyed durable history (CL-6453's symptom).
  • bun run check green.

Fixes CL-6451
Fixes CL-6453

An @mention or workflow command naming a definition already resident in
the room must route into the existing participant's run — never mint a
sibling — while an explicit re-invite still deliberately places a second
instance. A turn row still running past the occurrence timeout must read
back failed instead of zombieing (CL-6451).
…ts; stale turns fail

An @name resolving to a workflow command is now checked against the
definitions the room's agents were launched from — a resident match
routes the message into the existing participant's run through the
ordinary turn pipeline (queueing behind an in-flight turn) instead of
dispatching a launch, and startWorkflowCommand applies the same rule to
slash/command dispatch. The explicit invite affordance still always
launches: a deliberate second instance of one definition stays possible,
which is what handle de-duplication exists for.

Turn rows still running past the per-occurrence timeout plus grace are
failed on the store's next read or write: a dispatch the supervisor's
terminal-or-park backstop killed (or a hub death mid-turn) never sends
the closing event, and previously left the turn — and its typing
indicator — running forever.

Fixes CL-6451
…urns

One real stack (scratch DB, real Ollama): invite 'assistant' (handle
'myra'), send '@Assistant' — the message is never intercepted as a
command, the room keeps exactly one agent participant, and the reply
lands as turn__0 of the invited run. A second '@Assistant' answers as
turn__1 of the same run and recalls a fact from turn__0, proving the
bootstrap exchange lives in the same stepId-keyed durable history.
@TheGreatAxios
TheGreatAxios merged commit 03ac7f8 into main Aug 21, 2026
0 of 2 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-6451-single-run branch August 25, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant