Repository navigation
refactor(gateway): add the OpenCode V2 run loop - #1985
Conversation
The gateway reaches a workspace OpenCode V2 server through the proxy, so it needs the V2 siblings of remote-client and remote-question-client: bearer headers, the permission/form reply routes, interrupt, session reads and the SSE subscription with its heartbeat. The event mapper now carries the inbox id on a subagent notice and maps session.inbox.delivered, so a drain gate can tell a queued notice from one the parent has taken into a turn. A replied form carries its answer for the echo.
Adds the V2 sibling of the gateway run core and its collaborators: root stop and quarantine watch, repo quiescence over the active-list join, the drain gate (notice delivered, root idle in the notice's generation, root not live), owned-session settlement, approval replies keyed by request id, question forms mapped onto the existing question flow, and provider-failure labelling. run.ts and program.ts pick V1 or V2 from the opencodeVersion the workspace reports on /readyz (major >= 2 is V2). A version that is present but unreadable selects neither loop and counts as not ready. V1 behaviour is unchanged. Tests run the real loop against captured v2.2.0 frames. An end-to-end test, gated on FRO_BOT_OPENCODE_V2_BIN, drives one prompt with a Code Mode call and an approval through the real stock server.
…able notices and frames, track child inbox delivery
A background `shell` is a server job, not a session: `session.active` never lists it, an interrupt does not stop it, and it outlives the run's lock while its completion notice later wakes the root. The V2 loop now treats it like a background subagent. - Mapper: `shell.created`/`shell.exited` and the job notice (`source: shell` with a `jobID`; a user command's notice has none and is ignored) map to `shell_started`/`shell_exited`/`shell_notice`. An unreadable job notice names its shell so the loop can fence it. - Ledger: shells share the ownership ledger under a `shell:<id>` key space. Adopted on `shell.created` and on a `running` `shell` tool result; settled on the notice; a foreground shell is released when its call ends. Reconcile settles a shell absent from `GET /api/shell` that reads stopped or 404, and keeps it unknown on any other read error. - Drain: an outstanding shell blocks admission, and its notice is observed and delivered like a child's. - Stop paths: before the root abort, every owned running shell is DELETEd and confirmed gone by re-listing. An unconfirmed kill keeps the stop unconfirmed; the quarantine watch keeps deleting through the hold. - Quiescence: the repo is not clear while any shell runs in its directory. - Remote client: `listShells`, `getShell`, `deleteShell`.
…can still wake the root - A stop that deleted any shell, tracked or not, never confirms the root: the DELETE's error notice wakes it. Any stop with a live owned shell therefore quarantines, even after a clean kill (accepted cost). - A final sweep after the root confirms deletes or reports a shell started during the barrier, and leaves the stop unconfirmed. - Follow-up work stays open until an adopted child's or shell's notice (or its fence) has been observed, not until reconciliation settles the entry. - The drain commit point reads the running owned shells from the server before the commit and does not admit while one runs or the list cannot be read. - A fatal move of an owned session is never confirmed. - The shell kill waits six rounds of 500 ms.
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: CONDITIONAL
The V2 lifecycle barriers have gaps that can release checkout ownership while adopted work can still execute.
Blocking issues
| Priority | Location | Issue and required change |
|---|---|---|
| P1 | drain-completion-v2.ts:240–244 | An awakened child can still be executing when drain admits. Adopted child C can finish its job while its background shell S remains running. S's completion wakes C; delivery clears the notice fence, but C's new execution is ignored and liveness checks only the root. A subsequent root success can release the checkout while C writes. Require quiescence of every owned session, including settled children, and include resumed descendants in teardown. |
| P1 | drain-completion-v2.ts:270–279 | REST-discovered shells lose their completion obligation. When shell creation and tool-result events precede child adoption, the shell is untracked. A listing blocks admission while it runs, but disappearance permits admission before its completion notice and owner follow-up. Retain discovered shell obligations across listings; absence must not substitute for notice/delivery/completion evidence. The test at drain-completion-v2.test.ts:579–591 currently expects this unsafe admission. |
| P1 | shell-jobs-v2.ts:243–268 | A committed DELETE with a lost response is treated as producing no wake. For an unadopted owned shell, DELETE can remove it and schedule its notice while returning a network error. An empty confirming list then returns confirmed:true, deleted:[]; settlement derives shellWake=false and can release without quarantine. Track potentially committed deletion attempts separately from acknowledged successes, and preserve their follow-up uncertainty. |
| P2 | run-core-v2.ts:506–510 | Notice receipt prematurely closes the stop barrier's follow-up window. Enqueued notices immediately settle entries and count as accounted for. If the root fails before drain, followUpPossible can become false. Upstream synthetic admission precedes a separate lookup/wake, so a producer paused between admission and wake can restart the root after interruption and confirmation. Keep queued/in-flight notice consumption in the barrier predicate independently of drain state. |
| P2 | settle-owned-sessions-v2.ts:274–284 | A timed-out final sweep retains stale confirmation flags. After initial shell/root confirmation, a final sweep can discover a late unadopted shell and stall. Timeout then reports both unconfirmed flags false for an empty/settled ledger. The run quarantines, but run-core-v2.ts:528 skips its cancellation watch, leaving no retries against surviving shells or root wakes. Invalidate shell confirmation before awaiting the final sweep; restore it only after a completed confirming result. |
Non-blocking concerns
| Priority | Location | Concern |
|---|---|---|
| P2 | remote-client-v2.ts:228 | CRLF normalization is chunk-local. A valid SSE delimiter split across chunks can remain unrecognized, dropping/buffering frames. Parse lines incrementally or normalize with cross-chunk CR handling. A minimal bun -e probe using the actual client and mocked ReadableStream confirmed: intact CRLF delivered one frame; splitting its delimiter across three chunks delivered none. |
| P2 | provider-failure-v2.ts:40 | V2 provider.auth is passed unchanged to a classifier recognizing ProviderAuthError. With an ordinary message such as bad key, authentication becomes an unclassified session error. Translate structured V2 discriminants explicitly instead of relying on message hints. |
Missing tests
- An adopted, settled child resumes from its shell notice and remains active when the root succeeds.
- A REST-discovered shell disappears before its notice/follow-up; revise the existing test that admits on disappearance alone.
- DELETE commits but returns a network error, followed by an empty shell list.
- An enqueued notice's producer reaches its deferred wake after a pre-drain failure barrier interrupts/confirms the root.
- The second/final shell sweep discovers work and exceeds the settlement deadline; assert quarantine monitoring starts.
- CRLF delimiters split at every chunk boundary, and
provider.authclassification with arbitrary message wording.
Risk assessment (HIGH): likelihood of regression, security exposure, or blast radius
The V2 path can admit completion or confirm termination before owned writers are quiescent, risking concurrent checkout mutation. Versionless workspaces continuing on V1 limits immediate rollout exposure. No credential disclosure or authorization bypass was identified.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 40909378263 |
| Cache | hit |
| Session | ses_e778d76bfffeJk5X44iYVa8hDB |
Reviewed the supplied 56-file scope against origin/main; independently source-checked blocking findings. No build, lint, typecheck, or test suites were run. One minimal bun -e transport probe was executed as described above. The skill's structural-validation helpers were inaccessible under the environment's directory policy.
The response-file protocol supports a review body, not inline-comment payloads; file-specific findings are linked to exact lines above.
A V2 run is released by its directory, not by its own accounting. Stock v2.2.0 gives no way to prove that nothing the run started can still wake a session: a shell's DELETE wakes its owner, a notice is admitted and the wake starts in a separate step, and an interrupt reply is a delivery receipt. The checkout is released after the directory shows no active session, no running shell and no queued inbox item for 11 readings 500 ms apart. Every success waits for that window, and a stop repeats kill, interrupt and read until it passes or the bound ends it, then quarantines. A move of an owned session always quarantines. A reading is bounded so its cost does not grow with the workspace's history: - Active sessions are placed with GET /api/session/:id, not by listing the directory. - Inboxes are read for the run's root, the sessions it owns, and sessions updated since the run began less 10 minutes. Admitting an inbox item bumps time.updated, so a notice queued during the run always lands inside the bound. The list is read newest-first and the walk stops at the cut-off, so the page bound applies to recent sessions only. - The acquire check uses the same placement of active sessions. Also folds in the provider.auth and CRLF fixes and the documented relaxed invariant in packages/gateway/AGENTS.md.
| expect(busy).toMatchObject({passed: false, streak: 0}) | ||
| expect(after.streak).toBe(1) | ||
| // and the full window is needed again | ||
| let state = after |
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: CONDITIONAL
The previous findings are addressed within the documented quiescence model. Four issues remain in the revised release and cancellation paths.
Blocking issues
| Priority | Location | Issue and required change |
|---|---|---|
| P1 | drain-completion-v2.ts:301–305 | A rejected final reading leaves the old quiet window reusable. If root activity and fresh idle arrive during the final observation, the wait returns passed without rechecking its predicate. Revision drift correctly rejects admission, but never resets the gate. The retry then admits after one quiet reading, without five seconds following the new activity. Recheck invalidation after every awaited observation, including the passing one, and reset obsolete evidence before retrying. Reproduced with the actual drain gate. |
| P1 | root-stop-v2.ts:81–99 | A stalled shell DELETE starves session interrupts throughout quarantine. All DELETE responses are awaited before any interrupt. Observation, deletion and interruption share the watch's five-second signal; when DELETE consumes that budget, subsequent interrupts receive an already-aborted signal. Every tick repeats this, leaving a responsive root/child running. Independently bound deletion and reserve a usable budget for session interrupts, retaining interruption after deletion-induced wakes. |
| P2 | directory-quiescence-v2.ts:298–301, 117–129 | Incomplete observations discard known cancellation targets. A successful shell/active read is replaced with empty activity when another branch times out. Separately, one failed lookup from the global active list discards every successfully located session—including this checkout's root/children. Barrier and watch consequently omit otherwise-healthy DELETE/interrupt requests. Preserve completed positive evidence on timeout and successful placements alongside lookup failures; keep release classified unreadable. |
| P2 | settle-owned-sessions-v2.ts:150–158 | A moved owned session escapes interruption. V2 supports moving an active execution and continuing the same session at its destination. Placement then excludes it from both the original-directory barrier and quarantine watch. Forced quarantine occurs, but the known live session is never interrupted. Retain moved owned IDs as explicit teardown/watch targets independent of directory placement. The current movement test masks this because its getSession() fixture always returns the original directory. |
Non-blocking concerns
None
Missing tests
- Hold the final directory observation, inject root activity and fresh idle, resolve it quiet, then assert the retry requires an entirely fresh window. Existing invalidation coverage changes evidence between readings.
- Combine a hanging DELETE with healthy root/child interrupt endpoints; assert interrupts are delivered with non-aborted signals during quarantine.
- Preserve and act on a known shell/session when another observation branch times out; also cover mixed successful placements and an unrelated failing session lookup.
- On an active root/child move, update its returned directory to the destination and verify interruption there during teardown and quarantine.
Risk assessment (HIGH): likelihood of regression, security exposure, or blast radius
The stale streak can shorten the release barrier, while degraded-read and movement cases can leave canceled executions running through quarantine. Versionless workspaces remaining on V1 limits immediate rollout exposure. The documented upstream producer-delay limitation is separate from these findings.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 40925310161 |
| Cache | hit |
| Session | ses_e778d76bfffeJk5X44iYVa8hDB |
Re-reviewed 2a52aa0..65aa2e4 and checked prior feedback. No build, lint, typecheck, or test suites were run. Blocking findings were independently source-validated.
One minimal bun -e probe exercised the actual drain gate with a shortened three-reading window and activity during its final observation. It returned reads:4, revision-drift, then admission—rather than requiring three fresh readings after invalidation. Command:
bun -e 'import {createDrainCompletionV2} from "./packages/gateway/src/execute/drain-completion-v2.ts"; import {createDirectoryStabilityGate} from "./packages/runtime/src/agent/directory-quiescence-v2.ts"; import {createOwnershipLedger} from "./packages/runtime/src/agent/ownership-ledger.ts"; let reads=0; let drain; const reasons=[]; let admit; const done=new Promise(resolve=>{admit=resolve}); const gate=createDirectoryStabilityGate({requiredObservations:3,minSpacingMs:0,minSpanMs:0,observe:async()=>{reads++; if(reads===3){drain.noteRootActivity();drain.noteRootIdle();drain.requestValidation()} return {kind:"quiescent"}}}); drain=createDrainCompletionV2({rootSessionId:"root",ledger:createOwnershipLedger(),adapter:{liveSessionIds:async()=>({success:true,data:new Set()}),children:async()=>({success:true,data:[]})},directoryGate:gate,directorySpacingMs:0,signal:new AbortController().signal,logger:{debug(){},info(){},warn(){},error(){}},onNotAdmitted:reason=>reasons.push(reason),onAdmitted:()=>admit()}); drain.noteRootIdle(); drain.beginDrain(); drain.requestValidation(); const timer=setTimeout(()=>{drain.dispose();throw new Error("probe timed out")},2000); try {await done; process.stdout.write(JSON.stringify({reads,reasons,streak:gate.state().streak}));} finally {clearTimeout(timer);drain.dispose()}'The response-file protocol supports a review body rather than inline-comment payloads; findings link to exact source lines above.
…op targets under degraded reads - The quiet-window wait asks whether its evidence is still current after every reading, the passing one included, and drops the streak when it is not. Root activity and a fresh idle that landed during the last reading used to leave the old streak for the retry to reuse. - A stop gives its shell DELETEs and its session interrupts bounds of their own, derived from the barrier's or the hold's life instead of the reading's. A DELETE that never answers no longer leaves the interrupts with a dead signal on every quarantine tick. Interrupts are issued together, root first. - An unreadable reading keeps the positive evidence its reads found (shells, active sessions placed in the directory, queued inboxes), and one failed placement lookup no longer discards the placements that succeeded. The release still counts it as unreadable; the barrier and the watch act on it. - An owned session that moved is an explicit target of the barrier and the watch: while it is on the active list it is interrupted wherever the server stores it now. A move still always quarantines.
fro-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS
All four outstanding findings are addressed with targeted regression coverage: fresh quiet-window retries, independent stop budgets, retained partial cancellation targets, and moved-session interruption through quarantine. No new blocking issues found.
Blocking issues
None
Non-blocking concerns
None
Missing tests
None
Risk assessment (MED): likelihood of regression, security exposure, or blast radius
The V2 lifecycle remains a substantial asynchronous change with the documented upstream producer-delay limitation. Current versionless workspace images continue selecting V1, limiting immediate rollout exposure.
Run Summary
| Field | Value |
|---|---|
| Event | pull_request |
| Repository | fro-bot/agent |
| Run ID | 40950595529 |
| Cache | hit |
| Session | ses_e778d76bfffeJk5X44iYVa8hDB |
Re-reviewed 65aa2e4..d82eee2, reconciled previous findings, and inspected the new regression tests and integration paths. Validation was source-based; no tests, build, lint, or type-check commands were run. Repository files remain unchanged.
This adds the gateway's OpenCode V2 run loop (#1805). Each run picks V1 or V2 from the
opencodeVersionthe workspace reports on/readyz:No current image reports a V2 version, so every run stays on V1 until a V2 image is deployed. That deploy is the switch.
Run loop
The V2 modules live next to the V1 ones in
packages/gateway/src/execute/:run-core-v2root-stop-v2drain-completion-v2settle-owned-sessions-v2repo-quiescence-v2permission-reply-v2question-client-v2opencode-attach-v2provider-failure-v2opencode-generationThe runtime adds the V2 remote and question clients and the directory quiescence check. V1's behaviour doesn't change.
run-core.tsgains two exports. The shared ownership ledger gains an additivereleaseop that V1 never calls, and V1 reconcile skipsshell:keys.The guarantees V1 has carry over:
{interrupted:false}reply never counts as a stop.server.connectedbefore sending the prompt. It never reconnects. Any of these goes through the barrier:server.connectedwaitReasonare kept.execute, and fail closed during the barrier.provider.autherrors classify as auth errors whatever the provider's message says.Release invariant
A V2 run releases the checkout, and the lock with it, only after the run's directory reads as quiet 11 times in a row, at least 500 ms apart and at least 5 s overall. Any read that fails or times out resets the count. Quiet means all three of these:
GET /api/session/activestored under the directory, including children, grandchildren and resumed descendants;GET /api/shellfor the directory, whoever started it;GET /api/session/:id/inboxfor this run's root, its owned sessions, or any session under the directory updated in the last 10 minutes.time.updated(core/src/session/projector.ts), so older sessions can't hold a pending wake.Success and stop both pass through the same check. That includes a run that never adopted anything.
On stop. The loop repeats three steps until the check passes within the barrier budget:
If the check doesn't pass in time, the run quarantines. The quarantine watch repeats the same kill-and-interrupt for the whole hold.
The rule is directory-wide on purpose. A gateway checkout belongs to the gateway, so any session or shell found in it is treated as the run's.
What this rule doesn't prove. Stock OpenCode has unbounded gaps between a shell exiting and its notice, between a notice being admitted and the wake, and inside notification producers. A producer that pauses for more than 5 s and then wakes a session would slip past. Quarantine on any unconfirmed stop and the next run's acquire check narrow this, but don't close it. The real fix is a location close-and-drain fence upstream (anomalyco/opencode#60286). The same gap appears in anomalyco/opencode#60107: an interrupt doesn't stop background shells.
Background shells
A background
shelljob runs on the server and isn't a session.shell.created, or from a shell tool result that moved to the background, when the owner is the root or an owned child. Its ledger key isshell:<id>. A foreground shell is released when its tool call ends.Verification
run-core*suites onto V2 fixtures built from frames captured from a real server. They cover each guarantee above, the release check, and the shell lifecycle, built from the realshell.created, notice and DELETE frames. They also cover every case raised in review:throwWithBarrieralso covers.FRO_BOT_OPENCODE_V2_BIN, so CI skips them. They cover:executecall with a nested approval answered by request id, through to completioncheck-types,lint,build,test:scripts