Skip to content

refactor(gateway): add the OpenCode V2 run loop - #1985

Merged
marcusrbrown merged 9 commits into
mainfrom
feat/opencode-v2-gateway
Oct 29, 2026
Merged

marcusrbrown merged 9 commits into
mainfrom
feat/opencode-v2-gateway

Conversation

@marcusrbrown

@marcusrbrown marcusrbrown commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

This adds the gateway's OpenCode V2 run loop (#1805). Each run picks V1 or V2 from the opencodeVersion the workspace reports on /readyz:

  • missing → V1
  • major version 2 or higher → V2
  • unparseable → not ready

No current image reports a V2 version, so every run stays on V1 until a V2 image is deployed. That deploy is the switch.

Run loop

The V2 modules live next to the V1 ones in packages/gateway/src/execute/:

  • run-core-v2
  • root-stop-v2
  • drain-completion-v2
  • settle-owned-sessions-v2
  • repo-quiescence-v2
  • permission-reply-v2
  • question-client-v2
  • opencode-attach-v2
  • provider-failure-v2
  • opencode-generation

The runtime adds the V2 remote and question clients and the directory quiescence check. V1's behaviour doesn't change. run-core.ts gains two exports. The shared ownership ledger gains an additive release op that V1 never calls, and V1 reconcile skips shell: keys.

The guarantees V1 has carry over:

  • Root stop. Every early stop interrupts the root. A stop counts as confirmed only when there's a terminal assistant reply to the latest turn and the release check below passes. An {interrupted:false} reply never counts as a stop.
  • Event stream. The loop subscribes eagerly and waits for server.connected before sending the prompt. It never reconnects. Any of these goes through the barrier:
    • a second server.connected
    • a lost stream
    • an unparseable data frame
    • 30 s of silence
  • Background work and drain.
    • Reconcile passes are bounded, signalled and guarded.
    • The drain gate and waitReason are kept.
    • An owned child's error marks that child unknown.
    • An owned session that moves location fails the run and always quarantines.
    • A notice that can't be read fences its child or shell.
  • Approvals. They're keyed by request id, including nested asks inside one execute, and fail closed during the barrier.
  • Questions. V2 has no question route, so a question arrives as a form and is answered through the form reply route. A form of an unknown kind is cancelled with a notice.
  • Replies. Reasoning text never reaches the reply. provider.auth errors classify as auth errors whatever the provider's message says.

Release invariant

A V2 run releases the checkout, and the lock with it, only after the run's directory reads as quiet 11 times in a row, at least 500 ms apart and at least 5 s overall. Any read that fails or times out resets the count. Quiet means all three of these:

  • no session in GET /api/session/active stored under the directory, including children, grandchildren and resumed descendants;
  • no running shell in GET /api/shell for the directory, whoever started it;
  • no queued item in GET /api/session/:id/inbox for this run's root, its owned sessions, or any session under the directory updated in the last 10 minutes.
    • An inbox admit bumps time.updated (core/src/session/projector.ts), so older sessions can't hold a pending wake.
    • The list walk stops at that cutoff, so the check's cost doesn't grow with stored session history.

Success and stop both pass through the same check. That includes a run that never adopted anything.

On stop. The loop repeats three steps until the check passes within the barrier budget:

  1. DELETE every running shell in the directory.
  2. Interrupt every active session in the directory, starting with the root.
  3. Read the directory again.

If the check doesn't pass in time, the run quarantines. The quarantine watch repeats the same kill-and-interrupt for the whole hold.

The rule is directory-wide on purpose. A gateway checkout belongs to the gateway, so any session or shell found in it is treated as the run's.

What this rule doesn't prove. Stock OpenCode has unbounded gaps between a shell exiting and its notice, between a notice being admitted and the wake, and inside notification producers. A producer that pauses for more than 5 s and then wakes a session would slip past. Quarantine on any unconfirmed stop and the next run's acquire check narrow this, but don't close it. The real fix is a location close-and-drain fence upstream (anomalyco/opencode#60286). The same gap appears in anomalyco/opencode#60107: an interrupt doesn't stop background shells.

Background shells

A background shell job runs on the server and isn't a session.

  • Adoption. The loop adopts one from shell.created, or from a shell tool result that moved to the background, when the owner is the root or an owned child. Its ledger key is shell:<id>. A foreground shell is released when its tool call ends.
  • Settling. A shell settles on its completion notice, or when reconcile shows it has stopped.
  • Waiting. The drain waits for owned shells, so the agent can background a build, end its turn, and pick up again when it's woken.
  • Never-ending shells. A shell that never exits keeps the run open until the wall-clock deadline.
  • Not yet owned. A shell started from a foreground subagent's own session isn't owned yet. The release check still sees it in the directory and kills it.

Verification

  • Ported behaviour tests. The tests are ported from V1's run-core* suites onto V2 fixtures built from frames captured from a real server. They cover each guarantee above, the release check, and the shell lifecycle, built from the real shell.created, notice and DELETE frames. They also cover every case raised in review:
    • a settled child woken again by its shell's notice
    • a shell found through REST that disappears before its wake
    • a DELETE that committed but whose response was lost
    • an enqueued notice whose wake is deferred
    • a final sweep that times out
    • a late-admitted prompt
    • a successor root turn woken by a cancel
    • CRLF delimiters split at every chunk boundary
    • a large stored history (5000 sessions, 3 recent) that reads as quiet using 3 inbox reads
  • Revert-checks. Every production change across the loop, the review fixes, shell ownership and the release check has one. Each mutation is caught, apart from one in a close guard that throwWithBarrier also covers.
  • End-to-end against stock v2.2.0, digest-verified. These tests are gated on FRO_BOT_OPENCODE_V2_BIN, so CI skips them. They cover:
    • an execute call with a nested approval answered by request id, through to completion
    • a backgrounded shell that completes after its notice and the turn that notice woke
    • a deadline stop that kills the shell, so its marker is never written
  • Gates. These pass on the branch, rebased onto current main:
    • check-types, lint, build, test:scripts
    • runtime (2144), gateway (6580), action (2711), opencode-plugin (62)

The gateway reaches a workspace OpenCode V2 server through the proxy, so it
needs the V2 siblings of remote-client and remote-question-client: bearer
headers, the permission/form reply routes, interrupt, session reads and the
SSE subscription with its heartbeat.

The event mapper now carries the inbox id on a subagent notice and maps
session.inbox.delivered, so a drain gate can tell a queued notice from one the
parent has taken into a turn. A replied form carries its answer for the echo.
Adds the V2 sibling of the gateway run core and its collaborators: root stop
and quarantine watch, repo quiescence over the active-list join, the drain gate
(notice delivered, root idle in the notice's generation, root not live),
owned-session settlement, approval replies keyed by request id, question forms
mapped onto the existing question flow, and provider-failure labelling.

run.ts and program.ts pick V1 or V2 from the opencodeVersion the workspace
reports on /readyz (major >= 2 is V2). A version that is present but unreadable
selects neither loop and counts as not ready. V1 behaviour is unchanged.

Tests run the real loop against captured v2.2.0 frames. An end-to-end test, gated
on FRO_BOT_OPENCODE_V2_BIN, drives one prompt with a Code Mode call and an
approval through the real stock server.
…able notices and frames, track child inbox delivery
A background `shell` is a server job, not a session: `session.active` never
lists it, an interrupt does not stop it, and it outlives the run's lock while
its completion notice later wakes the root. The V2 loop now treats it like a
background subagent.

- Mapper: `shell.created`/`shell.exited` and the job notice (`source: shell`
  with a `jobID`; a user command's notice has none and is ignored) map to
  `shell_started`/`shell_exited`/`shell_notice`. An unreadable job notice
  names its shell so the loop can fence it.
- Ledger: shells share the ownership ledger under a `shell:<id>` key space.
  Adopted on `shell.created` and on a `running` `shell` tool result; settled
  on the notice; a foreground shell is released when its call ends. Reconcile
  settles a shell absent from `GET /api/shell` that reads stopped or 404, and
  keeps it unknown on any other read error.
- Drain: an outstanding shell blocks admission, and its notice is observed and
  delivered like a child's.
- Stop paths: before the root abort, every owned running shell is DELETEd and
  confirmed gone by re-listing. An unconfirmed kill keeps the stop unconfirmed;
  the quarantine watch keeps deleting through the hold.
- Quiescence: the repo is not clear while any shell runs in its directory.
- Remote client: `listShells`, `getShell`, `deleteShell`.
…can still wake the root

- A stop that deleted any shell, tracked or not, never confirms the root: the
  DELETE's error notice wakes it. Any stop with a live owned shell therefore
  quarantines, even after a clean kill (accepted cost).
- A final sweep after the root confirms deletes or reports a shell started
  during the barrier, and leaves the stop unconfirmed.
- Follow-up work stays open until an adopted child's or shell's notice (or its
  fence) has been observed, not until reconciliation settles the entry.
- The drain commit point reads the running owned shells from the server before
  the commit and does not admit while one runs or the list cannot be read.
- A fatal move of an owned session is never confirmed.
- The shell kill waits six rounds of 500 ms.
@marcusrbrown
marcusrbrown requested a review from fro-bot as a code owner October 29, 2026 16:13

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CONDITIONAL

The V2 lifecycle barriers have gaps that can release checkout ownership while adopted work can still execute.

Blocking issues

Priority Location Issue and required change
P1 drain-completion-v2.ts:240–244 An awakened child can still be executing when drain admits. Adopted child C can finish its job while its background shell S remains running. S's completion wakes C; delivery clears the notice fence, but C's new execution is ignored and liveness checks only the root. A subsequent root success can release the checkout while C writes. Require quiescence of every owned session, including settled children, and include resumed descendants in teardown.
P1 drain-completion-v2.ts:270–279 REST-discovered shells lose their completion obligation. When shell creation and tool-result events precede child adoption, the shell is untracked. A listing blocks admission while it runs, but disappearance permits admission before its completion notice and owner follow-up. Retain discovered shell obligations across listings; absence must not substitute for notice/delivery/completion evidence. The test at drain-completion-v2.test.ts:579–591 currently expects this unsafe admission.
P1 shell-jobs-v2.ts:243–268 A committed DELETE with a lost response is treated as producing no wake. For an unadopted owned shell, DELETE can remove it and schedule its notice while returning a network error. An empty confirming list then returns confirmed:true, deleted:[]; settlement derives shellWake=false and can release without quarantine. Track potentially committed deletion attempts separately from acknowledged successes, and preserve their follow-up uncertainty.
P2 run-core-v2.ts:506–510 Notice receipt prematurely closes the stop barrier's follow-up window. Enqueued notices immediately settle entries and count as accounted for. If the root fails before drain, followUpPossible can become false. Upstream synthetic admission precedes a separate lookup/wake, so a producer paused between admission and wake can restart the root after interruption and confirmation. Keep queued/in-flight notice consumption in the barrier predicate independently of drain state.
P2 settle-owned-sessions-v2.ts:274–284 A timed-out final sweep retains stale confirmation flags. After initial shell/root confirmation, a final sweep can discover a late unadopted shell and stall. Timeout then reports both unconfirmed flags false for an empty/settled ledger. The run quarantines, but run-core-v2.ts:528 skips its cancellation watch, leaving no retries against surviving shells or root wakes. Invalidate shell confirmation before awaiting the final sweep; restore it only after a completed confirming result.

Non-blocking concerns

Priority Location Concern
P2 remote-client-v2.ts:228 CRLF normalization is chunk-local. A valid SSE delimiter split across chunks can remain unrecognized, dropping/buffering frames. Parse lines incrementally or normalize with cross-chunk CR handling. A minimal bun -e probe using the actual client and mocked ReadableStream confirmed: intact CRLF delivered one frame; splitting its delimiter across three chunks delivered none.
P2 provider-failure-v2.ts:40 V2 provider.auth is passed unchanged to a classifier recognizing ProviderAuthError. With an ordinary message such as bad key, authentication becomes an unclassified session error. Translate structured V2 discriminants explicitly instead of relying on message hints.

Missing tests

  • An adopted, settled child resumes from its shell notice and remains active when the root succeeds.
  • A REST-discovered shell disappears before its notice/follow-up; revise the existing test that admits on disappearance alone.
  • DELETE commits but returns a network error, followed by an empty shell list.
  • An enqueued notice's producer reaches its deferred wake after a pre-drain failure barrier interrupts/confirms the root.
  • The second/final shell sweep discovers work and exceeds the settlement deadline; assert quarantine monitoring starts.
  • CRLF delimiters split at every chunk boundary, and provider.auth classification with arbitrary message wording.

Risk assessment (HIGH): likelihood of regression, security exposure, or blast radius

The V2 path can admit completion or confirm termination before owned writers are quiescent, risking concurrent checkout mutation. Versionless workspaces continuing on V1 limits immediate rollout exposure. No credential disclosure or authorization bypass was identified.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 40909378263
Cache hit
Session ses_e778d76bfffeJk5X44iYVa8hDB

Reviewed the supplied 56-file scope against origin/main; independently source-checked blocking findings. No build, lint, typecheck, or test suites were run. One minimal bun -e transport probe was executed as described above. The skill's structural-validation helpers were inaccessible under the environment's directory policy.

The response-file protocol supports a review body, not inline-comment payloads; file-specific findings are linked to exact lines above.

A V2 run is released by its directory, not by its own accounting. Stock
v2.2.0 gives no way to prove that nothing the run started can still wake a
session: a shell's DELETE wakes its owner, a notice is admitted and the wake
starts in a separate step, and an interrupt reply is a delivery receipt.

The checkout is released after the directory shows no active session, no
running shell and no queued inbox item for 11 readings 500 ms apart. Every
success waits for that window, and a stop repeats kill, interrupt and read
until it passes or the bound ends it, then quarantines. A move of an owned
session always quarantines.

A reading is bounded so its cost does not grow with the workspace's history:

- Active sessions are placed with GET /api/session/:id, not by listing the
  directory.
- Inboxes are read for the run's root, the sessions it owns, and sessions
  updated since the run began less 10 minutes. Admitting an inbox item bumps
  time.updated, so a notice queued during the run always lands inside the
  bound. The list is read newest-first and the walk stops at the cut-off, so
  the page bound applies to recent sessions only.
- The acquire check uses the same placement of active sessions.

Also folds in the provider.auth and CRLF fixes and the documented relaxed
invariant in packages/gateway/AGENTS.md.
expect(busy).toMatchObject({passed: false, streak: 0})
expect(after.streak).toBe(1)
// and the full window is needed again
let state = after

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CONDITIONAL

The previous findings are addressed within the documented quiescence model. Four issues remain in the revised release and cancellation paths.

Blocking issues

Priority Location Issue and required change
P1 drain-completion-v2.ts:301–305 A rejected final reading leaves the old quiet window reusable. If root activity and fresh idle arrive during the final observation, the wait returns passed without rechecking its predicate. Revision drift correctly rejects admission, but never resets the gate. The retry then admits after one quiet reading, without five seconds following the new activity. Recheck invalidation after every awaited observation, including the passing one, and reset obsolete evidence before retrying. Reproduced with the actual drain gate.
P1 root-stop-v2.ts:81–99 A stalled shell DELETE starves session interrupts throughout quarantine. All DELETE responses are awaited before any interrupt. Observation, deletion and interruption share the watch's five-second signal; when DELETE consumes that budget, subsequent interrupts receive an already-aborted signal. Every tick repeats this, leaving a responsive root/child running. Independently bound deletion and reserve a usable budget for session interrupts, retaining interruption after deletion-induced wakes.
P2 directory-quiescence-v2.ts:298–301, 117–129 Incomplete observations discard known cancellation targets. A successful shell/active read is replaced with empty activity when another branch times out. Separately, one failed lookup from the global active list discards every successfully located session—including this checkout's root/children. Barrier and watch consequently omit otherwise-healthy DELETE/interrupt requests. Preserve completed positive evidence on timeout and successful placements alongside lookup failures; keep release classified unreadable.
P2 settle-owned-sessions-v2.ts:150–158 A moved owned session escapes interruption. V2 supports moving an active execution and continuing the same session at its destination. Placement then excludes it from both the original-directory barrier and quarantine watch. Forced quarantine occurs, but the known live session is never interrupted. Retain moved owned IDs as explicit teardown/watch targets independent of directory placement. The current movement test masks this because its getSession() fixture always returns the original directory.

Non-blocking concerns

None

Missing tests

  • Hold the final directory observation, inject root activity and fresh idle, resolve it quiet, then assert the retry requires an entirely fresh window. Existing invalidation coverage changes evidence between readings.
  • Combine a hanging DELETE with healthy root/child interrupt endpoints; assert interrupts are delivered with non-aborted signals during quarantine.
  • Preserve and act on a known shell/session when another observation branch times out; also cover mixed successful placements and an unrelated failing session lookup.
  • On an active root/child move, update its returned directory to the destination and verify interruption there during teardown and quarantine.

Risk assessment (HIGH): likelihood of regression, security exposure, or blast radius

The stale streak can shorten the release barrier, while degraded-read and movement cases can leave canceled executions running through quarantine. Versionless workspaces remaining on V1 limits immediate rollout exposure. The documented upstream producer-delay limitation is separate from these findings.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 40925310161
Cache hit
Session ses_e778d76bfffeJk5X44iYVa8hDB

Re-reviewed 2a52aa0..65aa2e4 and checked prior feedback. No build, lint, typecheck, or test suites were run. Blocking findings were independently source-validated.

One minimal bun -e probe exercised the actual drain gate with a shortened three-reading window and activity during its final observation. It returned reads:4, revision-drift, then admission—rather than requiring three fresh readings after invalidation. Command:

bun -e 'import {createDrainCompletionV2} from "./packages/gateway/src/execute/drain-completion-v2.ts"; import {createDirectoryStabilityGate} from "./packages/runtime/src/agent/directory-quiescence-v2.ts"; import {createOwnershipLedger} from "./packages/runtime/src/agent/ownership-ledger.ts"; let reads=0; let drain; const reasons=[]; let admit; const done=new Promise(resolve=>{admit=resolve}); const gate=createDirectoryStabilityGate({requiredObservations:3,minSpacingMs:0,minSpanMs:0,observe:async()=>{reads++; if(reads===3){drain.noteRootActivity();drain.noteRootIdle();drain.requestValidation()} return {kind:"quiescent"}}}); drain=createDrainCompletionV2({rootSessionId:"root",ledger:createOwnershipLedger(),adapter:{liveSessionIds:async()=>({success:true,data:new Set()}),children:async()=>({success:true,data:[]})},directoryGate:gate,directorySpacingMs:0,signal:new AbortController().signal,logger:{debug(){},info(){},warn(){},error(){}},onNotAdmitted:reason=>reasons.push(reason),onAdmitted:()=>admit()}); drain.noteRootIdle(); drain.beginDrain(); drain.requestValidation(); const timer=setTimeout(()=>{drain.dispose();throw new Error("probe timed out")},2000); try {await done; process.stdout.write(JSON.stringify({reads,reasons,streak:gate.state().streak}));} finally {clearTimeout(timer);drain.dispose()}'

The response-file protocol supports a review body rather than inline-comment payloads; findings link to exact source lines above.

…op targets under degraded reads

- The quiet-window wait asks whether its evidence is still current after every
  reading, the passing one included, and drops the streak when it is not. Root
  activity and a fresh idle that landed during the last reading used to leave
  the old streak for the retry to reuse.
- A stop gives its shell DELETEs and its session interrupts bounds of their
  own, derived from the barrier's or the hold's life instead of the reading's.
  A DELETE that never answers no longer leaves the interrupts with a dead
  signal on every quarantine tick. Interrupts are issued together, root first.
- An unreadable reading keeps the positive evidence its reads found (shells,
  active sessions placed in the directory, queued inboxes), and one failed
  placement lookup no longer discards the placements that succeeded. The
  release still counts it as unreadable; the barrier and the watch act on it.
- An owned session that moved is an explicit target of the barrier and the
  watch: while it is on the active list it is interrupted wherever the server
  stores it now. A move still always quarantines.

@fro-bot fro-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS

All four outstanding findings are addressed with targeted regression coverage: fresh quiet-window retries, independent stop budgets, retained partial cancellation targets, and moved-session interruption through quarantine. No new blocking issues found.

Blocking issues

None

Non-blocking concerns

None

Missing tests

None

Risk assessment (MED): likelihood of regression, security exposure, or blast radius

The V2 lifecycle remains a substantial asynchronous change with the documented upstream producer-delay limitation. Current versionless workspace images continue selecting V1, limiting immediate rollout exposure.


Run Summary
Field Value
Event pull_request
Repository fro-bot/agent
Run ID 40950595529
Cache hit
Session ses_e778d76bfffeJk5X44iYVa8hDB

Re-reviewed 65aa2e4..d82eee2, reconciled previous findings, and inspected the new regression tests and integration paths. Validation was source-based; no tests, build, lint, or type-check commands were run. Repository files remain unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants