Skip to content

feat(desktop): codex app-server driver, and the input shapes codex rejects (L4c) - #571

Merged
physercoe merged 2 commits into
mainfrom
feat-vision-l4c-codex-driver
Aug 16, 2026
Merged

feat(desktop): codex app-server driver, and the input shapes codex rejects (L4c)#571
physercoe merged 2 commits into
mainfrom
feat-vision-l4c-codex-driver

Conversation

@physercoe

Copy link
Copy Markdown
Owner

Vision-parity L4c — the codex driver. L4a chose the rung, L4b opened the byte path, and neither was imported by main.ts. This is what makes the lane user-visible: pick codex in the Companion's local picker and the dock drives a real app-server thread.

Continues #568 (L4a) and #569 (L4b). Plan: docs/plans/desktop-companion-vision-parity.md, lane L.

What ships

  • localagent/codexwire.ts — pure JSON-RPC shapes: thread params, turn input, server-request classification, per-method response shapes.
  • localagent/codexdriver.ts — the session: handshake, queued input, turn/interrupt, delta throttling, parked approvals, resume.
  • localagent/driver.ts — the engine-neutral vocabulary (posture, input kinds, DriverEvent, LocalDriver), moved out of claudewire.ts. With two engines, a type named after claude that codex has to import is a name that lies to the next reader.
  • service.ts picks a driver per family and routes rebind by resume mechanism.

Notifications go through the L2 interpreter with the hub's own codex frame profile, already shipped in agent_families.generated.json — so a local codex transcript and a hub-driven one are the same rows. No second vocabulary.

★★ Two of the shapes we ship are rejected by codex

Measured against a live codex-cli 0.147.0:

sent answer
{type:"input_image", image_url:"data:…"} -32600 Invalid request: unknown variant 'input_image', expected one of 'text', 'image', 'localImage', 'audio', 'localAudio', 'skill', 'mention'
{type:"input_file", file_data:"data:…"} the same error
{type:"image", url:"data:…"} accepted — the model described the pixel

Those are OpenAI responses-API content types, not app-server ones. This is not only the new driver's problem: the hub's AppServerDriver.startTurn has sent both since W4.3, so E4's image passthrough has never reached a codex M2 agent, and a PDF was worse than a no-op — an unknown variant fails the whole turn/start, taking the director's message with it. Fixed here, with the strip-and-warn shape driver_exec_resume.go already uses for what gemini cannot carry.

The registry asserted it too: codex.prompt_pdf.M2 was true (what the F3 composer gate reads) and is now false — 0.147.0's UserInput union has no file variant at all. prompt_image.M2 stays true; only the shape was wrong.

Why a green suite missed it: the Go test's fake app-server accepts any params, so it pinned the shipped shape rather than the protocol — a fixture that cannot disprove the rule it tests. Hence an opt-in live e2e.

The plan line that was wrong

"resume argv from the N1 table" names a rung this driver does not use — and N1's own table says so: the codex family row reads mechanism: appserver_thread_resume, with a note that the codex resume <id> CLI recipe is a different rung. service.rebind now routes by mechanism: argv families get a splice, the rest hand their handle to a driver that knows what to do with it.

Other measurements the driver is built on

  • thread/resume restores memory and emits no replay — the only notifications after it are configWarning, remoteControl/status/changed, thread/status/changed, thread/tokenUsage/updated, thread/goal/cleared. Same shape as claude's --resume, so L3b's durable log restores the view here too. (Its response does carry thread.turns[], unlike claude's — recorded, not used.)
  • sandbox: read-only + approvalPolicy: never is a real write barrier. Asked to create a file in its cwd, codex tried, failed, said "the environment is read-only", and nothing appeared on disk. The response echoes {"type":"readOnly","networkAccess":false}. converse cannot be kept exactly (codex has no tool-disable switch), so the lifecycle row carries a posture_note rather than implying a boundary that is not there.

Approvals, with no attention table

Server requests become approval_request events in R1's own shapes, answered by the cards that already ship — which is exactly what R1's comment predicted. Four kinds are refused at once rather than parked, because a card nobody can answer holds the engine open forever: a form-mode elicitation, a url elicitation, item/permissions/requestApproval (its response is a granted profile, not a verdict), and — the one worth naming — a question codex marks isSecret, since service.input() records every input into the durable on-disk transcript before sending it. Each refusal uses that method's own response shape; an empty {} fails to deserialize on every one of them.

Config seeding rides thread/start.config, so configuring a session never leaves a .codex/config.toml in the director's project.

Verification

  • 716 electron tests, 0 fail, 6 skipped (the two live e2e skip without their env var). 49 new.
  • go build ./..., go vet ./..., go test ./... — all clean.
  • CI's full lint set clean, including lint-openapi.sh and lint-desktop-tokens.sh.
  • Live e2e passes against a real codex app-server (TERMIPOD_CODEX_DRIVER_E2E=1): a real turn round-trips with streamed partials, thread/resume finds the thread and the engine remembers, and an image reaches the model.

The e2e earned itself on its first run by failing two of my assertions: one asserted a streamed partial on a reply too short to produce one (a race), and the other asserted the wrong colour — the widely-copied 1×1 "red dot" PNG is RGBA(255,0,0,127), and a half-transparent pixel gets described differently depending on what it is composited against. It now uses an opaque pure-green pixel, since red and blue are what a model guesses when it cannot see the image at all.

Left in the lane, named rather than skipped

No posture maps to approvalPolicy: on-request, so codex's own command/file-change gates never fire from a Companion session today. Adding one is a posture decision, not a driver change — the approval path is built and tested, and MCP elicitations already exercise it.

🤖 Generated with Claude Code

Ubuntu and others added 2 commits August 16, 2026 14:48
…jects (L4c)

L4a chose the rung and L4b opened the byte path; neither was imported by
main.ts. This is the driver that makes the lane user-visible: pick `codex`
in the Companion's local picker and the dock drives a real app-server
thread — streaming as it writes, folding tool calls the way a claude
session does, taking approvals inline, and reattaching after a restart.

Notifications go through the L2 interpreter with the hub's OWN codex frame
profile, which already ships to the desktop in agent_families.generated.json,
so a local codex transcript and a hub-driven one are the same rows.

Measured against a live codex-cli 0.147.0, not read off documentation — and
two of the measurements contradict what we ship:

  * `{type:"input_image", image_url:…}` and `{type:"input_file", file_data:…}`
    are answered `-32600 unknown variant`. They are OpenAI responses-API
    content types, not app-server ones. The hub's AppServerDriver has been
    sending both since W4.3, so E4's image passthrough has never reached a
    codex M2 agent, and a PDF was worse than a no-op: an unknown variant
    fails the whole turn/start, losing the director's message with it.
    Images now use `{type:"image", url:"data:…"}` (confirmed end to end —
    the model described the picture); PDFs are stripped with a system row,
    the strip-and-warn shape driver_exec_resume.go already uses.
    `codex.prompt_pdf.M2` in the registry was `true` and is now `false`:
    0.147.0's UserInput union has no file variant at all.

  * The plan's "resume argv from the N1 table" names a rung this driver
    does not use, and N1's own table says so — the codex family's mechanism
    is `appserver_thread_resume`. `service.rebind` now routes by mechanism:
    argv families get a splice, the rest hand their handle to the driver.

Why a green suite missed the first one: the Go test's fake app-server
accepts any params, so it pinned the shipped shape rather than the protocol.
Hence an opt-in live e2e (TERMIPOD_CODEX_DRIVER_E2E=1) that runs real turns
against a real app-server — it caught a wrong assertion of my own on the
first run.

Also here:

  * Posture is measured, not named. `read_local` lowers to
    `{sandbox:"read-only", approvalPolicy:"never"}`, probed by asking codex
    to write a file: it tried, failed, said so, and nothing landed on disk.
    `converse` cannot be kept exactly (codex has no tool-disable switch), so
    the lifecycle row carries a posture_note instead of implying a boundary
    that is not there.
  * Approvals become `approval_request` events in R1's own shapes, answered
    by the cards that already ship — which is what R1's comment predicted.
    Four kinds are refused AT ONCE rather than parked, because a card nobody
    can answer holds the engine open forever; the one worth naming is a
    question codex marks `isSecret`, since service.input() records every
    input into the durable on-disk transcript before sending it.
  * Config seeding rides `thread/start.config`, so configuring a session
    never leaves a .codex/config.toml in the director's project.
  * The engine-neutral half of claudewire.ts moves to driver.ts: with two
    engines, a type named after claude that codex must import is a name that
    lies to the next reader.

Verify: 716 electron tests (0 fail, 6 skipped — the two live e2e skip
without their env var), go build/vet/test ./... clean, CI's full lint set
clean, and both live e2e tests pass against a real codex app-server.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…unrace the image e2e

The rebind guard in the approval request id never guarded: `#epoch` was
bumped once per driver instance (there is no reconnect path inside one), so
every id was `codex-1-<n>` — and a rebind is a NEW instance counting from
the same start. The server's JSON-RPC id counter restarts with every
connection, and `#onClose` leaves parked asks unanswered, so the collision
lands exactly where it matters: a card left dangling by a dead engine could
answer whatever request reached the same number after the rebind — approving
a command the director never saw. The prefix is now a per-instance random
token (transcripts outlive the process, so a counter shared across instances
would still collide across app restarts); a stale click lands as
`codex_answer_unmatched`, pinned by a two-instance test.

The image e2e asserted /green/ on the FIRST completed text, but a turn is
free to open with a preamble ("i'll inspect the image…") before the answer —
it did, on this box. The wait now looks for a completed text that names the
colour, timing out with every collected text in hand — the same lesson the
e2e's other two assertions already record.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@physercoe
physercoe merged commit f35e03e into main Aug 16, 2026
8 checks passed
@physercoe
physercoe deleted the feat-vision-l4c-codex-driver branch August 16, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants