Skip to content

feat(desktop): live command output + agent-produced media (R4) - #565

Merged
physercoe merged 1 commit into
mainfrom
feat-vision-r4-live-output-media
Aug 16, 2026
Merged

feat(desktop): live command output + agent-produced media (R4)#565
physercoe merged 1 commit into
mainfrom
feat-vision-r4-live-output-media

Conversation

@physercoe

Copy link
Copy Markdown
Owner

Last wedge of vision-parity W3. Consumes E3 (#563) and E4 (#564): a running command's output now appears while it runs, and an image a tool returns renders as an image.

What was actually broken

E3's streamed output was already reaching the client and being dropped by the renderer. useToolMaps folds tool_call_update onto its parent call (latest-wins on toolCallId), but ToolCallBody took only (p, result) — the folded update had no way in — and isToolCallUpdateHidden suppresses the standalone card whenever a visible parent exists. So the output had nowhere to render on either path.

ToolResultBody did str(content) ?? jsonText(content), so an image-bearing result printed a screen of base64.

And blob: refs were skipped with a comment calling them "a future hub". payload_externalize.go already swaps every string leaf over 64 KiB for a blob:sha256/ ref on ingest (handlers_agent_events.go:118) — so for any real screenshot the blob path is the normal one, and images were vanishing at exactly the size where they mattered.

Three of the plan line's five specifics were wrong

Recorded in the plan, because the corrections are the wedge:

  1. "fold via streamingPartials.ts" — no. That module is in ui/ not state/, its FOLD_KINDS is text|thought|plan, and its chains key on message_id; a tool_call_update carries toolCallId and no message_id. Nothing needed folding.
  2. "MCP image blocks" — only half. claude M2 forwards claude's own tool_result content verbatim (driver_stdio.go:446), i.e. the Anthropic dialect source:{type:"base64",media_type,data}. Measured by running claude --print --output-format stream-json over a PNG rather than recalled — it returns a LIST, with is_error absent rather than false. Both dialects render.
  3. "termipod-att:// refs" — a category error. Tauri-only note plumbing for the Read surface (state/attachments.ts:200); no agent event has ever carried one.

Two consequences the plan could not have anticipated, both silent-failure shaped:

  • the MIME must come from the event block, since the hub stores an externalized leaf as application/octet-stream — using the blob's mime paints nothing;
  • the fetched body is base64 text, so it needs one decode. getBlobDataUrl would double-encode it. No client had ever resolved an externalized payload leaf (mobile's blob code is all artifact viewers, a different case).

A url image source is deliberately not painted — it would make the renderer fetch a host the agent chose.

Also fixed

A pre-existing Companion bug found while threading the new prop: result and callName were handed to the wrong branches, so the dock's call cards folded in nothing and every standalone result was labelled "Result". The full transcript surface was always correct.

Verification

  • 17 new pure-module tests; fixtures are captured producer output, not invented shapes.
  • 9 mutations checked, all killed. One survived first and exposed a test passing for the wrong reason — a url source was being rejected for carrying no data, not by the source-type guard. Now pinned by a block claiming both.
  • 876 frontend tests, tsc, vite build, token ratchet (from root), and the full doc lint set incl. openapi — all green.

Not visually verified — no display on this machine, and the E2E suite does not exercise the transcript. Adds to the standing render debt (#166).

⚠️ Touches the same plan status block as #563/#564 did, so expect a doc conflict if another vision-parity PR lands first.

Two halves of one gap: the transcript could describe what a tool did but
never show it. A long build sat as a silent spinner until it exited, and
an image a tool returned printed as a screen of base64.

E3's streamed output was already reaching the client and being dropped by
the renderer: `useToolMaps` folds `tool_call_update` onto its parent call
(latest-wins on toolCallId), but `ToolCallBody` took only `(p, result)`,
and `isToolCallUpdateHidden` suppresses the standalone card whenever a
visible parent exists. The fold was never the missing piece — the plan's
"fold via streamingPartials.ts" was wrong twice over (that module is in
ui/, its FOLD_KINDS is text|thought|plan, and its chains key on
message_id, which a tool_call_update does not carry). The live block
stands down when the tool_result lands, because codex's finalizing result
carries `aggregatedOutput` — measured during E3 as byte-identical to the
reassembled deltas.

Images render in both dialects our engines actually emit. claude M2
forwards claude's own tool_result content verbatim (driver_stdio.go:446),
which is Anthropic's `source:{type:"base64",media_type,data}` — measured
here by running `claude --print --output-format stream-json` over a PNG,
not assumed. The MCP/ACP shape (`mimeType`+`data`) is the other. A `url`
source is deliberately not painted: it would make the renderer fetch a
host the agent chose.

`blob:sha256/` refs are resolved rather than skipped. The old comment
called them "a future hub"; payload_externalize.go already swaps every
string leaf over 64 KiB for a ref on ingest, so for any real screenshot
the blob path is the normal one and images were vanishing at exactly the
size where they mattered. Two consequences: the MIME must come from the
event block (the hub stores the leaf as application/octet-stream), and
the fetched body is base64 TEXT, so it needs one decode — getBlobDataUrl
would double-encode it. No client had ever resolved an externalized
payload leaf; mobile's blob code is all artifact viewers.

`termipod-att://`, the plan's fourth item, is Tauri-only note plumbing
for the Read surface and has never ridden an agent event. Out of scope.

Also fixes a pre-existing Companion bug found while threading the new
prop: `result` and `callName` were handed to the wrong branches, so its
call cards folded in nothing and every standalone result was labelled
"Result". The full transcript surface was always correct.

Verification: 17 new pure-module tests (fixtures are captured producer
output, not invented shapes); 9 mutations checked, all killed — one
survived first and exposed a test passing for the wrong reason (a url
source was rejected for carrying no `data`, not by the source-type
guard), now pinned by a block claiming both. 876 frontend tests, tsc,
vite build, token ratchet, and the full doc lint set all green.

NOT visually verified — no display here, and the E2E suite does not
exercise the transcript. Adds to the standing render debt (#166).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@physercoe
physercoe merged commit db8ce27 into main Aug 16, 2026
8 checks passed
physercoe pushed a commit that referenced this pull request Aug 16, 2026
Both PRs edited the vision-parity status block and the desktop changelog's
Unreleased section. The status line now records both wedges: R4 shipped (W3
complete) and F4 shipped (first W4 wedge), leaving L3c as the only remaining
item. Both changelog entries kept, F4 above R4 in land order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant