feat(desktop): live command output + agent-produced media (R4) - #565
Merged
Conversation
Two halves of one gap: the transcript could describe what a tool did but
never show it. A long build sat as a silent spinner until it exited, and
an image a tool returned printed as a screen of base64.
E3's streamed output was already reaching the client and being dropped by
the renderer: `useToolMaps` folds `tool_call_update` onto its parent call
(latest-wins on toolCallId), but `ToolCallBody` took only `(p, result)`,
and `isToolCallUpdateHidden` suppresses the standalone card whenever a
visible parent exists. The fold was never the missing piece — the plan's
"fold via streamingPartials.ts" was wrong twice over (that module is in
ui/, its FOLD_KINDS is text|thought|plan, and its chains key on
message_id, which a tool_call_update does not carry). The live block
stands down when the tool_result lands, because codex's finalizing result
carries `aggregatedOutput` — measured during E3 as byte-identical to the
reassembled deltas.
Images render in both dialects our engines actually emit. claude M2
forwards claude's own tool_result content verbatim (driver_stdio.go:446),
which is Anthropic's `source:{type:"base64",media_type,data}` — measured
here by running `claude --print --output-format stream-json` over a PNG,
not assumed. The MCP/ACP shape (`mimeType`+`data`) is the other. A `url`
source is deliberately not painted: it would make the renderer fetch a
host the agent chose.
`blob:sha256/` refs are resolved rather than skipped. The old comment
called them "a future hub"; payload_externalize.go already swaps every
string leaf over 64 KiB for a ref on ingest, so for any real screenshot
the blob path is the normal one and images were vanishing at exactly the
size where they mattered. Two consequences: the MIME must come from the
event block (the hub stores the leaf as application/octet-stream), and
the fetched body is base64 TEXT, so it needs one decode — getBlobDataUrl
would double-encode it. No client had ever resolved an externalized
payload leaf; mobile's blob code is all artifact viewers.
`termipod-att://`, the plan's fourth item, is Tauri-only note plumbing
for the Read surface and has never ridden an agent event. Out of scope.
Also fixes a pre-existing Companion bug found while threading the new
prop: `result` and `callName` were handed to the wrong branches, so its
call cards folded in nothing and every standalone result was labelled
"Result". The full transcript surface was always correct.
Verification: 17 new pure-module tests (fixtures are captured producer
output, not invented shapes); 9 mutations checked, all killed — one
survived first and exposed a test passing for the wrong reason (a url
source was rejected for carrying no `data`, not by the source-type
guard), now pinned by a block claiming both. 876 frontend tests, tsc,
vite build, token ratchet, and the full doc lint set all green.
NOT visually verified — no display here, and the E2E suite does not
exercise the transcript. Adds to the standing render debt (#166).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
physercoe
pushed a commit
that referenced
this pull request
Aug 16, 2026
Both PRs edited the vision-parity status block and the desktop changelog's Unreleased section. The status line now records both wedges: R4 shipped (W3 complete) and F4 shipped (first W4 wedge), leaving L3c as the only remaining item. Both changelog entries kept, F4 above R4 in land order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last wedge of vision-parity W3. Consumes E3 (#563) and E4 (#564): a running command's output now appears while it runs, and an image a tool returns renders as an image.
What was actually broken
E3's streamed output was already reaching the client and being dropped by the renderer.
useToolMapsfoldstool_call_updateonto its parent call (latest-wins ontoolCallId), butToolCallBodytook only(p, result)— the folded update had no way in — andisToolCallUpdateHiddensuppresses the standalone card whenever a visible parent exists. So the output had nowhere to render on either path.ToolResultBodydidstr(content) ?? jsonText(content), so an image-bearing result printed a screen of base64.And
blob:refs were skipped with a comment calling them "a future hub".payload_externalize.goalready swaps every string leaf over 64 KiB for ablob:sha256/ref on ingest (handlers_agent_events.go:118) — so for any real screenshot the blob path is the normal one, and images were vanishing at exactly the size where they mattered.Three of the plan line's five specifics were wrong
Recorded in the plan, because the corrections are the wedge:
streamingPartials.ts" — no. That module is inui/notstate/, itsFOLD_KINDSistext|thought|plan, and its chains key onmessage_id; atool_call_updatecarriestoolCallIdand nomessage_id. Nothing needed folding.tool_resultcontent verbatim (driver_stdio.go:446), i.e. the Anthropic dialectsource:{type:"base64",media_type,data}. Measured by runningclaude --print --output-format stream-jsonover a PNG rather than recalled — it returns a LIST, withis_errorabsent rather than false. Both dialects render.termipod-att://refs" — a category error. Tauri-only note plumbing for the Read surface (state/attachments.ts:200); no agent event has ever carried one.Two consequences the plan could not have anticipated, both silent-failure shaped:
application/octet-stream— using the blob's mime paints nothing;getBlobDataUrlwould double-encode it. No client had ever resolved an externalized payload leaf (mobile's blob code is all artifact viewers, a different case).A
urlimage source is deliberately not painted — it would make the renderer fetch a host the agent chose.Also fixed
A pre-existing Companion bug found while threading the new prop:
resultandcallNamewere handed to the wrong branches, so the dock's call cards folded in nothing and every standalone result was labelled "Result". The full transcript surface was always correct.Verification
urlsource was being rejected for carrying nodata, not by the source-type guard. Now pinned by a block claiming both.tsc,vite build, token ratchet (from root), and the full doc lint set incl. openapi — all green.Not visually verified — no display on this machine, and the E2E suite does not exercise the transcript. Adds to the standing render debt (#166).