Skip to content

feat(hub): forward desktop bridge tool results instead of re-wrapping (E4) - #564

Merged
physercoe merged 2 commits into
mainfrom
feat-vision-e4-relay-image-passthrough
Aug 16, 2026
Merged

feat(hub): forward desktop bridge tool results instead of re-wrapping (E4)#564
physercoe merged 2 commits into
mainfrom
feat-vision-e4-relay-image-passthrough

Conversation

@physercoe

Copy link
Copy Markdown
Owner

Vision-parity E4 (asymmetry #3). A screenshot taken through the hub reached a remote agent as base64 prose.

routeTunnelInvoke passed the desktop's reply through mcpResultJSON, which renders it as JSON inside one text block — so {"content":[{"type":"image","data":"<base64 PNG>"}]} arrived as text no model can look at.

It was never image-specific

Reading the far end widened the fix. dispatchHubInvoke answers a relayed call with the object callTool returns, and every bridge tool returns an MCP tool result — 16 textContent() calls plus the literal {content:[{type:'image',…}]} returns. The relay was re-wrapping an already-conformant result on every call; the image case is just where the extra layer destroys the payload instead of merely quoting it.

The relay now forwards a well-formed tool result verbatimisError and extra keys included, since "what a local caller sees" is the point — and keeps the JSON wrapper for anything else. Local and relayed calls now return the same thing.

The guard errs toward the wrapper, deliberately

content must be a non-empty array of objects each carrying a string type.

That rejects {"content":[]}, which is a legal MCP result — a false negative costing a reader one layer of quoting. The other direction would hand the agent something malformed, and nothing distinguishes "an empty tool result" from "an arbitrary object with an empty list under that key". Losing nothing beats forwarding garbage.

Two things checked rather than assumed

  • The Go fixtures were describing a desktop that does not exist. okBrowserEnvelope was being handed bare JSON ({"tabs":…}, {"clicked":true}) — a shape no bridge tool has ever returned. So the existing tests could not have caught the double-wrap and would not have noticed the fix either. The read-routing test now asserts against the real reply shape, and okBrowserToolResult names the distinction for the callers that still exercise the fallback on purpose.
  • The rest of the chain forwards it. The host-runner gateway returns a tools/call result verbatim (gwIdentity only stamps _meta server info), so the image block survives desktop → hub → gateway → agent.

Verification

  • Full go test ./...exit 0, all 33 packages (server suite 290s); build + vet clean.
  • New tests on both relayed classes: browser_invoke (image block survives; text stops being double-wrapped; five non-tool-result shapes keep the wrapper) and ui_screenshot (the path the asymmetry is named for), plus a 15-case table for the shape predicate including the {"content":[]} boundary it deliberately gets "wrong".
  • Four mutations checked, all fail: removing the passthrough, dropping the shape guard, skipping the per-block type check, accepting an empty content array.
  • Full lint set green, including lint-openapi (venv) and lint-desktop-tokens from the repo root. NUL + CJK scans clean.

Note for the merge

Branched off main, not off #563 (E3) — the code paths don't overlap. Both edit the plan's status line, so whichever lands second needs a one-line rebase there.

🤖 Generated with Claude Code

Ubuntu and others added 2 commits August 16, 2026 04:05
… (E4)

A screenshot taken through the hub reached a remote agent as base64
prose. `routeTunnelInvoke` passed the desktop's reply through
`mcpResultJSON`, which renders it as JSON inside one text block — so
`{"content":[{"type":"image","data":"<base64 PNG>"}]}` arrived as text
the model cannot look at.

Reading the far end showed the defect was never image-specific.
`dispatchHubInvoke` answers a relayed call with the object `callTool`
returns, and every bridge tool returns an MCP tool result — 16
`textContent()` calls plus the literal image returns. The relay was
re-wrapping an already-conformant result on every call; the image case
is just where the extra layer destroys the payload rather than merely
quoting it.

So the relay now forwards a well-formed tool result verbatim — isError
and extra keys included, since "what a local caller sees" is the whole
point — and keeps the JSON wrapper for anything else.

The shape guard errs toward the wrapper on purpose: `content` must be a
non-empty array of objects each carrying a string `type`. That rejects
the legal-but-indistinguishable `{"content":[]}`, costing a reader one
layer of quoting; the other direction — an arbitrary object with an
empty list under that key, forwarded as a tool result — hands the agent
something malformed, and no discriminator separates the two.

Two things checked rather than assumed:

- The Go fixtures were describing a desktop that does not exist.
  `okBrowserEnvelope` was being handed bare JSON (`{"tabs":...}`,
  `{"clicked":true}`), a shape no bridge tool has ever returned, so the
  tests could not have caught the double-wrap and would not have
  noticed the fix. The read-routing test now asserts against the real
  reply shape; `okBrowserToolResult` names the distinction for the
  callers that still exercise the fallback deliberately.
- The rest of the chain forwards it. The host-runner gateway returns a
  `tools/call` result verbatim (`gwIdentity` only stamps `_meta`), so
  the image block survives desktop to hub to gateway to agent.

Four mutations checked, all fail: removing the passthrough, dropping
the shape guard, skipping the per-block type check, and accepting an
empty content array.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review finding on E4: the passthrough forwarded the desktop's tool
result without `resultType`. The desktop stamps the additive 2026-07-28
fields only on its own HTTP leg (stampMcpResult runs in the JSON-RPC
serving path, which the tunnel bypasses), so the relayed reply was the
one hub result missing the stamp the tools/call wrapper's comment says
every dispatch path carries (ADR-063 D3) — and a LOCAL caller does see
`resultType`, so "what a local caller sees" was off by exactly that key.
Spec-legal either way (a missing resultType reads as complete), which is
why this is consistency, not breakage.

`routeTunnelInvoke` now wraps the passthrough in mcpwire.StampResult;
the headline image test asserts the stamp, mutation-checked (stamping
"nope" fails it). The predicate itself still returns the map verbatim.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@physercoe
physercoe merged commit ea9cecd into main Aug 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants