Skip to content

feat(browser): call WebMCP page tools before clicking (LAC-6315) - #923

Draft
lacymorrow wants to merge 1 commit into
mainfrom
LAC-6315-juno-call-webmcp-page-tools-before-clicking-document-modelcontext
Draft

lacymorrow wants to merge 1 commit into
mainfrom
LAC-6315-juno-call-webmcp-page-tools-before-clicking-document-modelcontext

Conversation

@lacymorrow

Copy link
Copy Markdown
Owner

Paperclip: LAC-6315

When a page publishes WebMCP tools, Juno's browser agent now calls them. The DOM path (browser_interact) stays as the fallback.

What changed

  • browser_navigate appends page_tools (names plus short descriptions) to its result when the page has tools. Nothing is added when it has none.
  • New browser_list_page_tools: full list with input schemas and the page's annotations. Empty list, never an error, when there is no API.
  • New browser_call_page_tool(name, input, timeout?): looks the tool up in getTools() (executeTool takes the tool object, not a name), runs it with an AbortSignal and a 30 s default timeout.
  • Risk, through the existing gate (risk_classifier and permission_policy):
    • consequentialHint is High, so it asks in "Ask me first" and "Ask when risky".
    • readOnlyHint without consequential is Low.
    • Anything unflagged or never listed is Medium.
    • The gate sees only a tool's name and input, so webmcp.rs keeps a small registry refreshed on every listing. If a consequential tool was not approved as consequential (stale registry), the call is refused and the model re-lists, so the gate asks next time.
  • untrustedContentHint output is wrapped as {untrusted_content: true, source, warning, content}.
  • Juno-launched Chrome (persistent profile, temp profile, fresh) gets --enable-features=WebMCP.
  • Tool toggles in settings: both tools in the Browser category.

Spike (Chrome 156.0.8078.74)

document.modelContext is undefined with no flags and an object with --enable-features=WebMCP, headless and headed, on a secure context (localhost). Full table in the issue thread. Consequence: a Chrome the person started themselves, without the flag, shows no API, and Juno silently uses the DOM.

Cut

  • No navigator.modelContext fallback (removed in Chrome 152).
  • No polyfill in the fixture; the flag works headless.
  • No user-facing message when tools are missing or a read fails.
  • No Juno-as-WebMCP-provider, no juno-www tools (out of scope).
  • No new settings UI.

Test plan

  • Unit tests in webmcp.rs (parsing, hint cap, untrusted wrapper, risk registry, JS literal embedding). CI.
  • risk_classifier drift tests cover both new tools (list is deliberately ungated, call has its own arm).
  • Live: webmcp_page_tools_are_listed_and_called_without_clicking in tests/browser_cdp_live.rs (#[ignore], Chrome on :9222 started with --enable-features=WebMCP).

Not done yet

Recording of Juno finishing a task on the fixture page by tool call. It needs this branch built by CI and an attended run. Will be committed to docs/changelog/media/<PR>/.

…Context)

Adds browser_list_page_tools and browser_call_page_tool, a page-tools hint on
browser_navigate, risk from the page's consequentialHint, untrusted wrapping
for untrustedContentHint, and --enable-features=WebMCP on Juno-launched Chrome.

LAC-6315
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant