Skip to content

agents.md

Latest commit

 

History

History
151 lines (115 loc) · 7.23 KB

agents.md

File metadata and controls

151 lines (115 loc) · 7.23 KB
Raw

Driving DeviceDeck with an agent

DeviceDeck mirrors a simulator or emulator's native accessibility tree as a real browser DOM: an element's type becomes an ARIA role, its accessibility label becomes the accessible name, its accessibility identifier becomes data-testid. Every web agent that acts by reading a page's accessibility tree therefore drives a device the same way it drives a website — by role and name, no plugin, no coordinates, no vision model.

Set up your agent

Every agent needs two things: the devicedeck binary installed and running (devicedeck in a terminal), and a browser tool — Playwright MCP. The devicedeck mcp server adds device tools (list, boot, launch, UI tree, device settings) and briefs the agent when it connects; the four skills (authoring, maestro, flows, triage) teach it the rest.

Claude Code — the plugin brings the skills and the devicedeck MCP server:

claude mcp add playwright npx @playwright/mcp@latest
claude plugin marketplace add devicelab-dev/DeviceDeck
claude plugin install devicedeck@devicedeck-marketplace

Gemini CLI — one extension brings both MCP servers and the skills:

gemini extensions install https://github.com/devicelab-dev/DeviceDeck

Codex CLI

codex mcp add playwright -- npx @playwright/mcp@latest
codex mcp add devicedeck -- devicedeck mcp
npx skills add devicelab-dev/DeviceDeck

Codex stops a tool call after 60 seconds by default, and booting a cold emulator or installing a large build can take longer. Raise it for DeviceDeck in ~/.codex/config.toml:

[mcp_servers.devicedeck]
tool_timeout_sec = 300

VS Code / GitHub Copilot

code --add-mcp '{"name":"playwright","command":"npx","args":["@playwright/mcp@latest"]}'
code --add-mcp '{"name":"devicedeck","command":"devicedeck","args":["mcp"]}'
npx skills add devicelab-dev/DeviceDeck

Cursor — add both servers to ~/.cursor/mcp.json (or .cursor/mcp.json in a project), then npx skills add devicelab-dev/DeviceDeck:

{
  "mcpServers": {
    "playwright": { "type": "stdio", "command": "npx", "args": ["@playwright/mcp@latest"] },
    "devicedeck": { "type": "stdio", "command": "devicedeck", "args": ["mcp"] }
  }
}

If you installed DeviceDeck with npm into a project, devicedeck is not on your PATH: use npx devicedeck mcp as the MCP command, or install it globally with npm install -g devicedeck.

Anything else (Windsurf/Devin, Cline, JetBrains AI, …) — the same two servers in that client's MCP settings, in its own format; npx skills add devicelab-dev/DeviceDeck installs the skills for most agents. Clients that install Agent Plugins can install this repository directly: plugin.json, mcp.json and skills/ at its root are that bundle.

Ask for a test

Write a Playwright test that logs in to my app.

The agent calls list_apps to find the build you registered with devicedeck --app, boots a simulator of its platform, launches the app (installing it first), drives it through the device page and writes a spec that selects by the app's accessibility ids. Review and run it like any other test.

Tested with

This page records that claim being tested. Each tool below is a stock release, pointed at a device URL, driving TestHive on an iOS 26.2 simulator. Nothing about any of them knows what a simulator is.

Results

Tool Version Reads the tree Acts by Full login Leaves an artifact
agent-browser 0.37.1 accessibility snapshot, @e refs ref ✓ end to end to the catalog no (imperative CLI)
playwright-cli 0.1.19 aria snapshot, e refs ref / getByRole / getByTestId ✓ yes — records a Playwright spec
browser-harness 0.1.13 Accessibility.getFullAXTree over CDP coordinate + insertText reads fully; typing fights the soft keyboard (see below) agent-written helpers, not a flow
Playwright (library) 1.x aria snapshot getByTestId / getByRole ✓ (the committed specs) yes — the spec you write
Stagehand LOCAL 4.0.2 trimmed accessibility tree act() / observe() not run here — needs an LLM key cached deterministic action

Transcripts are under examples/agents/; the recorded Playwright spec is examples/playwright/tests/recorded/login-recorded.spec.ts.

What each proof shows

agent-browser — snapshot -i lists the four login controls with their roles and names; fill @e1 / fill @e2 reach the device, Sign In enables, click @e3 lands on the catalog where every product's Add button is a distinct, addressable control. This is the plain case: the tool that 42k projects use to drive a browser drives a simulator with zero changes.

playwright-cli — the same login by ref, wrapped in recording-start / recording-stop. The recorder emitted:

await page.getByTestId('username-input').fill('devicelab');
await page.getByTestId('password-input').fill('robustest');
await page.getByTestId('login-button').click();

Those three lines are a durable, reviewable mobile test, produced by a web recorder that never knew it was looking at a phone. The committed spec replays them green on a live simulator.

browser-harness — Accessibility.getFullAXTree returns the mirror's real tree, roles and names intact, which is the substrate claim: browser-harness reads exactly what DeviceDeck manufactures. Its typing path is the one caveat. It drives by coordinate and types with CDP insertText into whatever the DOM has focused, so once the on-screen keyboard opens over the next field, a coordinate tap can miss it. Tools that focus a field by selector before typing (agent-browser, Playwright) do not hit this. It is a property of coordinate-first driving, not of the mirror.

Stagehand — its Playwright substrate is the same one proven above, so the deterministic half drives the mirror. Its act() / observe() need a model key, which this machine does not have, so that half is not exercised here.

The one thing every agent needs to know

Every action waits for the device. click() and fill() (what browser_click and browser_type emit) return only once the device has acted and the screen has settled, so the next snapshot shows the result — no sleeps, no polling. Two things still need a second look: a button the app keeps disabled until a form is filled can appear only as text, with no ref, until then — fill the form and snapshot again; and an app that changes on a timer (a debounced search) is asserted with expect, which retries. See behaviors.md.

Coordinate agents still get a selector

Some agents (the Anthropic and OpenAI and Google computer-use tools) act by pixel, not by ref. A pixel click on the mirror's video resolves to the DOM element under the point, so Flow Capture still records it as a durable tapOn: id step, not a coordinate. A coordinate agent driving DeviceDeck produces a selector-based artifact anyway.