feat: add mobile_observe/mobile_act/mobile_session_end for host-agent-driven manual mode - #114
MrAk47Anand007 wants to merge 9 commits into
Conversation
Adds mobile_observe/mobile_act/mobile_session_end MCP tools so any MCP host with its own model (Claude Code, Codex, Antigravity, etc.) can drive a device through Artemis's existing observation/action-executor internals without configuring an LLM credential in Artemis itself.
Task-by-task TDD plan for mobile_observe/mobile_act/mobile_session_end, grounded in the existing McpActionExecutor/DeviceExecutionLock internals.
ruff format reflow on two lines, and bump the broad-exception-handler quality-ratchet baseline by one for mobile_observe's tool-boundary except Exception, matching the existing mobile_get_device_state pattern.
Verified against a real connected device (observe -> act(press_key) -> observe -> session_end, with the device lock confirmed released afterward).
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
5efea31 to
bc13683
Compare
…y failure mobile_observe/mobile_act only caught DeviceBusyError around get_or_create(), letting the generic Exception _get_controller raises for a missing/wrong device_serial crash the tool call instead of returning the structured error shape every other tool in this file uses. mobile_observe also overwrote indexed_elements/indexed_points from obs.elements unconditionally, clobbering a valid element index when obs.hierarchy_ok is False (screenshot ok, hierarchy parse failed) -- contrary to the contract documented on ObserveResult and already honored by action_executor.py's own observation-refresh path.
713ea6e to
37d4616
Compare
|
Tried this branch out end-to-end against a physical Pixel 10 Pro with no Google/Gemini credential configured at all — One small bug worth fixing:
|
…eleased
ManualSessionRegistry.end() ran _reap_idle() before _end_key(), so when the
target session was idle past DEFAULT_IDLE_TIMEOUT_S, the reap already
popped it and released its lock -- leaving _end_key() nothing to find.
The caller got {"ended": false} for the exact call that released the
device, which reads as "nothing happened" and can cause a host to report
the device as still locked or retry needlessly.
End the requested key first, then reap the rest opportunistically.
Reported by @Dor-bl on google#114.
|
@Dor-bl Thanks for testing this end-to-end and catching that you were right, |
|
Follow-up stacked on this PR: MrAk47Anand007#2 It builds a Jev-driven device loop on top of the manual-mode session added here ( |
Summary
mobile_observe/mobile_act/mobile_session_endMCP tools so any MCP host with its own model (Claude Code, Codex, Antigravity, etc.) can drive a connected Android device through Artemis's existing observation/action-executor internals, without configuring any LLM credential in Artemis itself.McpActionExecutor(Flash's exact action vocabulary and post-action observation) and the existingDeviceExecutionLockper-device mutex, guarded by a new in-processManualSessionRegistrywith opportunistic idle-session reaping.mobile_run_task(Flash/Pro) is unchanged; this is a new, third mode alongside it. Design spec and implementation plan are included underdocs/superpowers/.Investigated wiring real MCP
sampling/createMessageintoFlashRunner/the Pro graph directly, but rejected it for this iteration (no native tool-calling in the sampling spec, andmobile_run_task's detached background-subprocess model is incompatible with a synchronous per-step callback to the calling client) — details in the design doc's "Rejected approach" section.Test Plan
uv run pytest(deterministic suite): 2124 passed, 0 regressions (80 pre-existing failures on this machine are unrelated — missingGOOGLE_API_KEY/GEMINI_API_KEYbreaks sometests/unit/agents/test_flash_*/test_video_analyzer.pytests onmaintoo, verified before this branch's changes)make lint(ruff format/check + quality ratchet) — cleanmake typecheck(pyright) — clean on the new filestests/e2e/test_manual_mode_device.py(-m android) run against a real connected device: observe → act(press_key) → observe → session_end, device lock confirmed released afterward