Skip to content

feat: add mobile_observe/mobile_act/mobile_session_end for host-agent-driven manual mode - #114

Open
MrAk47Anand007 wants to merge 9 commits into
google:mainfrom
MrAk47Anand007:feat/manual-observe-act-mode
Open

MrAk47Anand007 wants to merge 9 commits into
google:mainfrom
MrAk47Anand007:feat/manual-observe-act-mode

Conversation

@MrAk47Anand007

@MrAk47Anand007 MrAk47Anand007 commented Sep 16, 2026

Copy link
Copy Markdown

Summary

  • Adds mobile_observe / mobile_act / mobile_session_end MCP tools so any MCP host with its own model (Claude Code, Codex, Antigravity, etc.) can drive a connected Android device through Artemis's existing observation/action-executor internals, without configuring any LLM credential in Artemis itself.
  • Reuses McpActionExecutor (Flash's exact action vocabulary and post-action observation) and the existing DeviceExecutionLock per-device mutex, guarded by a new in-process ManualSessionRegistry with opportunistic idle-session reaping.
  • mobile_run_task (Flash/Pro) is unchanged; this is a new, third mode alongside it. Design spec and implementation plan are included under docs/superpowers/.

Investigated wiring real MCP sampling/createMessage into FlashRunner/the Pro graph directly, but rejected it for this iteration (no native tool-calling in the sampling spec, and mobile_run_task's detached background-subprocess model is incompatible with a synchronous per-step callback to the calling client) — details in the design doc's "Rejected approach" section.

Test Plan

  • uv run pytest (deterministic suite): 2124 passed, 0 regressions (80 pre-existing failures on this machine are unrelated — missing GOOGLE_API_KEY/GEMINI_API_KEY breaks some tests/unit/agents/test_flash_*/test_video_analyzer.py tests on main too, verified before this branch's changes)
  • make lint (ruff format/check + quality ratchet) — clean
  • make typecheck (pyright) — clean on the new files
  • tests/e2e/test_manual_mode_device.py (-m android) run against a real connected device: observe → act(press_key) → observe → session_end, device lock confirmed released afterward

Adds mobile_observe/mobile_act/mobile_session_end MCP tools so any MCP
host with its own model (Claude Code, Codex, Antigravity, etc.) can drive
a device through Artemis's existing observation/action-executor internals
without configuring an LLM credential in Artemis itself.
Task-by-task TDD plan for mobile_observe/mobile_act/mobile_session_end,
grounded in the existing McpActionExecutor/DeviceExecutionLock internals.
ruff format reflow on two lines, and bump the broad-exception-handler
quality-ratchet baseline by one for mobile_observe's tool-boundary
except Exception, matching the existing mobile_get_device_state pattern.
Verified against a real connected device (observe -> act(press_key) ->
observe -> session_end, with the device lock confirmed released
afterward).
@google-cla

google-cla Bot commented Sep 16, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@MrAk47Anand007
MrAk47Anand007 force-pushed the feat/manual-observe-act-mode branch from 5efea31 to bc13683 Compare September 16, 2026 06:55
…y failure

mobile_observe/mobile_act only caught DeviceBusyError around
get_or_create(), letting the generic Exception _get_controller raises
for a missing/wrong device_serial crash the tool call instead of
returning the structured error shape every other tool in this file
uses.

mobile_observe also overwrote indexed_elements/indexed_points from
obs.elements unconditionally, clobbering a valid element index when
obs.hierarchy_ok is False (screenshot ok, hierarchy parse failed) --
contrary to the contract documented on ObserveResult and already
honored by action_executor.py's own observation-refresh path.
@MrAk47Anand007
MrAk47Anand007 force-pushed the feat/manual-observe-act-mode branch from 713ea6e to 37d4616 Compare September 16, 2026 09:01
@Dor-bl

Dor-bl commented Sep 17, 2026

Copy link
Copy Markdown

Tried this branch out end-to-end against a physical Pixel 10 Pro with no Google/Gemini credential configured at allmobile_observemobile_act (manage_app, click, input_text) → mobile_session_end all worked, driving a real app through a launch-and-search flow. The credential-free premise holds up, and targeting by element index rather than coordinates made it straightforward to drive from the host side. Nice work.

One small bug worth fixing:

mobile_session_end returns ended: false after successfully releasing the lock

ManualSessionRegistry.end() reaps before it ends:

def end(self, device_serial: str | None) -> bool:
    """Releases the device lock and drops the session. Idempotent."""
    self._reap_idle()
    return self._end_key(self._key(device_serial))

When the target session has been idle for >= DEFAULT_IDLE_TIMEOUT_S (600s), _reap_idle() pops it and releases its lock, so the following _end_key() finds nothing and returns False. The caller gets:

{"status": "success", "device_serial": "", "ended": false}

even though that same call is what released the device.

Why it matters: ended is the only signal the host agent gets. false reads as "nothing was released", so a host may report to the user that the device is still locked, or retry. I hit this after ~12 minutes of idle time and had to check /tmp/artemis/device-locks/ on the filesystem to confirm the lock had in fact been dropped.

Suggested fix — end the explicitly requested key first, then reap the rest:

def end(self, device_serial: str | None) -> bool:
    ended = self._end_key(self._key(device_serial))
    self._reap_idle()
    return ended

That keeps reaping opportunistic and makes the return value mean "this call released a session", which is what the docstring implies.


Edited to correct an earlier note in this comment. I originally wrote that omitting device_serial fails schema validation and suggested it might be a repo-wide issue with optional parameters. That was my mistake — it is not an Artemis bug. The server publishes a correct schema for these parameters:

"device_serial": {
  "anyOf": [{"type": "string"}, {"type": "null"}],
  "default": null
}

with the parameter properly excluded from required. The validation failure I saw came from schema handling on my MCP client's side, which dropped the anyOf before validating. Nothing to fix here. Apologies for the noise.

…eleased

ManualSessionRegistry.end() ran _reap_idle() before _end_key(), so when the
target session was idle past DEFAULT_IDLE_TIMEOUT_S, the reap already
popped it and released its lock -- leaving _end_key() nothing to find.
The caller got {"ended": false} for the exact call that released the
device, which reads as "nothing happened" and can cause a host to report
the device as still locked or retry needlessly.

End the requested key first, then reap the rest opportunistically.

Reported by @Dor-bl on google#114.
@MrAk47Anand007

MrAk47Anand007 commented Sep 17, 2026

Copy link
Copy Markdown
Author

@Dor-bl Thanks for testing this end-to-end and catching that you were right, _reap_idle() running before _end_key() meant the call that released an idle session's lock reported ended: false. Fixed in a7355c4: end() now ends the requested key first, then reaps the rest opportunistically, so the return value actually reflects whether that call released a session. Added a regression test (test_end_reports_true_when_the_call_itself_reaped_an_idle_session) covering the idle-timeout race deterministically.

@MrAk47Anand007

Copy link
Copy Markdown
Author

Follow-up stacked on this PR: MrAk47Anand007#2

It builds a Jev-driven device loop on top of the manual-mode session added here (mobile_observe / mobile_act), so a run needs no screenshots and no Artemis LLM credential. Verified on a real device (open Gallery + home; Settings search typing). Not yet verified: the opt-in FlashRunner hook in a live loop, or any cost/speed comparison against a VLM. No changes to this PR's code; the stacked PR is based on this branch and can be retargeted to main after this merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants