Skip to content

feat(windows): the spike, juno-cua presses Calculator through UI Automation without the cursor, but takes the foreground (LAC-6321) - #928

Draft
lacymorrow wants to merge 7 commits into
mainfrom
LAC-6321-windows-spike
Draft

lacymorrow wants to merge 7 commits into
mainfrom
LAC-6321-windows-spike

Conversation

@lacymorrow

@lacymorrow lacymorrow commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Result: half of the spike's claim holds. On Windows 11, juno-cua pressed 7 × 6 = in Calculator through UI Automation and read back "Display is 42" without moving the cursor. But the first press brought Calculator to the front, and so did the other presses I sampled starting from behind another window. So "works behind the window you're using" isn't true yet. Please review the recording before this merges.

Calculator run

The clip is the real run at 2 frames a second (QEMU screendumps from the host, so nothing in the guest records). Notepad starts in front, covering the keypad. On the first press, Calculator comes forward, and each pressed button shows a focus ring. QEMU draws the pointer as a hardware cursor, so it isn't in the frames. The cursor check below comes from GetCursorPos. MP4 · before · after

The test (scripts/windows-spike.ps1, Windows 11 Pro 23H2, build 22631)

Check Result
All five presses succeeded, through UIA Invoke yes, 75 to 142 ms each
Display reads 42 yes, "Display is 42"
Cursor unchanged yes, (1279, 799) before and after
Notepad was in front before the first press yes
7, ×, 6 and = sat inside Notepad's rectangle (bounds from juno-cua). C was left out of this check and was visible above Notepad yes, but only until the first press brought Calculator forward
Foreground unchanged no, it ended on "Calculator"

Raw output: result.json

Is it only Calculator?

Probably not, but the sample is small: three presses, one each. A second probe (script, output) pressed one button per case:

  • num7Button, fully covered by Notepad: Calculator took the foreground.
  • num6Button, a second press: same.
  • Character Map's Select button (classic Win32, not XAML): Character Map took the foreground, but this press started with Calculator in front, not Notepad.

The probe doesn't settle whether a background process can hand the foreground back. SetForegroundWindow(Notepad) from the test script worked before the second case and had no effect before the third. Windows' foreground lock is a likely cause, but I didn't confirm it. An earlier version of this description said the handback was refused outright, which was wrong.

Measurements (no pass or fail)

  • File Explorer tree read: 497 to 616 ms for 176 nodes, about 560 to 680 ms end to end with process start.
  • VS Code (Electron) tree read: the first read got 14 nodes in 47 ms because Chromium hadn't built its tree yet. Later reads got 193 nodes in 406 to 429 ms, with VS Code covered by Notepad.
  • Rename in a covered Explorer window: works. ValuePattern.SetValue on the list item renamed rename-me to renamed-by-uia while Notepad stayed on top. But keyboard focus left Notepad (its caret disappeared and GetForegroundWindow returned an untitled window). Renames should go through the filesystem, not Explorer's UI.

Two bugs the VM found, fixed here

  • A fresh Windows 11 has no vcruntime140.dll, so the first build exited with 0xC0000135 before main. .cargo/config.toml now links the C runtime statically on Windows. The CI artifact runs on the clean VM.
  • A Descendants search from the desktop root returns nothing (FindAll gave 0 elements). The engine now searches each top-level window's subtree when it's asked to search the whole desktop.

What this means for the plan

The plan's selling point in docs/plans/windows.md is that Juno works without taking over the computer, unlike ChatGPT's foreground-only Windows agent. The pointer half is proven. The foreground half isn't, and it decides the pitch. What to try next, cheapest first:

  1. Hand the foreground back after each press, and measure how often that sticks. Doing it from the Juno bar process may be more reliable, since the bar is where the person just typed, but that's untested. At best the person would see a flicker. Repeated trials for this and for option 2 are written but haven't run (see below).
  2. For Win32 controls, send BM_CLICK straight to the control's window instead of using Invoke.
  3. MXC isolation sessions, already in the plan for later. They give the agent its own desktop, but they need Insider build 26300.8553.

Until one of those works, the copy should promise "never moves your pointer", not "works behind your windows".

What changed

  • mcp-server-os-level: Mac-only dependencies moved to a cfg(target_os = "macos") table. Mac behaviour is unchanged.
  • platforms/windows.rs: the stub, which didn't compile, is replaced by a UI Automation engine (uiautomation 0.25.1). It finds elements by automation id, name or role, presses them through Invoke, Toggle or SelectionItem, and reads trees with timing. Anything that would need real input returns UnsupportedPlatform.
  • .cargo/config.toml: static C runtime on Windows.
  • .github/workflows/windows.yml: builds juno-cua on windows-latest and uploads it. Not a required check.
  • scripts/windows-spike.ps1: the test above.

Test rig

A new "Juno Windows 11" VM on library, currently stopped: plain QEMU with OVMF and a software TPM, no GPU, an 80 GB thin disk, unattended install. Two changes from the kickoff prompt:

  • It runs under plain QEMU instead of the Unraid VM service, so the service and the Windows 10 VMs were never touched.
  • It has 6 GB of RAM instead of 8, because library had 11 GB free.

Not run yet

The out-of-memory killer ended the VM at 15:54:57 on 10-31, after these results were captured, while the Windows 10 VM was starting on the same host. It stays off until there's enough memory for both, so the repeated foreground trials haven't run.

DRI: Founding Engineer. Tracking: LAC-6321

…CI (LAC-6321)

The spike from docs/plans/windows.md. The Mac-only dependencies move to a
macOS target table, the dead Windows stub is replaced with an engine that
finds elements and presses them through UI Automation patterns (no input
events), and a non-required Windows workflow builds juno-cua.
A fresh Windows 11 VM has no vcruntime140.dll, so juno-cua.exe exited with
STATUS_DLL_NOT_FOUND (0xC0000135) before main. LAC-6321.
Notepad now covers Calculator's keypad but not its display, so a recording
shows 7, 6 and 42 arrive while every button juno-cua presses stays hidden.
A new check reads each button's bounds through juno-cua and confirms it sits
inside Notepad's rectangle.
On Windows 11 a Descendants search from the desktop root returns nothing:
FindAll with TrueCondition gave 0 elements, and every Calculator button
lookup failed. The same search scoped to the Calculator window finds them.
A search from the desktop now runs in each top-level window's subtree.

Also pass [NullString]::Value to FindWindow in the spike script, since
PowerShell turns $null into an empty string. LAC-6321.
@lacymorrow lacymorrow changed the title feat(windows): the spike, juno-cua presses a background Calculator through UI Automation (LAC-6321) feat(windows): the spike, juno-cua presses Calculator through UI Automation without the cursor, but takes the foreground (LAC-6321) Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant