Repository navigation
feat(windows): the spike, juno-cua presses Calculator through UI Automation without the cursor, but takes the foreground (LAC-6321) - #928
Draft
lacymorrow wants to merge 7 commits into
Draft
lacymorrow wants to merge 7 commits into
lacymorrow wants to merge 7 commits into
Conversation
…CI (LAC-6321) The spike from docs/plans/windows.md. The Mac-only dependencies move to a macOS target table, the dead Windows stub is replaced with an engine that finds elements and presses them through UI Automation patterns (no input events), and a non-required Windows workflow builds juno-cua.
…he desktop session (LAC-6321)
A fresh Windows 11 VM has no vcruntime140.dll, so juno-cua.exe exited with STATUS_DLL_NOT_FOUND (0xC0000135) before main. LAC-6321.
Notepad now covers Calculator's keypad but not its display, so a recording shows 7, 6 and 42 arrive while every button juno-cua presses stays hidden. A new check reads each button's bounds through juno-cua and confirms it sits inside Notepad's rectangle.
On Windows 11 a Descendants search from the desktop root returns nothing: FindAll with TrueCondition gave 0 elements, and every Calculator button lookup failed. The same search scoped to the Calculator window finds them. A search from the desktop now runs in each top-level window's subtree. Also pass [NullString]::Value to FindWindow in the spike script, since PowerShell turns $null into an empty string. LAC-6321.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Result: half of the spike's claim holds. On Windows 11,
juno-cuapressed 7 × 6 = in Calculator through UI Automation and read back "Display is 42" without moving the cursor. But the first press brought Calculator to the front, and so did the other presses I sampled starting from behind another window. So "works behind the window you're using" isn't true yet. Please review the recording before this merges.The clip is the real run at 2 frames a second (QEMU screendumps from the host, so nothing in the guest records). Notepad starts in front, covering the keypad. On the first press, Calculator comes forward, and each pressed button shows a focus ring. QEMU draws the pointer as a hardware cursor, so it isn't in the frames. The cursor check below comes from
GetCursorPos. MP4 · before · afterThe test (
scripts/windows-spike.ps1, Windows 11 Pro 23H2, build 22631)UIA InvokeRaw output: result.json
Is it only Calculator?
Probably not, but the sample is small: three presses, one each. A second probe (script, output) pressed one button per case:
num7Button, fully covered by Notepad: Calculator took the foreground.num6Button, a second press: same.The probe doesn't settle whether a background process can hand the foreground back.
SetForegroundWindow(Notepad)from the test script worked before the second case and had no effect before the third. Windows' foreground lock is a likely cause, but I didn't confirm it. An earlier version of this description said the handback was refused outright, which was wrong.Measurements (no pass or fail)
ValuePattern.SetValueon the list item renamedrename-metorenamed-by-uiawhile Notepad stayed on top. But keyboard focus left Notepad (its caret disappeared andGetForegroundWindowreturned an untitled window). Renames should go through the filesystem, not Explorer's UI.Two bugs the VM found, fixed here
vcruntime140.dll, so the first build exited with0xC0000135beforemain..cargo/config.tomlnow links the C runtime statically on Windows. The CI artifact runs on the clean VM.Descendantssearch from the desktop root returns nothing (FindAllgave 0 elements). The engine now searches each top-level window's subtree when it's asked to search the whole desktop.What this means for the plan
The plan's selling point in
docs/plans/windows.mdis that Juno works without taking over the computer, unlike ChatGPT's foreground-only Windows agent. The pointer half is proven. The foreground half isn't, and it decides the pitch. What to try next, cheapest first:BM_CLICKstraight to the control's window instead of using Invoke.Until one of those works, the copy should promise "never moves your pointer", not "works behind your windows".
What changed
mcp-server-os-level: Mac-only dependencies moved to acfg(target_os = "macos")table. Mac behaviour is unchanged.platforms/windows.rs: the stub, which didn't compile, is replaced by a UI Automation engine (uiautomation0.25.1). It finds elements by automation id, name or role, presses them through Invoke, Toggle or SelectionItem, and reads trees with timing. Anything that would need real input returnsUnsupportedPlatform..cargo/config.toml: static C runtime on Windows..github/workflows/windows.yml: buildsjuno-cuaonwindows-latestand uploads it. Not a required check.scripts/windows-spike.ps1: the test above.Test rig
A new "Juno Windows 11" VM on library, currently stopped: plain QEMU with OVMF and a software TPM, no GPU, an 80 GB thin disk, unattended install. Two changes from the kickoff prompt:
Not run yet
The out-of-memory killer ended the VM at 15:54:57 on 10-31, after these results were captured, while the Windows 10 VM was starting on the same host. It stays off until there's enough memory for both, so the repeated foreground trials haven't run.
DRI: Founding Engineer. Tracking: LAC-6321