Skip to content

macos-host: say which phase the UI root search is in, so slow stops looking hung - #494

Merged
smerwin merged 1 commit into
mainfrom
host-root-search-progress
Sep 28, 2026
Merged

smerwin merged 1 commit into
mainfrom
host-root-search-progress

Conversation

@smerwin

@smerwin smerwin commented Sep 12, 2026

Copy link
Copy Markdown
Owner

The Spotlight autopilot app was reported broken. It was not — the root search took 81 seconds and the bot then flew its route perfectly, eight jumps and counting.

For those 81 seconds the only thing on screen was the bot's status line, reprinted unchanged every tick:

EVE Online framework status:
Search the address of the UI root in process 87630

That line is re-derived each tick from a setup state that has genuinely not moved, so it says nothing about whether the host is alive. And this repo has shipped a hang that presents identically — #455, where the framework rebuilt the volatile process forever. "Working, just slow" and "wedged" looked the same, and an operator already burned by the second reasonably read this as the second.

The change

Each phase announces itself on stderr with an elapsed clock, plus a rate-limited heartbeat that marks time inside a phase — the phase lines alone leave a silence as long as the longest phase, which is the whole gap. The heartbeat lives on the request side because that is what runs once per tick; the worker sits inside one phase for tens of seconds with no loop to hang a clock on.

Live, cold:

# UI root search (pid N, 0s): checking the cached root
# UI root search (pid N, 0s): dumping process memory
# UI root search (pid N, 10s): indexing the dump
# UI root search (pid N, 10s): scanning for a seed object
# UI root search (pid N, 21s): still scanning for a seed object -- ... not a hang
# UI root search (pid N, 34s): bootstrapping the str type
# UI root search (pid N, 54s): walking up to the UI root
# UI root search (pid N, 93s): still walking up to the UI root -- ... not a hang
# UI root search (pid N, 95s): found the UI root

It disproved a belief in this file on its first run

Which is the argument for real phases over a spinner:

phase elapsed
dumping process memory 10s
indexing the dump ~0s
scanning for a seed object 24s
bootstrapping the str type 20s
walking up to the UI root 41s
total 95s

The dump was labelled "the long one" and is the shortest substantive phase. _cached_ui_root still calls it "(~20-40s) ... the whole cost of starting the bot" — it is a tenth of the wall clock. The cost is the two repr scans and the walk to the root, which is not where either comment pointed. Label corrected and the breakdown recorded.

Safety

Reporting can never break the search it reports on — the reporter is wrapped, because a search dying for want of describing itself would be this change causing the failure it exists to explain. The protocol answer is untouched: SearchUIRootAddressInProgress still keeps the framework waiting rather than tearing the volatile process down.

Verified

14 cases in test_root_search_reports_progress.py, no client needed. Five mutations, each failing a named case: rate limit removed so it floods; dump phase unannounced; cached path silent; interval cut to a second; phase no longer recorded for the heartbeat to repeat. Plus the live cold search above.

…ooking hung

The Spotlight autopilot app was reported broken on 2026-09-12. It was not: the
root search took 81 seconds on that client and the bot then flew its route
perfectly, eight jumps and counting. What the operator had on screen for those
81 seconds was the bot's own status line, reprinted unchanged every tick:

    EVE Online framework status:
    Search the address of the UI root in process 87630

That line is the bot's, re-derived each tick from a setup state that has
genuinely not moved, so it says nothing about whether the host is alive. And
this repository has shipped a hang that presents identically -- #455, where
`character_name` on the wrong object turned every client-list request into
ProcessNotFound and the framework rebuilt the volatile process forever. So
"working, just slow" and "wedged" looked the same, and an operator who had
already been burned by the second read this as the second.

Each phase now announces itself on stderr with an elapsed clock, and a
rate-limited heartbeat marks time *inside* a phase -- the phase lines alone
leave a silence as long as the longest phase, which is the gap this is about.
The heartbeat is on the request side because that is what runs once per tick;
the worker is inside a single phase for tens of seconds with no loop to hang a
clock on.

**It corrected a belief in this file on its first run**, which is the argument
for real phases over a spinner. Timed cold:

    dumping process memory        10s
    indexing the dump             ~0s
    scanning for a seed object    24s
    bootstrapping the str type    20s
    walking up to the UI root     41s
    total                         95s

The dump was labelled "the long one" and is the shortest substantive phase;
`_cached_ui_root` still calls it "(~20-40s) ... the whole cost of starting the
bot" and it is a tenth of the wall clock. The cost is the two repr scans and the
walk to the root, which is not where either comment pointed.

Reporting can never break the search it reports on: the whole reporter is
wrapped, because a search dying for want of describing itself would be this
change causing the failure it exists to explain. The protocol answer is
untouched -- `SearchUIRootAddressInProgress` still keeps the framework waiting.

Verified in tools/macos-host/tests/test_root_search_reports_progress.py (14
cases, no client needed), confirmed by five mutations each failing a named case:
the rate limit removed so it floods, the dump phase unannounced, the cached path
silent, the interval cut to a second, and the phase no longer recorded for the
heartbeat to repeat. Confirmed live on a cold search: every phase reported, the
clock visibly moving, 95s end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@smerwin
smerwin merged commit 8ff509c into main Sep 28, 2026
2 of 3 checks passed
@smerwin
smerwin deleted the host-root-search-progress branch September 28, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant