test(android-e2e): reveal automation canaries by visibility probe

[thymikee]

Summary

smoke:automation-system revealed its canaries with a single blind scroll down <amount> and then acted on them immediately. scroll is a gesture, not an offset — the command's own metadata says "app scroll physics determine the final content offset" — so one amount cannot guarantee the target is on screen, least of all right after the landscape/portrait round-trip has relaid the list out.

Both open Android smoke failures sit on that seam:

	reveal	then	outcome
step 34	scroll down 0.7	press id="automation-press"	selector_not_found
post-alert	scroll down 1	snapshot -i for the diff baseline	missing Open automation alert

Step 34 is not the orientation family. It reproduces on #2360 and #2361 (disjoint diffs), on main at bd08e6e, and — decisively — on #2356 at the identical line live-automation-scenario.ts:130, so the orientation fix does not cover it. #2359 records the device as being in portrait at that failure, which rules out the "still landscape" reading. The post-alert one is the family #2356 lists under "What this does not explain".

The iOS scenario already does this correctly — it probes is visible and scrolls again until the element is on screen, with the comment "an offscreen node can exist in the accessibility tree, so a successful wait <selector> is not sufficient evidence to skip scrolling." That policy was iOS-only. This moves it to the shared live-device layer as createVisibilityScroll and gives the Android scenario the same reveal; the unit test moves with it.

No production code changes. This is not a suppression: the scroll budget stays bounded at 4 attempts, and the press and diff assertions still have to pass against a real element, so a genuinely absent canary still fails.

Validation

At 120d8ad: pnpm check:tooling passes (format, lint, typecheck, layering, depgraph, gate-manifest, production-exports, tmpdir-leaks, xctest-selection, mcp-metadata, build, bundle-owner-files, package). pnpm check:fallow — no issues in the 5 changed files. pnpm check:affected --run — all runnable checks passed. The moved unit test passes under node --test. No baseline, .fallowrc or gate-manifest edits were needed.

Not validated on a device from here — this container has no emulator, so the Android lane in CI on this PR is the first real exercise of the change. If step 34 still fails after it, the visibility probes now name which canary was off screen and after how many scrolls, which the blind scroll never did.

Refs #2350, #2356, #2359. Unblocks #2360 and #2361, whose only red check is this scenario.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jqfa11D8QsCMuL17SsLvDz

---

Generated by Claude Code

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.50 MB	4.50 MB	-503 B
Package (unpacked)	4.50 MB	4.50 MB	-503 B
Package (download)	1.33 MB	1.33 MB	-294 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.4 ms	27.2 ms	-0.1 ms
CLI --help	78.9 ms	80.0 ms	+1.1 ms

[thymikee]

The first run refuted the hypothesis this PR was built on

Android smoke on 120d8ad (job 101638359133):

AssertionError: id="automation-press" did not become visible after scrolling
last visibility probe: { "details": { "command": "is", "predicate": "visible",
                          "reason": "selector_not_found", "selector": "id=\"automation-press\"" } }

selector_not_found, after four probes and three 0.75-viewport scrolls — the canary was not in the accessibility tree at all, not merely off screen. So "one blind scroll amount undershot" is wrong, and I should not have stated it as the cause. What the search did buy is a failure that says which canary, which predicate and after how many attempts, which the blind scroll never did.

Two things follow, both pushed in a2c4368.

1. The search only scrolls down. A canary the rotation round-trip left above the viewport is unreachable however many attempts it spends. Both Android reveals now anchor with scroll top first — the idiom the scenario already uses at restore automation route top after Android system UI — so the search starts from a known position and the target can only be below it. This is the reading the evidence still permits: automation-window reads fine just before, but that proves nothing about position, since (as this helper's own comment says) an offscreen node can exist in the tree.

2. I regressed the evidence, and that is on me. The probes run with allowFailure, so the budget-exhausted assert.fail never reached the harness's failed-step capture. The Android artifact fell from 143,676 to 12,086 bytes and lost failed-step-N.png, -snapshot.json and -device.txt — exactly the files that would settle this. The search now spends one final probe as a real step before failing, restoring all three.

So the next Android failure, if there is one, arrives with a snapshot of what was actually on screen and the dumpsys/rotation facts alongside it. That is the point of this push as much as the anchor is.

Validated at a2c4368: check:tooling 0, check:fallow clean, check:affected --run all passed, the moved unit test now 3/3 (added one covering the evidence probe). Still not exercised on a device from here — no emulator in this environment.

---

Generated by Claude Code

[thymikee]

Superseded by #2369, which merged as 5ba4ac7 while this was in flight. Closing.

#2369 reaches the same conclusion and covers more ground: scroll top to re-anchor after the rotation, then a visibility probe before every canary interaction rather than only the two that had failed, at a finer 0.25 step. The Android lane is green on main at 5ba4ac7 (run 34089138374), where it failed at 233a34d. That is the outcome that matters, and this branch would only conflict with it now.

Worth recording from this PR's one CI run, since it cost an emulator cycle to learn:

• The failure was selector_not_found — the canary was not in the accessibility tree at all, not merely off screen. My original framing ("the blind scroll down 0.7 undershot") was wrong, and test(android): reveal smoke canaries by visibility #2369's scroll top anchor is the part that actually matters, because a downward-only search cannot reach a canary the rotation left above the viewport.

• One thing test(android): reveal smoke canaries by visibility #2369 inherits that is worth a follow-up: scrollToVisibleSelector probes with allowFailure, so its closing assert.fail never reaches the harness's failed-step evidence capture. I hit exactly this — the Android artifact dropped from 143,676 to 12,086 bytes, losing failed-step-N.png, -snapshot.json and -device.txt, which are the files that would have settled the diagnosis on the first run instead of the second. Spending one final probe as a real step before failing restores all three. The same gap exists in the iOS searchForVisibleElement. Happy to send that as a small standalone PR if it is wanted.

---

Generated by Claude Code

[thymikee]

The Android reveal fix already landed in #2369, which also checks the input results and alert canaries. This PR now conflicts with that change. Its shared helper also scrolls after any non-stalled failure, losing the typed error check now on main; transport or permission failures must not be treated as an off-screen element. I suggest closing this as superseded. Any later helper extraction should preserve all current visibility checks and fail immediately on unexpected errors.