Repository navigation
feat(native): the tree view and aria-refs, into cross-site iframes too (phase 4) - #11
Merged
Merged
Conversation
unohee
force-pushed
the
feat/native-locators
branch
from
October 25, 2026 03:57
7ef0ae0 to
38fe0db
Compare
…o (phase 4) read_page(mode="tree") now runs on the native driver, and click / type_text / any element tool take its aria-refs. - snapshot: Accessibility.getFullAXTree frame by frame, written as Playwright's aria_snapshot(mode="ai", boxes=True) so compact_snapshot reads it unchanged. Chrome's internal roles are mapped back (MenuListOption is an option, StaticText a text), a link's /url is its href as written (Chrome's tree has the resolved URL), boxes are in the element's own frame's viewport as Playwright gives them, and refs go to every rendered node with a box: e<n> in the top frame, f<k>e<n> in the k-th frame met. A snapshot replaces the page's ref table, so refs belong to the latest read. - frames: out-of-process iframes are auto-attached (no waitForDebuggerOnStart, as for tabs: cdp_guard holds them) and every frame maps to the session that speaks for its document; nested OOPIFs included. Coordinates from an OOPIF session get the owner <iframe> content boxes added on the way out, so input lands where it should in the window. - locator: elements carry their frame (and session) through every action; an aria-ref selector switches to the ref's frame, an unknown or stale one matches nothing, and one into a frame that is gone raises Playwright's "Invalid frame in aria-ref selector" (read as not_found). Verified: pytest 1745 passed (2 pre-existing macOS sha256sum failures deselected); tests/native 82; testing/native_tree_parity_261023_v1.py, headless and headful (headful twice): read_page(mode="tree") on the native and the playwright driver gives the same 75 elements in the same in-view order; refs click a button in the top frame, in a same-site iframe and in a cross-site OOPIF, type Korean into an OOPIF field and submit it, and a ref from before a navigation answers not_found. The phase 1-2 and phase 3 smokes still pass.
unohee
force-pushed
the
feat/native-snapshot
branch
from
October 25, 2026 03:59
1f9d938 to
95aa236
Compare
unohee
marked this pull request as ready for review
October 25, 2026 03:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #10 (phase 3), which is stacked on #9 (phases 1–2). This is phase 4 of the plan.
What
native/snapshot.pybuilds the tree fromAccessibility.getFullAXTree, one frame at a time. Its output follows Playwright'saria_snapshot(mode="ai", boxes=True)format, socompact_snapshotreads it unchanged./urlis thehrefattribute as written.e<n>in the top frame andf<k>e<n>inside frames. Each snapshot replaces the page's ref table.native/frames.pyauto-attaches out-of-process iframes and maps each frame to the session that controls it, nested OOPIFs included. For an element inside an OOPIF, the offset of the owner<iframe>is added to its coordinates, so input lands in the right place.native/locator.pyaria-refresolves in the ref's own frame.Invalid frame in aria-ref selector, which the tools report asnot_found.Verified
pytest: 1745 passed. The two pre-existing macOSsha256sumfailures were deselected.tests/native: 82 tests.testing/native_tree_parity_261023_v1.pypasses headless and headful (headful run twice):read_page(mode="tree"), native vs. playwright drivernot_foundThe phase 1–2 smoke and the phase 3 smoke still pass.