feat(native): docs, gate wiring, CodeQL triage, and fixes the gates found (phase 6)

README.md

@@ -770,6 +770,46 @@ Windows are not measured. `python scripts/verify_npx_e2e.py` reproduces it. See
 **Deploying it as a remote server (Cloudflare Containers):** see
 [docs/CLOUDFLARE.md](https://github.com/Intrect-io/lyra-browser/blob/master/docs/CLOUDFLARE.md).
 
+## Drivers
+
+lyra drives the browser through one of three drivers, picked with `LYRA_BROWSER_DRIVER`:
+
+| Driver | Needs | Notes |
+|---|---|---|
+| `auto` (default) | `patchright` or `playwright` | `patchright` when installed, else `playwright`. |
+| `patchright` / `playwright` | that package | Pins one. |
+| `native` (opt-in) | Chrome, Edge or Chromium, nothing else | lyra's own driver (`lyra_browser.native`): Chrome over raw DevTools on lyra's own WebSocket. No Playwright, no Node, no chromedriver. Measured: the real-browser runs pass with `playwright` and `patchright` blocked from importing, and no Playwright module loads. |
+
+What `native` does differently, on purpose:
+
+- **No `Runtime.enable`.** That is the one DevTools call Kasada refuses a browser on.
+  Scripts run in an isolated world the page cannot see or tamper with. A script that
+  reads the page's own globals (`window.CKEDITOR`) asks for the main world. The UAT
+  recorder's console and page-error observers are the only thing that turns
+  `Runtime.enable` on, and only for the pages they observe.
+- **The sandbox stays on.** Playwright always passes `--no-sandbox`. This browser holds
+  real logins, so `native` keeps Chrome's renderer sandbox unless `LYRA_BROWSER_SANDBOX`
+  is off (a container without user namespaces).
+- **The `cdp` guard, always.** `native` has no `context.route`. Asking for it selects
+  `LYRA_BROWSER_GUARD=cdp`, which judges every redirect hop.
+- **The same switches as Playwright otherwise.** The list was measured from Playwright
+  1.64's launch. It keeps a shared profile's cookies readable across drivers and keeps
+  the back/forward cache off, which the guard relies on.
+
+Status: opt-in, not yet at parity. Selectors are css (open shadow roots pierced),
+`text=`, `role=…[name=…]` (from Chrome's accessibility tree), `xpath=`, `id=`,
+`data-testid=`, `nth=`, `:visible` and `:has-text()`; any other engine is refused by name.
+`read_page(mode="tree")` matched Playwright's on 75 of 75 elements in the same in-view
+order, and its refs reach into same-site and cross-site iframes
+(`python scripts/verify_native_parity_e2e.py`). The repo's own gates run on it with
+`LYRA_BROWSER_DRIVER=native python scripts/verify_<gate>_e2e.py` (or `--driver native`
+where a gate has that flag). On 2026-10-23 they found gaps the parity check does not
+cover: Playwright calls the gates make directly (`page.content`, `page.input_value`,
+`context.set_default_timeout`), the `internal:control=enter-frame` selector, a key press
+after some flows, a navigation away from a `beforeunload` page, a download a popup starts,
+and an unapproved popup's tab. Until those gates pass, keep the default driver.
+
+
 ## Configuration (env)
 
 | Var | Default | Meaning |

docs/VEGA_INTEGRATION.md

@@ -32,6 +32,23 @@ moving parts, two answers:
    VEGA's UI should detect this status and show a one-click "Install Chrome"
    prompt — the UI-first onboarding path, not a CLI error.
 
+**Without Playwright at all (opt-in, `LYRA_BROWSER_DRIVER=native`):** lyra's own driver
+drives the installed Chrome or Edge over raw DevTools. Point 1 then goes away:
+VEGA does not have to bundle `playwright`, and the refs work whatever Playwright version
+VEGA's runtime would have shipped. The tree view comes from Chrome's own accessibility
+tree. Point 2 is unchanged (the user's Chrome or Edge, or `browser_unavailable`).
+Set it in the server's `env`:
+
+```json
+"env": {"LYRA_BROWSER_DRIVER": "native"}
+```
+
+It runs with the `cdp` guard, keeps Chrome's sandbox on and never sends
+`Runtime.enable`; see the README's "Drivers". `playwright` is still a declared
+dependency of the package until `native` becomes the default. A frozen VEGA runtime
+that leaves it out works with `native` (measured with the import blocked), but not with
+`auto`.
+
 The pip + `playwright install chromium` commands below are the **developer** path
 for working on this repo, never something an end user runs.
 

scripts/verify_browser_e2e.py

@@ -104,6 +104,9 @@ async def build_tools(data_dir: Path, *, headless: bool, **overrides):
         if "LYRA_BROWSER_GUARD" in os.environ
         else {}
     )
+    # LYRA_BROWSER_DRIVER=native runs it on lyra's own driver (no Playwright), the same way.
+    if "LYRA_BROWSER_DRIVER" in os.environ:
+        backend["driver"] = os.environ["LYRA_BROWSER_DRIVER"]
     cfg = Config(data_dir=data_dir, headless=headless, **backend, **overrides)
     # The gate promises to leave the user's data alone; with VEGA_DATA_DIR set
     # the capture dir would otherwise resolve into the real uploads root.

scripts/verify_cdp_guard_e2e.py

@@ -2359,7 +2359,9 @@ def main() -> int:
     group = parser.add_mutually_exclusive_group()
     group.add_argument("--headless-only", action="store_true")
     group.add_argument("--headful-only", action="store_true")
-    parser.add_argument("--driver", choices=["auto", "playwright", "patchright"], default="auto")
+    parser.add_argument(
+        "--driver", choices=["auto", "playwright", "patchright", "native"], default="auto"
+    )
     parser.add_argument(
         "--popup-runs", type=int, default=10, help="runs of every popup scenario (scenarios)"
     )

scripts/verify_click_refusal_e2e.py

@@ -725,7 +725,9 @@ def main() -> int:
     group = parser.add_mutually_exclusive_group()
     group.add_argument("--headless-only", action="store_true")
     group.add_argument("--headful-only", action="store_true")
-    parser.add_argument("--driver", choices=["auto", "playwright", "patchright"], default="auto")
+    parser.add_argument(
+        "--driver", choices=["auto", "playwright", "patchright", "native"], default="auto"
+    )
     args = parser.parse_args()
     asyncio.run(async_main(args.headless_only, args.headful_only, args.driver))
     return 0

scripts/verify_downloads_e2e.py

@@ -191,14 +191,23 @@ def everything_in(directory: Path) -> list[str]:
     return sorted(p.name for p in directory.rglob("*")) if directory.exists() else []
 
 
+# Where each driver parks a download before lyra decides on it: Playwright's artifacts
+# dir, or the native driver's private download dir (lyra_browser.native.context).
+_PARKING_DIRS = ("playwright-artifacts-*", "lyra-native-downloads-*")
+
+
 def browser_temp_files(root: Path) -> set[str]:
-    """Files in this run's Playwright artifacts dir: where the browser parks a download.
+    """Files in this run's download parking dirs: where the browser parks a download.
 
     ``root`` is the private TMPDIR the run gave the browser (``isolate_tmpdir``), so
     a browser some other process is running cannot put a file in this answer.
     """
     return {
-        str(p) for d in root.glob("playwright-artifacts-*") for p in d.rglob("*") if p.is_file()
+        str(p)
+        for pattern in _PARKING_DIRS
+        for d in root.glob(pattern)
+        for p in d.rglob("*")
+        if p.is_file()
     }
 
 

scripts/verify_native_parity_e2e.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""Compare the native driver's tree view with Playwright's, then act through its refs.
+
+Same fixture page, read through lyra's tools on both drivers (``read_page(mode="tree")``):
+the elements and their in-view order have to be the same. Then, on the native driver
+only, refs are used the way a model uses them: a button in the top frame, in a same-site
+iframe and in a cross-site iframe (an out-of-process frame), Korean typed into a field
+of that cross-site frame, and a ref from before a navigation, which must answer
+``not_found``.
+
+Needs Google Chrome and ``playwright>=1.60`` (the baseline side only). Not part of the
+unit suite: ``python scripts/verify_native_parity_e2e.py [--headful]``.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import re
+import sys
+import tempfile
+import threading
+from collections import Counter
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+
+from fastmcp import Client
+
+from lyra_browser.config import Config
+from lyra_browser.server import build_server
+
+MAIN = """<!doctype html><title>Tree</title>
+<h1>Tree page</h1>
+<nav><a href="/next">Next page</a> <a href="/logo"><img alt="Logo" width="20" height="20"
+  src="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs="></a></nav>
+<label>Email <input type="email"></label>
+<select aria-label="Country"><option>Korea</option><option selected>Japan</option></select>
+<label><input type="checkbox" checked> Remember me</label>
+<button onclick="this.textContent='top hit'">Top button</button>
+<button disabled>Disabled button</button>
+<div role="tablist"><div role="tab" aria-selected="true">Tab one</div>
+  <div role="tab" aria-selected="false">Tab two</div></div>
+<h2>한국어 제목</h2>
+<shadow-host></shadow-host>
+<script>customElements.define('shadow-host', class extends HTMLElement {
+  constructor() { super(); this.attachShadow({mode: 'open'}).innerHTML =
+    '<button>Shadow button</button>'; } });</script>
+<iframe src="/inner" title="same" width="400" height="80"></iframe>
+<iframe src="http://localhost:CROSS/cross" title="cross" width="400" height="80"></iframe>
+<div id="list"></div>
+<script>for (let i = 0; i < 60; i++) { const b = document.createElement('button');
+  b.textContent = 'Item ' + i; document.getElementById('list').appendChild(b); }</script>"""
+INNER = """<!doctype html><button onclick="this.textContent='inner hit'">Inner button</button>"""
+CROSS = """<!doctype html>
+<button id="cb" onclick="this.textContent='cross hit'">Cross button</button>
+<input aria-label="Cross field" onkeydown="if(event.key==='Enter')
+  document.getElementById('cb').textContent='typed:'+this.value">"""
+
+
+def serve(pages: dict) -> ThreadingHTTPServer:
+    class Handler(BaseHTTPRequestHandler):
+        def do_GET(self):  # noqa: N802 — http.server API
+            body = pages.get(self.path.split("?")[0], "<title>other</title>other")
+            body = body() if callable(body) else body
+            self.send_response(200)
+            self.send_header("Content-Type", "text/html; charset=utf-8")
+            self.end_headers()
+            self.wfile.write(body.encode())
+
+        def log_message(self, *args):
+            pass
+
+    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+    threading.Thread(target=server.serve_forever, daemon=True).start()
+    return server
+
+
+_REF = re.compile(r"^aria-ref=\S+ ")
+
+
+async def tree_for(driver: str, url: str, keep_open: bool = False):
+    data = Path(tempfile.mkdtemp(prefix=f"lyra-tree-{driver}-"))
+    cfg = Config(
+        client="generic",
+        headless="--headful" not in sys.argv,
+        driver=driver,
+        guard_backend="cdp",
+        consent_channel="off",
+    )
+    cfg.data_dir = data
+    cfg.__post_init__()
+    cfg.consent_channel = "off"
+    client = Client(build_server(cfg))
+    await client.__aenter__()
+
+    async def call(tool: str, **args):
+        res = await client.call_tool(tool, args, raise_on_error=False)
+        return res.structured_content or {
+            "text": [c.text for c in res.content if hasattr(c, "text")]
+        }
+
+    await call("open_browser")
+    await call("navigate", url=url, reason="parity", wait_until="load")
+    await asyncio.sleep(0.8)
+    tree = await call("read_page", mode="tree", max_chars=200000)
+    if not keep_open:
+        await call("close_browser", reason="parity")
+        await client.__aexit__(None, None, None)
+        return tree, None, None
+    return tree, call, client
+
+
+async def main() -> int:
+    cross = serve({"/cross": CROSS})
+    site = serve(
+        {"/": lambda: MAIN.replace("CROSS", str(cross.server_address[1])), "/inner": INNER}
+    )
+    url = f"http://127.0.0.1:{site.server_address[1]}/"
+    failures: list[str] = []
+
+    def check(name: str, ok: bool, detail) -> None:
+        if not ok:
+            failures.append(name)
+        print(
+            ("PASS " if ok else "FAIL ")
+            + name
+            + " :: "
+            + json.dumps(detail, ensure_ascii=False, default=str)[:400]
+        )
+
+    pw_tree, _, _ = await tree_for("playwright", url)
+    nat_tree, call, client = await tree_for("native", url, keep_open=True)
+    pw_lines = [_REF.sub("", ln) for ln in pw_tree.get("text", "").splitlines()]
+    nat_lines = [_REF.sub("", ln) for ln in nat_tree.get("text", "").splitlines()]
+    only_pw = Counter(pw_lines) - Counter(nat_lines)
+    only_nat = Counter(nat_lines) - Counter(pw_lines)
+    check(
+        "tree_elements_match",
+        not only_pw and not only_nat,
+        {
+            "playwright": len(pw_lines),
+            "native": len(nat_lines),
+            "only_playwright": list(only_pw.elements()),
+            "only_native": list(only_nat.elements()),
+        },
+    )
+    check(
+        "tree_refs_on",
+        nat_tree.get("refs") is True and pw_tree.get("refs") is True,
+        {"pw": pw_tree.get("refs"), "native": nat_tree.get("refs")},
+    )
+    check(
+        "in_viewport_count_matches",
+        pw_tree.get("in_viewport") == nat_tree.get("in_viewport"),
+        {"pw": pw_tree.get("in_viewport"), "native": nat_tree.get("in_viewport")},
+    )
+    first_pw = [ln for ln in pw_lines[: pw_tree.get("in_viewport") or 0]]
+    first_nat = [ln for ln in nat_lines[: nat_tree.get("in_viewport") or 0]]
+    check(
+        "in_viewport_order_matches",
+        first_pw == first_nat,
+        {"pw": first_pw[:6], "native": first_nat[:6]},
+    )
+
+    def ref_of(text: str, needle: str) -> str:
+        for ln in text.splitlines():
+            if needle in ln:
+                return ln.split(" ", 1)[0].removeprefix("aria-ref=")
+        return ""
+
+    text = nat_tree.get("text", "")
+    for label, needle, after in (
+        ("ref_click_top", 'button "Top button"', 'button "top hit"'),
+        ("ref_click_same_site_iframe", 'button "Inner button"', 'button "inner hit"'),
+        ("ref_click_cross_site_oopif", 'button "Cross button"', 'button "cross hit"'),
+    ):
+        ref = ref_of(text, needle)
+        r = await call("click", selector=f"aria-ref={ref}", reason="parity")
+        tree = await call("read_page", mode="tree", max_chars=200000)
+        check(
+            label,
+            r.get("status") == "ok" and after in tree.get("text", ""),
+            {"ref": ref, "click": r},
+        )
+        text = tree.get("text", "")
+
+    ref = ref_of(text, 'textbox "Cross field"')
+    r = await call(
+        "type_text", selector=f"aria-ref={ref}", value="안녕 OOPIF", submit=True, reason="parity"
+    )
+    tree = await call("read_page", mode="tree", max_chars=200000)
+    check(
+        "ref_type_korean_into_oopif",
+        r.get("status") == "ok" and "typed:안녕 OOPIF" in tree.get("text", ""),
+        {"ref": ref, "type": r},
+    )
+
+    stale = ref_of(tree.get("text", ""), "Item 5")
+    await call("navigate", url=url + "other", reason="parity", wait_until="load")
+    r = await call("click", selector=f"aria-ref={stale}", reason="parity", timeout_ms=1500)
+    check("stale_ref_not_found", r.get("status") == "not_found", r)
+
+    await call("close_browser", reason="parity")
+    await client.__aexit__(None, None, None)
+    site.shutdown()
+    cross.shutdown()
+    print("\nFAILED:" if failures else "\nALL PASSED", failures or "")
+    return 1 if failures else 0
+
+
+if __name__ == "__main__":
+    sys.exit(asyncio.run(main()))

scripts/verify_redirect_hop_e2e.py

@@ -1302,7 +1302,9 @@ def main() -> int:
     group = parser.add_mutually_exclusive_group()
     group.add_argument("--headless-only", action="store_true")
     group.add_argument("--headful-only", action="store_true")
-    parser.add_argument("--driver", choices=["auto", "playwright", "patchright"], default="auto")
+    parser.add_argument(
+        "--driver", choices=["auto", "playwright", "patchright", "native"], default="auto"
+    )
     parser.add_argument(
         "--guarantee",
         choices=["after", "before"],

scripts/verify_scriptredirect_e2e.py

@@ -262,7 +262,9 @@ def main() -> int:
     group = parser.add_mutually_exclusive_group()
     group.add_argument("--headless-only", action="store_true")
     group.add_argument("--headful-only", action="store_true")
-    parser.add_argument("--driver", choices=["auto", "playwright", "patchright"], default="auto")
+    parser.add_argument(
+        "--driver", choices=["auto", "playwright", "patchright", "native"], default="auto"
+    )
     args = parser.parse_args()
     asyncio.run(async_main(args.headless_only, args.headful_only, args.driver))
     return 0

scripts/verify_snapshot_e2e.py

@@ -633,7 +633,9 @@ def main() -> int:
     group = parser.add_mutually_exclusive_group()
     group.add_argument("--headless-only", action="store_true")
     group.add_argument("--headful-only", action="store_true")
-    parser.add_argument("--driver", choices=["auto", "playwright", "patchright"], default="auto")
+    parser.add_argument(
+        "--driver", choices=["auto", "playwright", "patchright", "native"], default="auto"
+    )
     args = parser.parse_args()
     asyncio.run(async_main(args.headless_only, args.headful_only, args.driver))
     return 0

src/lyra_browser/native/context.py

@@ -15,6 +15,7 @@
 
 import asyncio
 import contextlib
+import os
 import shutil
 import tempfile
 from collections.abc import Callable
@@ -64,7 +65,11 @@ def __init__(self, process: BrowserProcess, conn: Connection, options: dict) ->
         self._handlers: dict[str, list[Callable]] = {}
         self._downloads: dict[str, Download] = {}
         self._download_log: list[tuple[str, str]] = []  # (page target id, guid), in order
-        self._download_dir = Path(tempfile.mkdtemp(prefix="lyra-native-downloads-"))
+        # TMPDIR as it is now, not as tempfile cached it at first use: a caller that
+        # points TMPDIR at a private dir (the downloads gate does) gets the files there.
+        self._download_dir = Path(
+            tempfile.mkdtemp(prefix="lyra-native-downloads-", dir=os.environ.get("TMPDIR") or None)
+        )
         self._started = False
         self._closing = False
         self._closed = False