feat(cli): restore cursor overlay through page.evaluate - #2869
Conversation
|
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Architecture diagram
sequenceDiagram
participant User as Browse CLI User
participant CLI as Browse CLI
participant Runtime as Runtime Command Handlers
participant Manager as Page Manager
participant Page as Active Browser Page
participant DOM as Page DOM Overlay
participant Browser as Browser Input and Renderer
Note over CLI,DOM: CLI-owned cursor visualization (no Stagehand core cursor API)
User->>CLI: browse cursor
CLI->>Runtime: Dispatch cursor command
Runtime->>Manager: Resolve activePage()
Manager-->>Runtime: Active page
Runtime->>Page: evaluate(CURSOR_OVERLAY_SCRIPT)
Page->>DOM: Ensure fixed overlay exists
alt Overlay already installed
DOM-->>Page: Reuse __browse_cursor_overlay__
else Overlay not installed
DOM->>DOM: Create fixed SVG cursor element
DOM->>DOM: Set aria-hidden, z-index, pointer-events none
DOM->>DOM: Append overlay to document root
DOM->>DOM: Install capture-phase mousemove listener
end
Page-->>Runtime: Evaluation complete
Runtime-->>CLI: Return enabled=true
CLI-->>User: Cursor overlay enabled
User->>CLI: browse mouse hover x y
CLI->>Browser: Move mouse to viewport coordinates
Browser->>DOM: Dispatch mousemove event
DOM->>DOM: Clamp clientX and clientY to non-negative values
DOM->>DOM: Update fixed overlay left and top
User->>CLI: browse screenshot --path cursor.png
CLI->>Page: Capture current page
Page->>Browser: Render DOM and overlay
Browser-->>CLI: Screenshot including visible cursor
CLI-->>User: Save cursor.png
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
dc60e5f to
8428e33
Compare
8428e33 to
638c86f
Compare
## Summary Import `packages/cli/**` exactly from the published `browse@0.9.6` V3 release, without changing its source or runtime behavior. This is intentionally a provenance checkpoint, not a line-by-line feature review. The imported source is kept runnable by a root, version-scoped pnpm override that resolves its unchanged Stagehand dependency to `3.7.1`. #2834 removes that override and starts the V4 migration. ## Exact-source provenance - Annotated tag: `browse@0.9.6` (`548c56407431db27823a212f53475443c7e8358d`) - Release commit: `1d49a95c0c230c346f8d50647e10303d6310fcd2` - Authoritative CLI tree: `b4048badce921cf54f199f96033d9a014ef977ec` - This PR's `HEAD:packages/cli` tree: `b4048badce921cf54f199f96033d9a014ef977ec` The tag's ignored README whitespace is retained too; formatting the import would invalidate the tree proof. ## Verification - Current remote head: `a77e1507b85e3c02553f36ead6ebd0237b0cccc6`, based on current `main`. - `HEAD:packages/cli` exactly equals the published V3 tree hash above. - pnpm 11 frozen install and the repository supply-chain release-age policy pass. - Browse lint, typecheck, and build pass; the full baseline suite passes: 25 files / 366 tests. - A fresh extension build still exactly matches the Go-embedded archive: SHA-256 `8efc7d171a625cca95c02d02d369b59435fae776cae6c7dd2f6fe72eb19785c0` on both files. This specifically verifies that adding the V3 dependency graph does not perturb the current V4 extension artifact. - This layer intentionally exercises V3 through the scoped Stagehand 3.7.1 override. V4 behavior starts in #2834. ## Stack (#2872) 1. **#2833 — exact Browse V3 baseline import** 2. #2834 — Stagehand V4 runtime and standard command parity 3. #2869 — CLI-owned cursor overlay 4. #2849 — CLI-private CDP sidecar; V3 network parity 5. #2835 — remove `--return-xpath`; supported V3 parity/release checkpoint 6. #2838 — eval and packaging integration 7. #2839 — managed Context names (fast-follow) 8. #2701 — shared Functions core consumer (fast-follow) ## Review and landing boundary Review this PR by verifying the tree hashes, dependency pin, root package wiring, and changeset—not by treating the imported V3 source as newly authored code. This head deliberately imports V3 code into the V4 repository and is not independently publishable. It lands only as the base of the complete stack. The framework network-event schema proposal in #2832 is intentionally outside this landing stack.
638c86f to
75d8e04
Compare
AzamAbdul
left a comment
There was a problem hiding this comment.
Feedback from Astra labeled with my take:
seems like a good idea to change to DOM readiness
- The overlay can disappear after navigation until the next mouse action.
The initialization script runs at document creation, when document.documentElement and document.body can still be absent. ensureCursor() then returns without scheduling another attempt. There is no DOMContentLoaded handler or retry, so a subsequent screenshot can have no cursor. V3 explicitly waited for DOM readiness and retried installation.
Fix: retry installation once the DOM is ready.
Reference:
not sure about this one
- A cursor rendering failure can prevent the actual mouse action.
Click, hover, scroll, and drag all await the overlay update before performing the action. If that extra page.evaluate() fails—for example, during navigation—the command aborts before sending its mouse input. The branch already treats the final drag overlay update as optional, but does not do so for these initial updates. V3 swallowed cursor-update errors throughout.
Fix: make visual position updates best-effort while continuing to propagate errors from the actual mouse operation.
Reference:
lets definitely do this to support backwards compatibility
- browse cursor changes its JSON output unnecessarily.
V3 returned { "cursor": "enabled" }; this branch returns { "enabled": true }. The command prints that result directly, so existing scripts checking .cursor break.
Fix: preserve the old response unless this is an intentional API change.
Reference:
…2834) ## Summary Migrate Browse's browser lifecycle and standard command surface together from Stagehand V3 to V4. - Replace the V3 constructor/init lifecycle with V4 browser factories and `Stagehand.create()`. - Support managed local, Browserbase remote, and attached CDP connection targets. - Preserve owned-versus-attached cleanup, daemon persistence, Browserbase session identity, and timeout handling. - Restore navigation, page information, deterministic locator actions, keyboard/mouse input, viewport/screenshot, snapshot, eval, and tab commands on V4 APIs. - Keep click/fill/select deterministic; this does not add a model-free structured `act()` path. - Make the remaining cursor, network, and coordinate-XPath gaps fail explicitly for the stack layers that restore or remove them. ## Stack (#2872) 1. #2833 — exact Browse V3 baseline import 2. **#2834 — Stagehand V4 runtime and standard command parity** 3. #2869 — CLI-owned cursor overlay 4. #2849 — CLI-private CDP sidecar; V3 network parity 5. #2835 — remove `--return-xpath`; supported V3 parity/release checkpoint 6. #2838 — eval and packaging integration 7. #2839 — managed Context names (fast-follow) 8. #2701 — shared Functions core consumer (fast-follow) ## Review shape The lifecycle and command migration remain two ordered implementation commits: 1. `389e2dae6` — V4 browser/session foundation and lifecycle ownership. 2. `b45167462` — standard command translation on that foundation. They are one PR because both commits rewrite the same nine command/test files. Reviewing their combined final diff avoids temporary deletion/stubbing followed by reimplementation, while the commits still provide useful lifecycle-versus-command checkpoints. Review follow-up `24178275f` adds narrowly scoped ownership, error-sanitization, and timeout guards. The resulting PR diff is 23 files, +988/−436. Cursor DOM injection and private CDP network transport remain separate because they are independently reviewable mechanisms and cleanly additive diffs. The legacy coordinate `returnXPath` request is still accepted here but fails explicitly until #2835 removes the option. This remains an intentionally non-publishable intermediate head. ## E2E Test Matrix Fresh post-flatten verification used the actual built CLI at final head `6f7e9c209`. Every daemon command used an isolated owner-only runtime directory. | Command / flow | Observed output | Confidence / sufficiency | | --- | --- | --- | | `pnpm install --frozen-lockfile` | Lockfile passed supply-chain policy, was already up to date, and installation completed | Proves the flattened stack resolves exactly from the committed lockfile | | `pnpm exec turbo run build --filter=browse` | Protocol, extension, Stagehand SDK, and Browse CLI built successfully (4/4 tasks) | Proves the CLI was tested against this head's protocol/extension/SDK artifacts, not stale workspace `dist` files | | Compare the rebuilt extension with `packages/sdk-go/internal/extensionassets/stagehand-extension.zip` | Exact byte match; both SHA-256 `8efc7d171a625cca95c02d02d369b59435fae776cae6c7dd2f6fe72eb19785c0`; archive manifest and package version both `1.0.2` | Confirms the TypeScript/CLI build and Go-embedded extension are synchronized | | Built CLI: `browse open <synthetic-data-url> --remote`; `browse status` | Remote browser connected and initialized; deterministic fixture loaded | Exercises production Browserbase provisioning plus the V4 daemon/session lifecycle on the exact final head | | `browse get text //h1`; `fill`; `select`; `click`; `is checked`; `wait selector`; `highlight`; `viewport`; `screenshot`; `snapshot --full` | XPath returned `Ready`; input became `Ada`; select became `b`; click produced `Clicked`; checkbox was true; PNG was 17,761 bytes; snapshot contained the fixture | Covers deterministic V4 reads, actions, waits, state, and rendering without an LLM | | Set a page marker; `tab new`; `tab list`; `tab close`; read the marker from a new CLI process | Tab count changed to 2 and the original page returned marker `yes` | Proves daemon persistence, active-tab handling, and state reuse across invocations | | Inspect the isolated runtime directory/PID; `browse stop`; poll the Browserbase session | Modes were `0700` / `0600`; the owned remote session reached `COMPLETED` | Proves owner-only daemon files and owned Browserbase resource cleanup | | `browse cursor`; `browse network on`; `browse mouse hover ... --return-xpath` | Each exited 1 with the intended explicit layer-boundary error | Confirms this intermediate layer fails honestly until the cursor, network, and flag-removal layers land | | `pnpm --filter browse test` | 25 files / 385 tests passed | Full Browse unit/integration suite on the exact final head | The runner has no Chrome/Chromium installation, so a fresh attached-CDP ownership smoke was not possible. Attached-browser non-ownership remains covered by the focused suite and is not claimed as a fresh live result here.
75d8e04 to
6a9d6aa
Compare
|
Addressed the three points from #2869 (review) in
I also reproduced the review findings on the prior head with the built CLI and a real Browserbase browser. The old code missed the overlay after 20/20 cross-origin navigations and aborted a coordinate click when the visual update was forced to fail. On this commit the overlay survived 20/20 navigations, the same forced visual failure still delivered the click, and the page recorded the click. Full Browse verification is green: lint passed and 26 test files / 390 tests passed. The PR description now contains the complete before/after matrix. |
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Addresses review feedback from AzamAbdul.
Addresses feedback from cubic-dev-ai[bot].
68f6fcb to
20f96c6
Compare
## Summary Restore Browse V3 network-capture behavior on Stagehand V4 through a CLI-private CDP sidecar, without committing core Stagehand, its protocol, or generated SDKs to a public network-event schema. ## Stack (#2872) 1. #2833 — exact Browse V3 baseline import 2. #2834 — Stagehand V4 runtime and standard command parity 3. #2869 — CLI-owned cursor overlay 4. **#2849 — CLI-private CDP sidecar; V3 network parity** 5. #2835 — remove `--return-xpath`; supported V3 parity/release checkpoint 6. #2838 — eval and packaging integration 7. #2839 — managed Context names (fast-follow) 8. #2701 — shared Functions core consumer (fast-follow) ## Architecture - Lazily open one CLI-private browser-level CDP WebSocket using the endpoint already held by the initialized Stagehand client. - Attach a flattened CDP session to the active V4 page target. - Present the unchanged V3 network writer with the same `on` / `off` / `send` session shape. - On `network off`, remove listeners, send `Network.disable`, and detach the page target. - Keep the browser-level sidecar WebSocket alive across off/on cycles; closing an auxiliary Browserbase socket during the live session can disconnect the browser. - Close the sidecar only during Browse session cleanup. - Never log or return the signed browser WebSocket endpoint. The request correlation and request/response JSON writer are inherited from the V3 CLI. No public protocol schema, extension, SDK, or generated-client changes are included. #2832 remains open outside the landing stack for that separate API-design discussion. ## User-visible behavior The V3 command and file surface is retained: ```bash browse network on browse network path browse network clear browse network off ``` Each request directory contains `request.json` and, when available, `response.json`, including the existing treatment of POST bodies, response bodies, failures, redirects, cache hits, and binary responses. ## E2E Test Matrix Full post-propagation implementation verification ran against exact clean #2849 head `adbe80d8bbfc337fea9dc38e41a88da18c287046`. Its seven-file network patch has the same stable patch ID (`f062ac789108ddf640bc912776dafa210411f50c`) as the previously stress-tested head, so the deterministic V3/V4 and MSN/CNN evidence below applies unchanged. Frozen install, fresh builds, focused/full tests, and the real Browserbase lifecycle were rerun on `adbe80d8b`. Current review-fix head `9887732b6f0c32cdef0966ebfd2ac2c454321d4b` changes only the test helper timeout diagnostic; on that exact head, the targeted network-capture tests passed 2/2 and Browse formatting, ESLint, and TypeScript checks passed. The comparison CLI was the exact built V3 implementation at `7365a20d52955c10d72606f2e6ddd74791609d13`. Every CLI flow used a unique daemon directory; no pre-existing daemon was reused or stopped. | Command / flow | Observed output | Confidence / sufficiency | | --- | --- | --- | | `pnpm install --frozen-lockfile`; build extension, local Stagehand SDK, then `browse` | Frozen install and all three builds passed; the CLI manifest was generated from the tested workspace. | Proves the clean stacked head installs from its lockfile and the subprocess tests exercised fresh V4 SDK/extension/CLI artifacts, not stale `dist` output. | | Built V3 vs built V4 deterministic fixture: `open → network on → eval(GET, cached GET twice, POST, redirect, binary, abort) → path → off → path → clear → stop` | V3: 8 requests / 7 responses. V4: 8 requests / 7 responses. Evaluation results matched; normalized request/response diff count was **0**; command-shape diff was `[]`; clear left 0 entries for both. | Exact parity for the stable V3 command and on-disk contract. Normalization was limited to request ID, timestamp/duration, HTTP `Date`, and fixture origin/port. | | Concurrent real CLI enables: two `browse network on` subprocesses launched together, followed by one marked navigation | One subprocess returned `{ enabled: true }`; the other returned `{ alreadyEnabled: true, enabled: true }`. The marked navigation produced exactly 1 request record with 1 unique request ID. | Proves overlapping `network on` calls serialize and attach listeners once rather than duplicating capture. | | Real CLI `on → navigate → off → on → navigate → off` without clearing | The two records were numbered `000` and `001`; both URLs remained present, and the SHA-256 of the first `request.json` was unchanged after the second cycle. | Proves off/on resumes at the next on-disk counter and does not overwrite retained captures. | | Real CLI permissions and lifecycle | Capture directories were `0700`; `request.json` and `response.json` were `0600`; `network clear` left 0 entries; `browse stop` reported stopped. | Proves the V3 private-file contract and cleanup behavior through the built CLI. | | Real Browserbase, two `network on → open Example Domain → off` cycles in one session, then navigation with capture off | Both cycles captured GET 200 and the complete Example Domain body; the Browserbase session stayed stable; post-off `open --wait networkidle` worked; final status was connected, initialized, and remote; stop completed. | Proves the auxiliary sidecar survives real remote off/on while the owning Stagehand browser remains usable. No session ID or signed endpoint is included here. | | Real Browserbase stress, V3 and V4, public MSN then CNN; each cycle ran `on → path → open → scroll → collect 12s → off → path → navigate while off → status → clear` | All 21 commands completed for each CLI. V3 captured MSN 415/401 and CNN 192/187 request/response records. V4 captured MSN 498/481 and CNN 194/187. Across all four cycles: 0 malformed request JSON, 0 malformed response JSON, 0 response-ID mismatches, 0 new records after off, and 0 entries after clear. Both sessions stayed remote/connected, stopped cleanly, and emitted no WebSocket endpoint. | Proves bounded high-concurrency, cross-origin Document/Script/Fetch/XHR/Image/Font/Media traffic. Live request counts are intentionally not expected to match because sites and ad auctions are nondeterministic; contract invariants and command shapes matched. | | `pnpm --filter browse lint` | Passed formatting, ESLint, and TypeScript checks at full-verification head `adbe80d8b`; passed again at current review-fix head `9887732b6`. | Static support for both the fully exercised implementation head and the exact current head. | | `pnpm --filter browse test` with isolated daemon directory | At full-verification head `adbe80d8b`: **27 files / 390 tests passed**; focused network subset: **3 files / 30 tests passed**. At current review-fix head `9887732b6`: targeted network-capture tests passed **2/2**. | Covers sidecar routing, response bodies, listener teardown, attach/detach, request-write races, counter reuse, and overlapping enable serialization in addition to the real flows above; the current-head rerun is scoped to the test-only diagnostic change. | The deterministic artifact comparison includes method, URL/path, request/response headers and bodies, status/status text, MIME type, resource type, error shape, counter naming, and file modes. The live stress test also observed GET/POST/OPTIONS, failures, base64 bodies, cross-origin documents, and request-only records for traffic still in flight at the bounded `off` point. This matrix does not claim WebSocket-frame, SSE-message, service-worker, or every out-of-process-iframe edge-case coverage; those are outside the V3 JSON request/response file contract proven here.
## Summary Remove the unsupported `--return-xpath` option from Browse's coordinate click, hover, scroll, and drag commands. The CLI rejects the flag, and strict driver schemas reject stale `returnXPath` payloads before performing a mouse action. Update the examples and the Browse minor changeset for the V4 migration and flag removal. XPath selectors and snapshot `xpathMap` remain supported. ## Scope The cursor overlay (#2869) and network sidecar (#2849) are already merged into `main`. This branch includes the final parent and reconciles its squash merge, preserving the current cursor fixes. The final diff against `main` is **8 files, +17/−62**, limited to the XPath flag removal, documentation, removal of the obsolete XPath test, and release metadata. ## Validation - At `db5e592ce4359e1340ef4e921aad4b98581ee94a`: focused driver-command suite **28 tests passed** with an isolated daemon directory; changed-file formatting and ESLint, Browse TypeScript checks, and `git diff --check` passed. - Production code is unchanged from `c15d1ef8ca8a53a25a2f5f05722ef45df9a89605`, which passed a frozen install, fresh extension/SDK/Browse builds, Browse formatting/lint/type checks, changeset validation, and the full **28-file / 397-test** Browse suite. - Built CLI smoke checks at that production-code head: `--return-xpath` was absent from all four mouse help pages; passing it to each command exited 2 with `Nonexistent flag`, without starting a daemon. - Confirmed the final network parent and its squash merge have identical trees, and the squash reconciliation did not change the resolved files.
Summary
Restore Browse's visible cursor as a CLI-owned DOM overlay, without adding a cursor API to core Stagehand V4.
cursor-overlay.tsfile.page.evaluate(CURSOR_OVERLAY_SCRIPT)and for future navigations throughpage.addInitScript(...).DOMContentLoadedwhen the init script runs before the document root exists.browse cursorJSON response:{ "cursor": "enabled" }.Stack (#2872)
--return-xpath; supported V3 parity/release checkpointWhy this is separate
The cursor is a self-contained optional visual feature with different review concerns from the combined V4 runtime/command migration: injected DOM/CSS, idempotency, event handling, and screenshot behavior. Keeping it additive on #2834 lets this feature be reviewed or reverted without disturbing browser lifecycle or commands.
E2E Test Matrix
Review-feedback verification compared the exact prior head
6a9d6aa09with fixed implementation head1035fbbf5through the built CLI and real Browserbase browsers. Final head68f6fcb2conly expands automated coverage and does not change runtime code. Targets were the publicexample.comandexample.orgpages.#__browse_cursor_overlay__before any mouse input0after 20/20 navigations1after 20/20 navigations (0 misses)DOMContentLoadedretry restores the overlay after navigation in the real browser pathbrowse mouse click 200 200against an oversized synthetic button1; the page's click state remainednull{ "clicked": true }; page click state became"yes"browse cursor{ "cursor": "enabled" }(prior head returned{ "enabled": true })pnpm --filter browse lintpnpm --filter browse test:clibrowse stopafter each live runThe already-uploaded screenshot below remains representative visual proof of the same overlay behavior.
No LLM path or customer data was involved.