Skip to content

feat(remote): proxy parity for Simulator observation (#2198 slice B) - #2351

Merged
thymikee merged 9 commits into
codex/2198-runner-demandfrom
codex/2198-proxy-parity
Sep 7, 2026
Merged

feat(remote): proxy parity for Simulator observation (#2198 slice B)#2351
thymikee merged 9 commits into
codex/2198-runner-demandfrom
codex/2198-proxy-parity

Conversation

@thymikee

@thymikee thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member

Stacked on #2329 (slice A). Part of #2198; tracker #2188.

What this closes

Slice B of #2198: a client behind agent-device proxy gets exactly what a client on the daemon host gets. One deterministic Simulator acquisition fixture, one script, run direct and proxied, compared with transport identity removed. Three places where the proxied leg diverged are fixed:

  1. Cancellation stopped at the proxy. The proxy's upstream fetch only carried AbortSignal.timeout, so a remote client that gave up mid-request left the daemon request (and any runner work it owned) running to completion. The upstream request is now bound to the client's connection: when the client vanishes before the response ends, the upstream socket is dropped and the daemon's own disconnect path fires markRequestCanceled for that one request, exactly as it does for a direct client. Response completion is guarded, so a normal end never aborts. Test: a daemon behind the proxy whose handler waits on its request signal; the client destroys its socket; the signal aborts. Red without the change (5 s timeout).
  2. Failure records were unreachable through the proxy. The proxy answered 404 for GET /sessions/<session>/requests/<id>/diagnostics (A remote caller cannot read the diagnostics log the error names #1801), so every proxied failure came back with logPathUnavailable: … HTTP 404 while the direct client localizes the record into logPath. The proxy now forwards that one route, GET only, client-authenticated first, upstream-authenticated by the daemon as before; the route still enumerates nothing (/sessions/<s>/requests stays 404). This forwards an existing daemon route the remote client already speaks; it adds no new endpoint semantics. Test: the parity suite (red without it: the logPathUnavailable line reappears) plus a proxy unit test pinning method, path, auth header and the 401/404 edges.
  3. Version skew behind the proxy failed at the RPC, not at health. The client compared only the proxy's rpcProtocolVersion. The proxy already nests the daemon's health as upstream; the client now reads that link and refuses the chain at health when any link speaks another protocol, naming the skewed link (remoteService: agent-device-daemon). Additive per ADR 0006: the field is optional, payloads without it parse as before, request shapes are unchanged; the wire ledger carries the digests and acks. Tests: loopback health servers for compatible chain, skewed daemon behind a compatible proxy, and a plain daemon payload.

Parity and lease evidence

test/integration/provider-scenarios/remote-proxy-parity.test.ts, over a scripted runner (repeat snapshot) and scripted simctl:

  • opensnapshot -isnapshot --rawdiff snapshotclick @e404 (typed failure) → appstatecloseopendiff snapshotclose, run direct through the harness request boundary and proxied through a real daemon HTTP server + proxy + the real remote client. Every response is deep-equal after removing request ids, diagnostic ids, timing (measuredAt, p50Ms…), the per-open refsGeneration, and per-world temp roots. On failures, every envelope field the direct leg carries (hint, details, diagnosticId, logPath) is present on the proxied leg.
  • Session cleanup does not carry comparison state: the diff after close/open reports baselineInitialized: true on both legs.
  • Two proxied clients (different sessions and workspaces) contending for one device: the second fails with DEVICE_IN_USE + hint at claim admission, with no simctl call beyond list and no runner call recorded after the refusal.
  • Lease heartbeat through the proxy (clock-driven LeaseRegistry): a leased session captures a diff baseline; an explicit lease_heartbeat RPC just before the lease's reported expiry moves the expiry forward; a request past the old expiry but inside the renewed window still finds the session and its baseline (red without the renewing heartbeat: LEASE_NOT_FOUND).
  • Lease expiry and reacquisition through the proxy (clock-driven LeaseRegistry): a leased session captures a diff baseline; its lease lapses past the proxy TTL without a heartbeat; the next proxied request is refused UNAUTHORIZED / LEASE_NOT_FOUND with a hint and the session is torn down; a freshly allocated lease reopens and its first diff reports baselineInitialized: true, so no capture or comparison state crosses the expired lease.

The provider scenario harness exposes handleRequest and token for this; nothing else in the harness changes.

Already covered, cited not re-added

  • Cancellation inside the daemon: the request signal reaches the Apple open probe (awaitObservable checks it between polls) and the runner client; suites session-test-suite-command-cancellation, snapshot-command-runtime, request-execution-scope.
  • Lease heartbeat and provider-side expiry release: lease-lifecycle.test.ts, provider-lease-expiry.test.ts, session-device-claims.test.ts.

Non-goals honored

No new authentication model, lease model, public flag, or RPC redesign. Bridge state stays private: nothing about the runner or the observation probe crosses the proxy.

Validation

  • Focused: daemon-proxy.test.ts (13), daemon-client-health-compat.test.ts (3), daemon-client.test.ts (36), remote-request-diagnostics.test.ts, remote-proxy-parity.test.ts (4), remote-daemon-client.test.ts, remote-proxy-artifact-tenant.test.ts, test/wire-compat (39), test-file size ratchet. Planted-red for all three fixes recorded above, and for the expiry regression (clock not advanced → the "expired" request succeeds and the test fails).
  • Stacked on perf(ios): derive runner demand for Simulator opens and stop observation from awaiting the runner #2329; this branch merges its head 325343d (slice A merged with main, the runner release with the deferred-start fix, and the discovery-wait fix).
  • pnpm typecheck green.
  • Controlled-RTT proxy bench (0 / 20 / 80 ms, persistent client and fresh-process CLI, 20 samples per cell) base 27a97ee vs head 318d510 on the dedicated Simulator: unchanged within noise at every point (largest move −8 ms on a 93 ms cell), identical response size, 0 failures; table and raw results in the PR comment and on evidence/ios-snapshot (2198-slice-b-rtt-318d510769/). test(ios): establish snapshot convergence baselines and permanent evidence #2189's ios-snapshot-proxy-71fb2483f is the reference profile (same cells and RTT points, different Simulator and runtime).
  • pnpm check:affected --run on the exact head: posted as a comment when it completes.

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.51 MB 4.51 MB +581 B
Package (unpacked) 4.51 MB 4.51 MB +581 B
Package (download) 1.34 MB 1.34 MB +219 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.1 ms 28.0 ms +0.9 ms
CLI --help 79.7 ms 80.9 ms +1.2 ms

…, cancellation, and version check

#2198 slice B. Direct-daemon and proxy execution over the same deterministic Simulator fixture
now publish the same responses, and the three places where they did not are closed:

- A client that disconnects mid-request behind the proxy now cancels the daemon request. The
  proxy's upstream fetch is bound to its client's connection, so the daemon's own disconnect
  cancellation (`markRequestCanceled`) fires exactly as it does for a direct client.
- The proxy forwards `GET /sessions/<session>/requests/<id>/diagnostics` (#1801), so a remote
  client behind it localizes a failure's diagnostics record instead of reporting
  `logPathUnavailable: HTTP 404`. GET only; the route still enumerates nothing.
- The client's ADR 0006 health check reads the `upstream` link a proxy's /health already nests:
  a proxy whose daemon speaks another RPC protocol fails at health, before the command RPC.

The provider scenario harness exposes its request boundary so a scenario daemon can sit behind a
real HTTP server and proxy; the new parity suite runs one script direct and proxied and compares
the published responses with transport identity removed, and proves two proxied clients
contending for one device fail at claim admission before any lifecycle call.
@thymikee
thymikee force-pushed the codex/2198-proxy-parity branch from f654a39 to dfc9149 Compare September 6, 2026 09:31
@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

The forwarding changes look sound, but the validation is not complete yet. The new tests cover contention and explicit close/reopen; they do not exercise proxy-path heartbeat, lease expiry and reacquisition with fresh generation state, as required by #2198. Add that case and show it fails when stale state is retained. The controlled 0/20/80 ms RTT comparisons and the remaining validation listed in the PR are also still pending. Keep this behind #2329 in merge order.

* codex/2198-runner-demand:
  feat(apple): release a speculative runner when the plan is proven observation-only
  refactor(daemon): move four pure leaves to their kits (#2347)
  fix(ios): stop charging every capture for a slow Simulator app discovery (#2331)
  refactor(commands): move commands-side rendering out of src/daemon and retire the doctor progress flag (#2349)
… a reacquired lease starts clean

#2198 acceptance: lease expiry, session cleanup, and device reacquisition do not reuse prior
capture or comparison state. The parity world takes a clock-driven LeaseRegistry; a leased
session behind the proxy captures a diff baseline, its lease lapses past the proxy TTL, the next
request is refused as UNAUTHORIZED/LEASE_NOT_FOUND with the session torn down, and a freshly
allocated lease reopens and reports baselineInitialized on its first diff.
@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

Lease expiry / reacquisition through the proxy is now covered at 20462bf (remote-proxy-parity.test.ts, clock-driven LeaseRegistry): the leased session's diff baseline exists before expiry, the first request after the proxy TTL lapses is refused UNAUTHORIZED / LEASE_NOT_FOUND with the session torn down, and a reacquired lease's first diff starts from no baseline. Planted-red: with the clock not advanced the "expired" request succeeds and the test fails.

This head also merges the slice A head that resolved the conflict with main (b36d2ce). Still outstanding here: the controlled 0/20/80 ms RTT bench against the #2189 baseline and the full check:affected gate on this exact head; both run as soon as the dedicated simulator finishes the slice A corpus.

@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

The new expiry/reacquisition regression at 20462bf closes the prior expiry gap. Please also add the proxy-path heartbeat renewal case required by #2198, then publish the controlled 0/20/80 ms RTT comparisons against the #2189 baseline. Keep this behind #2329 in merge order.

@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

At ddfbd9d, the proxy changes have no additional code finding, but the heartbeat-renewal case and controlled 0/20/80 ms RTT evidence requested above are still missing. This branch also lacks the latest parent benchmark correction from #2329; bring that in before final validation. #2329 remains the prerequisite, and Android smoke is still failing.

* codex/2198-runner-demand:
  fix(ios): keep observation on the bridge while app discovery is pending and no runner is live
  bench(ios): press an unambiguous target on the catalog and Settings screens
@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

The stack-base issue is resolved at 318d510: this now includes the current #2329 head. The proxy heartbeat-renewal test and controlled RTT results remain outstanding, and merge still depends on #2329. Android smoke has now failed on this revision; other checks are still running.

@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

Controlled-RTT proxy bench: base vs head

Tested commits: base 27a97ee, head 318d510 (this branch's head, which merges slice A e729321); dist built from each. Harness scripts/ios-snapshot-benchmark --mode proxy: the repository proxy behind the harness's deterministic conditioner at RTT 0 / 20 / 80 ms, unlimited bandwidth, 0 % loss; screen quiet, warm state, 20 samples per cell, persistent Node client and fresh-process CLI. Same dedicated Simulator (ad-bench-2198, iOS 26.2) and host as the slice A corpus, 1-minute load 8–9 during the runs. Wall-clock ms.

Raw results and provenance: 2198-slice-b-rtt-318d510769/ on evidence/ios-snapshot (base, head, SHA256SUMS).

RTT Execution N Base median Head median Δ Base p95 Head p95 Daemon median base / head Response bytes Failures
0 persistent client 20/20 36 32 −4 (−10%) 38 38 27 / 23 1764 0/0
0 fresh-process CLI 20/20 164 163 −2 (−1%) 170 174 29 / 28 1764 0/0
20 persistent client 20/20 93 85 −8 (−9%) 105 107 34 / 28 1765 0/0
20 fresh-process CLI 20/20 219 216 −3 (−1%) 226 231 32 / 31 1764 0/0
80 persistent client 20/20 233 227 −6 (−3%) 240 237 46 / 42 1756 0/0
80 fresh-process CLI 20/20 478 484 +6 (+1%) 495 504 37 / 40 1760 0/0

Reading: the proxy path is unchanged within noise at every RTT point and execution mode (the largest move is −8 ms on a 93 ms cell), response size is identical, and no cell failed. The persistent-client wall tracks RTT at roughly 2.5–3 round trips per snapshot (36 → 93 → 233 ms), the fresh-process CLI adds its fixed process cost on top; neither slope changes between base and head, which is what a pass-through proxy with the same published payload should show.

Reference profile: #2189's ios-snapshot-proxy-71fb2483f (bench-golden-v2, iOS 27.0) covers the same cells and RTT points on a different Simulator and runtime, so it is the profile these runs follow, not a same-host comparison; its walls are higher across the board (persistent client 273 / 561 / 806 ms at 0 / 20 / 80 ms), which reflects everything that landed on the capture path since that revision, not this branch.

@thymikee

thymikee commented Sep 6, 2026

Copy link
Copy Markdown
Member Author

The controlled 0/20/80 ms RTT evidence at 318d510 closes the benchmark gap. The same-host base/head comparison shows no material regression; the older #2189 run is correctly identified as a reference profile, not a directly comparable baseline. The proxy heartbeat-renewal test, #2329 prerequisite and failing Android smoke remain the merge blockers.

* codex/2198-runner-demand:
  fix(apple): queue a speculative-runner release behind a start that is still in flight
…e session's comparison state

#2198 acceptance: lease heartbeat through the proxy. A leased session behind the proxy captures a
diff baseline; an explicit lease_heartbeat RPC just before the lease's reported expiry moves the
expiry forward; a request past the old expiry but inside the renewed window still finds the
session and reports the baseline. Red without the renewing heartbeat (LEASE_NOT_FOUND).
@thymikee

thymikee commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

Heartbeat renewal through the proxy is covered at f557142 (remote-proxy-parity.test.ts): a leased session behind the proxy captures a diff baseline, an explicit lease_heartbeat RPC just before the lease's reported expiry moves the expiry forward, and a request past the old expiry but inside the renewed window still finds the session and reports the baseline. Red without the renewing heartbeat (LEASE_NOT_FOUND). It pins renewal against the expiry the lease itself reports rather than a TTL constant, so it holds whichever TTL the admission applies.

This head also carries the parent's latest fixes (325343d: the deferred-start release race and the discovery-wait regression). The full check:affected gate on this exact head follows once the simulator finishes the parent's re-measurements.

@thymikee

thymikee commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

pnpm check:affected --run on f557142: every stage green except three assertions in packages/host-kit/src/code-signature.test.ts (a workspace subpath is walked…, retargeting the exports map…, an installed dependency is not followed into), which this branch does not touch and which fail identically on main itself (233a34d, five failures there): the fixture root under /tmp resolves to /private/tmp on macOS and the stamped labels come back as ../../../private/tmp/... paths. They pass on CI (Linux), where this head's Coverage lane is green. Nothing else in the gate moved: 1,329 related files, 9,812 tests.

@thymikee
thymikee marked this pull request as ready for review September 7, 2026 06:12
@thymikee

thymikee commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

Merged the parent at fdbc70d, which brings in the current main (#2360 moved the daemon client to src/daemon-client, #2361 the session event journal; the wire ledger and the health-compat test follow the move, and #2369 fixes the Android smoke canary). No conflicts; typecheck, wire-compat, the proxy suites, and fallow are green locally. Out of draft.

@thymikee

thymikee commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

The heartbeat regression at fdbc70d closes the remaining parity gap: renewal preserves the session and its comparison baseline, while expiry and reacquisition reset it. The recorded RTT results and updated lower branch cover the earlier review requests. Ready for human review; merge after #2329.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 7, 2026
@thymikee
thymikee merged commit 7bcbf13 into main Sep 7, 2026
18 checks passed
@thymikee
thymikee deleted the codex/2198-proxy-parity branch September 7, 2026 08:13
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-07 08:13 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant