perf(ios): derive runner demand for Simulator opens and stop observation from awaiting the runner - #2329
Conversation
Size Report
Startup median (7 runs, lower is better):
|
|
Not ready at bc68c7e.
Please clarify the runner-retention claim: demand none skips a new prewarm but does not release an existing speculative runner. Either prove the promised release through the existing lifecycle owner, or narrow the claim and explicitly settle that acceptance requirement. |
Thermo-nuclear code quality reviewThis slice should not land in its current shape. The demand pipeline is the right idea, but the model still treats optimizations as requirements, misreads default Do not approve. Behavior on the happy observation batch is better; the implementation preserves incidental complexity and gets the policy model wrong in ways that will keep growing special cases. 1. Demand treats preferred operations as “must start a runner”
That fights ADR 0019: preferred is a measured fast path, not a requirement. The tests pin the wrong model ( That also makes the 80-row host table earn its keep: it classifies what a required operation actually executes, instead of laundering optimizations into prewarm. 2.
|
…ion from awaiting the runner Local Simulator opens now decide how much the XCTest runner is needed from the runtime operations declared by the steps still ahead in the same batch: an observation-only plan starts no runner, an unknown plan keeps the speculative prewarm without ever awaiting it, and a plan with an interaction prepares readiness for that step. open --relaunch no longer waits for runner readiness on a Simulator and resets the runner target only when a session is already alive. The Apple find ports report not-proven instead of starting a runner on a Simulator without a live session, so wait and read-only find observe through the canonical AX-bridge tree. Physical devices keep their lifecycle unchanged. The plan travels through the server-private internal request channel, never the wire; the Apple owner maps declared operations to a runner demand through a record complete over the runtime operation union. Refs #2198
…runtime fixtures The Apple and Android runtime fixtures carried identical audio-probe doubles; host-kit now owns the one copy and both fixtures import it. Also folds the two Apple native-find ports onto one admission helper and lifts the Simulator runner prewarm policy out of the open sequence, keeping both under the complexity gate.
The find ports and the relaunch target reset asked the local session registry whether a runner was alive, which misreads scripted and request-scoped runner providers as absent. Liveness is now a provider question: the local provider consults its session registry, a provider without startup cost counts as live, and an awaited prewarm proves liveness without asking.
…targets a bounded bridge grace A snapshot, diff, or find step now selects the runtime uses its structured input reaches, the way its handler does, so a plain snapshot no longer counts the custom-actions alternative and an observation-only batch resolves runner demand none. The descriptor declares the selector next to its alternatives; the daemon plan derivation honors it and keeps the union for every other command. Without the runner wait, the first snapshot after an open reached the AX bridge while the app was still becoming the primary foreground owner or registering its accessibility server, and the typed fallback then started the runner the plan had just avoided. A target younger than ten seconds is re-read for a bounded grace measured from the first such failure: five seconds for a missing AX server, one second for an ownership miss so a launch-time system dialog still reaches the fallback quickly. Established targets get no grace.
… answered A session record exists while xcodebuild is still connecting, so an alive child pid is not a runner that can answer. Treating it as live sent the relaunch target reset into a starting runner, queued behind its connection retries, and the failed reset invalidated the very session the prewarm was building. Liveness now also requires the session's readiness flag, which the first successful runner response sets.
… runner-session tests within the size ratchet A Simulator open schedules a best-effort runner prewarm that outlives its request. The replay test-runner suite opened a Simulator with the real Apple tools, so the prewarm's deferred import resolved after the file finished and spawned into whichever file the worker ran next, where the hermetic signal guard failed an unrelated test.
…steps the way their handlers do Runner demand now counts a command's required operations only: a preferred or conditional operation is a measured fast path the command succeeds without, so get, wait, and read-only find stay observation-only. The step selectors for snapshot, diff, and find live next to the registry and read the daemon step exactly as the handlers do: the daemon flag for custom actions, and find's positionals through the same parser, where a missing action is a click and an unparseable step keeps every declared alternative. The handler and the selector share one action-to-intent map. The batch runner hands each step its remaining steps in handler shape, and the derived operations reach the platform as a typed list on the lifecycle execution instead of an untyped plan on every open.
…nd make runner liveness explicit The snapshot route no longer infers a launch from process start text and retries inside its own capture. Open owns launch timing instead: a local Simulator open asks the AX bridge whether the launched app is observable, bounded by per-code windows measured from the first typed launch-transition failure and never extended, so an ownership miss seen after an AX-server miss shrinks the deadline to the ownership window and a launch-time system dialog still reaches the typed fallback quickly. Any other device, or a bridge that cannot answer, keeps the fixed settle. The open response reports what it learned. Every runner provider now states whether it can answer without a startup wait; a bare executor answers directly by construction and scripted providers say so. The runner prewarm policy and the observation settle move out of the open sequence into their own module, and the native find admission is named for what it admits.
…budget The enumeration and the no-public-flag rule live on the contract type that owns them; CONTEXT.md keeps the term itself, and two neighbouring entries lose words that carried no meaning.
…operations union The lifecycle execution carries the operations a plan requires, but typing that list with the operations union closed a 36-file type cycle: the operations types depend on the lifecycle types. The vocabulary now lives as a const list below both, proven equal to the union by a type test, so the plan is typed end to end, the Apple host table indexes it without casts, and the daemon narrows descriptor names through a guard instead of a cast.
Every Apple tool port loads the runner operations through the one memoized loader (#2314): a port that opens its own dynamic import can resolve the unmocked module while a test's mock factory is still loading and let a real local runner escape. The liveness port now uses the loader like its siblings; the facade members consumed only through the loader are declared to fallow, and the plan resolver reads one step per helper to stay under the complexity threshold.
The launch observation, the runner-free find admission, and the relaunch policy apply only where the host AX bridge exists: iOS Simulators. A tvOS Simulator keeps its awaited prewarm and asks for no observation, which the tvOS provider scenario now pins.
|
Addressed both reviews at Demand model
Launch timing
Retain/release
Types and seams
CI
Evidence
|
bc68c7e to
d654cc3
Compare
…harness An open that defers runner readiness moves its cost to the first runner-dependent command. The cell starts each sample like cold, opens the fixture untimed, then times the first press that follows (the deep-link confirmation when the launch URL raises it, otherwise the screen anchor).
…de type The open response carries no tree and regular snapshots publish the node type, so the confirmation iOS raises for a launch URL was never seen on this runtime and every deep-linked cell failed its anchor check.
The eager-closure ratchet counts every module the registry loads; the selectors need nothing the registry does not already import, so they live beside find's recording-effect reader instead of adding a module to every entry that loads the registry.
|
The earlier demand-selection, find parsing, launch-ownership and typing findings are addressed at 7616ba2, and CI is green. Two acceptance items remain:
Holding readiness until those are resolved. |
…ervation-only #2198 requires a `none` runner demand to retain no runner, not only to start none. A runner a prewarm started that no command has used yet is speculative: the session records that mark at creation, the first command that is not a readiness probe clears it, and a Simulator open whose plan is proven observation-only asks the runner owner to release a speculative session in the background, so the observation path never waits for a runner to stop either. A runner that has served a command is the session's working runner and stays under the existing idle-stop policy, so a mixed workload does not pay a cold runner start at every observation-only open. The release goes through the runner provider seam: the local provider stops its own speculative session; a provider that never starts speculative work omits the operation and releases nothing.
|
CI is green on 7616ba2, but this head now conflicts with main. Resolve the conflict before landing the stack. The two acceptance items from the previous review are still open: publish the cold/warm/relaunch and first-interaction results, and settle the runner-retention scope in #2198. No new implementation review is needed for this unchanged head. |
|
Conflict with main resolved by merging main at b36d2ce (two list unions: the host-kit Runner retention: implemented rather than waived, at 9189275. A runner a prewarm started that no command has used is speculative; a Simulator open whose plan is proven observation-only releases it through the runner owner, in the background. A runner that has served a command stays under the existing idle-stop owner. The explicit scope decision and its rationale are on #2198. Benchmarks: the base legs are complete at 27a97ee (cold, first-interaction, warm, relaunch, six screens); the head legs are running now against dist built from 7616ba2 on the dedicated simulator. The full comparison, tested commits, and raw JSON land here as soon as the last leg finishes. Two first-interaction cells ( |
|
The new release path misses a pending prewarm: a possible open can still be blocked before runnerSessions.set, then a later none open sees no session and its release returns false; when startup completes, that speculative runner remains retained. Coordinate cleanup with the runner owner’s pending-start state and add a deferred-start regression. The benchmark/corpus evidence is also still pending. |
Corpus: base vs head, cold / first-interaction / warm / relaunchTested commits: base production code 27a97ee, head production code 7616ba2 (the Raw results and provenance:
What the cells say
(Both cells run under the same preconditions; the sum pairs their medians.)
|
Main moved the command descriptor registry into @agent-device/command-registry (#2348) and split src/daemon/types.ts (#2346): the planned-operations module and its test follow the registry into the package (exported as ./planned-operations), the executionPlan carrier moves to daemon-request.ts, and the batch step context lands on the new BatchInvoke signature.
…creens The first-interaction cell pressed the screen's anchor text, which on the catalog and iOS Settings screens names two actionable elements (the native tab and the screen title); the CLI refuses that as AMBIGUOUS_MATCH by design, so those two cells could never measure anything. Each such screen now names the element the cell presses.
|
The corpus now shows useful cold-open gains. The pending-prewarm release issue in the earlier comment still applies at adec882. Please rerun the two invalid first-interaction cells with the corrected selectors and document the additional production code by responsibility, including why a smaller design was ruled out. Android smoke is still failing; iOS smoke is running. |
…ng and no runner is live #2331 bounds one capture's wait for the Simulator app discovery and takes the XCTest fallback past it; #2198 stops a Simulator open from awaiting the runner. Together, a `wait` right after a relaunch on a loaded host fell back to XCTest while the runner was still starting, spent its poll budget on that start, and timed out (the iOS smoke lane after the main merge). A capture with no live runner now stays on the single-flight discovery, one wait slice at a time, until the discovery's own deadline or the request signal ends it; a runner that is already live still takes the fallback at once, the cheaper route #2331 chose.
|
CI after the main merge: the Android smoke lane fails on main itself at the same live-emulator E2E (main bd08e6e, Root cause: two merged changes met in Fix: a capture with no live runner stays on the single-flight discovery, one wait slice at a time, until the discovery's own 15 s deadline or the request signal ends it; a runner that is already live still takes the fallback at once (the cheaper route #2331 chose). Pinned in |
|
The new e729321 discovery fix addresses the iOS wait regression without adding another discovery path. The pending-prewarm release race and the two corrected first-interaction measurements are still outstanding. Android smoke has now failed on this revision; other checks are still running. |
… still in flight A `possible` open's prewarm registers its session only when the start completes, so a `none` open that released in that window found nothing and the runner it meant to release survived as a retained speculative session. The release now takes the runner session lock: it queues behind the in-flight start, sees the registered speculative session, and stops it; a start a command asked for is left alone. Two deferred-start regressions pin both outcomes.
|
Round at 325343d:
|
|
|
|
The pending-prewarm release race is fixed at a23f495. Release now waits on the startup owner’s lock, and the deferred-start regression fails without that ordering while preserving a demanded runner. The recorded live release and benchmark evidence address the earlier gap. Ready for human review; remaining CI status is separate. |
thymikee
added
the
ready-for-human
|
…erge (#2375) `no APPROVED_OVER_CEILING row is stale` reads the introduced-entry set, which is derived from `git merge-base origin/main HEAD`. On a push to main the merge-base IS the head, so nothing is first-introduced and every approval row reads as stale whatever its real state. That is exactly the shape of the approving PR's own merge commit: #2329 added the `packages/command-registry/src/planned-operations.ts` row to merge, and the merge that followed it called the row dead. Coverage has been red on main since (run 34099687663), and every branch cut from main after it inherits the same failure. - `staleApprovalRows` makes the verdict a named rule and defers it when the merge-base is the head, where no row is readable at all. Enforcement is not lost: a row that outlives its PR is still reported on the first branch whose merge-base could have read it, and the rule is pinned in both directions. - The `planned-operations.ts` row goes, which is what the rule asks for now that main carries the entry: its closure (74) is governed by the no-growth rule from here on, not by the domain-facade ceiling. Claude-Session: https://claude.ai/code/session_01SfQqXj7JKQVgBA8eg9SMVB Co-authored-by: Claude <noreply@anthropic.com>
The APPROVED_OVER_CEILING row for packages/command-registry/src/planned-operations.ts was consumed by #2329. Now that main carries the entry, the no-growth rule governs it and the row can no longer change any verdict, so the "no row is stale" probe fails on main and on every PR based on it. Remove the row, as the probe instructs.
1 participant
Summary
Slice A of #2198. A local iOS Simulator
opennow derives how much the XCTest runner is needed from the runtime operations declared by the steps still ahead of it in the samebatch, and observation stops waiting for runner readiness.runBatchhands every step its place in the plan through the server-privateinternalrequest channel (never the wire). The daemon flattens the remaining steps' declaredplatformExecutionuses;snapshot,diff, andfindselect their alternative from the step input the way their handlers do (selectUseson the descriptor). The Apple owner maps the operations through a record complete over the runtime-operation union. Observation-only plan →none(no runner started or retained). Unknown plan, including anopenthat ends its batch →possible(today's speculative prewarm, never awaited). Any runner-served operation →required(prewarm now; the first runner-dependent step awaits the same startup under the runner session lock with today's typed errors and deadlines).open --relaunchno longer awaits runner readiness on a Simulator. The runner re-resolves a new app PID at its next command, sotargetResetis sent only to a runner that is already alive and ready, or that an awaited prewarm just proved. Physical devices keep their relaunch lifecycle unchanged.waitand read-onlyfindobserve runner-free. The Apple native find ports reportfound: false("not proven here") instead of starting a runner on a Simulator without a ready session, so the poll consults the canonical AX-bridge tree (ADR 0019 §2). A ready runner keeps answering; provider-backed runners count as live through the runner provider seam.timinggainsrunnerDemandnext to the existing prewarm fields (additive; ADR 0006 untouched). Vocabulary added to CONTEXT.md.Also folds the two Apple native-find ports onto one admission helper, lifts the runner prewarm policy out of the open sequence (complexity gate), and shares one inert audio-probe fixture through
@agent-device/host-kit/audio-probe-fixturesinstead of two identical platform copies.Scope: 41 files, ~1,370 gross lines. Over the 1,000-line budget because the liveness seam, the plan derivation, and the Simulator policy only make sense together with the regressions that fell out of live validation; I can split it into a three-PR stack (liveness seam → plan derivation → Simulator policy) if preferred.
Proxy parity, lease isolation, cancellation, version skew and controlled-RTT measurements remain slice B of #2198.
Production code by responsibility
Production lines vs main (tests, fixtures, scripts, docs and config excluded): 29 files, +918 / −114.
packages/contracts/src/runtime-operation-names.ts(new),platform-runtime-operations.ts(findRuntimeIntent),command-platform-execution.ts(selectUses)PlatformRuntimeOperationstype (that import formed a 36-file type cycle, R9).selectUseslets a descriptor state which of its uses a concrete step needs (snapshot --custom-actions,findby action) instead of declaring the union.packages/command-registry/src/planned-operations.ts(new),registry.ts(step-use selectors onsnapshot,diff,find)src/core/batch.ts(BatchStepContext),src/daemon/handlers/session-batch.ts,src/daemon/execution-plan.ts(new),src/daemon/daemon-request.ts(internal.executionPlan),src/daemon/application-lifecycle-execution.ts,src/daemon/interaction/internal/find.tsreq.internal(stripped on the wire) and the lifecycle execution hands it to the platform asplannedOperations. No public flag, no new RPC field.packages/platform-apple/src/runner-demand.ts(new),open-policy.ts(new),lifecycle.ts,contracts/application-lifecycle-runtime.ts(plannedOperations,runnerDemand,postOpenObservation,hasLiveRunnerSession,releaseSpeculativeRunner)none/possible/required, never awaits the prewarm on a Simulator with the bridge, notifies the runner only when one is live, releases a speculative runner on a proven observation-only plan, and lets the open settle on the bridge's own observability instead of a fixed sleep. Physical devices and tvOS keep their lifecycle.packages/platform-apple/src/snapshot-observability.ts(new),snapshot-route.ts,runtime-snapshot.tsfindadmission and the discovery-pending case ask whether the runner can answer now instead of assuming it; with no live runner, observation stays on the bridge.packages/platform-apple/src/runner/runner-provider.ts,runner-client.ts,runner-session.ts,runner-session-types.ts,runner-lifecycle.ts,client.ts,core/runner-client.ts,runner-operations-facade.ts,src/platform-runtime-apple-application-tools.tsWhy a smaller design was ruled out:
find(its default action isclick); the review caught both, and the selector-based derivation is what fixed them.ps-parsed heuristic in the wrong owner and made the first post-open capture fall back onforeground-owner-unverified; moving it into the open as a bounded probe removed both.Validation
Head:
325343d9a7; base for the measurements:27a97ee619(the merge-base of the measured head with main).none/possible/requiredfrom the plan; anoneopen releases a speculative runner and leaves a served one; a release that arrives while the start is in flight stops it once it completes (deferred-start regression, red without the lock); relaunch resets the target only on a live runner; physical iOS and tvOS keep the awaited prewarm; the observation probe's windows, cap and deadline shrink; a pending app discovery keeps observation on the bridge while no runner is live, and takes the fallback at once when one is (red without the fix); nativefindadmission by liveness; plan derivation counts required operations only, keepsfindpositionals, and classifies unknown names as unknown plans.pnpm check:affected --runon the exact head posted as a comment when it completes.evidence/ios-snapshot, table in the PR comment): cold open −27 % to −40 %; relaunch −9 % to −12 % (system-surface −56 %); warm snapshot unchanged (xctest-stress −56 %); first interaction after a cold open unchanged on two screens, +0.3–0.5 s on the two fastest (the exposed tail of the runner start), with cold open + first click −18 % to −37 % on every measurable screen.list,system-surfacefirst interaction, head e729321):list1372 → 1359 ms (−1 %),system-surface990 → 1053 ms (+6 %, one 1.9 s sample), 0 failures on either leg; raw JSON onevidence/ios-snapshot(2198-slice-a-first-interaction-fixed-e729321dcc/).--debug): a loneopen --relaunch(demandpossible) starts the runner; an observation-only batch (open --relaunch,snapshot -i,wait text, demandnone) releases it within a second; the same batch issued immediately after a lone open, while that start is still in flight, releases it once it completes; after a lone open and a servedclick, an observation-only batch leaves the runner alone. Twoios_runner_speculative_releasedrecords in the daemon diagnostics, one per release; every batch and click succeeded.