Give each concurrent test action its own simulator via a per-(device, OS) pool

[erneestoc]

Summary

Split out from #3021 per review feedback, redesigned as a simulator pool after further discussion: instead of serializing concurrent tests on one shared simulator, each concurrent test action gets its own.

Problem

A simulator can only service one test session at a time. Two failure modes follow whenever Bazel actually executes several simulator tests concurrently (CI runs --local_test_jobs=2):

1. Session collisions: concurrent xcodebuild test-without-building / xctest sessions against the same reused simulator disrupt each other — 300s hangs, or Failed to initialize for UI testing: … kAXErrorCannotComplete.
2. Creation races: concurrent tests racing through the create action each fail to find an existing simulator and create same-named duplicates. CI logs from Add screen_capture_format attribute to ios_xctestrun_runner #3021 show two BAZEL_TEST_iPhone 16_26.0 devices created 3 seconds apart, with every simulator test on that machine hanging from then on.

These are the dominant source of iOS test flakiness on this repo's CI. They stay invisible on main because the affected tests are almost always remote-cache hits; any PR that touches the runner invalidates those caches and forces real concurrent executions (#3021 hit this for 6 consecutive builds across 8 different agents until the interim locks landed on its branch).

Design

• Pool slot claim (runner template): before invoking the simulator creator, the runner claims an exclusive slot in a machine-wide pool by atomically creating a shlock(1) pid lockfile in $TMPDIR. Slots are not keyed on device type or OS version, so a single invocation running tests across several simulator types still hands every concurrent test its own device.
• Simulator naming: slot 0 keeps the historical simulator name, so existing simulators keep being reused; higher slots are only claimed while tests actually run concurrently and get their own suffixed simulator (…_1, …_2), which stays booted between test actions for warm reuse like slot 0.
• Where the locks live: ${SIMULATOR_POOL_LOCK_DIR:-${TMPDIR:-/tmp}}. Bazel points TMPDIR at the per-user temp dir (/var/folders/…/T/), which is shared across sandboxed test actions and writable there — unlike TEST_TMPDIR, which is per-action. Verified under darwin-sandbox with two concurrent test actions.
• Remote execution escape hatch: TMPDIR is not part of the action's declared environment (bazel aquery shows only PATH), so under remote execution the worker decides what it is, and a worker handing each action its own TMPDIR would silently put every test back on slot 0. --test_env=SIMULATOR_POOL_LOCK_DIR=… points the pool at a directory the execution environment actually shares. Set-but-unusable warns and falls back rather than failing the test.
• Robustness: a claim is valid only while the claiming test process is alive. A test killed for any reason — including SIGKILL from a Bazel test timeout — never leaves a slot permanently claimed: the next prober validates the recorded pid and shlock atomically reclaims dead slots. (First attempt used a kernel flock held on an fd via lockf(1); abandoned because macOS lockf's fd mode misbehaves under try-lock probing — after one failed probe, every subsequent probe in that process fails even on free lock files.)
• Session-wedge recovery (runner): a CI attempt log showed the dominant hang: the runner finds its pooled simulator booted, the readiness probe passes, and xcodebuild test-without-building sits silent until the Bazel timeout — the device is carrying a dead test session from a previously terminated test, and every new session against it hangs. Two complementary defenses: (1) the runner traps TERM/INT after claiming its simulator and shuts the device down (Bazel delivers SIGTERM with a grace period before SIGKILL), so the next attempt cold-boots clean; (2) for deaths the trap cannot catch — SIGKILL, grace overruns — the runner keeps a session marker holding its pid (same pid-liveness pattern as the pool locks), removed on any controlled exit, and the creator reboots any booted device whose marker names a dead pid before reusing it. Normal completions leave the device booted for warm reuse, in every slot. This also self-heals agents whose devices are already wedged: the first attempt still times out, but the device is recycled and the retry passes.
• Bounded boot (creator): simctl bootstatus -b has no deadline of its own, so a device wedged mid-boot consumed the whole test timeout and was SIGKILLed — leaving it wedged for the next attempt. The boot is now bounded against the test's own TEST_TIMEOUT (minus a 30s cleanup reserve, 30s floor — the bound exists to fail cleanly before SIGKILL, not to ration boot time, so slow first-boot data migrations still get nearly the whole budget), and a timed-out boot is deliberately left in progress: the boot is owned by CoreSimulatorService and continues in the background, so a flaky-test retry re-enters bootstatus and resumes where the previous attempt left off — a slow first-boot data migration completes across attempts instead of restarting from zero each time.
• Reused-simulator health check (creator): a simulator can report state booted while being unusable — if the test that started its initial boot was killed partway through, the device stays booted but never becomes able to run tests, and every later test reusing it hangs. Reused booted devices are now probed for readiness (SpringBoard in simctl spawn <udid> launchctl list, bounded at 30s) and rebooted if not responding.
• Unaffected: ios_test_runner (doesn't set SIMULATOR_POOL_SLOT → slot 0 → today's behavior), reuse_simulator = False, device runs, and machines without shlock(1) (degrades to today's single-simulator behavior).

Review feedback addressed

• Pool is no longer device/OS-version specific — it's machine-wide.
• Reads $TMPDIR instead of shelling out to getconf DARWIN_USER_TEMP_DIR.
• The machine-wide boot gate is removed entirely rather than having its timeout shortened: the creation race it partly covered is already fixed by per-test slots, and --local_test_jobs already bounds overlapping boots.
• SIMULATOR_POOL_SLOT now parses as a hard error instead of silently falling back to slot 0.
• Added SIMULATOR_POOL_LOCK_DIR so the lock directory is configurable for execution environments that do not share $TMPDIR.

Test plan

• Concurrent-claim simulation: concurrent runners get distinct slots; a freed slot is reclaimed by the next test; a SIGKILLed holder's slot is atomically reclaimed.
• Locking verified inside Bazel's darwin-sandbox (shared $TMPDIR is writable and lockable there, and identical across concurrent test actions, under both darwin-sandbox and --spawn_strategy=local).
• SIMULATOR_POOL_LOCK_DIR: three concurrent claimers with differing $TMPDIRs still get distinct slots when pointed at a shared dir; unwritable dir, missing dir and unset $TMPDIR all fall back to slot 0 immediately instead of hanging; propagation through --test_env confirmed.
• //test:ios_xctestrun_runner_ui_test behaves identically to main; single-test runs claim slot 0 and reuse the historically-named simulator.