fix(tests): root-cause wake/watcher suite flakiness

[cm-maple7]

Intent

Fix two pre-existing flaky test scripts, tests/fm-wake-queue.test.sh and tests/fm-watch-triage.test.sh, by root-causing each failure with evidence rather than papering over it with sleeps or retries. Make the tests hermetic where the cause is in the test, and fix the code where the cause is a real defect in the code under test. Where a test genuinely depends on an environment the sandbox cannot provide, make it skip loudly with the exact reason printed, never silently pass and never weaken an assertion. Prove the result with repeated consecutive green runs recorded in the PR body.

What Changed

• Fixed a real defect in bin/fm-wake-lib.sh's lock-reclaim logic: fm_lock_try_acquire compared a recorded holder pid against ${BASHPID:-$$} inline, which on shells with no BASHPID (stock macOS bash 3.2, where $$ is identical in every subshell) could let a subshell mistake its parent's still-live lock hold for its own abandoned one. Extracted a new fm_pid_is_this_frame helper that uses BASHPID when available and otherwise only matches on the main shell (BASH_SUBSHELL = 0), so deeper frames on BASHPID-less shells correctly keep waiting instead of reclaiming a live hold.
• Consolidated the duplicated is_live_non_zombie/reap fixture-process helpers (previously redefined per test file with an unbounded kill; wait) into single owners in tests/lib.sh, and made reap bounded: it now sends SIGTERM, polls for exit, and escalates to SIGKILL instead of hanging indefinitely when a watcher's EXIT trap gets stuck re-acquiring a contended lock. tests/wake-helpers.sh and the fm-*.test.sh files now source these instead of defining their own.
• Made tests/lib.sh's fm_test_tmproot resolve and return the physical (symlink-free) path, fixing fixtures that were spuriously rejected on macOS because $TMPDIR resolves through the /var -> /private/var symlink and production code refuses a state root reached through a symlink.
• Raised wait_for_exit's default budget from 50 to 100 ticks (5s to 10s) in tests/wake-helpers.sh, based on measured watcher startup cost (up to 2.5s under load), and removed the now-redundant explicit 40/100 tick arguments scattered across tests/fm-wake-queue.test.sh and tests/fm-watch-triage.test.sh in favor of that shared default.
• Added an FM_TEST_ONLY-gated single-test entry point to tests/fm-wake-queue.test.sh so CI can run the frame-identity lock-reclaim regression in isolation, and added a step in .github/workflows/ci.yml that runs it under /bin/bash on the macOS lane and asserts exactly one passing result, since that half of the reclaim contract only exercises the BASHPID-less path on a shell that lacks it.

Risk Assessment

✅ Low: Each of the three claimed root causes (macOS symlinked TMPDIR tripping the process-event state-root refusal, bash 3.2's shared $$ making a subshell indistinguishable from its parent in lock reclaim, and an insufficient wait_for_exit budget vs. measured watcher startup cost) is fixed at a verifiable source location with reasoning that checks out against the actual code (fm-procevent-lib.sh's pwd -P identity check, fm-wake-lib.sh's BASHPID-based reclaim, fm-watch.sh's startup work), a bonus fourth fix (bounded reap replacing unbounded kill; wait) is applied consistently everywhere the old unbounded pattern existed, all consumers of the deduplicated reap/is_live_non_zombie helpers transitively source tests/lib.sh so nothing breaks, the new FM_TEST_ONLY gate is added after all test function definitions so normal full-suite runs are unaffected, and no assertions were weakened or silently skipped - matching the user intent's requirements.

Testing

Reproduced the actual pre-fix lock-reclaim defect on this bash 3.2/no-BASHPID macOS host (rc=13 with the old code, rc=0 with the fix) and independently confirmed the TMPDIR-symlink root cause the tests/lib.sh fix addresses; then ran tests/fm-wake-queue.test.sh and tests/fm-watch-triage.test.sh three consecutive times each plus 20 consecutive isolated runs of the frame-identity regression, all green with no flakes, and one clean run of the touched tests/fm-inactive-reconcile.test.sh - no test failures or evidence gaps found.

Root causes

Three distinct causes, none of them the same bug. Each has a minimal repro against unmodified files.

1. macOS symlinked TMPDIR vs the process-event claim (deterministic, fm-watch-triage.test.sh)

fm_procevent_claim_state_root_identity (bin/fm-procevent-lib.sh:349) deliberately refuses a state root reachable through a symlink: it compares the physically resolved path against a lexical normalization and rejects any mismatch.

fm_test_tmproot handed back whatever mktemp -d "$TMPDIR/..." returned. On macOS TMPDIR is /var/folders/.../T/ and /var is a symlink to private/var, so every fixture built from it is reached through a symlink and the claim refuses it. No runner ever claims the source, no result is ever captured, and the six procevent cases fail. On Linux CI /tmp and $RUNNER_TEMP are real directories, so CI never saw it.

This is a fixture defect, not a product defect: a real firstmate home is a real path.

Minimal repro (unmodified files, before the fix)

$ mkdir -p /tmp/pe/state          # /tmp -> /private/tmp, same shape as macOS $TMPDIR
$ FM_PROCEVENT_CLAIM_ROOT=/tmp/pe/claims FM_HOME=/tmp/pe bin/fm-procevent.sh \
    register lavish delivery-src -- /bin/sh -c 'printf "session:\n  status: waiting\n"'
registered: delivery-src (lavish)
$ FM_PROCEVENT_CLAIM_ROOT=/tmp/pe/claims FM_HOME=/tmp/pe bin/fm-procevent.sh reconcile
reconciled: published=0 started=1 stopped=0 uncertain=0
$ cat /tmp/pe/state/.wake-queue            # empty: the runner died claiming
$

The runner's own trace shows the exact refusal:

+ '[' /private/tmp/pe/state = /tmp/pe/state ']'
+ return 1
...
error: cannot claim source: delivery-src

Same commands under a physically-resolved root capture the result:

$ FM_PROCEVENT_CLAIM_ROOT=/private/tmp/pe7/claims FM_HOME=/private/tmp/pe7 ... reconcile
$ cat /private/tmp/pe7/state/.wake-queue
1788255278	1	check	procevent:delivery-src:1	check: procevent lavish delivery-src 1

tests/fm-procevent.test.sh was broken by the same cause (2 ok, then not ok - reconcile never claimed the registered source); it now passes 62/62.

Fix: fm_test_tmproot resolves the root with cd -P && pwd -P before handing it out.

2. Lock reclaim could not tell a subshell from its parent (deterministic, fm-wake-queue.test.sh)

fm_lock_try_acquire's reclaim branch asked [ "$pid" = "${BASHPID:-$$}" ]. Stock macOS bash 3.2 has no BASHPID, and its $$ is the same value in every subshell, so the fallback makes a subshell look exactly like an abandoned frame of its own parent - and it reclaims the lock the parent is still holding.

/bin/bash is 3.2.57 and precedes Homebrew's bash 5 on the default macOS PATH, so #!/usr/bin/env bash resolves to 3.2 in any shell that has not put Homebrew first. That is the whole difference between the environments: an interactive terminal with Homebrew's shellenv runs bash 5, the sandbox runs 3.2.

Minimal repro (unmodified files, before the fix)

$ /bin/bash --version | head -1
GNU bash, version 3.2.57(1)-release (arm64-apple-darwin25)
$ FM_STATE_OVERRIDE=/private/tmp/s /bin/bash -c '
    . bin/fm-wake-lib.sh
    lock=/private/tmp/s/.l
    fm_lock_acquire_wait "$lock"
    ( fm_lock_try_acquire "$lock" && echo "SUBSHELL STOLE THE PARENT'"'"'S LIVE HOLD" )'
SUBSHELL STOLE THE PARENT'S LIVE HOLD

10 of 10 runs of the whole file failed at this case under bash 3.2; the other 28 cases in the file pass there.

Fix: fm_pid_is_this_frame is now the single owner of "does this recorded pid name the frame asking". BASHPID answers directly where it exists. Where it does not, $$ names a frame only where there is no subshell to confuse it with - the main shell, the one place BASH_SUBSHELL is 0 - so a subshell can never take the branch.

Why the obvious alternatives were rejected (both tried, both measured)

• Refusing to reclaim at all without BASHPID. Safe against the steal, but it reintroduces the deadlock the reclaim exists to prevent. Measured: with that version in place, test_stale_terminal_status_overridden_by_active_run hung for over 8 minutes under bash 3.2 with bin/fm-watch.sh alive and unreapable; the unmodified file passes the same case in seconds.
• Recovering a true frame pid from $(sh -c 'echo $PPID'). It relies on bash exec'ing a one-command substitution in place, which is an optimization, not a guarantee - and it does not hold where this code actually lives. Measured under bash 3.2: exact in a top-level script (a=b=$$), but forking, and so wrong, inside a sourced file (a=70804 b=70806 self=70803), under bash -c, and any time a redirection is added to the substitution (on bash 5 too). A shell that forks answers with a fresh pid every call, which would write an already-dead owner into every lock.

BASH_SUBSHELL is a shell-maintained fact rather than an optimization, needs no fork, and is exactly the distinction the branch was missing.

3. Watcher-exit budget below the work inside the window (rare, fm-wake-queue.test.sh)

tests/fm-wake-queue.test.sh gave its five one-shot watcher waits 40 ticks (4s), while tests/fm-watch-triage.test.sh gives the same watcher 100 ticks and already documented why a tighter budget reports a spurious "did not surface": fm-watch.sh does bounded startup work before its first poll, and a budget close to that cost reaps the process while it is still starting.

Measured cost of that one poll on this machine: 1.1s idle, 2.5s at load 18 - a 1.6x margin against 4s, versus 4x for the sibling suite's budget.

Observed rate is low: 1 failure in 3 full-file runs when first seen, then 0 failures in 39 further pre-fix runs. Reported honestly rather than dressed up; the budget is fixed because it is measurably under-provisioned against the repo's own documented standard, not because a reproduction rate demanded it.

Fix: wait_for_exit in tests/wake-helpers.sh now owns the standard budget and the rationale, and defaults to it. fm-wake-queue.test.sh's five waits take that default. fm-watch-triage.test.sh's 65 copies of the literal 100 are replaced by the same default, so its "every budget here is the standard one" comment is true by construction instead of by 65 coincidences; its three deliberately larger away-mode budgets stay explicit.

No assertion was weakened and nothing was papered over with a sleep or a retry: a watcher that never surfaces its wake still fails its assertion when the budget runs out.

4. An unbounded stop-and-collect turned a wedged watcher into a whole-suite hang (rare, fm-watch-triage.test.sh)

The most severe failure of the three files was not a failing assertion at all.
A 20-run loop had run 11 stop producing output at ok=58 and never return; it had to be killed from outside at a 900-second bound, with no not ok line and nothing naming the case it died in.

The mechanism is the way these suites stop a process they spawned.
Both spellings ended in an unbounded wait: reap() { kill "$1"; wait "$1"; }, and the budget-exhausted tail of wait_for_exit.
The process being stopped is bin/fm-watch.sh, which leaves through trap watcher_cleanup EXIT, and watcher_cleanup itself takes a lock (fm_recovery_transition ... "$WATCH_LOCK" downtime).
A watcher signalled while that lock is contended can sit inside its own exit path, and an unbounded wait sits there with it - forever, taking the run's whole budget and reporting nothing.

To be exact about what is and is not established: the wedge itself is rare and was not reproduced on demand, and it is not caused by this PR's production change.
That was tested rather than assumed, twice: 25 isolated rounds of the case the loop died near give mine 0/25, head 0/25, and eight instrumented full-file runs logging every reclaim the new frame check newly refuses give refusals=0 across all eight (rc=0 ok=92 each).
The defect being fixed here is the fixture's, and it is deterministic and provable on its own terms: whatever wedges the exit path, an unbounded wait converts it into a silent hang instead of a result.

Minimal repro (the two spellings, against an exit path that does not complete on TERM)

$ /bin/bash -c 'trap "sleep 9999" TERM; sleep 9999' &   # a watcher stuck in its EXIT trap
$ kill "$pid"; wait "$pid"          # the old spelling, bounded at 30s from outside
old  rc=137                          # never returned; killed from outside
$ reap "$pid"                        # the new one
new  rc=0   new returned after 6s

Fix: tests/lib.sh now owns one bounded reap: signal, allow a bounded number of ticks to leave, then SIGKILL, which cannot be trapped, so the bound holds whatever the exit path is doing.
wait_for_exit's exhausted-budget tail routes through it, and the two verbatim copies of the old spelling - in tests/fm-watch-triage.test.sh and in tests/fm-inactive-reconcile.test.sh, which had the identical latent hang - are replaced by that single owner.

No assertion is weakened: reap is cleanup that runs after a case has already made its assertions, and wait_for_exit still returns 124 for a watcher that never surfaced its wake.
What changes is only that a wedged exit path now costs a bounded wait and a reported result instead of an unbounded hang.

5. Nothing needed a loud skip

Both suites now pass in full on both shells present here, so no case needed an environment skip.

5. What CI caught that 14 local runs could not

The bounded reap above shipped with a 5 second allowance, and CI failed on it: not ok - could not acknowledge the intentional phase-A watcher stop, one assertion, in tests/fm-watch-triage.test.sh.

That is a real regression introduced by this branch, found by CI and worth stating plainly.
The reasoning error is visible in the original comment, which called reap "cleanup that runs after a case has already made its assertions".
That is true of most callers and false of the one that mattered: ack_stopped_cycle depends on the EXIT trap itself completing, because that trap is what publishes the downtime recovery marker the following drain must find.
Escalating to SIGKILL there does not merely skip cleanup - it erases a side effect the next assertion reads.
Under CI contention the watcher's exit path exceeded 5 seconds, the marker was never written, and the drain had nothing to acknowledge.

Fix: the default is now the same 10 second standard budget wait_for_exit uses, and the comment says why: both bound the same fm-watch.sh under the same load doing comparably sized work, so one standard serves both rather than two unexplained numbers.
It passes locally because the exit path finishes well inside 5 seconds here; only a contended runner exposes it.

Proof

All runs from the pooled task worktree, each wrapped in a hard wall-clock bound so a hang fails loudly instead of hanging the run.

	before	after
fm-wake-queue.test.sh (bash 3.2)	0 ok, fails at the lock case, 10/10 runs	29 ok
fm-wake-queue.test.sh (bash 5)	29 ok, 1 flake seen in 3 runs then 0 in 39	29 ok
fm-watch-triage.test.sh (bash 3.2)	79 ok then not ok - the fixture captured no process-event result	92 ok
fm-watch-triage.test.sh (bash 5)	79 ok then the same failure	92 ok
fm-procevent.test.sh (collateral)	2 ok then not ok - reconcile never claimed the registered source	62 ok

Consecutive full-file runs

Run under stock /bin/bash 3.2, the shell where these failed before the fix, each run bounded so a hang fails loudly instead of eating the budget silently.

tests/fm-wake-queue.test.sh     20 / 20 green    every run: ok=29
tests/fm-watch-triage.test.sh   14 / 14 green    every run: ok=92, notok=0

The wake-queue loop ran its full 20.
The triage loop was stopped deliberately at 14 rather than run to 20: the two failures it was there to catch are both root-caused and fixed deterministically, with a minimal repro each, so further identical runs re-measure a rate that no longer has a mechanism behind it.
14 consecutive clean full-file runs alongside a found-and-fixed cause is the evidence; a 20th run would not have added a different kind.

Two details worth stating rather than smoothing over.
Run 11 is the position where the pre-fix loop hung and had to be killed at 900 seconds; it now passes in 274 seconds like every other run.
The loop driver was killed from outside between runs 13 and 14 and restarted; no test failed, nothing was left running, and the files under proof were verified byte-identical across the restart, so the 14 are 14 runs of one unchanged tree.

Per-run wall clock was 269-291 seconds, tightly clustered, which is itself evidence that nothing is sitting near a timing boundary.

Both suites also pass under bash 5, where the full 175-script serial sweep records fm-wake-queue.test.sh exit=0, fm-watch-triage.test.sh exit=0, and fm-procevent.test.sh exit=0.

The one suite whose result needed a controlled comparison

tests/fm-control-relaunch.test.sh failed during the sweep, so it was compared against pristine HEAD in interleaved rounds. It is a pre-existing load-sensitive flake and this change is not distinguishable as a cause:

run 1  (real TMPDIR)              mine 3/8   head 0/8
run 2  (real TMPDIR, busier)      mine 7/10  head 5/10
calibration (canonical TMPDIR,
  where this diff is a no-op and
  both trees run identical code)  mine 4/10  head 6/10

The calibration is the control: with identical code on both sides the spread is still +/-2 at n=10 and the direction reverses, which accounts for the earlier runs. Pooled over 28 interleaved rounds: mine 14/28, head 11/28. Its assertion is a 2s wall-clock budget, the same under-provisioned shape as cause 3.

Both regressions were proven load-bearing by reverting only the production line and re-running:

frame check reverted, bash 3.2 : not ok - a subshell reclaimed its parent's live hold (rc=13, bash 3.2.57(1)-release)
frame check reverted, bash 5   : ok   (BASHPID present; behavior identical either way)
frame check restored, bash 3.2 : ok

bin/fm-lint.sh clean (ShellCheck 0.11.0 pinned, actionlint 1.7.12 pinned, 3 workflows valid).
bin/fm-doc-audience-check.sh clean (89 surfaces, 300 local links).

Pre-existing failures found on the way, deliberately not fixed here

All four reproduce on unmodified origin/main and none is caused by this change. They are listed so they are not lost, not folded in, because each is a deterministic or environmental defect in a file this task does not name, and bundling them would make a flakiness fix harder to review.

• tests/fm-muse-harness.test.sh (macOS, stock bash only). The fixture does cp "$(command -v bash)" and renames the copy to muse-bin-<version> so harness detection can find it in the process tree. A copied macOS system /bin/bash will not execute at all - code signing - so the probe returns empty. It passes wherever bash resolves to a copyable build, which is why Linux CI and a Homebrew-first PATH never see it. A symlink in place of the copy fixes it on macOS (verified: detection returns muse, and the anchored negatives still refuse musescore, amuse, notmuse-bin, muse-binary, muse-bind), but the Linux half of that change cannot be verified from here.
• tests/fm-composer-lib.test.sh (stock bash only): not ok - a half-block rule row must count as a structural edge. Passes under bash 5. Undiagnosed.
• tests/fm-backend-herdr-presentation-e2e.test.sh: error: herdr presentation recovery could not acquire its session lock; refusing a concurrent resume. A race between two concurrent resumes in a real-Herdr end-to-end case, on a machine also running another firstmate session against the same Herdr daemon.
• tests/fm-control-relaunch.test.sh: relaunch did not reach trace delivery. That assertion gives the relaunch 2s (200 x 0.01s) to reach trace delivery - the same class of under-provisioned wait as cause 3 above.

Notes

• Two files outside the two named ones are touched, both for cause 4 and both deliberately.
  tests/lib.sh gains the bounded reap because it is the generic helper that the wake, watcher, and reconcile suites all already source, so it is the one place the contract can be stated once.
  tests/fm-inactive-reconcile.test.sh loses its verbatim copy of the old unbounded spelling, because leaving it would leave a local definition shadowing the fixed one and silently keeping the same hang; it is a one-line deletion and that suite passes (17 ok).
• Based on current origin/main (6aa4beb), which already contains the watcher change from fix(bin): absorb background-run stale wakes and trust declared pauses over ci-monitoring #2 that edited tests/fm-watch-triage.test.sh. The budget cleanup here sits on top of those cases rather than around them; the new test_crew_run_step_paused_classifier and the pause-trust cases pass unchanged.
• bin/fm-wake-lib.sh's behavior on any bash with BASHPID - every bash 4.0 and later, which is every shell CI and the fleet actually run - is unchanged by this PR.
• CI's stock macOS Bash lane now runs the lock regression under real /bin/bash 3.2 via FM_TEST_ONLY, so the half of the contract the Linux lanes cannot reach is enforced rather than assumed.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• /bin/bash tests/fm-wake-queue.test.sh x3 consecutive runs (29 assertions, exit 0 each)
• /bin/bash tests/fm-watch-triage.test.sh x3 consecutive runs (92 assertions, exit 0 each)
• /bin/bash tests/fm-inactive-reconcile.test.sh (17 assertions, exit 0)
• FM_TEST_ONLY=test_self_held_lock_reclaims_instead_of_deadlocking /bin/bash tests/fm-wake-queue.test.sh x20 consecutive isolated runs of the frame-identity lock regression (all ok)
• Manual repro: ran the pre-fix bin/fm-wake-lib.sh lock-reclaim logic against the subshell-reclaim scenario directly, confirming it fails (rc=13) before the fix and the current code passes (rc=0) - fail-before/pass-after regression proof
• Manual verification that TMPDIR on this host resolves through a /var -> private/var symlink, confirming the root cause tests/lib.sh's fm_test_tmproot fix addresses

⚠️ **Document** - 1 info

• ℹ️ The new stock-macOS-bash CI lane check for the wake-lock frame-identity regression (ci.yml) has no maintainer-verification record, unlike the analogous existing pattern in docs/verification/public-followup.md for its own FM_TEST_ONLY bash-3.2 lane. This is new evidence, not a stale fact this change broke, so out of scope here; flagging as an optional follow-up if the team wants CI-visible bash-3.2-only guarantees consistently recorded.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[cm-maple7]

Closing on captain instruction. The lock-reclaim bash-3.2 edge is not worth more pipeline or load-test theater on this machine.