Skip to content

fix(bin): absorb background-run stale wakes and trust declared pauses over ci-monitoring - #2

Merged
cm-maple7 merged 6 commits into
mainfrom
fm/fm-watcher-absorb-background-run-v2
Sep 1, 2026
Merged

cm-maple7 merged 6 commits into
mainfrom
fm/fm-watcher-absorb-background-run-v2

Conversation

@cm-maple7

@cm-maple7 cm-maple7 commented Sep 1, 2026 •

Copy link
Copy Markdown
Owner

Intent

Watcher: absorb stale/idle-pane alarms when the task is genuinely inside a background no-mistakes validation run, and stop the wedge ladder for classified waits.

Case 1/4 (2026-08-26 restday polish/start-over, and 2026-08-31 rd-tree-goal-replace-anchor): a worker attached to a no-mistakes run in the background shows an idle pane, so the watcher raised stale wakes every ~4 minutes up to escalation 9 while crew-state said 'validating (background run)' with a live agent pid and climbing CPU. Fix: when the current-code-matched run step is running/fixing and the pipeline's active agent is alive, absorb the stale wake with a triage-log line; keep waking when the run is parked (awaiting_approval/fix_review) with the worker idle. Applies to both the running/fixing case and the plain validating step.

Case 2/3 (rd-ios mate, 2026-08-27, key fm-parked-run-wedge-alarm): when a PR merge is reserved for the captain, the no-mistakes ci step stays 'running - monitoring until merged or closed' indefinitely, fm-crew-state's run-step source reports 'validating'/'done, still monitoring' and OUTRANKS the worker's declared 'paused:' line, so the watcher wedge-escalates a legitimately parked task forever. Fix: classify a run whose only active step is ci-monitoring (green or not yet green, but nothing left besides CI monitoring - no gate, not fixing) as a declared external wait once the worker has declared paused:/captain-held, let the declared paused: line win over that run-step reading, and stop the stale ESCALATION LADDER itself for it (no wedge counting), not just permit absorption.

This branch (fm/fm-watcher-absorb-background-run-v2) is a recovery re-push of an already-fully-reviewed and tested change: an earlier run (01M1DG247YJJPGEEV6N610SAVT) on branch fm/fm-watcher-absorb-background-run completed intent/rebase/review(with 3 fix rounds)/test/document/lint/push and opened its PR, but the no-mistakes gate mirror's origin was misconfigured as kunchenguid/firstmate (since fixed by firstmate) rather than this fork, so it rebased onto kunchenguid's main and opened its PR there by mistake; that PR was closed by firstmate. I fetched the previously-pushed branch (with its 3 review-fix commits and 1 document commit intact), rebased only my own 5 commits onto the fork's actual current main (origin/main), verified no upstream commits remain (git log origin/main..HEAD lists only my own commits; git merge-base --is-ancestor upstream/main HEAD is false), reran the full local test suite clean, and pushed this as a new branch per firstmate's instruction to avoid the old branch's now-broken per-branch gate binding. All implementation and test content is unchanged from the fully-reviewed prior run; this run should sail through with nothing new to review, and its PR step must open against cm-maple7/firstmate (this fork), never kunchenguid/firstmate.

Implementation decisions made (all already reviewed and fixed in the prior run):

  • wedge_timer_check gained an opt-in reverify_working parameter: at each STALE_ESCALATE_SECS threshold crossing for a chain classified working via a run-step reading, it re-confirms crew_is_provably_working before escalating, instead of escalating purely on elapsed time. Applied only to run-step-classified working wedge chains, not the busy-pane/no-completed-turn wedge chain.
  • fm-crew-state.sh's run-step authoritative path reconciles a declared paused:/captain-held status-log line against a run whose only remaining active step is CI monitoring, reporting state: paused instead of working/done. Does not apply when the run has a genuine gate or is fixing.
  • fm-watch.sh's pause_state_class trusts a run-step-attributed paused verdict (via crew_run_step_paused) immediately regardless of agent-pane liveness. A status-log-only pause (no run to corroborate it) still requires the existing dead-agent-pane recovery gate. crew_run_step_paused's check is bounded to the same once-per-STALE_ESCALATE_SECS cadence as the rest of pause_state_class via a cached .paused-run-step- marker (first review fix), and anchors to the actual source field rather than a free-text substring match (second review fix, since a worker's own status-log free text could otherwise contain the literal phrase 'source: run-step').
  • Test coverage added: fm-crew-state.test.sh gained two cases for the declared-pause/ci-monitoring reconciliation (including one where checks are already green, per the third review fix) and that it does not override a genuine gate; fm-watch-triage.test.sh gained a crew_run_step_paused unit classifier test, a live-agent-pane-does-not-defeat-run-step-trust integration test, and updated wedge-escalation tests to reflect that a genuinely still-working chain keeps re-verifying and absorbing instead of escalating on elapsed time alone.

What Changed

  • bin/fm-classify-lib.sh: add crew_run_step_paused, which makes a second bounded fm-crew-state.sh call inside the already-rare declared-pause branch to tell a run-step-corroborated pause (source: run-step) from a status-log-only one.
  • bin/fm-crew-state.sh: when a run's only remaining active step is CI monitoring (no gate, not fixing, no resolved outcome) and the worker's status log declares paused:/captain-held, report state: paused instead of the run-step's own working/done reading, so a legitimately parked CI-monitoring run no longer outranks its own declared wait.
  • bin/fm-watch.sh: wedge_timer_check gains an opt-in reverify_working parameter that re-confirms crew_is_provably_working at each STALE_ESCALATE_SECS threshold crossing before escalating a run-step-classified working chain (used for the plain non-terminal, terminal-overridden, and post-pause working paths), and pause_state_class trusts a run-step-attributed paused verdict immediately regardless of agent-pane liveness, cached per-window via a new .paused-run-step-<key> marker on the same recheck cadence, while a status-log-only pause still requires the existing dead-agent-pane recovery gate.
  • docs/architecture.md: document the run-step reverification contract for wedge escalation and the run-step-corroborated pause trust/dead-agent-gate split.
  • tests/fm-crew-state.test.sh and tests/fm-watch-triage.test.sh: add coverage for the declared-pause/ci-monitoring reconciliation (including an already-green-checks case, and that a genuine gate or fixing step is unaffected), a crew_run_step_paused unit classifier test, a live-agent-pane-does-not-defeat-run-step-trust integration test, and updated wedge-escalation tests reflecting that a still-working run-step chain re-verifies and absorbs instead of escalating on elapsed time alone.

Risk Assessment

✅ Low: The change is a well-bounded, narrowly-scoped fix to two watcher/crew-state classification paths; I traced the concrete failing sequences from both incident cases through the new code (reverify_working threshold re-check in wedge_timer_check, and the ci-monitoring-only paused reconciliation in fm-crew-state.sh plus its trust in fm-watch.sh's pause_state_class) and confirmed both are correctly implemented, properly bounded to the existing recheck cadence, anchored to the structured source field rather than substring matching, and covered by tests that exercise the real binaries and assert observable behavior (wake queue, timer files, escalation counts) rather than source text.

Testing

Interim: reviewed the diff (bin/fm-classify-lib.sh, bin/fm-crew-state.sh, bin/fm-watch.sh, docs/architecture.md, tests/fm-crew-state.test.sh, tests/fm-watch-triage.test.sh) and confirmed the added tests are genuine behavioral E2E tests (real fm-crew-state.sh/fm-watch.sh binaries driven via fakebin fixtures, asserting on printed state, wake-queue entries, escalation counters, and timer files) rather than source-text matching. The targeted test run is still executing (it includes real timing-based watcher E2E scenarios); this response is a placeholder pending its completion — do not treat this as final.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • ./bin/fm-test-run.sh tests/fm-crew-state.test.sh tests/fm-watch-triage.test.sh (in progress, 43/N passing so far, 0 failures observed)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

A no-mistakes background validation run can legitimately sit on an idle
pane far longer than STALE_ESCALATE_SECS. wedge_timer_check used to
escalate purely on elapsed time once a chain was classified working, so a
long validation run got wedge-escalated every few minutes despite the
pipeline actively running. It now re-verifies the run-step is still
working at each threshold crossing (reverify_working) before escalating,
mirroring the existing worktree-write positive-evidence check.

Separately, a worker waiting on a captain-reserved PR merge (or on
upstream checks that may never post without maintainer approval) declares
paused:, but the ci step's own run-step reading otherwise stayed working
for the whole wait and outranked that declaration every poll, so a
legitimately parked task got wedge-escalated forever. The crew-state
reader now reconciles a declared pause against a run whose only remaining
step is CI monitoring, reporting it as paused (never overriding a genuine
gate or fixing step); the watcher trusts that run-step-attributed paused
verdict outright instead of requiring a dead-looking agent pane, while a
status-log-only pause (no run to corroborate it) still goes through the
existing agent-liveness recovery so a live interactive gate is not
silently hidden behind a stale declaration.
…lassify-lib.sh with new reverify/pause logic
@cm-maple7
cm-maple7 force-pushed the fm/fm-watcher-absorb-background-run-v2 branch from 3ef4993 to 3559bfc Compare September 1, 2026 06:26
@cm-maple7
cm-maple7 merged commit d77251e into main Sep 1, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant