Skip to content

fix: clean up AFK launcher lock on signal - #517

Closed
ViktorTsvetkov wants to merge 3 commits into
kunchenguid:mainfrom
ViktorTsvetkov:fm/fix-afklaunch-race
Closed

ViktorTsvetkov wants to merge 3 commits into
kunchenguid:mainfrom
ViktorTsvetkov:fm/fix-afklaunch-race

Conversation

@ViktorTsvetkov

Copy link
Copy Markdown

Intent

Fix a signal-handling race in the away-mode launcher. fm_afk_launch_main armed its EXIT/INT/TERM traps AFTER calling fm_afk_launch_lock_acquire, so a TERM/INT delivered in that window killed the process by default disposition, the EXIT trap never ran, and the lifecycle lock was orphaned. Move the three trap lines before the lock acquisition; fm_afk_launch_lock_release is already pid-guarded ([ pid = $$ ] || return 0), so arming the EXIT trap before acquisition is a safe no-op if a signal fires pre-acquire and a correct release otherwise. Pure cross-platform bug fix, affects POSIX and Windows equally, not gated. Deflakes tests/fm-afk-launch.test.sh unit_signal_exits_with_lock_cleanup (measured ~40 percent to ~0.4 percent failure).

What Changed

  • Captain, armed the AFK launcher EXIT/INT/TERM traps before lifecycle lock acquisition so interrupted starts release any owned .afk-launch.lock and preserve signal exit status.
  • Added a regression unit case for TERM delivered during lock-directory publication, alongside the existing launcher signal cleanup coverage.
  • Documented the AFK launcher lock signal-safety behavior in the herdr backend notes.

Risk Assessment

✅ Low: Captain, the change is tightly scoped to AFK launcher signal cleanup, preserves the existing lock ownership guard, and adds a focused regression case for the previously reported publication-window race.

Testing

The pre-run full baseline was already green; I then ran the relevant AFK launcher suite and a focused signal-window check for TERM and INT, both of which demonstrated correct signal exit and lock cleanup with evidence saved under /tmp/no-mistakes-evidence/01KXD28SMG8QQMACD1NKHAA3N9.

Evidence: AFK launcher test log
ok - clear-stale: removes escalations buffer, sidecar, and wedge marker
ok - clear-stale: leaves the durable wake-queue intact (no pending work dropped)
ok - refresh: daemon already alive - stale artifacts preserved (current session's buffer kept)
ok - stop-ordering: daemon SIGTERM'd while .afk still present (flush is not a no-op)
ok - stop-ordering: .afk cleared last
ok - stop-ordering: daemon-terminal record removed
ok - stop identity: stale lock cannot signal an unrelated live process
ok - failed start: away flag and delivery artifacts roll back
ok - concurrent start: one serialized daemon terminal remains tracked
ok - launcher lock: incomplete publication receives initialization grace
ok - launcher signal: TERM exits and releases the lifecycle lock
ok - launcher signal: TERM during lock publication releases the lifecycle lock
ok - herdr create: malformed response recovers durable exact ownership
ok - herdr create error: unconfirmed exact id is persisted for reconciliation
ok - herdr run failure: unconfirmed exact id remains reconcilable
ok - record failure: newly created terminal is closed by exact id
ok - readiness failure: exact terminal and durable record roll back
ok - readiness failure: unconfirmed terminal retains its reconciliation id
ok - tmux absence: clean missing differs from transport probe failure
ok - native lifecycle: launcher owns state with no terminal
ok - native lifecycle: uniform stop clears state without closing a terminal
ok - native entry: launcher-prepared lifecycle state is not rewritten
ok - teardown failure: exact terminal record is preserved
ok - record publication: failed atomic rename preserves the complete prior record
ok - record read: malformed record fails closed without acting on a partial id
ok - stop: malformed terminal record preserves away state and fails closed
ok - tmux launch: planned exact target is recorded before creation and removed on failure
ok - tmux launch: unique names eliminate collision teardown
ok - stop validation: malformed record causes no daemon or state side effects
ok - launcher lock: incomplete metadata fails acquisition and releases lock
ok - stop state: away-flag removal failure is surfaced
ok - stop liveness: captured live daemon preserves lifecycle state after lock release
ok - refresh record: malformed terminal identity fails closed
ok - clear failure: native entry aborts and restores prior state
ok - confirmed absence: cleanup succeeds and removes the stale record
ok - rollback restore: incomplete restoration retains its recovery backup
ok - flag failure: lifecycle aborts without active state
ok - herdr e2e: captain tab pane count unchanged after start (no split)
ok - herdr e2e: daemon launched in a separate non-visible workspace
ok - herdr e2e: daemon pane is NOT in the captain's tab
ok - herdr e2e: daemon terminal scoped to the lab session
ok - herdr e2e: captain tab pane count restored after stop
ok - herdr e2e: daemon workspace removed by exact id on stop
ok - herdr e2e: record + .afk cleared on stop
ok - tmux e2e: captain window pane count unchanged after start (no split-window)
ok - tmux e2e: daemon launched in a separate detached session
ok - tmux e2e: captain window pane count unchanged after stop
ok - tmux e2e: daemon session killed by exact id on stop
ok - tmux e2e: record + .afk cleared on stop
Evidence: Signal-window transcript

signal=TERM exit_code=143 expected=143 lock_exists_after_exit=no verdict=pass signal=INT exit_code=130 expected=130 lock_exists_after_exit=no verdict=pass

signal=TERM
home=/tmp/fm-afk-signal-window.GioLbG
exit_code=143 expected=143
lock_exists_after_exit=no
verdict=pass

signal=INT
home=/tmp/fm-afk-signal-window.jZz8Dr
exit_code=130 expected=130
lock_exists_after_exit=no
verdict=pass

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • ⚠️ bin/fm-afk-launch.sh:601 - The remaining lock-acquisition critical section can still orphan .afk-launch.lock: the traps now run before fm_afk_launch_lock_acquire, but fm_afk_launch_lock_release only removes the directory when pid already equals $$. If TERM/INT lands after mkdir "$FM_AFK_LAUNCH_LOCK" succeeds but before pid/pid-identity are fully written, the EXIT trap sees no matching pid and leaves the just-created lock behind. Make acquisition publish ownership in a signal-safe cleanup path, or let the release path remove an incomplete lock known to be created by this process.

🔧 Fix: Captain, harden AFK lock signal cleanup
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • Baseline already provided as passing: command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • bash tests/fm-afk-launch.test.sh | tee /tmp/no-mistakes-evidence/01KXD28SMG8QQMACD1NKHAA3N9/fm-afk-launch-test.log
  • Manual evidence check invoking fm_afk_launch_main start with injected TERM and INT during mkdir "$FM_AFK_LAUNCH_LOCK", recorded in /tmp/no-mistakes-evidence/01KXD28SMG8QQMACD1NKHAA3N9/afk-launch-signal-window-transcript.log
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@ViktorTsvetkov

Copy link
Copy Markdown
Author

Some extra evidence for this one, in case it helps prioritise it: while validating #506 I saw this race reproduce on a clean main checkout, with none of my changes applied.

tests/fm-afk-launch.test.sh intermittently fails its own assertion:

not ok - launcher signal: interrupted lifecycle resumed or retained its lock

That is the existing unit_signal_exits_with_lock_cleanup case — it TERMs the launcher mid-lifecycle and asserts state/.afk-launch.lock is released. When the race hits, the lock is left behind. It surfaced twice for me during unrelated full-suite runs on main, which is what sent me looking.

To check it on your side:

git checkout main
for i in $(seq 1 20); do
  ./tests/fm-afk-launch.test.sh 2>&1 | grep '^not ok - launcher signal' || true
done

I measured roughly 40% failures before this fix and 0/150 after — but that was on my hardware, and the window is timing-dependent, so the rate is likely to differ on yours. Worth confirming rather than taking my number at face value; the failure mode itself should reproduce.

@kunchenguid

kunchenguid commented Jul 24, 2026 •

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#517 at 535c294f.

@kunchenguid

Copy link
Copy Markdown
Owner

Automated reminder: this PR still looks blocked on a rebase or merge conflict fix.

If you are still interested, please rebase onto the current base branch, resolve the conflict, and push.

If I do not hear back, I may close this as inactive.

@kunchenguid

Copy link
Copy Markdown
Owner

I am closing this because it has been waiting on a rebase or merge-conflict fix since 2026-07-24, and I have not seen a comment or push since then.

If you still want to keep working on this, please reopen it or open a new PR and mention this one.

Happy to take another look when there is an update.

@kunchenguid kunchenguid closed this Aug 7, 2026
@fhhcdde588

Copy link
Copy Markdown

Scheduling approved by David, 2026-09-09. The overnight-campaign session that owns this card is approved to start, which supersedes the earlier intake-only hold recorded on this issue.

Ownership and overlap fences for that session come from the campaign's bundling and overlap audit completed before launch. Item-level acceptance stays per card, and a review finding on any card in a bundle holds the whole bundle.

Not released by this scheduling approval: specific design selections, protected-region decisions, and any recurring production cost increase. Those remain reserved for David.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants