fix(bin): preserve process identity across host clock steps - #52
Merged
Merged
Conversation
* fix(bin): keep pending-reply sender and lab viewer identity stable across clock steps The pending-reply recovery sender check and the Herdr lab viewer ownership check identified processes by ps lstart text, which on Linux is the wall-clock-derived boot time plus start ticks. A host clock step (WSL2 steps about every 30 seconds) re-renders it, so a live recovery sender read as dead and a running lab viewer pair read as not owned. Both now use /proc/<pid>/stat start ticks where readable, like fm_pid_identity and task_process_identity, and keep the ps form elsewhere. Records already written in the legacy lstart form are still compared through ps, so an upgrade does not strand an in-flight recovery or a running viewer. * no-mistakes(document): Document clock-stable process identity and legacy records
…#46) * fix(bin): keep Linux remote job worker identity stable across clock steps The remote job worker identified its own processes (lock owner, staging owner, job claims, lanes, command groups) by `ps -o lstart=` text. On Linux, procps renders lstart from the current boot time, which moves whenever the wall clock is stepped (NTP, VM or WSL2 time sync, resume). After a step a healthy worker no longer matched its own lock record, so every remote call started another detached supervisor beside it, the losers restarted for minutes, the serving loop blocked on live lanes it thought had exited, a competing worker reclaimed the live lock, and running jobs were published as "remote job worker stopped before this job completed". - Record Linux process identity as starttime=<stat field 22>, which no clock step moves; Darwin keeps ps lstart, unchanged. - Keep records written by earlier workers comparable: an lstart record is compared as lstart, and a Linux lock owner still recorded as lstart is identified by pid and exact command, so an update replaces it in place and drains supervisors already piled beside it instead of stranding it. - A serving worker that has lost its ownership lock now stops its own active execution and exits on a stop signal instead of re-arming, and never writes quarantine into a lock it does not own. * no-mistakes(document): Document Linux remote worker identity and shutdown behavior
knowttl
force-pushed
the
fm/fm-reland-clock-fixes
branch
from
September 30, 2026 18:15
4b6eb9f to
9223540
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Re-land two fixes that were dropped when the fork's local main was hard-reset to match upstream: "fix(bin): keep process identity stable across host clock steps" (#45, commit 59c9f6d) and "fix(bin): prevent Linux remote job worker pileups after clock changes" (#46, commit a97b7cd). Context: after the reset, the remote second mate host's remote job worker no longer starts on current code.
fm-remote-doctor.sh --fixfails with "remote job worker did not report ready after startup", and each attempt leaves another stuckfm-remote-job-worker.shprocess behind (23 on the host at last count). The captain approved dispatching this fix to get that host working again.What Changed
ps lstartrecords and recognize remote worker lock owners by PID and exact command, preventing supervisor pileups and allowing replacement when worker code changes.Risk Assessment
✅ Low: The changes are bounded to clock-stable process identity and legacy-record compatibility, with no substantiated correctness or intent-conformance defects.
Testing
Three targeted test files and live worker/watcher checks passed, including a pre-fix ownership counterexample. Clock effects were simulated because namespace isolation was denied. Herdr preparation was blocked, preventing real viewer and visual evidence. Disposable state was cleaned up.
Evidence: Live worker and watcher transcript
Source: Live worker and watcher transcript
Evidence: Herdr live preparation blocker
Source: Herdr live preparation blocker
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
⏭️ **Test** - skipped
TMPDIR="$PWD/.test-clock-tmp" bash tests/fm-remote-job.test.shTMPDIR="$PWD/.test-clock-tmp" bash tests/fm-pending-reply.test.shTMPDIR="$PWD/.test-clock-tmp" bash tests/fm-herdr-lab.test.shbash ~/.no-mistakes/evidence/01M3SNGYYS8919CG59S3FMWEW3/live-clock-check.shExecuted the base commit's worker ownership interface against the same live worker: it rejected the drifted record; current code accepted it.unshare --user --map-root-user --time truerefused with Operation not permitted; clock effects were simulated without changing host time.Ran isolatedbin/fm-herdr-lab.sh prepare fm-lab-clock-gate; the required default-session tripwire was unavailable.Stopped disposable worker trees, removed transient worktree files, and confirmed clean git status.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.