Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ Work routed to a secondmate is recorded in that secondmate home's own backlog, n
A decision is simply a task held for the captain: create the task with `bin/fm-tasks-axi.sh add` when needed, then always hold it through `bin/fm-captain-hold.sh hold <id> --reason "<reason>"`, with `--until <date>` when the captain defers it.
When a main-side thread such as a pending captain decision or relay reminder is worth durable tracking, file it as its own work item and hold it through that wrapper.
Captain calls discovered by investigations or visual reviews follow `captain-hold-lifecycle`, which owns their completion gate and recorded-answer rules.
When the automatic transition gate applies, dispatch and completion move the item themselves - `bin/fm-spawn.sh` and `bin/fm-teardown.sh` own those transitions and refuse rather than report success without them - so what remains yours is filing the item before dispatch, recording decisions, and keeping notes current; `docs/configuration.md` owns gate applicability and the manual-backend exception.
When the automatic transition gate applies, dispatch and completion move the item themselves - `bin/fm-spawn.sh` and `bin/fm-teardown.sh` own those transitions and refuse rather than report success without them - so what remains yours is filing each item as soon as its work is authorized - including every later phase gated on another item (`blocked-by`) or a date, so teardown and session-start re-evaluation can find it - recording decisions, and keeping notes current; `docs/configuration.md` owns gate applicability and the manual-backend exception.
Re-evaluate queued work after every teardown and heartbeat, dispatching items only when dependencies and time gates have cleared.

`.tasks.toml`, `docs/configuration.md`, and current `tasks-axi --help` own the backlog schema, compatibility, retention, and routine command syntax.
Expand Down
1 change: 1 addition & 0 deletions bin/fm-brief.sh
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,7 @@ Delegate project work to your own crewmates with the normal firstmate lifecycle:
Do not invent a second delegation system.
You do not generate your own work.
Act only on tasks the main firstmate routes to you.
Later phases the main firstmate authorizes in a routed message are routed work: file each one in your backlog when it arrives, with its dependencies, and dispatch it when it becomes ready without waiting to be asked again.
Never start a survey, audit, or "find improvements" sweep on your own initiative; that is not your job and it is unwanted.

# The captain and the parent channel
Expand Down
15 changes: 15 additions & 0 deletions bin/fm-herdr-lab-viewer.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,21 @@ def _child(slave, master, session):


def _process_start(pid):
# Must match bin/fm-herdr-lab.sh's fm_herdr_lab_process_start: /proc start
# ticks count from boot, so a host clock step cannot change them the way it
# re-renders ps lstart.
try:
with open("/proc/%d/stat" % pid, encoding="utf-8") as handle:
stat = handle.read()
except OSError:
return _process_lstart(pid)
fields = stat.rpartition(")")[2].split()
if len(fields) < 20 or not fields[19].isdigit():
raise RuntimeError("process start ticks unavailable")
return "proc-starttime=%s" % fields[19]


def _process_lstart(pid):
result = subprocess.run(
["ps", "-p", str(pid), "-o", "lstart="],
check=True,
Expand Down
41 changes: 36 additions & 5 deletions bin/fm-herdr-lab.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@
# bin/fm-herdr-lab-viewer.py owns the pty mechanics.
# Start succeeds only when that session reports a foreground client and the
# recorded viewer process still matches its launch identity.
# When /proc stat is readable, the viewer records start ticks for both processes;
# existing ps lstart records remain readable for a running viewer.
# Stop signals only identity-matched recorded processes and retains its
# ownership record until detach is confirmed or the session is stopped or
# absent; teardown refuses when that stop cannot be confirmed.
Expand Down Expand Up @@ -207,10 +209,41 @@ fm_herdr_lab_viewer_reason() { # <session>
printf '%s' "$out" | jq -r '.result.reason // empty' 2>/dev/null
}

# Prints the process start identity the viewer launcher records. /proc stat
# field 22 counts clock ticks since boot, so a host clock step cannot change it;
# ps lstart re-renders those ticks against the wall-clock boot time (WSL2 steps
# it about every 30 seconds) and would disown a running viewer.
fm_herdr_lab_process_start() { # <pid>
local stat_line starttime
local -a stat_fields
if [ -r "/proc/$1/stat" ]; then
stat_line=$(cat "/proc/$1/stat" 2>/dev/null) || return 1
# After the final comm delimiter, array index 19 is proc stat field 22.
read -r -a stat_fields <<< "${stat_line##*)}"
[ "${#stat_fields[@]}" -ge 20 ] || return 1
starttime=${stat_fields[19]}
case "$starttime" in ''|*[!0-9]*) return 1 ;; esac
printf 'proc-starttime=%s' "$starttime"
return 0
fi
fm_herdr_lab_process_lstart "$1"
}

fm_herdr_lab_process_lstart() { # <pid>
LC_ALL=C ps -p "$1" -o lstart= 2>/dev/null | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
}

fm_herdr_lab_process_start_matches() { # <pid> <recorded-start>
local current
case "$2" in
proc-starttime=*) current=$(fm_herdr_lab_process_start "$1") || return 1 ;;
# A record written before start-tick identity holds ps lstart text; keep
# honoring it so an upgrade does not strand a running viewer.
*) current=$(fm_herdr_lab_process_lstart "$1") || return 1 ;;
esac
[ -n "$current" ] && [ "$current" = "$2" ]
}

fm_herdr_lab_process_parent() { # <pid>
LC_ALL=C ps -p "$1" -o ppid= 2>/dev/null | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
}
Expand All @@ -225,18 +258,16 @@ fm_herdr_lab_viewer_recorded_value() { # <session> <key>
}

fm_herdr_lab_viewer_owned_pair() { # <session>
local launcher_pid viewer_pid launcher_start viewer_start current_start parent_pid
local launcher_pid viewer_pid launcher_start viewer_start parent_pid
launcher_pid=$(fm_herdr_lab_viewer_recorded_value "$1" launcher_pid) || return 1
viewer_pid=$(fm_herdr_lab_viewer_recorded_value "$1" viewer_pid) || return 1
case "$launcher_pid:$viewer_pid" in
*[!0-9:]*) return 1 ;;
esac
launcher_start=$(fm_herdr_lab_viewer_recorded_value "$1" launcher_start) || return 1
viewer_start=$(fm_herdr_lab_viewer_recorded_value "$1" viewer_start) || return 1
current_start=$(fm_herdr_lab_process_start "$launcher_pid") || return 1
[ -n "$current_start" ] && [ "$current_start" = "$launcher_start" ] || return 1
current_start=$(fm_herdr_lab_process_start "$viewer_pid") || return 1
[ -n "$current_start" ] && [ "$current_start" = "$viewer_start" ] || return 1
fm_herdr_lab_process_start_matches "$launcher_pid" "$launcher_start" || return 1
fm_herdr_lab_process_start_matches "$viewer_pid" "$viewer_start" || return 1
parent_pid=$(fm_herdr_lab_process_parent "$viewer_pid") || return 1
[ "$parent_pid" = "$launcher_pid" ] || return 1
printf '%s %s' "$launcher_pid" "$viewer_pid"
Expand Down
36 changes: 34 additions & 2 deletions bin/fm-pending-reply-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@
# request_turn_completed_epoch=
# recovery_attempted_epoch=
# recovery_sender_pid=
# recovery_sender_identity=
# recovery_sender_identity= Linux /proc start ticks plus full cmdline hex;
# ps lstart plus command where /proc is unavailable.
# Existing ps-form records remain readable.
# recovery_sent_epoch=
# recovery_delivery_outcome=
# recovery_turn_seen_busy=
Expand Down Expand Up @@ -984,6 +986,31 @@ fm_pending_reply_send_recovery() { # <state-dir> <corr_id>
}

fm_pending_reply_pid_identity() { # <pid>
local pid=$1 proc_root stat_line starttime cmdline_hex
local -a stat_fields
case "$pid" in ''|*[!0-9]*) return 1 ;; esac
proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc}
# /proc stat field 22 counts clock ticks since boot, so a host clock step
# cannot change it; ps lstart re-renders those ticks against the wall-clock
# boot time (WSL2 steps it about every 30 seconds) and would read a live
# sender as dead. Start ticks distinguish reused PIDs; the full cmdline
# preserves the sender command identity.
if [ -r "$proc_root/$pid/stat" ] && [ -r "$proc_root/$pid/cmdline" ]; then
stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1
# After the final comm delimiter, array index 19 is proc stat field 22.
read -r -a stat_fields <<< "${stat_line##*)}"
[ "${#stat_fields[@]}" -ge 20 ] || return 1
starttime=${stat_fields[19]}
case "$starttime" in ''|*[!0-9]*) return 1 ;; esac
cmdline_hex=$(od -An -v -tx1 "$proc_root/$pid/cmdline" 2>/dev/null | tr -d '[:space:]') || return 1
[ -n "$cmdline_hex" ] || return 1
printf 'proc-starttime=%s cmdline-hex=%s' "$starttime" "$cmdline_hex"
return 0
fi
fm_pending_reply_ps_identity "$pid"
}

fm_pending_reply_ps_identity() { # <pid>
local pid=$1 identity
case "$pid" in ''|*[!0-9]*) return 1 ;; esac
identity=$(COLUMNS=10000 LC_ALL=C ps -p "$pid" -o lstart= -o command= 2>/dev/null) || return 1
Expand All @@ -996,7 +1023,12 @@ fm_pending_reply_sender_alive() { # <record-path>
pid=$(fm_pending_reply_get "$rec" recovery_sender_pid)
expected=$(fm_pending_reply_get "$rec" recovery_sender_identity)
[ -n "$expected" ] || return 1
actual=$(fm_pending_reply_pid_identity "$pid") || return 1
case "$expected" in
proc-starttime=*) actual=$(fm_pending_reply_pid_identity "$pid") || return 1 ;;
# A record written before start-tick identity holds the ps form; keep
# honoring it so an upgrade does not strand an in-flight recovery.
*) actual=$(fm_pending_reply_ps_identity "$pid") || return 1 ;;
esac
[ "$actual" = "$expected" ]
}

Expand Down
55 changes: 52 additions & 3 deletions bin/fm-remote-job-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,17 @@
# it to stop itself once its root is pruned, and
# bin/fm-remote-job-reap-orphans.sh uses it to reap workers that were already
# orphaned that way.
#
# fm_remote_job_process_start is the one process-identity reader behind the
# worker lock, staging, and claim start records. Where /proc/<pid>/stat is
# readable (Linux) it records starttime=<clock ticks since boot, stat field
# 22>, which no wall-clock step moves; ps lstart is rendered from the current
# boot time there, so every NTP, VM or WSL2 time-sync, or resume step would
# make a live worker stop matching its own records. Elsewhere (Darwin) it
# records ps lstart, which a clock step does not move. A Linux record still in
# lstart form was written before this contract: fm_remote_job_process_start_for_record
# compares it as lstart, and a lock owner in that form is identified by pid and
# exact command so ensure replaces it in place.

FM_REMOTE_JOB_LABEL=dev.firstmate.remote-job
FM_REMOTE_JOB_MAX_BYTES=${FM_REMOTE_JOB_MAX_BYTES:-1048576}
Expand Down Expand Up @@ -767,7 +778,7 @@ fm_remote_job_stage_owner_alive() { # <stage-dir>
case "$pid" in ''|*[!0-9]*) return 1 ;; esac
[ "$pid" -gt 1 ] || return 1
recorded_start=$(fm_remote_job_read_single_line "$stage/.owner-start" 256 2>/dev/null) || return 1
actual_start=$(fm_remote_job_process_start "$pid" 2>/dev/null) || return 1
actual_start=$(fm_remote_job_process_start_for_record "$pid" "$recorded_start" 2>/dev/null) || return 1
[ "$recorded_start" = "$actual_start" ]
}

Expand Down Expand Up @@ -902,7 +913,24 @@ fm_remote_job_worker_ready_path() { printf '%s\n' "$FM_REMOTE_JOB_STATE/worker.r
fm_remote_job_worker_identity_path() { printf '%s\n' "$FM_REMOTE_JOB_STATE/worker.identity"; }
fm_remote_job_worker_lock_path() { printf '%s\n' "$FM_REMOTE_JOB_STATE/worker.lock"; }

fm_remote_job_process_start() {
fm_remote_job_process_start() { # <pid>
local pid=$1 proc_root stat_line
local -a stat_fields
case "$pid" in ''|*[!0-9]*) return 1 ;; esac
proc_root=${FM_PROC_ROOT_OVERRIDE:-/proc}
if [ -r "$proc_root/$pid/stat" ]; then
stat_line=$(cat "$proc_root/$pid/stat" 2>/dev/null) || return 1
# After the final comm delimiter, array index 19 is proc stat field 22.
read -r -a stat_fields <<< "${stat_line##*)}"
[ "${#stat_fields[@]}" -ge 20 ] || return 1
case "${stat_fields[19]}" in ''|*[!0-9]*) return 1 ;; esac
printf 'starttime=%s\n' "${stat_fields[19]}"
return 0
fi
fm_remote_job_process_lstart "$pid"
}

fm_remote_job_process_lstart() { # <pid>
local pid=$1 ps_bin value
if [ -x /bin/ps ]; then ps_bin=/bin/ps; elif [ -x /usr/bin/ps ]; then ps_bin=/usr/bin/ps; else return 1; fi
value=$("$ps_bin" -p "$pid" -o lstart= 2>/dev/null) || return 1
Expand All @@ -911,6 +939,19 @@ fm_remote_job_process_start() {
printf '%s\n' "$value"
}

# The current start of <pid> in the form <recorded> was written in, so a record
# a pre-upgrade worker wrote as ps lstart text is still compared as lstart
# rather than never matching the starttime= form.
fm_remote_job_process_start_for_record() { # <pid> <recorded-start>
local current
current=$(fm_remote_job_process_start "$1") || return 1
case "$current:$2" in
starttime=*:starttime=*) ;;
starttime=*:*) current=$(fm_remote_job_process_lstart "$1") || return 1 ;;
esac
printf '%s\n' "$current"
}

fm_remote_job_process_command() {
local pid=$1 ps_bin value
if [ -x /bin/ps ]; then ps_bin=/bin/ps; elif [ -x /usr/bin/ps ]; then ps_bin=/usr/bin/ps; else return 1; fi
Expand Down Expand Up @@ -1012,7 +1053,15 @@ fm_remote_job_lock_owner_matches_process() {
[ "$pid" -gt 1 ] || return 1
recorded_start=$(fm_remote_job_read_single_line "$lock/start" 256) || return 1
actual_start=$(fm_remote_job_process_start "$pid") || return 1
[ "$recorded_start" = "$actual_start" ] || return 1
# A Linux owner recorded as ps lstart text is a worker from before start ticks
# were recorded. Any clock step since has re-rendered its lstart, so its pid
# and exact command identify it, which lets ensure replace it in place
# instead of starting a second supervisor beside it.
case "$actual_start:$recorded_start" in
starttime=*:starttime=*) [ "$recorded_start" = "$actual_start" ] || return 1 ;;
starttime=*:*) ;;
*) [ "$recorded_start" = "$actual_start" ] || return 1 ;;
esac
recorded_command=$(fm_remote_job_read_single_line "$lock/command" 8192) || return 1
actual_command=$(fm_remote_job_process_command "$pid") || return 1
[ "$recorded_command" = "$actual_command" ] || return 1
Expand Down
8 changes: 4 additions & 4 deletions bin/fm-remote-job-worker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ worker_signal_process_or_group() { # process|group <signal> <pid>
worker_supervisor_identity_status() { # <job-dir> <pid>
local job=$1 pid=$2 recorded_start actual_start
recorded_start=$(fm_remote_job_read_single_line "$job/.claim/supervisor_start" 256 2>/dev/null) || return 2
actual_start=$(fm_remote_job_process_start "$pid" 2>/dev/null) || {
actual_start=$(fm_remote_job_process_start_for_record "$pid" "$recorded_start" 2>/dev/null) || {
worker_process_or_group_alive process "$pid" && return 2
return 1
}
Expand All @@ -350,7 +350,7 @@ worker_group_identity_status() { # <job-dir> <pid>
local job=$1 pid=$2 recorded_start actual_start file="$1/.claim/group_start"
[ -e "$file" ] || [ -L "$file" ] || return 3
recorded_start=$(fm_remote_job_read_single_line "$file" 256 2>/dev/null) || return 2
actual_start=$(fm_remote_job_process_start "$pid" 2>/dev/null) || {
actual_start=$(fm_remote_job_process_start_for_record "$pid" "$recorded_start" 2>/dev/null) || {
kill -0 "$pid" 2>/dev/null && return 2
worker_process_or_group_alive group "$pid" && return 0
return 1
Expand Down Expand Up @@ -446,7 +446,7 @@ worker_stop_recorded_execution() { # <job-dir>
worker_lane_identity_matches() { # <pid> <start>
local pid=$1 start=$2 actual_start
[ -n "$start" ] || return 1
actual_start=$(fm_remote_job_process_start "$pid" 2>/dev/null) || return 1
actual_start=$(fm_remote_job_process_start_for_record "$pid" "$start" 2>/dev/null) || return 1
[ "$actual_start" = "$start" ]
}

Expand Down Expand Up @@ -571,7 +571,7 @@ worker_claim_owner_alive() { # <job-dir>
case "$pid" in ''|*[!0-9]*) return 1 ;; esac
if [ -e "$claim/owner_start" ] || [ -L "$claim/owner_start" ]; then
recorded_start=$(fm_remote_job_read_single_line "$claim/owner_start" 256 2>/dev/null) || return 1
actual_start=$(fm_remote_job_process_start "$pid" 2>/dev/null) || return 1
actual_start=$(fm_remote_job_process_start_for_record "$pid" "$recorded_start" 2>/dev/null) || return 1
[ "$recorded_start" = "$actual_start" ]
return
fi
Expand Down
2 changes: 2 additions & 0 deletions docs/remote-secondmates.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ On macOS the worker is `dev.firstmate.remote-job`, an Aqua-scoped LaunchAgent at
After that bootstrap, every non-doctor `fm-on.sh` target runs through that worker in the remote account's GUI session.
It never runs in the SSH process or a Herdr pane.
Linux uses the same queue and worker protocol without the Aqua-session requirement.
On Linux, where `/proc/<pid>/stat` is readable, the worker uses kernel start ticks for process identity so host clock steps do not make a healthy worker appear stale.
During an upgrade, a worker with an older `ps lstart` lock record is recognized by its PID and exact command and replaced when its code changes; [`bin/fm-remote-job-lib.sh`](../bin/fm-remote-job-lib.sh) owns that identity contract.
When idle, the worker checks for newly staged work about once per second; after a lane starts or finishes it checks more frequently for a short period.

### Job lanes and preemption
Expand Down
4 changes: 4 additions & 0 deletions tests/fm-brief.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,10 @@ test_secondmate_no_projects_charter() {
"secondmate charter did not close a quietly ended routed-work phase"
assert_grep 'use the same key on its later' "$brief" \
"secondmate charter did not supersede working phases with later states"
assert_grep 'Later phases the main firstmate authorizes in a routed message are routed work' "$brief" \
"secondmate charter did not treat authorized later phases as routed work"
assert_grep 'file each one in your backlog when it arrives, with its dependencies' "$brief" \
"secondmate charter did not require filing authorized later phases on arrival"
if grep -nE '^-[[:space:]]*$' "$brief" >/dev/null; then
fail "project-less charter left a stray empty project bullet"
fi
Expand Down
Loading