From f604a1863c1561479697b3bee48e7f0160d339a6 Mon Sep 17 00:00:00 2001 From: Gopinath K Date: Thu, 17 Sep 2026 00:48:41 +0530 Subject: [PATCH 1/7] fix(bin): give every Firstmate home its own Treehouse pool root Treehouse names a pool after the repository it serves, so the primary home and a persistent secondmate holding clones of one remote resolved to a single shared pool and could be handed each other's live slots. Only Claude's workspace-trust check caught the observed cross-owner allocation; no runtime-independent guard existed. Every ship and scout spawn now acquires its slot with `treehouse get --root /fm-home-`, a deterministic root per canonical FM_HOME owned by fm_treehouse_home_root in bin/fm-wake-lib.sh, and records it as treehouse_root= in the task meta. Teardown reconciles that record against the slot's actual root through fm_treehouse_task_root and returns the slot with the same --root; a record whose root does not contain its worktree refuses before any mutation, for the task itself and for a secondmate's child tasks alike. A record without the field predates it and returns through the pool that allocated it, so existing slots drain with no migration, move, or reinterpretation. Secondmate homes stay the primary's own lease under Treehouse's default root. Spawn also refuses a slot whose owner claim names a task that still has a task record in its home, on every harness and backend, and replaces only a claim whose task record is gone. tests/fm-treehouse-pool-isolation.test.sh pins the contract: two homes with clones of one repo allocate to distinct roots, a task returns through its recorded root, a legacy record returns through its original root, a mismatched root refuses without mutation at spawn and teardown, and a live foreign claim refuses while a stale one is replaced. --- .../skills/stuck-crewmate-recovery/SKILL.md | 3 +- bin/fm-home-seed.sh | 6 +- bin/fm-spawn.sh | 81 ++++- bin/fm-teardown.sh | 57 +++- bin/fm-test-run.sh | 3 +- bin/fm-wake-lib.sh | 91 +++++ docs/architecture.md | 1 + docs/configuration.md | 4 + tests/fm-spawn-pool-base-freshen.test.sh | 13 +- tests/fm-tangle-guard.test.sh | 2 +- tests/fm-treehouse-pool-isolation.test.sh | 314 ++++++++++++++++++ 11 files changed, 559 insertions(+), 16 deletions(-) create mode 100755 tests/fm-treehouse-pool-isolation.test.sh diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index c5209051a44..6a416554611 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -32,7 +32,8 @@ Read the targeted current state with `bin/fm-crew-state.sh ` before deciding A no-mistakes run matched to the crew's branch and current code remains authoritative when the endpoint is dead: handle a terminal or parked run through the normal lifecycle, and keep supervising an active run instead of creating a duplicate worker. When no authoritative run accounts for the task, inspect only its recorded backend and worktree inventory. -Use `treehouse status` for treehouse-backed tmux, herdr, zellij, or cmux tasks, and use the recorded `orca_worktree_id=` and `terminal=` for Orca tasks. +Use `treehouse status --root ` for treehouse-backed tmux, herdr, zellij, or cmux tasks, where `` is the task's recorded `treehouse_root=` or, for a record without one, the pool root its `worktree=` path sits under; a bare `treehouse status` reads only Treehouse's default root and misses every home-scoped pool. +Use the recorded `orca_worktree_id=` and `terminal=` for Orca tasks. Do not sweep another home's endpoints or infer ownership from a matching window label. Before relaunch, prove that no live agent still owns the recorded task and that the existing worktree remains available. diff --git a/bin/fm-home-seed.sh b/bin/fm-home-seed.sh index 6693ab1df74..ac8be0a6054 100755 --- a/bin/fm-home-seed.sh +++ b/bin/fm-home-seed.sh @@ -7,7 +7,11 @@ # a fresh firstmate worktree via "treehouse get --lease", which durably # leases the worktree under the secondmate so the home survives with # no live process and is never recycled until the lease is released with -# "treehouse return". Projects are cloned +# "treehouse return". That lease is the PRIMARY's own, taken from the +# firstmate repository's pool under Treehouse's default root, and is +# deliberately not one of the per-home child project pools that +# bin/fm-wake-lib.sh's fm_treehouse_home_root gives ship and scout +# slots, so registered homes keep resolving. Projects are cloned # from the active home into the secondmate home's projects/ directory. # That project list is non-exclusive provisioning data. Pass --no-projects # instead of a project list to seed a project-less home for a domain whose diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 88fa2fcfabe..2e187aaf2e8 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -347,6 +347,15 @@ # A ship task records the explicit mode/yolo it was passed; a secondmate spawn records # mode=secondmate, yolo=off, home=, and projects=; a scout records neither, and both the # success line and state/.meta omit them. +# A ship or scout spawn on a Treehouse-backed backend (every backend except orca) +# acquires its slot with `treehouse get --root `, where is this +# home's own Treehouse root (bin/fm-wake-lib.sh's fm_treehouse_home_root: a +# deterministic /fm-home- per canonical FM_HOME), and records that +# root as treehouse_root= in state/.meta. A slot that lands outside that root, +# or that another task's record still holds (judged from the slot's owner claim +# and that claimant's state/.meta, independent of harness or backend), refuses +# the launch. A relaunch keeps the recorded treehouse_root= line; a record without +# one predates the field and resolves its root from the slot path itself. # Every fresh spawn or relaunch records a new spawn_gen= incarnation token so durable # consumers can distinguish a replacement worker that reuses the same task id. # When the home session's frozen trace-context decision is enabled (see @@ -1046,6 +1055,7 @@ SPAWN_TASK_SET_LOCK_HELD=0 SPAWN_TREEHOUSE_PROJECT_LOCK= SPAWN_TREEHOUSE_PROJECT_LOCK_HELD=0 SPAWN_SLOT_CLAIMED=0 +SPAWN_TREEHOUSE_ROOT= RELAUNCH_REPLACEMENT_PENDING=0 RELAUNCH_REPLACEMENT_BUSY_GEN= RELAUNCH_REPLACEMENT_HARNESS= @@ -2703,6 +2713,42 @@ spawn_worktree_isolated() { # return 0 } +# Refuse a pool slot that another task still holds, judged from the slot's own +# claim (bin/fm-wake-lib.sh owns the claim and its states) and that claimant's +# task record: the record exists exactly while its task is between spawn and +# teardown, so its presence means the slot is still that task's whatever +# Treehouse's process lease currently says. A claim naming a task whose record +# is gone is a leftover of a teardown that never ran and is replaced by the +# caller's claim; a plain claim file that cannot be read refuses, because it +# may name a live task. Anything else in the claim's place (a directory, a +# symlink) is left to fm_treehouse_slot_owner_claim, whose own refusal names +# the unclaimable slot. +spawn_refuse_live_foreign_claim() { # + local worktree=$1 inspect_target=$2 owner_id owner_home owner_meta marker + fm_treehouse_slot_owner_state "$worktree" "$ID" + case "$FM_TREEHOUSE_SLOT_OWNER" in + mine | absent) return 0 ;; + other) + owner_id=$FM_TREEHOUSE_SLOT_OWNER_ID + owner_home=$FM_TREEHOUSE_SLOT_OWNER_HOME + if [ -z "$owner_home" ]; then + echo "error: Treehouse handed task $ID slot $worktree, which task $owner_id claims without naming its home, so that task cannot be proved finished; refusing to launch into another task's slot; inspect window $inspect_target" >&2 + exit 1 + fi + owner_meta="$owner_home/state/$owner_id.meta" + if [ -e "$owner_meta" ] || [ -L "$owner_meta" ]; then + echo "error: Treehouse handed task $ID slot $worktree, which task $owner_id of home $owner_home still holds (its record $owner_meta exists); refusing to launch into another task's slot; inspect window $inspect_target" >&2 + exit 1 + fi + return 0 + ;; + esac + marker=$(fm_treehouse_slot_owner_marker "$worktree" 2>/dev/null) || return 0 + { [ -f "$marker" ] && [ ! -L "$marker" ]; } || return 0 + echo "error: Treehouse handed task $ID slot $worktree, whose slot-owner claim $marker cannot be read, so it cannot be proved free; refusing to launch into a slot that may be another task's; inspect window $inspect_target" >&2 + exit 1 +} + validate_spawn_worktree() { # local source=$1 inspect_target=$2 if ! spawn_worktree_isolated "$WT"; then @@ -3464,7 +3510,22 @@ if [ "$RELAUNCH" -eq 1 ]; then fi [ "$KIND" = secondmate ] || validate_spawn_worktree "relaunch" "$T" elif [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then - spawn_send_text_line "$WT_TARGET" 'treehouse get' + # Acquire the slot from THIS home's Treehouse root (bin/fm-wake-lib.sh's + # fm_treehouse_home_root owns the contract). Treehouse names a pool after the + # repository, so without an explicit root two homes holding clones of one + # remote share a single pool and can be handed each other's slots. The root + # is passed as --root, which overrides TREEHOUSE_ROOT and any treehouse.toml + # in the pane's own environment, and is recorded below as treehouse_root= so + # every later call on the slot uses the root it was actually taken from. + SPAWN_TREEHOUSE_ROOT=$(fm_treehouse_home_root "$FM_HOME") || { + echo "error: could not resolve this home's Treehouse root for $FM_HOME; refusing to allocate from a pool another home may share" >&2 + exit 1 + } + mkdir -p -- "$SPAWN_TREEHOUSE_ROOT" 2>/dev/null || { + echo "error: could not create this home's Treehouse root $SPAWN_TREEHOUSE_ROOT" >&2 + exit 1 + } + spawn_send_text_line "$WT_TARGET" "treehouse get --root $(shell_quote "$SPAWN_TREEHOUSE_ROOT")" # Wait for the treehouse subshell: the pane's cwd moves from the project to the worktree. # Target the stable window id, not the name: if the name is ever lost (e.g. an @@ -3538,6 +3599,19 @@ elif [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then # Written under the Treehouse project lock held from before slot allocation # through metadata publication, so no other spawn or return sees a half-claim. if fm_treehouse_pool_slot "$PROJ_ABS" "$WT"; then + fm_treehouse_task_root "$SPAWN_TREEHOUSE_ROOT" "$WT" || { + echo "error: treehouse get entered $WT outside this home's Treehouse root ($FM_TREEHOUSE_ROOT_REASON); refusing to launch into a slot another home may own; inspect window $T" >&2 + exit 1 + } + # Runtime-independent ownership assertion: Treehouse's own lease is a live + # process lease and cannot say which TASK a slot belongs to, so a slot + # whose previous holder's worker has exited is handed out again even while + # that task's record - in this home or another - still names it. Only a + # harness-specific guard (Claude's workspace trust) caught that live; this + # check reads the slot's claim and the claimant's record, so it holds for + # every harness and backend. A claim whose task record is gone is stale + # and replaced; a claim naming a task that still has a record refuses. + spawn_refuse_live_foreign_claim "$WT" "$T" if ! fm_treehouse_slot_owner_claim "$WT" "$ID" "$FM_HOME"; then echo "error: could not claim Treehouse pool slot $WT for task $ID; refusing to launch a worker whose slot cannot later be proved to be its own; inspect window $T" >&2 exit 1 @@ -4074,6 +4148,11 @@ preserve_relaunch_meta() { echo "endpoint_task_id=$ID" echo "worktree=$WT" echo "project=$PROJ_ABS" + # The Treehouse root the slot was taken from (fm_treehouse_home_root). Written + # only by the fresh spawn that ran `treehouse get`; a relaunch passes the + # recorded line through untouched, and a record without it predates the field + # and resolves its root from the slot's own path (fm_treehouse_task_root). + [ "$RELAUNCH" -eq 1 ] || [ -z "$SPAWN_TREEHOUSE_ROOT" ] || echo "treehouse_root=$SPAWN_TREEHOUSE_ROOT" echo "harness=$HARNESS" echo "kind=$KIND" [ -z "$MODE" ] || echo "mode=$MODE" diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index dcdac9ef2db..b7465b112df 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -161,6 +161,16 @@ # leased home releases its durable treehouse lease so the pool slot is freed, # never left leased forever. If the treehouse return fails, teardown leaves the # leased home and state in place instead of hiding a still-held lease. +# Treehouse root (per-home pools): every `treehouse return` on a task's slot +# passes `--root `, where is reconciled by bin/fm-wake-lib.sh's +# fm_treehouse_task_root between the record's treehouse_root= line and the root +# the slot actually sits under. A record that predates the field (no +# treehouse_root=) resolves to the slot's own root, so legacy slots drain through +# the pool that allocated them with no migration; a record whose treehouse_root= +# does not contain its worktree REFUSES before any mutation, because returning +# through the wrong root would act on another home's slot. A secondmate home is +# the primary's own durable lease from the firstmate repository's pool and is +# returned without --root, exactly as it was leased. # Usage: fm-teardown.sh [--force] [--legacy-record] # --force skips ordinary-task dirty and landed-work checks, skips scout report # checks, and discards secondmate child work for kind=secondmate. Only use it @@ -1008,6 +1018,17 @@ elif [ "$TREEHOUSE_SLOT_LOCK_REQUIRED" = 1 ]; then echo "REFUSED: task $ID stopped naming a live Treehouse slot while teardown acquired its locks; nothing was changed" >&2 exit 1 fi +# The root every Treehouse call on this task's slot uses (script header, +# "Treehouse root"): the recorded treehouse_root= when it contains the slot, the +# slot's own root for a record that predates the field, and a refusal otherwise. +TEARDOWN_TREEHOUSE_ROOT= +if [ -n "$EXPECTED_TREEHOUSE_PROJECT_LOCK" ]; then + fm_treehouse_task_root "$(fm_meta_get "$META" treehouse_root)" "$WT" || { + echo "REFUSED: task $ID's worktree $WT is not under its recorded Treehouse root ($FM_TREEHOUSE_ROOT_REASON); returning it through that root would act on another home's slot, so nothing was changed - not even with --force." >&2 + exit 1 + } + TEARDOWN_TREEHOUSE_ROOT=$FM_TREEHOUSE_TASK_ROOT +fi MODE=$(grep '^mode=' "$META" | cut -d= -f2- || true) [ -n "$MODE" ] || MODE=no-mistakes @@ -1635,13 +1656,18 @@ cleanup_stale_lock_for_safety_check() { # Return a worktree/home via `treehouse return --force`, tolerating a transient or # stale git index.lock left by a killed crew process. See the script header. -teardown_treehouse_return() { - local dir=$1 cd_dir=$2 label=$3 post_cleanup_check=${4:-} +# A nonempty is passed as `--root `, so the return acts on the pool +# the slot was taken from (script header, "Treehouse root"); an empty +# leaves the pool to Treehouse's own resolution, as a secondmate home lease needs. +teardown_treehouse_return() { #