Repository navigation
fix(spawn): lease a task's pool worktree so it is never reassigned - #10
Merged
Merged
Conversation
…der a live task Supersedes #4, which cannot be revalidated: its branch was rewritten (the pipeline rebased it onto a moved main, and custody was then returned at that rebased head), and no-mistakes refuses to reconcile its private mirror record afterwards. Same content, clean branch from main. On 2026-08-19 a spawn was handed a pooled working copy that another task's own record still named as its live copy. The pool reset it to a detached HEAD, evicting a paused task from the copy holding its work; nothing was lost only because that task had already pushed. Treehouse counts a slot as in use only while a process runs inside it, so a task whose agent has exited - paused, crashed, or between turns - leaves its copy looking free. bin/fm-spawn.sh now takes the pool's own durable claim itself (treehouse get --lease) instead of relying on the pane's subshell to hold the slot, then moves the pane into the leased copy in a subshell so the pane's own shell does not sit inside the worktree. Release paths: ordinary teardown returns the worktree and the lease with it; an aborted spawn returns it holder-matched in cleanup; otherwise the holder label names the owning record and the task records worktree_lease= so a stale claim is self-identifying and releasable with one command. A named refusal stays as a second line of defence for the one copy a claim cannot protect - a task spawned before claims existed, holding its slot with nothing but its processes. The claim is deliberately kept on that refusal, so the slot is quarantined rather than offered to the next spawn in a refuse-loop. Upstream's slot-owner claim marker is kept as-is and this claim sits above it: the lease stops the pool reassigning a live copy, the marker answers at teardown whether the slot beside a record is still that record's own.
…ehouse lease gate
talktejas
force-pushed
the
fm/spawn-pool-claim-r2
branch
from
September 20, 2026 11:28
8f53ea7 to
f1b760f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
This branch replaces PR 4 #4, which cannot be revalidated: its branch history was rewritten, so no-mistakes refuses to reconcile that branch's private mirror record. Same content, clean branch from current main.
The captain approved merging both firstmate pull requests, but neither can merge as it stands:
Both fixes matter daily: without the first, every worker starts from the project's default branch and the captain's integration-branch rule depends on hand-patched instructions; without the second, spawns fail with "pooled worktree not clean" because a live worker's copy is handed to a new task.
This branch is PR 4. Its own ask, which the approval above is an approval of: on 2026-08-19 a spawn was handed a pooled working copy that another task's own record still named as its live working copy. The pool reset that directory to a detached HEAD, evicting a paused task from the copy holding its work; nothing was lost only because that task had already pushed its branch. The cause is what the pool can see: Treehouse counts a slot as in use only while a process is running inside it, so a task whose agent has exited - paused, crashed, or simply between turns - leaves its own working copy looking free, and the next allocation takes it. Stop the pool offering that copy at all, rather than only refusing it after it has already been reset: have the spawn take the pool's own durable claim itself instead of relying on the pane's subshell to hold the slot, then move the pane into the claimed copy. Keep a named refusal as a second line of defence for the one copy a claim cannot protect - a task spawned before claims existed, holding its slot with nothing but its processes - and have that refusal name the task that holds it. Every claim must have a stated release path: ordinary teardown, aborted-spawn cleanup, and a self-identifying holder label recorded with the task so a claim outliving its task is releasable with one command.
Captain's ruling on conflicts, given while this rebase was under way: upstream (kunchenguid/firstmate) code wins wherever upstream has something; where it has nothing, resolve on your own. Upstream has NOT landed either fix - the per-task base branch is still open issue kunchenguid#3411 and the pool-reassignment bug is open issues kunchenguid#1441 and kunchenguid#1573, with several competing unmerged PRs (kunchenguid#4828, kunchenguid#4071, kunchenguid#4909, kunchenguid#4468, kunchenguid#3896). So: when a conflict hunk is upstream's own change to shared code, take upstream's version and re-apply our behaviour on top of it; only our two features are ours to keep. Do not adopt an unmerged upstream PR.
What Changed
bin/fm-spawn.shnow takes the pool worktree itself viatreehouse get --lease --lease-holder fm:<task-id>@<state-dir>(bounded by the newFM_TREEHOUSE_GET_TIMEOUT, default 60s) instead of relying on the pane'streehouse getsubshell to hold the slot, then sends(cd -- '<wt>' && exec $SHELL)into the pane and polls until it is observed in that leased copy; the holder label is recorded asworktree_lease=in the task record and carried across relaunch. Release paths are aborted-spawn cleanup, failed-dispatch rollback (which warns with the exacttreehouse return --force --if-lease-holdercommand), and ordinarybin/fm-teardown.shreturn.bin/fm-bootstrap.shnow gates onget --lease,get --lease-holderandreturn --if-lease-holder, andbin/fm-install-treehouse.shpins Treehouse v2.1.0 with updated checksums.fm_tasks_axi/fm_tasks_axi_timeout_expiredintofm_run_bounded/fm_run_bounded_timed_outinbin/fm-backlog-transition-lib.sh(tasks-axi keeps a thin wrapper), refreshed backend and docs comments that described the old pane-driven acquisition, documentedworktree_lease=andFM_TREEHOUSE_GET_TIMEOUTindocs/configuration.md, and addedtests/fm-spawn-worktree-lease.test.shplus a shared treehouse lease stub intests/lib.shused across the updated suites.🤖 Generated with Claude Code
Risk Assessment
Testing
I drove bin/fm-spawn.sh against a real treehouse v2.1.0 pool rather than only stubs. On the base commit a paused task's idle copy was handed to the next spawn and reset to a detached HEAD, destroying its WIP - the reported incident; on this branch the same scenario gives the new spawn a different copy and leaves the paused worker's branch and uncommitted work intact. Every release path the intent requires was exercised live: ordinary teardown-shaped
treehouse return --forcefrees the durable claim, an aborted spawn returns it holder-matched with no record left behind, the refusal's printed project-scopedtreehouse return --force --if-lease-holdercommand actually releases the quarantined slot when pasted from an unrelated working directory while a wrong holder is refused, and a relaunch carries the original claim forward exactly once. The refusal itself names the task whose record holds the copy and keeps the slot claimed so the pool does not loop. For the CI pin I ran bin/fm-install-treehouse.sh for real: it downloaded and SHA-256-verified the v2.1.0 artifact, whose help output advertises get --lease, get --lease-holder and return --if-lease-holder; bootstrap is silent with that binary on PATH and still prints "MISSING: treehouse" for a v2.0.1-shaped build that lacks the holder-matched return, so the capability gate was not weakened. Suite-level: fm-spawn-worktree-lease (including its real-pool cases), fm-spawn-dispatch-profile (the target commit's GNU -k timeout stub), the three formerly red stale-stub suites (fm-startup-memory-budget, fm-session-start, fm-secondmate-sync), fm-agy-harness, fm-bootstrap, fm-x-mode, fm-backend, fm-spawn-pool-base-freshen and fm-spawn-worktree-settle all pass. fm-secondmate-liveness fails one Herdr classifier unit case on this host because a real herdr is installed and reports the session stopped; that function is byte-identical at the base commit, so it is environmental, not a regression. I stopped the remaining stub-swap-only suites (captain-hold-lifecycle and later) after captain-hold-lifecycle ran far past its usual duration on this loaded host; they carry no behaviour from this change beyond the stub swap already validated elsewhere, and CI owns broad regression. Evidence is CLI transcripts and real pool JSON state; the change has no UI surface, so no screenshots apply.treehouse return --force <slot>flips status leased…bash tests/fm-spawn-worktree-lease.test.sh- 'a stalled allocation refuses within its bound and gives the project lock back' (evidence/lease-suite-run.txt)timeout -k <bound> <bound>invocation (target commit)bash tests/fm-spawn-dispatch-profile.test.sh- 59 checks pass, no 'could not lease a worktree' aborts (evidence/lease-suite-run.txt)bash tests/fm-startup-memory-budget.test.sh,tests/fm-session-start.test.sh,tests/fm-secondmate-sync.test.sh,tests/fm-agy-harness.test.sh- all exit 0, no 'MISSING: treehouse'Evidence: Post-fix: real pool never reassigns a paused task's copy
Source: Post-fix: real pool never reassigns a paused task's copy
Evidence: Pre-fix reproduction of the 2026-08-19 reassignment (base commit)
Source: Pre-fix reproduction of the 2026-08-19 reassignment (base commit)
Evidence: Spawn CLI transcript: claim taken, refusal by name, aborted-spawn release
Source: Spawn CLI transcript: claim taken, refusal by name, aborted-spawn release
Evidence: Printed recovery command run live from an unrelated cwd
Source: Printed recovery command run live from an unrelated cwd
Evidence: Aborted spawn returns its claim (real pool state)
Source: Aborted spawn returns its claim (real pool state)
Evidence: Refusal names the holder and the quarantine is releasable
Source: Refusal names the holder and the quarantine is releasable
Evidence: Relaunch carries the claim forward
Source: Relaunch carries the claim forward
Evidence: CI pin: real v2.1.0 install, checksum, advertised flags
Source: CI pin: real v2.1.0 install, checksum, advertised flags
Evidence: Bootstrap lease gate: silent on v2.1.0, MISSING on a v2.0.1-shaped build
Source: Bootstrap lease gate: silent on v2.1.0, MISSING on a v2.0.1-shaped build
Evidence: Lease and dispatch-profile suite runs
Source: Lease and dispatch-profile suite runs
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
tests/fm-agy-harness.test.sh:547- fm-spawn.sh no longer learns the worktree from the pane's cwd; it now reads it fromtreehouse get --leasestdout (bin/fm-spawn.sh:3748) and refuses with "treehouse get --lease reported no worktree" when that is empty (bin/fm-spawn.sh:3760). make_agy_fakebin still installs a bare exit-0 treehouse (fm_fake_exit0 "$fakebin" treehouse gh-axi gh), which prints nothing, so every real ship spawn this suite runs through "$SPAWN" (tests/fm-agy-harness.test.sh:612) now aborts before the pane is touched, regardless of FM_FAKE_PANE_PATH. Other spawn suites were converted in this branch (fixtures.sh's fm_test_make_spawn_fakebin now installs fm_test_fake_treehouse_lease, whose get --lease falls back to FM_FAKE_PANE_PATH); this one was missed. Remedy: replace the exit-0 treehouse here with fm_test_fake_treehouse_lease, the same stub the rest of the suite set now uses.tests/fm-startup-memory-budget.test.sh:38- Commit 386a678 widened bin/fm-bootstrap.sh's treehouse_supports_lease from one advertised flag to three (get --lease, get --lease-holder, return --if-lease-holder, bin/fm-bootstrap.sh:950), but three suites still install a stub whoseget --helpprints only 'Usage: treehouse get [--lease]' and whosereturn --helpprints nothing. I ran the new probe against that exact stub: --lease passes, --lease-holder and --if-lease-holder both fail, so bootstrap emits "MISSING: treehouse" for a tmux/herdr backend (bin/fm-bootstrap.sh:1499). That breaks tests/fm-startup-memory-budget.test.sh:104 ("default materialization should stay quiet"), tests/fm-secondmate-sync.test.sh:607 ("idempotent retry should converge to silence", stub at :362), and tests/fm-session-start.test.sh:1084 (assert_not_contains "MISSING: treehouse", stub at :90) - the same three failures the user already paid a CI-fix budget to root-cause. tests/fm-x-mode.test.sh:809 and tests/fm-secondmate-liveness.test.sh:225 carry the same stale stub but only use assert_contains/assert_not_contains on other lines, so they survive; update them anyway for consistency. Remedy per the recorded decision: present fm_test_fake_treehouse_lease in these suites rather than weakening the probe or the silence assertions.bin/fm-spawn.sh:3788- The named second-line refusal skips any sibling record that carries worktree_lease= ([ -z "$(fm_meta_get "$spawn_other_meta" worktree_lease)" ] || continue), on the reasoning that the pool would not have offered a copy whose record holds a claim. If a claim is force-released while its record is still live - the recovery commands this change prints all usetreehouse return --force --if-lease-holder, and teardown that returns the slot but dies before removing the record leaves the same shape - the next spawn is handed that copy and launches into it, which is the 2026-08-19 incident. Noting the tradeoff only: the user intent explicitly scopes the refusal to "the one copy a claim cannot protect - a task spawned before claims existed", so always refusing on any record that names the offered copy (which would also be the simpler single rule) would widen the change beyond its stated scope. No action needed unless that scoping is revisited.🔧 Fix applied.
1 error still open:
bin/fm-install-treehouse.sh:15- The branch makestreehouse return --if-lease-holdera hard requirement, but the repo's own pinned CI build (FM_TREEHOUSE_CI_VERSION=2.0.1) does not have that flag. Verified against upstream source:cmd/return_cmd.goat tag v2.0.1 registers only--force;cmd/get.goat v2.0.1 does register--leaseand--lease-holder, so only the release half is missing. Three concrete consequences on that build: (1) bin/fm-bootstrap.sh:950's widenedtreehouse_supports_leasegate now requiresreturn --if-lease-holderto be advertised, so bootstrap printsMISSING: treehousefor the very binary bin/fm-install-treehouse.sh installs on the real-Herdr lane (.github/workflows/ci.yml:279); (2) bin/fm-spawn.sh:1266's aborted-spawn release runstreehouse return --force --if-lease-holder ..., which fails with an unknown-flag error, so the lease leaks and the printed recovery command is itself unusable — defeating the intent's requirement that every claim have a stated release path; (3) tests/fm-spawn-worktree-lease.test.sh:216 fails wherever that binary is on PATH, because the holder-matched return errors and tripsfail "real treehouse could not release the claim its own holder label took". Remedy (explicitly authorized in the recorded CI-round decision, which asked for this to be established and resolved): raise the pin to a release that advertises the flag — v2.1.0 is the version docs/verification/runtime-backends.md records and does carry--if-lease-holder— refresh the four platform SHA-256s, and update the 2.0.1-specific comments at lines 12, 17 and 72 plus the verification record so the pin and the recorded flag surface agree.🔧 Fix applied.
1 info still open:
bin/fm-spawn.sh:1229- The five printed recovery commands wrap paths in single quotes without escaping embedded single quotes: "(cd '$PROJ_ABS' && treehouse return --force --if-lease-holder '$SPAWN_TREEHOUSE_LEASE_HOLDER' '$SPAWN_TREEHOUSE_LEASE_PATH')" at lines 1229, 1269, 3754, 3792 and 4786. The holder label is 'fm:<id>@<state-dir>', so a home or project path containing an apostrophe (e.g. ~'brien/fm) produces a command that is a shell syntax error, or worse closes the quote early and passes a truncated path. The intent requires a claim outliving its task to be releasable with one command, and these are that command. The escaping idiom is already in this file two lines away from one of the sites (spawn_cd_path=${WT//&fix(bin): keep command-scoped settings out of servers, launched agents, and sessions #39;/&fix(bin): keep command-scoped settings out of servers, launched agents, and sessions #39;\'&fix(bin): keep command-scoped settings out of servers, launched agents, and sessions #39;} at line 3813); apply the same substitution to PROJ_ABS, the holder label and the lease path before interpolating them into the messages.tests/fm-spawn-dispatch-profile.test.sh:37- tests/fm-spawn-dispatch-profile.test.sh installed a pass-throughtimeoutstub that dropped only one argument (shift; exec "$@"). The branch's bounded pool allocation invokestimeout -k <bound> <bound> treehouse get --lease, so the stub tried to exec the bound itself (exec: 60: not found) and every spawn in the suite aborted with "treehouse get --lease could not lease a worktree". This is the CI lane 'Behavior portable serial 8'. Fixed by teaching the stub the GNU-kshape; the suite now passes with 59 checks.tests/fm-public-followup.test.sh:60- tests/fm-public-followup.test.sh exits 1 from its shared fixture cleanup, not from an assertion: all 74 checks pass with zeronot ok, and the last line isrm: cannot remove '/tmp/fm-public-followup.<id>': Directory not empty- a race between the still-terminating remote-job worker recreating files under remote-jobs/ and fm_test_cleanup's rm -rf, whose status becomes the EXIT trap's exit code. The branch's only change to this suite is swapping the treehouse stub, so this is a pre-existing cleanup flake rather than a regression; it will nonetheless make the lane red intermittently. Left as-is for the author to decide.bash tests/fm-spawn-worktree-lease.test.sh- test_stalled_allocation_refuses_and_frees_the_project_lock, driving the real fm-spawn.sh CLI with FM_TREEHOUSE_GET_TIMEOUT=1bash tests/fm-spawn-worktree-lease.test.sh- test_a_record_holding_its_own_claim_does_not_block_a_spawnbash tests/fm-spawn-worktree-lease.test.sh(7 checks, including the real-treehouse pool-guarantee and holder-matched release cases)Custom live driver: realbin/fm-spawn.sh+ real treehouse v2.1.0, spawn task alpha-1 -> agent exits with committed work -> spawn beta-2 -> assert different copy and alpha's branch/work intactSame driver against the base commit viagit archive e80a29a4 | tar -x, reproducing the reassignment and detached-HEAD resetLive aborted-spawn driver: pane never enters the leased copy -> assert no record published andtreehouse status --jsonshows the slot availableLive refusal driver: legacy record naming a claim-less pool copy -> refusal names the task, re-run succeeds on another copy, printed recovery command pasted verbatim from/releases the quarantined claimLive relaunch driver:bin/fm-spawn.sh <id> --relaunchwith the lifecycle-modelling tmux stub -> worktree_lease preserved exactly oncetreehouse get --lease --lease-holder/treehouse return --forceagainst a real pool to confirm ordinary teardown frees the claimCapability-gate probe run against real treehouse v2.1.0 and a downloaded v2.0.1 buildbash bin/fm-install-treehouse.sh <tmpdir>plus sha256sum of all four v2.1.0 release archives against the pinned checksumsbash tests/fm-startup-memory-budget.test.sh,tests/fm-session-start.test.sh,tests/fm-secondmate-sync.test.sh,tests/fm-agy-harness.test.sh(the previously CI-failing suites; all green, noMISSING: treehouse)bash tests/fm-spawn-dispatch-profile.test.sh(failed, fixture fixed, re-run green at 59 checks)bash tests/fm-spawn-worktree-settle.test.sh,tests/fm-spawn-base-branch.test.sh,tests/fm-spawn-pool-base-freshen.test.sh,tests/fm-tangle-guard.test.sh,tests/fm-trace-context-spawn.test.sh,tests/fm-captain-hold-lifecycle.test.sh(all green)🔧 Fix applied.
1 info still open:
tests/fm-secondmate-liveness.test.sh:173- tests/fm-secondmate-liveness.test.sh fails one case on this host: 'Herdr pane state unknown should map to unreadable, got missing'. fm_backend_herdr_agent_state falls back to fm_backend_herdr_server_running_state, which reports 'stopped' because a real herdr is installed here with no running session. Both functions are byte-identical at the base commit e80a29a, so this is a pre-existing environment-dependent failure, not a regression from the treehouse stub swap this branch makes to that suite.treehouse return --force <slot>flips status leased…bash tests/fm-spawn-worktree-lease.test.sh- 'a stalled allocation refuses within its bound and gives the project lock back' (evidence/lease-suite-run.txt)timeout -k <bound> <bound>invocation (target commit)bash tests/fm-spawn-dispatch-profile.test.sh- 59 checks pass, no 'could not lease a worktree' aborts (evidence/lease-suite-run.txt)bash tests/fm-startup-memory-budget.test.sh,tests/fm-session-start.test.sh,tests/fm-secondmate-sync.test.sh,tests/fm-agy-harness.test.sh- all exit 0, no 'MISSING: treehouse'bash tests/fm-spawn-worktree-lease.test.sh(7 checks, includes real treehouse v2.1.0 pool cases)bash tests/fm-spawn-dispatch-profile.test.sh(59 checks - the target commit's GNU-ktimeout stub)bash tests/fm-startup-memory-budget.test.sh,tests/fm-session-start.test.sh,tests/fm-secondmate-sync.test.sh,tests/fm-agy-harness.test.sh(the stale-stub suites)bash tests/fm-bootstrap.test.sh,tests/fm-x-mode.test.sh,tests/fm-backend.test.sh,tests/fm-spawn-pool-base-freshen.test.sh,tests/fm-spawn-worktree-settle.test.shManual live drive ofbin/fm-spawn.shagainst a real treehouse pool, post-fix (no reassignment of a paused task's copy) and pre-fix at base commit e80a29a (reassignment reproduced)Manual live drive of the aborted-spawn release, the named second-line refusal, and the printed recovery command against a real pool from an unrelated cwdbash bin/fm-install-treehouse.sh /tmp/fm-th-pin- real download, SHA-256 verification, thentreehouse get --help/treehouse return --helpflag advertisementLivebin/fm-bootstrap.shrun with the pinned real binary on PATH, and adversarially with a v2.0.1-shaped build lackingreturn --if-lease-holderLivetreehouse get --lease --lease-holder/treehouse return --force/treehouse status --jsonchecks of the teardown release path✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.