fix(daemon): event-driven worktree reconcile check so cleanup self-heals

[tbrownio]

Summary — What / Why / How

What. Adds an event-driven worktree reconcile check (WorktreeReconciler) to the Linear-agent daemon, plus preserve-then-remove semantics for every worktree close-out, so cleanup no longer depends on a single manually configured webhook subscription.

Why. The cleanup pipeline has never fired once: the Issue completed trigger (daemon/src/eventlog.ts append path) requires Issues data-change webhooks, which were never enabled on the live bloom-implementer app — the events table holds 193 AgentSessionEvent rows and zero Issue rows, cleanup_jobs is empty since provisioning, and 32 worktrees (18 GB) have accumulated on a 48 GB root volume now at 93%. A full disk halts every agent pipeline on the host (#128, artifact bundle linked from the issue).

How. A new worker sweeps at daemon startup, on turn completion, and on Issue-completed events — no standing timer. It enumerates on-disk worktrees plus retained cleanup jobs, confirms each issue's workflow state via a new LinearGateway query, and:

• resolved (completed/canceled) + session rows → enqueues through the existing CleanupWorker finalize-then-remove path;
• resolved + no session rows → closes out directly, with an operator notification (structured log + ntfy);
• no confirmed issue → persisted first-observed grace clock (default 14 days, WORKTREE_UNLINKED_GRACE_DAYS), removal only after expiry;
• lookup error → skipped that round, always — deletion requires positive confirmation;
• every close-out of dirty or PR-less state preserves first: commit + push to the existing agents/<id> branch, git-bundle fallback (WORKTREE_BUNDLES_DIR) — force removal is always reversible. Terminal retention now happens only when preservation itself fails.

SessionWorker, CleanupWorker, and the reconciler now share one WorktreeManager instance, so creation and removal serialize on a single mutation lock.

Done means: a resolved issue's worktree and agents/<id> branch are removed on the next trigger even with the Issues webhook unsubscribed; nothing is deleted without positive confirmation or grace expiry; nothing dirty or unpushed is deleted without a preserved copy; the webhook fast path is unchanged; all proven in pnpm test.

Visual overview

Before → after lifecycle of the cleanup path (the .excalidraw source lives in the run's artifact bundle):

User journeys

Operator/lifecycle change — no end-user UI. The journeys are daemon flows:

J#	Journey	Risk	Manual test
J1	Issue-completed webhook (subscription restored) → immediate cleanup enqueue	Must	M1
J2	Resolved issue, session-backed worktree, no webhook ever → reconciler → CleanupWorker → worktree + branch gone	Must	M2
J3	Resolved issue, dirty / PR-less worktree → preserve (push or bundle) → force remove → destination named in notification	Must	M3
J4	Unlinked directory → grace clock persisted → removed (with preservation) only after 14 days	Must	M4
J5	Linear degraded (error/timeout) → worktree skipped, no clock written	Must	M5
J6	Foreign/non-repo directory under the worktrees root → never touched	Must	M5
J7	Deploy restart against the live 32-worktree backlog → resolved subset reclaimed, unlinked clocks started	Important	M6

Coverage gaps: none — every branch above has an automated test; J7's live half is deploy-time.

Verification

Independently re-proven by a fresh verifier pass over the final tree (pnpm typecheck, pnpm build, pnpm test: 27 files passed, 392 tests passed / 1 skipped — the skip is the pre-existing proxy-integration test, unrelated):

• AC1 (reconcile closes worktree+branch without any Issue webhook) — ✅ worktree-reconciler.test.ts › "closes a resolved session-backed worktree end-to-end without an Issue webhook": job reaches done, worktree path absent, refs/heads/agents/ENG-1 absent.
• AC2 (webhook fast path unchanged) — ✅ server.test.ts (18 tests) incl. signed Issue-completed POST → persisted event + callback; non-completed events don't fire it.
• AC3 (preserve before force removal, destination named) — ✅ "commits and pushes dirty state before force removal", "falls back to a newly created bundle directory when push fails", "preserves and removes a dirty worktree and durably posts its destination".
• AC4 (persisted, env-configurable grace for unlinked) — ✅ "starts and expires only the persisted unlinked grace clock" (injected clock at expiry-1/expiry); config.test.ts (20 tests) covers WORKTREE_UNLINKED_GRACE_DAYS.
• AC5 (lookup failure skips; foreign dirs never touched) — ✅ "skips lookup errors without writing the grace clock", "never touches a foreign directory".
• AC6 (unresolved issues never enqueue) — ✅ "leaves unresolved worktrees and clears stale unlinked observations" (started state).
• AC7 (startup / turn-completion / Issue-completed triggers, no timer; backlog first-run semantics) — ✅ "start performs one immediate coalesced sweep with no standing timer", "stop awaits an in-flight lookup", server callback tests, sessions terminal-activity callback test.
• AC8 (decision matrix in pnpm test) — ✅ full suite green including the 11-case reconciler matrix.
• Background-job rubric — ✅ sequential double-sweep idempotency ("is sequentially idempotent after a completed-state preservation and removal"), retry/terminal failure path ("retains only when preservation exhausts its retry window"), preservation-guarded removal, structured logs with ids.
• Run gates: plan review 3 passes (7 Must Fix found and resolved), verifier pass 1 fail → fix → pass 2 clean; build gate pnpm typecheck && pnpm build && pnpm test + bash -n over the 7 ops scripts, all exit 0.

Manual tests

Must (breaks data if wrong):

• [J2] M2 — After deploy, restart the daemon with Issues webhooks still unsubscribed and a Done issue's worktree on disk → worktree and agents/<id> branch removed on startup sweep — left to human: live-host deploy step
• [J3] M3 — Pick a dirty backlog worktree with a Done issue → after its close-out, confirm the preserved state is reachable (git log origin/agents/<id> or the bundle file) and the notification names it — left to human: live-host deploy step
• [J4] M4 — Confirm an unlinked directory is still present after deploy and its observation row exists (grace clock started, not expired) — left to human: live-host deploy step
• [J5] M5 — With Linear unreachable (or during a 503 window), trigger a sweep → no removals, worktree_reconcile log shows skips — left to human: live-host deploy step
• [J1] M1 — After enabling the Issues webhook category, mark a throwaway issue Done → cleanup enqueues from the webhook without waiting for a sweep trigger — left to human: live-host deploy step

Important:

• [J7] M6 — After the first live sweep: df -h / improved; remaining worktrees are only unresolved issues + within-grace unlinked dirs — left to human: live-host deploy step
• M7 — sudo daemonctl status healthy after deploy; no error spam from the reconciler in service logs — left to human: live-host deploy step

Areas not affected: webhook signature verification, session scheduling/turn execution, artifact store, OTLP relay, ack worker, Linear MCP monitor.

QA results

Command-shaped QA executed post-review over the final tree (evidence in the QA proof comment): 0 of 7 Manual-test items are machine-runnable (all are live-host deploy steps — left to the human); the full automated proof re-ran green — pnpm typecheck, pnpm build, pnpm test (394 passed / 1 pre-existing skip), all 8 ACs re-quoted including both review-fix regressions (fresh grace clock on recreated worktrees; live session worktree survives an interleaved cleanup). Review loop: 3 passes, 3 Must Fix found → fixed → verified, final verdict Approve with zero findings. No bugs surfaced by QA.

Deploy notes

Ordered; nothing here blocks verification (all evidence above ran against local fixtures).

1. Deploy + restart (red — human). On the linear-agent host (gcloud compute ssh linear-agent --project=bloom-agents --zone=us-central1-a): fetch/review/fast-forward /opt/orchestra-source to the merged commit, then sudo daemonctl reload (runbook procedure). The restart fires the first reconcile sweep, which reclaims the resolved subset of the 32-worktree backlog and starts grace clocks for unlinked directories. Expect a burst of worktree_reconciled log lines and preservation pushes to agents/<id> branches.
2. Enable the Issues webhook category (red — human, ~2 min). Linear settings → bloom-implementer app → webhooks → check Issues under Data change events (docs/linear-agent-daemon-setup.md §Data change events; this restores the fast path — the reconciler is the safety net either way).
3. Schema (self-applying — no action). New daemon-private SQLite table worktree_unlinked_observations in events.db, created by CREATE TABLE IF NOT EXISTS at daemon startup. Additive; no external migration, no consumer-facing schema.
4. New optional env vars (no action needed; defaults apply). WORKTREE_UNLINKED_GRACE_DAYS (default 14) and WORKTREE_BUNDLES_DIR (default <db-dir>/worktree-bundles) — documented in daemon/ops/runbook.md.
5. Disk headroom note. Preservation pushes create/update remote agents/<id> branches on the target repo's origin; prune them at leisure once inspected (they exist so every deletion is reversible).

Residual risks

• Linear's issue(id: <identifier>) error shape for "entity not found" vs transient failures is classified defensively (anything non-definitive skips); if Linear's wording shifts, the failure mode is "worktree retained longer", never "worktree deleted".
• The first live sweep pushes potentially many agents/<id> branches in one burst; a credential problem on the host's checkout degrades to bundle files beside the DB (disk-cheap, logged), not to data loss.
• Branch name (feat/frontend-verifier-ios-simulator) predates this item — the branch had no open PR and its previous PR is merged; kept per pipeline rules (no new branches). Content is solely this fix.

Closes #128

🤖 Generated with Claude Code

[tbrownio]

QA evidence — command-shaped pass (post-review, final tree @ 3dd4b02)

Independent verifier run from daemon/; all fixtures local (temp git repos, bare origins, file-backed SQLite, loopback HTTP). No live system touched.

Gates

• pnpm typecheck && pnpm build && pnpm test → exit 0 — Test Files 27 passed | 1 skipped (28), Tests 394 passed | 1 skipped (395) (the skip is the pre-existing proxy-integration test, unrelated)
• pnpm vitest run test/worktree-reconciler.test.ts test/cleanup.test.ts --reporter=verbose → Tests 25 passed (25)
• bash -n over the 7 ops scripts → exit 0

Per-AC evidence (quoted test names)

• AC1 — ✓ closes a resolved session-backed worktree end-to-end without an Issue webhook
• AC2 — ✓ persists a signed Issue webhook without acking or creating a turn (+ server.test.ts 18 tests)
• AC3 — ✓ preserves and removes a dirty worktree and durably posts its destination · preserves and removes a clean present worktree when no pull request URL was recorded
• AC4 — ✓ starts and expires only the persisted unlinked grace clock · review regression: gives a recreated unlinked worktree a fresh grace clock
• AC5 — ✓ skips lookup errors without writing the grace clock · never touches a foreign directory · review regression: AC5 retries instead of removing a session worktree reused after cleanup is claimed
• AC6 — ✓ leaves unresolved worktrees and clears stale unlinked observations
• AC7 — ✓ start performs one immediate coalesced sweep with no standing timer + trigger-seam tests (onIssueCompleted on signed completed-Issue POST, onTurnComplete on terminal activity, onIssueFinalized on cleanup completion)
• AC8 — ✓ the decision matrix runs inside pnpm test

Split

• Passed automated: everything above — the complete AC set and both review-fix regressions.
• Remaining for the human: all 7 Manual-test checkboxes (M1–M7) — they are live-host deploy steps (deploy + restart, webhook category enablement, backlog reclaim observation) and intentionally not machine-run; start from the unchecked boxes in the body's Manual tests section and the Deploy notes.

[tbrownio]

---

type: wrap-up-report
item: daemon-worktree-cleanup
pr: #129

Wrap-Up Report — daemon-worktree-cleanup

What was built

The daemon's worktree cleanup had never fired once — the Issues data-change
webhook was never subscribed on the live bloom-implementer app, so 32
worktrees (18 GB) piled up and the host hit 93% disk. This run shipped the
brief's D1/D2 design: a new WorktreeReconciler sweeping at daemon startup,
on turn completion, and on Issue-completed events (no standing timer) that
positively confirms each worktree's issue state via a new LinearGateway
query and closes out resolved ones through the existing CleanupWorker
path; preserve-then-remove (commit + push to agents/<id>, git-bundle
fallback) makes every force removal reversible; unlinked directories get a
persisted 14-day grace clock (WORKTREE_UNLINKED_GRACE_DAYS); lookup
errors always skip; SessionWorker/CleanupWorker/reconciler share one
WorktreeManager mutation lock. Terminal retention now exists only for
preservation failures, and legacy retained jobs are re-pended once their
issue re-confirms resolved.

Verification evidence

All 8 ACs proven automated, twice: Step 3 verify (fail → fix → pass) and a
post-review QA re-run over the final tree — pnpm typecheck, pnpm build,
pnpm test: 394 passed / 1 pre-existing skip; per-AC quoted test evidence in
the PR's Verification section and QA proof comment. Review-fix regressions
covered: a recreated unlinked worktree gets a fresh grace clock (AC4), and a
live session worktree reused after cleanup is claimed survives — the job
retries instead (AC5). The live halves (backlog reclaim at deploy, webhook
enablement) are deploy-time human steps, not AC owners. No app-only ACs
existed; frontend_verifier: false on the item left nothing unproven.

Review outcome

Must Fix: 0 · passes used: 3/3 (plan loop also 3/3 — 7 plan Must Fix found
and resolved pre-implementation). Post-PR loop found 3 Must Fix total
(pass 1: stale-observation grace bypass on recreated worktrees + a
creation-vs-removal race; pass 2: the race persisting on the session-backed
cleanup path), each fixed in a resumed implementer round and verified fixed;
pass 3 verdict: Approve, zero findings. QA pass: 0 of 7 manual items
machine-runnable (all live-host deploy steps — left to the human with the
Deploy notes); no Should Fix / Nice to Have survivors, so no inline comments.

Human action required

• ⛔ Blocks verification / QA (prerequisite): none — all evidence ran
  against local fixtures.
• ⛔ You must do (deploy / external):
  1. Deploy + restart on the linear-agent host:
     gcloud compute ssh linear-agent --project=bloom-agents --zone=us-central1-a,
     fast-forward /opt/orchestra-source to the merged commit, then
     sudo daemonctl reload. The startup sweep reclaims the resolved subset
     of the 32-worktree backlog (expect worktree_reconciled log lines and
     pushes to agents/<id> branches).
  2. Enable Issues under Data change events on the bloom-implementer
     Linear app (~2 min; restores the webhook fast path).
  3. Then run the PR's Manual tests M1–M7 (checkbox list in the body).
• ✅ Done for you (applied in-run): nothing needed applying — the only
  schema change (worktree_unlinked_observations + nullable identity
  column) is daemon-private SQLite that self-applies at startup.

Residual risks / follow-ups

• Linear's "entity not found" error wording is classified defensively;
  drift degrades to longer retention, never deletion.
• First live sweep pushes many agents/<id> branches in one burst; a
  credential problem degrades to local bundles, logged.
• Follow-up candidate (named in the item): disk-usage alerting on the host —
  93% was found by chance.
• Branch name feat/frontend-verifier-ios-simulator predates the item
  (no-new-branches rule; previous PR merged).

Dial record

zone: 1
lanes: single-codex
passes: {plan: 3/3, post_pr: 3/3}
findings: {plan: {pass1: {codex: 5, claude: 0}, later: {codex: 4, claude: 0}},
           post_pr: {pass1: {codex: 2, claude: 0}, later: {codex: 1, claude: 0}}}
verifiers: {frontend: disabled_by_item, qa_pass: ran}  # command-shaped; no app-only ACs existed
qa_findings: 0
wall_clock: 1:02   # transcript JSONL 18:13:28Z → 19:15:16Z, single session
deviations: none
pr_size: {files_changed: 19, additions: 1401, deletions: 42}
tokens:
  codex: {total: 1064032, by_role: {code_researcher: 104767, plan_reviewer: 156756,
          implementer: 406024, backend_verifier: 192589, code_reviewer: 203896}}
  claude_subagents: 0    # no Claude-lane sub-agents dispatched (single lane, no frontend verifier)
  overseer: 20062993     # transcript message.id dedup, includes cache reads; output tokens 114855
  total: 21127025
spend_ratio: 14641.0
agents:
  - {role: code-researcher, model: gpt-5.6-sol, effort: low, dispatches: 1, wall_clock: unknown, tokens: 104767}
  - {role: plan-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: unknown, tokens: 156756}
  - {role: implementer, model: gpt-5.6-sol, effort: medium, dispatches: 4, wall_clock: unknown, tokens: 406024}
  - {role: backend-verifier, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: unknown, tokens: 192589}
  - {role: code-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: unknown, tokens: 203896}

Deltas vs plan

• daemon/test/linear.test.ts gained gateway-contract tests (plan noted it
  late; implementer recorded the delta).
• Review rounds added durable worktree identity + generation snapshots
  (worktrees.ts, eventlog.ts identity column) and CleanupWorker
  snapshot revalidation — extensions of the plan's shared-lock decision, not
  reversals. Everything else landed per the plan's Files-changed table.

[tbrownio]

---

type: postmortem
item: daemon-worktree-cleanup
pr: #129
anchor: #129

Postmortem — daemon-worktree-cleanup (ops-only)

Run operations (always)

Wall-clock 1:03 (18:13:28Z → 19:16:49Z, single session, no compaction).
Agent-active ≈ 63 min; human-idle 0 min (0%) — the transcript contains no
genuine human message after the invocation and no "continue" nudges;
post-completion idle: none. The run chained Steps 0→6 unattended, using two
scheduled self-wakeups as fallbacks (neither was needed — background waiters
notified first both times).

Per-step timing (scripted from transcript + dispatch launch epochs; times are
minutes from 18:13Z; costs fold into the token table):

Step / dispatch	Start–End (min)	Dur	Tokens	Note
Step 0 preflight + load	0–3	3m	overseer	bundle pull, hygiene scan (89 PRs, 0 Fixes lines)
code-researcher	3–13.5	10.5m	104,767	overseer read sources in parallel
Plan draft	13.5–17	3.5m	overseer
plan-review p1→p3 (+fixes)	17–25.5	8.5m	156,756	3 passes: 4+2+1 MF, all folded
implementer r1	25–34	9m	228,862	full slice, suite green
backend-verifier v1	34.5–36.3	2m	79,717	fail: AC1 e2e gap + idempotency blocker
implementer r2	36.3–38.5	2m	Δ38,864	two new tests
backend-verifier v2	38.6–42.5	4m	49,202	pass
Step 4 gate/commit/PR	42.5–49.5	7m	overseer	PR #129 open at ~min 50
code-review p1	43.2–47	4m	80,479	overlapped Step 4; 2 MF (races)
PR diagram (author+fix+host)	44–56.5	~9m	overseer	fully overlapped with review/fix lanes
implementer r3	48–50.5	2.5m	Δ63,556	MF-1 fixed; MF-2 partially
code-review p2	52–57	5m	74,865	MF-1 verified; MF-2 persists (session path)
implementer r4	54–57.5	3.5m	Δ74,742	snapshot guard in CleanupWorker
code-review p3	57.7–59.2	1.5m	48,552	Approve, 0 findings
QA backend-verifier	59.6–62	2.5m	63,670	final-tree re-proof, all ACs
Step 6 wrap-up	62–63.5	1.5m	overseer	label, notify, bundle

Aggregates: plan phase ≈ 40% of wall-clock (research-heavy by design at
zone 1 full lane), implement+verify ≈ 27%, post-PR review+QA ≈ 30%, wrap-up
≈ 3%. Summed overseer turnaround between dispatches ≈ 4 min (report pickup +
next dispatch authoring). Overlap discipline was good: the PR diagram and
Step 4 ran inside review/fix lanes' shadow.

Stalls: none. Blocker inventory: no AskUserQuestion gates, no rate
limits; all waits were legitimate detached-dispatch waits (the two long
foreground polls — 10 min researcher, 15 min implementer window — were
tracked waiters, not idle). Two minor operational blemishes: (1) one Codex
dispatch launch failed silently-ish because the overseer's cwd had drifted
to the excalidraw skill directory and the relative .codex-dispatches/...
paths didn't resolve — caught immediately, ~1 min lost; (2) the Step 3
verifier round-trip (fail→fix→re-verify) cost ~8 min for two test-coverage
gaps a sharper implementer test contract could have prevented.

Single change with the biggest payoff: pin dispatch artifact paths to the
repo root so cwd drift can't break a launch (proposal below).

What we asked for

Fix the never-fired worktree cleanup (Issues webhook never subscribed; 32
worktrees / 18 GB / disk 93%): enable-webhook as a human task plus an
event-driven reconcile check with preserve-then-remove and a 14-day grace
for unlinked worktrees, per brief D1/D2 and AC1–AC8.

Outcome vs intended

On-target — no outcome gap known at wrap-up (all 8 ACs proven; review ended
Approve). The outcome half defers to the human's PR review.

What to change so it doesn't recur

1. claude/skills/codex/SKILL.md — the Execute section's launch
   instructions assume the harness cwd is the repo root, but other skills
   (e.g. the excalidraw renderer) legitimately cd elsewhere; one dispatch
   this run wrote its prompt/launcher to a nonexistent relative path and
   died at launch. Proposed edit — in "### 2. Execute", after "Launch every
   dispatch fully detached from the harness, from the repo root.", add:

   Resolve the owner directory to an absolute path once
   (dir="$(git rev-parse --show-toplevel)/.codex-dispatches/$own") and
   use it in the prompt/launcher/marker paths — a previous tool may have
   moved the shell's cwd, and a relative .codex-dispatches/... then
   fails the launch silently.

2. references/agents/implementer/instructions.md — both Step 3
   verifier failures were the same shape: the implementer's tests proved a
   stage's enqueue but not the end state, and proved concurrent
   coalescing but not sequential idempotency. Proposed addition to its
   testing guidance:

   When a pipeline hands off between workers, at least one test drives the
   complete chain to its observable end state (not the handoff), and
   any at-least-once worker gets a sequential double-invoke test (a
   second run over settled state is a no-op) — concurrency coalescing
   tests don't cover it.

Neither change is applied here; both await the human via /postmortem-loop.

Dial record & right-sizing

zone: 1
lanes: single-codex
passes: {plan: 3/3, post_pr: 3/3}
findings: {plan: {pass1: {codex: 5, claude: 0}, later: {codex: 4, claude: 0}},
           post_pr: {pass1: {codex: 2, claude: 0}, later: {codex: 1, claude: 0}}}
verifiers: {frontend: disabled_by_item, qa_pass: ran}
qa_findings: 0
wall_clock: 1:03
deviations: none
pr_size: {files_changed: 19, additions: 1401, deletions: 42}
tokens:
  codex: {total: 1064032, by_role: {code_researcher: 104767, plan_reviewer: 156756,
          implementer: 406024, backend_verifier: 192589, code_reviewer: 203896}}
  claude_subagents: 0
  overseer: 20062993   # includes cache reads; output tokens 114855
  total: 21127025
spend_ratio: 14641.0
agents:
  - {role: code-researcher, model: gpt-5.6-sol, effort: low, dispatches: 1, wall_clock: "10:30", tokens: 104767}
  - {role: plan-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "8:30", tokens: 156756}
  - {role: implementer, model: gpt-5.6-sol, effort: medium, dispatches: 4, wall_clock: "17:00", tokens: 406024}
  - {role: backend-verifier, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "8:30", tokens: 192589}
  - {role: code-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "10:30", tokens: 203896}

Right-sizing judgment: review effort right-sized — every pass in
both loops yielded findings (plan: 5/3/1; post-PR: 2/1/0 with the final pass
the required convergence proof), so no pass was pure spend; the dial that
would have changed the outcome most is none — zone 1 single-lane was exactly
enough for a change whose two real bugs (a grace-clock reuse and a
deletion race) were both caught by the Codex lane alone.

System changes

Posted as comments on: PR #129 and issue #128 (URLs recorded after posting).