diff --git a/.agents/skills/afk/SKILL.md b/.agents/skills/afk/SKILL.md index 97c49d11c1d..a00beb35038 100644 --- a/.agents/skills/afk/SKILL.md +++ b/.agents/skills/afk/SKILL.md @@ -1,8 +1,8 @@ --- name: afk description: >- - Enter away-mode supervision when the captain invokes /afk, says they are going afk, `state/.afk` exists, an incoming message starts with `FM_INJECT_MARK`, or any `state/.subsuper-*` marker is involved. - It sets a durable away-mode flag so the sub-supervisor daemon can self-handle routine wakes and escalate captain-relevant events plus bounded declared-external-wait rechecks as batched digests during walk-away stretches, then exits automatically when any real unmarked message returns firstmate to full per-wake responsiveness. + Enter the away posture when the captain invokes /afk, says they are going afk, `state/.afk-contract` or `state/.afk` exists, an incoming message starts with `FM_INJECT_MARK`, or any `state/.subsuper-*` marker is involved. + It reads the captain's away words back as a mandate, writes the durable away-posture record after their go, announces hold-for-return only at entry, keeps the one supervision session running in the away posture (the daemon delivers batched digests while Pi and OMP extensions stand by), and on the first unmarked message renders the return brief from durable records before ordinary work resumes. user-invocable: true metadata: internal: true @@ -10,83 +10,89 @@ metadata: # afk -Away-mode supervision. When invoked, `/afk` makes the daemon's token-saving -tradeoff **consented** and **explicit**: the captain is stepping away, so the -sub-supervisor may triage routine wakes in bash instead of waking firstmate's -LLM for each one. Escalations still reach the captain, but as one pre-read, -batched digest rather than per-wake injections. - -## What it does - -1. **Enter the lifecycle through `bin/fm-afk-launch.sh`.** - This owns the durable state write, session-scoped stale-artifact clearing, - terminal record, and rollback. - The flag survives a firstmate restart, so recovery re-enters afk when it is present. - -2. **Ensure the sub-supervisor daemon is running as a tracked background process.** - Its hosting differs by harness. - Pick the right path: - - **Harness WITH a native in-pane tracked-background tool** (e.g. claude's - background bash, grok's background tool): first run - `bin/fm-afk-launch.sh start-native`, then run - `FM_AFK_STATE_PREPARED=1 bin/fm-afk-start.sh` through that native tool. +Away mode is a POSTURE of the one supervision session, not a second architecture. +Being away changes exactly two things: how the captain is informed, and what happens at a captain-owned decision point (hold for return, or later a pre-answered clause). +It never changes the authority set. +The posture is a file, `state/.afk-contract`, written only by `bin/fm-afk-contract.sh` after the captain confirms a read-back; nothing infers the posture from chat. +Hold-for-return is the default and the only reach profile this release records: there is no phone channel, and the entry announcement says so aloud every time. + +## Entering: `/afk [words]` + +1. **Translate the captain's words into mandate clauses.** + The words are recorded verbatim; the clauses are your reading of them as explicit fields `bin/fm-afk-contract.sh` records: an action from its fixed verb list, the object in the captain's words, and the stated precondition in the captain's words, plus an optional stop. + Read `bin/fm-afk-contract.sh --help` for the field flags, verb list, and coarse best-effort never-set flag rather than memorizing them. + No static parser reads the object or precondition text, by the captain's mandate: you supply the fields, the script records them verbatim, checks structural presence and the verb list, and may flag obvious never-set concepts without treating that best-effort scan as authoritative. + A flagged clause is still recorded, never refused, and the read-back and return brief show the flag; the flag can miss spellings, including joined compounds such as `oneTimeCode`, never fires on unrelated names such as `ping-service`, and authoritative never-set, forbidden-action, and precondition judgment belongs to the supervision session at execution time in phase 4. + Forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, and no recorded clause is authority by itself. + Write only clauses the words actually support; a wish with no object or no stated precondition is not a clause. + Plain `/afk` with no words has no clauses. +2. **Propose and read back.** + Run `bin/fm-afk-launch.sh propose --words-file [--action --object --when [--stop ]]... [--expected-return ] [--spend ]` (or `--words `), and relay its read-back to the captain in `AGENTS.md` section 9 language: the accepted clauses as a numbered list, every refused clause with the part it is missing, the expected return, the spend cap, and the one-sentence reach announcement. + A refused clause does not fail the proposal; the captain can restate it or leave it refused. + Exit 3 only means a clause was refused; the proposal stands. +3. **Confirm on the captain's go.** + Run `bin/fm-afk-launch.sh confirm`; it promotes the proposal into the record and prints the entry announcement. + Relay that announcement verbatim in spirit: hold-for-return only, no phone channel, anything that needs the captain waits for their return, N clauses recorded and M refused, recorded clauses are held for the return brief and are not executed by this release, and forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text because no recorded clause is authority by itself. + With no words, run `propose` and `confirm` back to back; the announcement is the same. + Re-invoking `/afk` while already away with no new words is a refresh and leaves the standing record untouched; new words replace the mandate after the same read-back, preserve the original session entry, and archive the superseded mandate for the return brief. +4. **Per harness, after the record exists:** + - **Harness WITH a native in-pane tracked-background tool** (claude's background bash, grok's background tool): run `bin/fm-afk-launch.sh start-native`, then run `FM_AFK_STATE_PREPARED=1 bin/fm-afk-start.sh` through that native tool. This is a deliberate no-separate-terminal exception because the harness-hosted job creates no terminal or layout mutation, and a shell launcher cannot invoke a harness-native background tool. - The launcher still owns lifecycle state and records the no-terminal mode, while the daemon inherits and auto-discovers the captain pane. If the native launch fails, run `bin/fm-afk-launch.sh stop` to roll back the prepared lifecycle. Do not wrap it in `nohup ... &` (Codex/herdr can reap fire-and-forget shell children after a tool call returns). - - **Harness WITHOUT one** (e.g. pi): run `bin/fm-afk-launch.sh start`. It is - the single owner of the daemon terminal: it creates a NON-VISIBLE tracked - terminal for the current backend (a herdr dedicated `--no-focus` workspace, - a detached tmux session), records its exact id, and passes the captain pane - in as `FM_SUPERVISOR_TARGET` so the daemon injects into the captain, not its - own new pane. **Never manufacture a terminal by splitting the captain's - active pane** (`herdr pane split`): a split co-tenants the tab and visibly - shrinks the captain's pane (docs/herdr-backend.md "Away-mode supervisor - support"). - Both paths share `bin/fm-afk-start.sh` as the daemon entry. - The native path tells it that the launcher already prepared lifecycle state; the terminal-backed path lets the entry perform its existing state setup inside the new terminal. - It exits immediately if the identity-backed daemon lock already names a live process, otherwise it execs `bin/fm-supervise-daemon.sh` in the foreground. - The daemon is **presence-gated**: it injects escalations only while - `state/.afk` exists, and stays quiet otherwise. - -3. **Do not separately arm `fm-watch.sh`.** - The daemon manages the watcher as its child; [`docs/watcher-continuity.md`](../../../docs/watcher-continuity.md) owns the adapter hand-off contract. - -4. **Acknowledge** in `AGENTS.md` section 9 language: "Captain, away mode is active; I will batch routine updates and surface only decisions, failures, credentials, or review-ready work until you return." - -## How to exit afk + - **Harness without native tracked-background support** (pi, pi-signed, codex, opencode, omp, kimi, cursor): run `bin/fm-afk-launch.sh start`. + It is the single owner of the daemon terminal: it creates a NON-VISIBLE tracked terminal for the current backend and passes the captain pane in as `FM_SUPERVISOR_TARGET` so the daemon injects into the captain, not its own new pane (docs/herdr-backend.md "Away-mode supervisor support"). + Both daemon paths require the already-confirmed record and share `bin/fm-afk-start.sh` as the daemon entry. + The daemon is **presence-gated**: it injects escalations only while `state/.afk` exists, and stays quiet otherwise. +5. **Do not separately arm `fm-watch.sh`.** + The daemon manages the watcher as its child; [`docs/watcher-continuity.md`](../../../docs/watcher-continuity.md) owns the Pi and OMP extension handoff. + +## While away + +- Declared external waits and verified held transfers retain their bounded, widening recheck cadence; an external wait may also name its clearing time (`bin/fm-watch.sh`, `bin/fm-classify-lib.sh`). +- Recorded clauses are not executed by this release. + Forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, no recorded clause is authority by itself, and merge authority plus ask-user findings keep exactly the rules they have when attended (`AGENTS.md` section 7 and `ask-user-authority`); anything that needs the captain holds for their return. +- The session-start digest reports the posture under its AFK subsection, so a restart re-enters the posture from the record, not from memory. + +## How to exit: the return No `/back` is needed. The first genuine message is the return signal: - A message **without** the current operational prefix or a legacy bare marker, and **not** starting with `/afk` -> the captain is back. Run `bin/fm-afk-return.sh` before acting on the message that brought the captain back. - That script owns correct-ordered daemon shutdown, durable wake presentation and post-handling acknowledgement, escalation and wedge evidence, and the return-catch-up gate. - If it reports a firstmate-actionable `blocked:` event, remediate it immediately through the normal lifecycle, or explicitly reclassify it with a durable reason and close its decision key with `resolved [key=...]`, then run `bin/fm-afk-return.sh check`. - Once the daemon stops, resume full per-wake responsiveness through the emitted primary-harness supervision protocol while blocker handling proceeds, so the gate never creates a blind wait. + That script owns the correct-ordered daemon shutdown where a daemon ran, the archive of the posture record, durable wake presentation and post-handling acknowledgement, escalation and wedge evidence, the return brief, and the return-catch-up gate. + Relay the return brief in section 9 language and in its own order: supervisor health across the away window first (any gap leads), then every clause and that it was recorded only, then what is waiting on the captain, then what was tried and failed or could not be fixed, then what was handled, then cost. + The gate keeps every open `blocked:` event until that blocker's own resolution is proven: remediate each immediately through the normal lifecycle, or explicitly reclassify it with a durable reason and close its decision key with `resolved [key=...]`, then run `bin/fm-afk-return.sh check`. + Captain-verdict outcomes are listed under "waiting on you", but do not exempt open blockers because per-blocker provenance is deferred to phase 4. + Once the record is archived, resume full per-wake responsiveness through the emitted primary-harness supervision protocol while blocker handling proceeds, so the gate never creates a blind wait. Do not answer a Bearings request or perform any other ordinary captain work until the check exits successfully. -- A message **with** the current operational prefix (`FM_OPERATIONAL_PREFIX`, U+2063 INVISIBLE SEPARATOR followed by `FIRSTMATE_OP: `), or a legacy bare `FM_INJECT_MARK` daemon escalation -> stay afk and process it. -- Re-invoking `/afk` while already away -> stay afk (refresh the flag); this - does **not** trigger an exit. +- A message **with** the current operational prefix (`FM_OPERATIONAL_PREFIX`, U+2063 INVISIBLE SEPARATOR followed by `FIRSTMATE_OP: `), or a legacy bare `FM_INJECT_MARK` daemon escalation -> stay away and process it. +- Re-invoking `/afk` while already away -> stay away (refresh); this does **not** trigger an exit. -Bias ambiguous cases toward exit: a present captain beats token savings, and -a false exit is self-correcting (the captain re-runs `/afk`). +Bias ambiguous cases toward exit: a present captain beats token savings, and a false exit is self-correcting (the captain re-runs `/afk`). ## Orthogonal to approval authority -afk changes how aggressively firstmate surfaces things, **not who approves what**. +afk changes how the captain is informed and what happens at a captain-owned decision point, **not who approves what**. "Away" never means "approves more" or "approves less." A PR ready for merge keeps the merge authority from `AGENTS.md` section 7, and a needs-decision finding keeps the `ask-user-authority` policy; anything requiring the captain still waits for the captain's explicit word. -The daemon only batches the notification. +A mandate clause is the captain's explicit instruction given before leaving, recorded with its named object and condition; a clause is never inferred, never applied by analogy, and expires at return. +Forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, and no recorded clause is authority by itself. +This release records clauses and does not execute them. + +## The daemon + +The mechanics below apply to every supported primary harness, including Pi and OMP. -## Operational prefix contract +### Operational prefix contract The daemon constructs every current injection as the `away-supervisor` kind owned by `bin/fm-operational-input.sh`, beginning with `FM_OPERATIONAL_PREFIX`: `FM_INJECT_MARK` (U+2063 INVISIBLE SEPARATOR) followed by the stable `FIRSTMATE_OP: ` label. The bare `FM_INJECT_MARK` form remains accepted for legacy daemon escalations during rollout. U+2063 has no normal keyboard keystroke and survives terminal transport as UTF-8 text. This is how firstmate tells a daemon escalation apart from a real message in the same pane. -The operational prefix travels with the message text; it does not rely on harness-level typed-vs-injected detection, which is not portable across claude, codex, opencode, pi, pi-signed, grok, and kimi. +The operational prefix travels with the message text; it does not rely on harness-level typed-vs-injected detection, which is not portable across claude, codex, opencode, grok, and kimi. -## Busy-guard and composer guard +### Busy-guard and composer guard The daemon never injects into an in-use pane. Two checks run before every injection, dispatched through `bin/fm-backend.sh` for the supervisor's own @@ -109,12 +115,11 @@ attempts one normal flush, which still requires an idle pane and an affirmativel The alarm is defense in depth rather than a substitute for keeping every genuinely idle supported composer injectable. If that submit cannot be confirmed, it raises a loud, rate-limited wedge alarm: an ERROR in the daemon log, a durable -`state/.subsuper-inject-wedged` marker (surface it on the "while you were out" -catch-up if present), a tmux status-line flash when applicable, and a configurable backend-independent active alert. +`state/.subsuper-inject-wedged` marker (the return brief's health line carries it), a tmux status-line flash when applicable, and a configurable backend-independent active alert. `docs/wedge-alarm.md` owns the alert channel setup, and `docs/verification/supervision.md` "Wedge-alarm channels" owns active evidence. So a guard false-positive becomes a visible stall, never an unbounded silent no-op. -## Submit model +### Submit model The digest is typed **once** (`send-keys -l` on tmux, `pane send-text` on herdr - both literal, non-submitting sends), then submitted with Enter and @@ -130,11 +135,11 @@ A bordered-empty or ghost-only composer is recognized as empty where that backen **Busy-queued Enter exception (opencode 1.18.4).** OpenCode keeps queued text visible while it is mid-turn, so tmux and herdr delegate the final delivery decision to `fm_composer_queued_enter_verdict` in `bin/fm-composer-lib.sh` rather than treating visible text alone as a swallowed Enter. The daemon still clears its buffer only on the backend's `empty` success verdict; [`docs/tmux-backend.md`](../../../docs/tmux-backend.md) and [`docs/herdr-backend.md`](../../../docs/herdr-backend.md) own the backend-specific confirmation signals. -## Classification policy +### Classification policy The daemon wraps `fm-watch.sh`, runs the watcher as a child, presents every durable wake after each actionable watcher close, classifies each presented record in bash, and acknowledges the presented generation only after routing completes. It self-handles the routine majority without consuming a firstmate turn. -Captain-relevant events, plus a bounded recheck of a declared wait that is still declared, escalate to firstmate's context as one pre-read, single-line, batched digest. +Captain-relevant events, plus a bounded recheck of a declared external wait that is still declared, escalate to firstmate's context as one pre-read, single-line, batched digest. The captain-relevant verb set, declared-wait vocabulary, status-span classifier, and presentation-marker contract live in shared `bin/fm-classify-lib.sh`, while each supervisor owns its routing and fleet scan as a consumer of that policy. While `state/.afk` exists the daemon owns the watcher, so the watcher reverts to one-shot and lets the daemon do the triage - the two never run their triage at the same time. @@ -169,7 +174,7 @@ operational prefix, carrying pre-read status summaries and a recommended action. The single-line format makes the submission unambiguous across harnesses, and the operational prefix lets firstmate distinguish it from a real captain message. -## Injection hardening +### Injection hardening - **Single-line digest** - embedded newlines are collapsed to a literal separator before injection, so submission is unambiguous regardless of @@ -220,21 +225,21 @@ the operational prefix lets firstmate distinguish it from a real captain message misapplying tmux primitives to a pane that isn't one (docs/herdr-backend.md "Away-mode supervisor support"). -## Stale-artifact lifecycle +### Stale-artifact lifecycle Treat `state/.subsuper-escalations`, its `.since` sidecar, and `state/.subsuper-inject-wedged` as session-scoped delivery artifacts, not as the durable work record. Always enter through `bin/fm-afk-launch.sh`, which clears prior-session artifacts only for a fresh entry and preserves the current session's buffer on refresh. -Always exit through `bin/fm-afk-launch.sh stop`, which keeps `state/.afk` present through the daemon's shutdown flush and clears it last. +Always exit through `bin/fm-afk-launch.sh stop`, which keeps `state/.afk` present through the daemon's shutdown flush, clears it, and archives the posture record last. `docs/herdr-backend.md` "Away-mode supervisor support" owns the current mechanism, and `docs/verification/runtime-backends.md` "Away-mode transport" owns active evidence. -## Reliability properties +### Reliability properties These properties must hold: - Nothing is lost after queue publication. The daemon leaves every presented wake durable until routing completes and post-handling acknowledgement succeeds, so interruption replays the same work to the daemon or its successor. - Wedge detection is bounded-latency, not lossy. -- Declared external waits are rechecked on a separate, bounded cadence rather than being mislabeled as wedges. +- Declared external waits are rechecked on a separate, bounded, condition-aware cadence rather than being mislabeled as wedges; items held for the captain are not rechecked while the posture record exists. - The catch-all scan backs up the keyword classifier. - The daemon preserves a single-instance portable lock, crash-loop backoff, a pane-gone guard, and a signal-trapped shutdown that flushes buffered diff --git a/.agents/skills/bearings/SKILL.md b/.agents/skills/bearings/SKILL.md index c28b7f4f4eb..12c08aa915f 100644 --- a/.agents/skills/bearings/SKILL.md +++ b/.agents/skills/bearings/SKILL.md @@ -128,7 +128,7 @@ After handling, rebuild the board from a fresh snapshot so acted-on items leave ### The merge-click ruling (captain-decided) A board "Merge now" answer IS the captain's explicit merge word for that one exact PR; ask no second confirmation. -The safeguards are mandatory, not optional: resolve the PR from the task's own `state/.meta` `pr=` record, never from board bytes; re-verify at wake time that the PR is still open and CI-green; refuse and report a red or changed PR rather than merging it; merge only through `bin/fm-pr-merge.sh`; and echo every merge in chat with the full PR URL. +The safeguards are mandatory, not optional: resolve the PR from the task's own `state/.meta` `pr=` record, never from board bytes; re-verify at wake time that the PR is still open and CI-green; refuse and report a red or changed PR rather than merging it; record the exact `merge` answer through `bin/fm-captain-hold.sh answer --decision-file --release` before invoking the merge; proceed only when that release succeeds; merge only through `bin/fm-pr-merge.sh`; and echo every merge in chat with the full PR URL. Only the exact answer value `merge` authorizes a merge; an answer carrying a freeform note is the captain's instruction text to read and act on with judgment, never an auto-merge. ## Chat-response contract diff --git a/.agents/skills/bootstrap-diagnostics/SKILL.md b/.agents/skills/bootstrap-diagnostics/SKILL.md index 62aa8c8ba96..e4dde858ea8 100644 --- a/.agents/skills/bootstrap-diagnostics/SKILL.md +++ b/.agents/skills/bootstrap-diagnostics/SKILL.md @@ -91,8 +91,8 @@ When any diagnostic needs captain attention, report the plain consequence and re - `SECONDMATE_SYNC: secondmate : skipped: ` - secondmate convergence left a live home on its existing checkout because the home was dirty, diverged, unsafe, on the wrong branch, missing its placement-specific target commit, unreachable, or otherwise not fast-forwardable, or because inherited local-material propagation failed; bootstrap continued, but inspect the reason because the secondmate's tracked instructions, inherited settings, or shared captain preferences may be stale after a primary update. - `SECONDMATE_LIVENESS: secondmate : skipped: |respawn failed after : ` - the session-start liveness sweep could not guarantee that the registered secondmate is running a real agent process. Investigate the reason because that secondmate is not guaranteed live. -- `SECONDMATE_HANDOFF: secondmate : pending delivery: item(s)` - queued work has already left the main dispatchable backlog and remains safe in the named remote route's backlog-format outbox, pending backlog receipt or receiver-wake confirmation. - Preserve that outbox and rerun `bin/fm-backlog-handoff.sh --resume-pending` after the route or endpoint problem is resolved; never re-add or dispatch the items from the main backlog. +- `SECONDMATE_HANDOFF: secondmate : pending delivery: item(s)` - queued work has already left the main dispatchable backlog and remains safe in the named remote route's backlog-format outbox because backlog receipt or local outbox cleanup has not completed; [`bin/fm-backlog-handoff.sh`](../../../bin/fm-backlog-handoff.sh) owns the release contract. + Preserve that outbox and rerun `bin/fm-backlog-handoff.sh --resume-pending` after the route, receipt, or cleanup problem is resolved; never re-add or dispatch the items from the main backlog. An unsafe-outbox variant requires path and file-type inspection before any retry. - `NUDGE_SECONDMATES: secondmate : send failed: ` - secondmate convergence changed a running home's loaded instructions or inherited config, but the deterministic `fm-send.sh fm-` re-read nudge failed. Inspect the reason, keep the pending marker under `state/.secondmate-nudge-pending/` intact, and rerun session start after the endpoint or metadata issue is fixed so bootstrap can retry the exact same marked send on the same local or remote route. diff --git a/.agents/skills/captain-hold-lifecycle/SKILL.md b/.agents/skills/captain-hold-lifecycle/SKILL.md index e84f68af72f..9fb5a4f7088 100644 --- a/.agents/skills/captain-hold-lifecycle/SKILL.md +++ b/.agents/skills/captain-hold-lifecycle/SKILL.md @@ -25,12 +25,14 @@ A completed investigation, a completed ADR design, and an ended visual review us Run the command in the originating work's authoritative `FM_HOME`; secondmate-owned work registers in that secondmate home's backlog, and a question already held anywhere is never re-registered as a second row. Do not close a captain-held task merely because the originating investigation completed, its report was archived, its visual review ended, or its task was torn down. Holding the work item the question gates is safe for exactly that reason: cleanup keeps such a row open with the finished work's deliverable recorded and returns it to the queue, so it still reads as the captain's own call. -Only `answer` with the captain's words or an evidence-backed `reconcile close` may close it. +Only `answer` with the captain's words or an evidence-backed `reconcile close` may resolve it. -Never close anything the captain owns without recording what he actually said: `bin/fm-captain-hold.sh answer` writes his exact words into the task and closes it in the same act, with `--release` when the answer frees a captain-gated work item to proceed instead of completing a question. +Never close anything the captain owns without recording what he actually said: `bin/fm-captain-hold.sh answer` writes his exact words into the task and closes a question-shaped call, while `--release` frees a captain-gated work item to proceed. +A merge approval uses that existing release path because approval permits the merge to proceed; cleanup closes the work only after it lands and records what shipped. +Closing a held row at merge approval instead records completion before landing, so the backlog claims completion before the work actually ships. When the answer changes what a task must build, follow `AGENTS.md` section 7's Validate contract to preserve the captain's words in the brief and steer the worker. When the captain says "later", that is an answer too: re-hold with `bin/fm-captain-hold.sh hold --reason "" --until ` so the item leaves the live Captain's Call and resurfaces on its date, instead of leaving a live-looking card or fabricating a closure. -"A keyed answer closes its matching captain-held task" is one capability with one owner, `bin/fm-captain-hold.sh answers`, and every channel that carries a captain answer feeds it the same task id and answer; a channel never maps keys to tasks, records a decision, or closes anything itself. +"A keyed answer resolves its matching captain-held task" is one capability with one owner, `bin/fm-captain-hold.sh answers`, and every channel that carries a captain answer feeds it the same task id and answer; a channel never maps keys to tasks, records a decision, or resolves anything itself. Chat already feeds it through `bin/fm-send.sh --resolve-key`, and a captured-answer source feeds it once bound with `bin/fm-captain-hold.sh bind `; bind before arming the source, and key each structured question by the held task's id. An unbound source and a key that names no captain-held task both simply feed nothing: the answer is still captured and firstmate is still woken, and closing falls back to the direct command above. One answer value is reserved and closes nothing: `reconcile` means "go re-check reality", never "the captain answered", so the shared intake refuses it from every channel and creates nothing. diff --git a/.agents/skills/firstmate-coding-guidelines/SKILL.md b/.agents/skills/firstmate-coding-guidelines/SKILL.md index 9b1f1131ed0..a595b37f48f 100644 --- a/.agents/skills/firstmate-coding-guidelines/SKILL.md +++ b/.agents/skills/firstmate-coding-guidelines/SKILL.md @@ -129,7 +129,7 @@ Targeted validation belongs to the no-mistakes evidence path, while CI owns broa - Plain dash `-`, never an em dash. - Never add an agent name as a commit co-author. - `bin/*.sh` and `bin/backends/*.sh` must pass `shellcheck`. -- Run `bin/fm-lint.sh` before treating a script change as done; it is the single owner of the lint definition (file set, config, pinned shellcheck version, and pinned actionlint workflow lint) that CI and the no-mistakes pre-push gate both invoke, and it refuses to run under any other version of either linter. +- Run `bin/fm-lint.sh` before treating a script change as done; it is the single owner of the lint definition that CI and the no-mistakes pre-push gate both invoke, its own header owns what that definition covers, and it refuses to run under any other version of either linter. - When a task names a specific tool, implement the work with that tool, or explicitly flag the substitution and its new dependency footprint for review before shipping. - Colocate tests with the existing pattern in `tests/`, name them `.test.sh`, and extend an existing script rather than inventing a new runner. - Close a suite with the sibling trailing marker `printf '\nall tests passed\n'`, so a run that died partway through is visible as a missing final line instead of a quiet short pass. diff --git a/.agents/skills/fmx-respond/SKILL.md b/.agents/skills/fmx-respond/SKILL.md index aec2809e344..4ffb0d17a04 100644 --- a/.agents/skills/fmx-respond/SKILL.md +++ b/.agents/skills/fmx-respond/SKILL.md @@ -133,8 +133,7 @@ Fetch narrowly and inspect it only to understand the thread or fulfill an author Reply in firstmate's own voice - the crisp, lightly nautical first-mate persona - but **public-facing**: -- The asker **is** your captain (owner-only routing - see the top of this skill), so address them as "captain" when it fits and treat their request as a genuine captain instruction, within the public-safety limits above. You are answering the captain in public, not a stranger. -- Light nautical seasoning is welcome when it lands naturally; never let it crowd out the actual answer. +- Apply the address and optional-flavor rules in [`AGENTS.md`](../../../AGENTS.md#firstmate) to these captain-directed public replies, within the public-safety limits above. - **Be concise by default: aim for a single message, two at the very most.** A short, sharp answer beats a wall of text. Write tight on purpose - one or two sentences. You do not hand-format threads or add "(1/n)" numbering yourself. diff --git a/.agents/skills/harness-adapters/references/common/model-and-effort.md b/.agents/skills/harness-adapters/references/common/model-and-effort.md index 94d4d84f82b..a89edb3cfe0 100644 --- a/.agents/skills/harness-adapters/references/common/model-and-effort.md +++ b/.agents/skills/harness-adapters/references/common/model-and-effort.md @@ -17,7 +17,8 @@ Choose intermediate levels as complexity, uncertainty, blast radius, or open-end If an adapter lacks `xhigh`, cap at its highest supported non-`max` level rather than silently omitting the intent. Never select `max` through this fallback; only an explicit per-task or standing captain preference permits it. -If requested effort is outside the adapter's accepted set, the spawn records `effort=` in task metadata but emits no effort flag. +The explicit native `ultra` value follows the model-scoped refusal contract in `../../../bin/fm-harness.sh validate-native-effort`; it is never silently omitted or mapped to a Pi level. +For other values, if requested effort is outside the adapter's accepted set, the spawn records `effort=` in task metadata but emits no effort flag. This preserves launch success instead of passing a known-bad value. A harness with no verified interactive effort flag follows the same record-and-omit contract. diff --git a/.agents/skills/harness-adapters/references/harness/pi.md b/.agents/skills/harness-adapters/references/harness/pi.md index 97136e0d171..b8bb3175cb6 100644 --- a/.agents/skills/harness-adapters/references/harness/pi.md +++ b/.agents/skills/harness-adapters/references/harness/pi.md @@ -15,6 +15,7 @@ Verified on 2026-07-27 with Pi and Pi-signed 0.82.0 unless a fact gives another | Effort flag | `--thinking `; both identities expose the same levels and completed the same model-qualified max-thinking smoke. | | Model discovery | Run the selected executable as ` --list-models [search]`; Pi's installed `docs/models.md` owns how built-in, extension-registered, and custom provider/model entries reach that list. | +Native Codex sessions may request `ultra` through the native extension flag described by `../../../bin/fm-spawn.sh`; it is separate from Pi's thinking levels. Pi has no permission system, so workers are always autonomous. Pi's installed `packages/coding-agent/docs/settings.md` UI and display section documents `regular` as the `tuiMode` default and `fullscreen` as experimental. Fullscreen can bury steering messages by rewriting scrollback, so Firstmate avoids it when the installed CLI supports the override. @@ -37,6 +38,7 @@ The decision persists per path in `~/.pi/agent/trust.json`, so later spawns in t `../../../bin/fm-spawn.sh` keeps the worker turn-end extension in `state/`, outside the worktree, because project-local extension files worsen the trust gate and pollute the project. The extension listens for Pi's `turn_end` event, not `agent_end`, so supervision is notified after each completed turn rather than only when the whole run exits. +Native-harness progress uses the separate generation-bound marker owned by `../../../bin/fm-busy-event.sh`; it never fabricates Pi turn completion. Pi sets `PI_CODING_AGENT=true` for its children as its harness-detection marker. ## Primary integration @@ -50,6 +52,7 @@ On native Windows, the extension runs its session-start, both PreToolUse, turn-e The primary watcher protocol also requires `.pi/extensions/fm-primary-pi-watch.ts`. The Pi engine auto-discovers both tracked project-local extensions once the project is trusted. The model arms through the `fm_watch_arm_pi` tool, never through a foreground shell arm. +Native-harness adapters can discover the same guarded FirstMate tools and operational message allowlist through the public Pi event-bus contract in `.pi/extensions/lib/fm-native-contract.ts`; no Pi built-in tools cross that contract. The tool result and clean-exit fallback are owned by `../../../docs/supervision-protocols/pi.md`. `../../../bin/fm-session-start.sh` reports when the live Pi-family session has not loaded both extensions and points at `/reload` or restarting the selected executable after project trust as the fix, with `-e` as a trust-free fallback. diff --git a/.agents/skills/secondmate-provisioning/SKILL.md b/.agents/skills/secondmate-provisioning/SKILL.md index ffc80f42be0..875ede69528 100644 --- a/.agents/skills/secondmate-provisioning/SKILL.md +++ b/.agents/skills/secondmate-provisioning/SKILL.md @@ -198,11 +198,10 @@ After seeding, run this handoff for the new secondmate's in-scope queued items. For an existing or inherited domain, complete record intake first so no already-shipped plan row is handed off as open work. For a local route, the helper resolves and validates the secondmate home from `data/secondmates.md`, then delegates the item move to `tasks-axi mv` (the single owner of the backlog format), which moves each named item - and a whole connected set, blocker plus dependents, atomically - from the main `data/backlog.md` into the secondmate home's `data/backlog.md`. For a remote route, the same helper first moves the dependency-closed set atomically from the main backlog into `data/handoff/.outbox.md`, then transfers that backlog-format outbox through `fm-on.sh` and lets the remote home's `fm-backlog-receive.sh` move every not-already-present key under the destination lock. -After a new local placement or a remote outbox receipt becomes durable, the helper sends one marked routed-work instruction through the receiving secondmate's recorded endpoint; missing or failed delivery makes the command fail loudly with the moved work intact, and the same handoff command retries known-undelivered wake intent without moving an already-present item again. -An unresolved delivery attempt is never blindly resent. -For a remote route, the outbox remains until both backlog receipt and receiver wake are confirmed; `--resume-pending` retries unfinished outboxes, while the script header owns its stable wake-correlation recovery state. +After a new local placement or a remote outbox receipt becomes durable, the helper attempts one marked routed-work instruction through the receiving secondmate's recorded endpoint. +[`bin/fm-backlog-handoff.sh`](../../../bin/fm-backlog-handoff.sh) owns route-specific wake outcomes, remote outbox release after durable receipt, and stable wake-correlation retry behavior. There is no two-phase handoff journal and no tasks-axi release beyond the already-required atomic `mv` capability. -Bootstrap retries pending outboxes when mutation is authorized and emits `SECONDMATE_HANDOFF:` for any that remain. +Bootstrap retries pending outboxes and wakes when mutation is authorized and emits `SECONDMATE_HANDOFF:` for any outboxes that remain. This delegated route remains required when `config/backlog-backend=manual`, which controls only routine firstmate backlog edits. It moves each queued item's whole block - the `- [ ] ...` header plus every following two-or-more-space-indented body line and blank separator, up to the next item or column-0 section heading - byte-exact under the same section, treating an indented `## ...` line as body rather than a section boundary, so neither the header nor its body is duplicated or orphaned. It refuses a selected item with a single-space or tab-indented continuation rather than risk leaving content orphaned in the main backlog. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d9e93158528..ab6442f625c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -135,10 +135,10 @@ jobs: tests-portable-serial: name: Behavior portable serial ${{ matrix.shard }} runs-on: ubuntu-latest - # The merged hint table estimates ~69 min of serial work; eight balanced - # shards keep the slowest near 8.8 min. The 20-minute upstream cap leaves - # room for setup and a bounded hung script to report its own verdict. - timeout-minutes: 20 + # Current runners can take ~20 min for a balanced shard. This 30-minute cap + # preserves the timeout as a hang tripwire while allowing runner-speed and + # job-setup margin; it is not the expected healthy end of the lane. + timeout-minutes: 30 strategy: # Every shard reports so one failure never hides another shard's result. fail-fast: false @@ -399,6 +399,12 @@ jobs: done < "$shell_inventory" [ "$parse_fail" -eq 0 ] || { echo "::error::stock macOS Bash 3.2 parse sweep failed"; exit 1; } + command -v npm >/dev/null || { echo "::error::npm is required to install tasks-axi"; exit 1; } + npm install -g tasks-axi@0.2.5 >/dev/null + PATH="$(npm prefix -g)/bin:$PATH" + export PATH + command -v tasks-axi >/dev/null || { echo "::error::tasks-axi is required for the stock Bash regressions"; exit 1; } + # Collapse floors, not tallies. set -e already catches a suite that # fails; these catch one that exits 0 having silently run far fewer # assertions under stock Bash 3.2. Both floors sit at the current @@ -415,17 +421,11 @@ jobs: bearings_output=$(/bin/bash tests/fm-bearings-snapshot.test.sh) printf '%s\n' "$bearings_output" bearings_count=$(printf '%s\n' "$bearings_output" | grep -c '^ok - ') - [ "$bearings_count" -ge 53 ] || { - echo "::error::expected at least 53 Bearings tests, got $bearings_count" + [ "$bearings_count" -ge 56 ] || { + echo "::error::expected at least 56 Bearings tests, got $bearings_count" exit 1 } - command -v npm >/dev/null || { echo "::error::npm is required to install tasks-axi"; exit 1; } - npm install -g tasks-axi@0.2.5 >/dev/null - PATH="$(npm prefix -g)/bin:$PATH" - export PATH - command -v tasks-axi >/dev/null || { echo "::error::tasks-axi is required for the public-followup bash 3.2 register regression"; exit 1; } - # The full public-followup suite is not a stock-bash snapshot; run only # the empty-lock register regression under real /bin/bash 3.2. pf_output=$(FM_TEST_ONLY=test_first_register_succeeds_with_empty_lock_list_under_bash32 \ diff --git a/.gitignore b/.gitignore index 254ffe8a44a..14423501ccb 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,5 @@ __pycache__/ *.pyc .env config/ + +.tools/ diff --git a/.pi/extensions/fm-branch-supervision.ts b/.pi/extensions/fm-branch-supervision.ts index d9cef8d7a43..682f0a087ab 100644 --- a/.pi/extensions/fm-branch-supervision.ts +++ b/.pi/extensions/fm-branch-supervision.ts @@ -88,6 +88,7 @@ import { } from "@earendil-works/pi-coding-agent"; import { Box, Container, fuzzyFilter, Input, SelectList, Text } from "@earendil-works/pi-tui"; import { Type } from "typebox"; +import { registerFirstmateTool } from "./lib/fm-native-contract.ts"; import { runCommandAsync } from "./lib/fm-async-exec.ts"; import { type CalmPresentationState, @@ -234,6 +235,9 @@ type BranchModel = NonNullable>; type BranchEffort = ReturnType>; type PinnedBranchModel = { model: BranchModel; modelRuntime: ModelRuntime }; type BranchModelResolution = { ok: true; selection: PinnedBranchModel } | { ok: false; reason: string }; +type FollowMainResolution = + | { ok: true; selection: PinnedBranchModel } + | { ok: false; reason: string; refusesBuild: boolean }; // Pi owns the effort vocabulary. The picker's options and every clamp still // come from Pi's own getSupportedThinkingLevels/clampThinkingLevel, so this @@ -758,6 +762,9 @@ export default function (pi: ExtensionAPI) { async function resolveBranchModel(provider: string, modelId: string): Promise { const label = `${provider}/${modelId}`; + if (provider === "codex-native") { + return { ok: false, reason: `${label} belongs to the main native session; choose an ordinary Pi provider for supervision` }; + } const modelRuntime = await ModelRuntime.create(); let model = modelRuntime.getModel(provider, modelId) as BranchModel | undefined; if (!model) { @@ -779,25 +786,50 @@ export default function (pi: ExtensionAPI) { return resolved.selection; } + // "Follow main" is ONE rule, shared by every unpinned branch build and by + // the /supervision-model report, so the report describes exactly what the + // next build does. An ordinary Pi provider is applied as main's own model; + // when the isolated runtime cannot run it, the build passes no override at + // all (refusesBuild false). A native provider owns a persistent main + // thread, so the branch instead selects the same model through Pi's + // independent openai-codex provider, and when that model is unavailable the + // build refuses (refusesBuild true) rather than inheriting the native + // thread or silently restoring a recorded native selection. + async function followMainModel(main: { provider: string; id: string }): Promise { + const native = main.provider === "codex-native"; + let resolved: BranchModelResolution; + try { + resolved = await resolveBranchModel(native ? "openai-codex" : main.provider, main.id); + } catch (error) { + resolved = { ok: false, reason: error instanceof Error ? error.message : String(error) }; + } + if (resolved.ok) return resolved; + if (!native) return { ...resolved, refusesBuild: false }; + return { + ok: false, + refusesBuild: true, + reason: `native main requires an independent Pi supervision model and ${resolved.reason}; the branch refuses to build until one is pinned with /supervision-model`, + }; + } + // The pin file's CURRENT state decides the model on every branch build, // create and reopen alike, and it overrides Pi's restore of whatever model // a reopened branch session recorded. With a pin, that model. With no pin, - // main's own model is applied EXPLICITLY - otherwise clearing the pin would - // report that the branch follows main while the reopened session quietly - // restored the model an earlier pin left behind. Only when main's model is - // genuinely unknown, or the isolated runtime cannot run it, does the build + // main's own model is applied EXPLICITLY through followMainModel - + // otherwise clearing the pin would report that the branch follows main + // while the reopened session quietly restored the model an earlier pin left + // behind. Only when main's model is genuinely unknown, or the follow rule + // says the isolated runtime cannot run an ordinary provider, does the build // fall back to passing no override at all, which is the pre-feature - // behavior; an unpinned branch is never refused over model choice alone. + // behavior. async function branchModelSelection(): Promise { const pin = readModelPin(); if (pin) return preparePinnedBranchModel(pin); if (!mainModel) return undefined; - try { - const resolved = await resolveBranchModel(mainModel.provider, mainModel.id); - return resolved.ok ? resolved.selection : undefined; - } catch { - return undefined; - } + const following = await followMainModel(mainModel); + if (following.ok) return following.selection; + if (following.refusesBuild) throw new Error(following.reason); + return undefined; } async function effectiveBranchModel(selected: BranchModel | undefined): Promise { @@ -1714,7 +1746,7 @@ ${context.command} await copyExtensionProviders(modelRuntime); available = ctx.modelRegistry .getAvailable() - .filter((model) => modelRuntime.getModel(model.provider, model.id) && modelRuntime.hasConfiguredAuth(model.provider)) + .filter((model) => model.provider !== "codex-native" && modelRuntime.getModel(model.provider, model.id) && modelRuntime.hasConfiguredAuth(model.provider)) .map(modelLabel); } catch (error) { ctx.ui.notify( @@ -1758,23 +1790,22 @@ ${context.command} modelReport = { message: `Supervision branch model: ${picked}.`, warning: false }; } else { // Clearing the pin only follows main if main's model can actually be - // applied to the branch; say what will really happen rather than - // reporting a state that did not take effect. - try { - const following = mainModel ? await resolveBranchModel(mainModel.provider, mainModel.id) : null; - if (following?.ok) branchModel = following.selection.model; - modelReport = following?.ok - ? { - message: `Supervision branch follows main's model (${modelLabel(following.selection.model)}).`, - warning: false, - } - : { - message: `Supervision branch pin cleared, but main's model could not be applied (${following ? following.reason : "main's model is not known yet"}); the branch keeps the model its own session recorded until that conversation is replaced.`, - warning: true, - }; - } catch (error) { + // applied to the branch; the same followMainModel rule the next build + // runs says what will really happen rather than reporting a state + // that did not take effect. + const following = mainModel ? await followMainModel(mainModel) : null; + if (following?.ok) { + branchModel = following.selection.model; + modelReport = { + message: `Supervision branch follows main's model (${modelLabel(following.selection.model)}).`, + warning: false, + }; + } else { + const consequence = following?.refusesBuild + ? "" + : "; the branch keeps the model its own session recorded until that conversation is replaced"; modelReport = { - message: `Supervision branch pin cleared, but main's model could not be applied (${error instanceof Error ? error.message : String(error)}); the branch keeps the model its own session recorded until that conversation is replaced.`, + message: `Supervision branch pin cleared, but main's model could not be applied (${following ? following.reason : "main's model is not known yet"})${consequence}.`, warning: true, }; } @@ -2030,7 +2061,7 @@ ${context.command} return shell; }; - pi.registerTool?.({ + registerFirstmateTool(pi, { name: "fm_branch_outcomes", label: "Read supervision branch outcomes", description: @@ -2092,7 +2123,7 @@ ${context.command} // cursor, never backwards), and refused outside lock ownership, so neither a // paraphrase, an empty reply, nor a stale generation can mark an outcome // processed. - pi.registerTool?.({ + registerFirstmateTool(pi, { name: "fm_branch_processed", label: "Acknowledge processed supervision outcomes", description: diff --git a/.pi/extensions/fm-primary-pi-watch.ts b/.pi/extensions/fm-primary-pi-watch.ts index 08c75b3755b..6752e86ade0 100644 --- a/.pi/extensions/fm-primary-pi-watch.ts +++ b/.pi/extensions/fm-primary-pi-watch.ts @@ -57,6 +57,7 @@ import { fileURLToPath } from "node:url"; import type { ExtensionAPI, Theme } from "@earendil-works/pi-coding-agent"; import { Box, Container, Text, type Component } from "@earendil-works/pi-tui"; import { Type } from "typebox"; +import { registerFirstmateTool } from "./lib/fm-native-contract.ts"; import { createBranchDispatchOffer, FM_BRANCH_DISPATCH_EVENT, @@ -1293,7 +1294,7 @@ export default function (pi: ExtensionAPI) { }, }); - pi.registerTool?.({ + registerFirstmateTool(pi, { name: "fm_watch_arm_pi", label: "Arm firstmate watcher", description: "Start the first required Pi watcher cycle, or repair one only after a notification says the cycle is missing, failed, or unhealthy. Do not call after ordinary work or ordinary notifications; the Pi extension re-arms automatically. Never run bin/fm-watch-arm.sh through bash.", diff --git a/.pi/extensions/lib/fm-native-contract.ts b/.pi/extensions/lib/fm-native-contract.ts new file mode 100644 index 00000000000..a00dece7ec2 --- /dev/null +++ b/.pi/extensions/lib/fm-native-contract.ts @@ -0,0 +1,36 @@ +import type { ExtensionAPI, ToolDefinition } from "@earendil-works/pi-coding-agent"; +import type { TSchema } from "typebox"; + +// Public Pi event-bus boundary for native-harness adapters. FirstMate owns the +// operational message allowlist and these tools; the adapter owns transport. +// Discovery is synchronous: emit { register(tool), allowMessageType(type) } on +// firstmate:native-tools. Only explicitly registered FirstMate controls cross +// this boundary, with the SAME execute callback and ownership checks as Pi. +// The native adapter supplies its current ExtensionContext when executing. +// Pi owns subscription cleanup with the extension runtime, including reload. +export function registerFirstmateTool( + pi: ExtensionAPI, + tool: ToolDefinition, +): void { + pi.registerTool?.(tool); + pi.events?.on?.("firstmate:native-tools", (request: unknown) => { + if (!request || typeof request !== "object") return; + const discovery = request as { + register?: (tool: unknown) => void; + allowMessageType?: (type: string) => void; + }; + if (typeof discovery.register === "function") { + discovery.register({ + name: tool.name, + description: tool.description, + inputSchema: tool.parameters, + execute: tool.execute, + }); + } + if (typeof discovery.allowMessageType === "function") { + for (const type of ["firstmate-sessionstart-nudge", "fm-branch-merge", "fm-branch-process"]) { + discovery.allowMessageType(type); + } + } + }); +} diff --git a/AGENTS.md b/AGENTS.md index dd8068f6fae..c65d043390e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,12 +7,12 @@ You are the first mate. The user is the captain. This file is your entire job description. -Address the user as "captain" at least once in every response. +Address the user as "captain" at least once in every chat message you send them, including public replies, without forcing it into every sentence. This is mandatory respectful address, not performance: it applies even when delivering bad news or relaying serious findings, such as "Captain, the build broke - ...". -Do not force it into every sentence, but never send a response with zero direct address. +The obligation applies to captain-directed chat under this supervisor role; crewmates retain their brief's reporting line through firstmate. +Never put direct address into a non-chat artifact such as a commit message, PR or issue description, brief, code, or comment. In a secondmate home that address is form only: section 9's parent-channel rule is the only way the captain is reached from there. -Use light nautical seasoning only when it fits: the occasional "aye", "on deck", "shipshape", "under way", or "ahoy" may land naturally. -Keep that seasoning optional and never let it obscure technical content; never use it in commits, briefs, PRs, or anything crewmates or other tools read; drop the playful flavor entirely when delivering bad news or relaying serious findings. +Use light nautical seasoning only when it fits: the occasional "aye", "on deck", "shipshape", "under way", or "ahoy" may land naturally, kept optional, never obscuring technical content, held to the same channel bound, and dropped entirely when delivering bad news or relaying serious findings. For captain-facing escalation style and outcome phrasing, see section 9. ## 1. Identity and prime directives @@ -75,7 +75,7 @@ README.md public overview and development notes .claude/skills symlink to .agents/skills for claude compatibility skills/ standalone public installer-facing skills, committed; not loaded by firstmate bin/ helper scripts, committed; read each script's header before first use -.env optional Relay pairing token; LOCAL, gitignored; presence-gates section 14 +.env optional Relay pairing token (presence-gates section 14) and mail-plane credentials (schema: docs/configuration.md "Mail plane"); LOCAL, gitignored config/crew-harness crewmate harness override; LOCAL, gitignored; inherited by secondmate homes (docs/configuration.md "Harness support"; section 4) config/launch-env-allowlist optional worker environment names; LOCAL, gitignored; inherited by secondmate homes (docs/configuration.md "Worker launch environment") config/crew-dispatch.json optional per-task crewmate dispatch profiles; LOCAL, gitignored; inherited by secondmate homes (docs/configuration.md "Crew dispatch profiles"; section 4) @@ -118,6 +118,7 @@ state/ runtime records and signals; gitignored .turn-ended turn-boundary wake notification, never current-state truth; producer contracts routed by harness-adapters .run-step last observed no-mistakes run step; private record owned by bin/fm-crew-state.sh and removed by teardown .agy-trust agy trust cleanup marker; exact lifecycle lives in bin/fm-agy-trust-lib.sh + .progress touched for observed native-harness activity inside one Pi turn; bin/fm-busy-event.sh owns its generation binding and bin/fm-watch.sh reads it beside turn-ended for the busy-age bound only, never as a completed turn .grok-turnend-token firstmate-owned grok hook registry token for the task; removed by teardown .kimi-turnend-token firstmate-owned Kimi hook registry token for the task; removed by teardown .pr-status cached normalized PR observation, refreshed only by bin/fm-pr-status.sh so read-only consumers never call a forge; removed by teardown @@ -146,6 +147,8 @@ state/ runtime records and signals; gitignored x-watch.check.sh generated Relay poll shim; present only when opted in (section 14) tool-updates.check.sh generated watched-tool update poll shim and its .check-trust binding; present only after bin/fm-tool-update-check.sh arm; its report record .tool-updates prevents repeat reporting of one pending update issue-status/ cached work-item enrichment results; safe to delete, rebuilt on demand (docs/configuration.md "Project issue trackers") + mail.check.sh generated received-mail poll shim and its .check-trust binding; present only after bin/fm-mail-check.sh arm; report record .mail-check (mail schema: docs/configuration.md "Mail plane") + .mail-seen .mail-woken .mail-retry .mail-retry-pos .mail-turn .mail-seen.lock mail-plane poll cursor, emission journal, transient-fetch retry set, retry-scan position, contended-slot turn flag, and overlapping-poll lock; written only by bin/fm-mail.sh (mail schema: docs/configuration.md "Mail plane") pending-replies/ parent-owned secondmate pending-reply records (correlation id, delivery vs reply, recovery, escalation); fm-pending-reply-lib.sh procevent/ registered process-to-event sources, one private record per canonical source id; written only by bin/fm-procevent.sh, and their presence alone keeps supervision required (section 13) procevent-inbox/ private captured results and their durable handled-acknowledgement markers; source output lives here and never in an event line @@ -167,7 +170,9 @@ state/ runtime records and signals; gitignored .watcher-down private generation-bound recovery state coupling watcher downtime, durable wake presentation, and post-handling acknowledgement; never touch ..open-decisions-cursor per-task byte cursor and folded open-decision set bounding the OPEN DECISIONS scan's cost to new status-log appends; written only by fm-classify-lib.sh's status_open_decisions_incremental, removed by teardown, safe to delete (forces one full re-fold) .status-presentation-cursor .status-presentation-lock fleet-wide per-task status identity plus independent annotation and outcome-backstop byte offsets, with a serialization lock preventing already-presented lines from replaying while preserving delayed signal annotations; owned by fm-classify-lib.sh, with each task's row retired by teardown - .afk durable away-mode flag; present = sub-supervisor may inject escalations (set by /afk, cleared on user return) + .afk-contract the away-posture record: the captain's verbatim away words, expected return, reach profile, spend cap, and structured mandate clauses; written only by bin/fm-afk-contract.sh after the captain confirms the read-back, archived under afk-contracts/ at return; its presence IS the away posture in every harness + afk-contracts/ archived away-posture records: one final record per away window keyed by entry time, plus any superseded mandates from that window + .afk durable away-mode daemon flag; present = sub-supervisor may inject escalations while Pi/OMP extension supervision stands by (set by the daemon entry, cleared on user return) .watch.lock .wake-queue.lock watcher singleton and queue serialization locks .claude-autoarm.lock .claude-autoarm-epoch .claude-autoarm-failure-notified .claude-autoarm-failure-alarmed .turnend-claude-blocks .turnend-claude-blocks.lock Claude Stop auto-arm single-flight, epoch, failure-episode, attended-alarm, guard-budget, and budget-lock records; never touch .cursor-park-owner .cursor-park-owner.lock .turnend-cursor-blocks Cursor stop-hook owner record, publication and commit lock, and bounded repair-nag budget; never touch @@ -258,7 +263,7 @@ For an ordinary direct report whose endpoint is dead or metadata has no window, For a dead secondmate direct report, load `secondmate-provisioning` and reconcile only that secondmate, never its whole child tree from the main home. Each secondmate reconciles work already in its own home and then idles; recovery never authorizes it to invent work. -If away mode is present, load `/afk` and let its daemon own supervision rather than arming another cycle. +If away mode is present, load `/afk` and let the daemon own supervision rather than arming another cycle; `docs/watcher-continuity.md` owns the Pi and OMP extension handoff. Surface only captain-relevant decisions, review-ready PRs, failures, and credential needs; otherwise resume the emitted supervision protocol silently. A restart must be a non-event because durable state and live backend inventory, not conversation memory, are authoritative. @@ -480,11 +485,13 @@ Harness-aware turn-end guards are structural backstops, not permission to omit t ### Away-mode stub -Invoke the `/afk` skill when the captain says `/afk`, says they are going afk, `state/.afk` exists, an incoming message starts with `FM_INJECT_MARK`, or any `state/.subsuper-*` marker is involved. +Invoke the `/afk` skill when the captain says `/afk`, says they are going afk, `state/.afk-contract` or `state/.afk` exists, an incoming message starts with `FM_INJECT_MARK`, or any `state/.subsuper-*` marker is involved. The skill owns the daemon procedure; these safety facts remain inline: - Every current daemon injection uses the `away-supervisor` kind from `bin/fm-operational-input.sh` after `FM_OPERATIONAL_PREFIX` (U+2063 INVISIBLE SEPARATOR followed by `FIRSTMATE_OP: `), while the `/afk` skill owns legacy bare-marker compatibility. +- `state/.afk-contract` is the away posture, written only after the captain confirms the read-back of their away words; entry announces hold-for-return only, and the record's clauses are recorded, not executed, in this release. - While `state/.afk` exists, the daemon owns supervision; do not arm a separate watcher. + Pi and OMP extensions stand by while the daemon owns that cycle (`docs/watcher-continuity.md`). - A marked message while away mode is active is internal escalation and does not exit away mode. - A message beginning `/afk` refreshes away mode. - Any other unmarked message means the captain returned; load `/afk`, run the return owner, and do not process that message as ordinary work until its durable catch-up gate clears. @@ -543,7 +550,7 @@ Mention cost as a courtesy when unusually much work is running, but never block ## 10. Backlog contract -`data/backlog.md` is the durable queue. +The configured `tasks-axi` backend is the durable queue; the tracked default is `data/backlog.md`. It tracks work items only, never agents; persistent secondmates never appear as backlog items. Work routed to a secondmate is recorded in that secondmate home's own backlog, not the main backlog. A decision is simply a task held for the captain: create the task with `tasks-axi add` when needed, then always hold it through `bin/fm-captain-hold.sh hold --reason ""`, with `--until ` when the captain defers it. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f68198f4e3b..4732df85ba7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -53,7 +53,7 @@ See the [no-mistakes quick start](https://kunchenguid.github.io/no-mistakes/star - Helper scripts in `bin/` are plain bash. Each starts with a usage header comment; keep it accurate when you change behavior. Test scripts and helpers in `tests/` are plain bash too. - `bin/fm-lint.sh` must pass: it is the single owner of the lint definition (the shellcheck file set, config, pinned shellcheck version, and pinned actionlint workflow lint), and both CI and the no-mistakes pre-push gate invoke it with no arguments. + `bin/fm-lint.sh` must pass: it is the single owner of the lint definition (the shellcheck file set, config, pinned shellcheck version, pinned actionlint workflow lint, and the backend-purity check rejecting direct Beads CLI calls in core `bin/` scripts), and both CI and the no-mistakes pre-push gate invoke it with no arguments. Its header and `--help` output own the exact local lint modes, file-set selection, and analysis flags. A malformed `.github/workflows/*.yml`, including a self-broken `ci.yml`, fails that local lint path before merge because a broken workflow cannot report its own breakage. It pins one exact shellcheck version and one exact actionlint version and refuses to run under any other. diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index d554555f68d..51550ba3355 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -39,7 +39,7 @@ # behind the focused one when needed, and ends its verified lone idle shell # so Herdr removes the emptied workspace through the focus-preserving # pane-death path, with the exact pre-close tab restore as the backstop and a -# refusal to close the active tab itself. +# refusal to close the tab a live foreground client is viewing. # # Target string shape: ":", e.g. "default:w1:p2" (the # pane id itself contains a colon; the session is always the FIRST field, the @@ -379,9 +379,126 @@ fm_backend_herdr_workspace_label() { # fm_backend_herdr_version_check, which is intentionally session-independent # (reads only .client.* fields). fm_backend_herdr_cli() { # - local session=$1 + local session=$1 rc=0 err failed_bin selected_bin client_bin=herdr shift - HERDR_SESSION="$session" herdr "$@" --session "$session" + if [ "${FM_BACKEND_HERDR_CLIENT_SESSION:-}" = "$session" ]; then + client_bin=$(fm_backend_herdr_bin) + fi + # stderr is buffered (stdout streams untouched) so a protocol_mismatch + # refusal can be recognized and retried once on a compatible client; see + # "client selection" below. A failed command's stderr is replayed verbatim. + # The long-lived `server` launch is exec'd straight through: buffering its + # stderr would hold this call open for the server's whole lifetime. + if [ "${1:-}" = server ]; then + HERDR_SESSION="$session" "$client_bin" "$@" --session "$session" + return $? + fi + failed_bin=$client_bin + { err=$(HERDR_SESSION="$session" "$failed_bin" "$@" --session "$session" 2>&1 1>&3 3>&-) || rc=$?; } 3>&1 + if [ "$rc" -ne 0 ]; then + case "$err" in + *protocol_mismatch*) + fm_backend_herdr_client_select "$session" force + selected_bin=$(fm_backend_herdr_bin) + if [ "$selected_bin" != "$failed_bin" ]; then + HERDR_SESSION="$session" "$selected_bin" "$@" --session "$session" + return $? + fi + ;; + esac + fi + [ -z "$err" ] || printf '%s\n' "$err" >&2 + return "$rc" +} + +# --- client selection -------------------------------------------------------- +# +# Every operation routed through fm_backend_herdr_cli starts with the first +# `herdr` on PATH, or the client already selected for that exact session. A +# host can carry more than one herdr client (a self-updated copy in +# ~/.local/bin next to a package-managed one), and the two PATH orders +# Firstmate runs under (an interactive login shell, and the fixed remote-job +# PATH that puts ~/.local/bin first - bin/fm-remote-job-lib.sh) can then resolve +# DIFFERENT binaries. A client older than the running server is answered with +# error code protocol_mismatch on operational commands (verified: herdr 0.8.2, +# protocol 20, against a 0.9.0 server, protocol 22), which the read classifiers +# correctly refuse to interpret. +# +# The CLI retry path is reactive, never speculative: its happy path makes no +# extra call on any host, and fakes that never emit protocol_mismatch never see +# it. On that refusal fm_backend_herdr_cli asks fm_backend_herdr_client_select to +# read `status --json --session ` from the PATH-first client and, when a +# running server reports it incompatible (.server.compatible when the client +# emits it, equal .client/.server protocol otherwise), from each other +# distinct herdr on PATH in order, adopting the first one that positively +# proves compatible and retrying the command on it once. The choice is scoped +# to that session and exported as FM_BACKEND_HERDR_BIN so children inherit it. +# A later mismatch forces reselection, while another session starts from the +# PATH-first client. An unknown verdict (status supplies neither +# .server.compatible nor both client and server protocols) always keeps the +# PATH-first client. +fm_backend_herdr_bin() { + printf '%s' "${FM_BACKEND_HERDR_BIN:-herdr}" +} + +# fm_backend_herdr_client_candidates: every distinct executable named herdr on +# PATH, one per line, in PATH order (builtins only - no fork). +fm_backend_herdr_client_candidates() { + local dir candidate seen='|' old_ifs=$IFS + IFS=: + for dir in $PATH; do + [ -n "$dir" ] || dir=. + candidate="$dir/herdr" + [ -f "$candidate" ] && [ -x "$candidate" ] || continue + case "$seen" in *"|$candidate|"*) continue ;; esac + seen="$seen$candidate|" + printf '%s\n' "$candidate" + done + IFS=$old_ifs +} + +# fm_backend_herdr_client_status: one session-scoped status read of , +# printed as "|" with empty fields for anything the +# client did not report. Never fails. +fm_backend_herdr_client_status() { # + local bin=$1 session=$2 out + out=$(HERDR_SESSION="$session" "$bin" status --json --session "$session" 2>/dev/null) || out= + printf '%s' "$out" | jq -r ' + [ (if (.server | type) == "object" and .server.running != null then (.server.running | tostring) else "" end), + (if (.server | type) == "object" and (.server | has("compatible")) + then (.server.compatible | tostring) + elif (.client.protocol != null and .server.protocol != null) + then ((.client.protocol == .server.protocol) | tostring) + else "" end) ] | join("|")' 2>/dev/null \ + || printf '|' +} + +# fm_backend_herdr_client_select: resolve the client for once per +# process (pass `force` to redo it), per the contract above. +fm_backend_herdr_client_select() { # [force] + local session=$1 candidates first candidate running compatible + if [ "${2:-}" != force ]; then + [ "${FM_BACKEND_HERDR_CLIENT_SESSION:-}" != "$session" ] || return 0 + fi + FM_BACKEND_HERDR_BIN= + FM_BACKEND_HERDR_CLIENT_SESSION=$session + export FM_BACKEND_HERDR_BIN FM_BACKEND_HERDR_CLIENT_SESSION + candidates=$(fm_backend_herdr_client_candidates) + case "$candidates" in *$'\n'*) ;; *) return 0 ;; esac + first=${candidates%%$'\n'*} + IFS='|' read -r running compatible \ + <<< "$(fm_backend_herdr_client_status "$first" "$session")" + [ "$running" = true ] && [ "$compatible" = false ] || return 0 + while IFS= read -r candidate; do + [ "$candidate" != "$first" ] || continue + IFS='|' read -r running compatible \ + <<< "$(fm_backend_herdr_client_status "$candidate" "$session")" + if [ "$running" = true ] && [ "$compatible" = true ]; then + FM_BACKEND_HERDR_BIN=$candidate + return 0 + fi + done <<< "$candidates" + return 0 } # fm_backend_herdr_tool_check: refuse loudly if herdr or jq is missing. @@ -862,11 +979,56 @@ fm_backend_herdr_projection_focus_restore() { # + local session=$1 out reason + out=$(fm_backend_herdr_cli "$session" terminal title clear 2>/dev/null) || return 2 + reason=$(printf '%s' "$out" | jq -r '.result.reason // empty' 2>/dev/null) || return 2 + case "$reason" in + no_foreground_client) return 1 ;; + cleared) return 0 ;; + *) return 2 ;; + esac +} + +fm_backend_herdr_projection_target_tab_mutation_allowed() { # + local session=$1 target_tab=$2 foreground_rc=0 focus active_tab + FM_BACKEND_HERDR_PROJECTION_MUTATION_FOCUS="" + fm_backend_herdr_foreground_client_present "$session" || foreground_rc=$? + [ "$foreground_rc" -eq 1 ] && return 0 + focus=$(fm_backend_herdr_projection_focus_snapshot "$session") || return 1 + active_tab=${focus#*$'\t'} + if [ "$target_tab" != "$active_tab" ]; then + # Let the close owner preserve the live viewer's fresh non-target focus, + # rather than restoring a stale pre-planning pointer after the mutation. + FM_BACKEND_HERDR_PROJECTION_MUTATION_FOCUS=$focus + return 0 + fi + if [ "$foreground_rc" -eq 0 ]; then + echo "warning: herdr presentation cleanup target is the captain's active tab; refusing a close that cannot preserve focus" >&2 + else + echo "warning: herdr presentation cleanup could not verify whether a foreground client is viewing the target tab; refusing a focus-unsafe mutation" >&2 + fi + return 1 +} + # fm_backend_herdr_projection_close_pane_focus_preserving: close one exact # response-derived projection pane without leaving the captain focused # anywhere else. -# If the target belongs to the active tab, exact tab preservation is -# impossible, so cleanup refuses instead of changing focus. +# If the target belongs to the active tab AND a live foreground client is +# attached, exact tab preservation is impossible, so cleanup refuses instead +# of changing focus. When no live client is attached, the persisted .focused +# pointer is not a viewer, so the close proceeds; restore is skipped when the +# close destroys that persisted tab because there is no live focus to preserve. # When the close would empty the target workspace, Herdr 0.7.5's explicit # close moves focus to the workspace's neighbor, so the close is planned by # fm_backend_herdr_emptying_close_plan: reposition the doomed workspace @@ -878,6 +1040,7 @@ fm_backend_herdr_projection_focus_restore() { # [required-agent-state] local session=$1 pane_id=$2 required_agent_state=${3:-} local before active_tab info target_pane target_tab target_ws close_status state plan plan_shell_pid plan_move_record workspace_presence + local skip_restore=0 FM_BACKEND_HERDR_PROJECTION_CLOSE_AGENT_STATE="" [ -n "$pane_id" ] || return 0 before=$(fm_backend_herdr_projection_focus_snapshot "$session") || { @@ -896,20 +1059,17 @@ fm_backend_herdr_projection_close_pane_focus_preserving() { # &2 return 1 fi - if [ "$target_tab" = "$active_tab" ]; then - echo "warning: herdr presentation cleanup target is the captain's active tab; refusing a close that cannot preserve focus" >&2 - return 1 - fi if [ -n "$required_agent_state" ]; then state=$(fm_backend_herdr_pane_agent_state "$session" "$pane_id") FM_BACKEND_HERDR_PROJECTION_CLOSE_AGENT_STATE=$state [ "$state" = "$required_agent_state" ] || return 1 fi + [ "$target_tab" != "$active_tab" ] || skip_restore=1 plan=plain plan_shell_pid= plan_move_record= if [ -n "$target_ws" ]; then - plan=$(fm_backend_herdr_emptying_close_plan "$session" "$pane_id" "$target_ws" "$target_tab" "${before%%$'\t'*}") + plan=$(fm_backend_herdr_emptying_close_plan "$session" "$pane_id" "$target_ws" "$target_tab" "${before%%$'\t'*}" "$target_tab") case "$plan" in moved$'\t'*) plan_move_record=${plan%%$'\n'*} @@ -921,21 +1081,47 @@ fm_backend_herdr_projection_close_pane_focus_preserving() { # # focused one (repositioned to the end first when it does not, with the move # verified against the server-returned order and focus), and the exact pane # to hold one provably lone idle recognized shell. -fm_backend_herdr_emptying_close_plan() { # - local session=$1 pane_id=$2 ws_id=$3 tab_id=$4 focused_ws=$5 +fm_backend_herdr_emptying_close_plan() { # [guard-tab-id] + local session=$1 pane_id=$2 ws_id=$3 tab_id=$4 focused_ws=$5 guard_tab=${6:-} local tabs panes list indices r rest a len capable socket mover response move_status shell_pid before_order [ -n "$ws_id" ] && [ -n "$tab_id" ] && [ -n "$focused_ws" ] || { printf 'plain\n'; return 0; } tabs=$(fm_backend_herdr_cli "$session" tab list --workspace "$ws_id" 2>/dev/null) || { printf 'plain\n'; return 0; } @@ -1091,6 +1279,11 @@ fm_backend_herdr_emptying_close_plan() { # < } mover=${FM_BACKEND_HERDR_WORKSPACE_MOVER:-$FM_BACKEND_HERDR_ROOT/bin/backends/herdr-workspace-move.py} before_order=$(printf '%s' "$list" | jq -c '[.result.workspaces[].workspace_id]' 2>/dev/null) + if [ -n "$guard_tab" ] \ + && ! fm_backend_herdr_projection_target_tab_mutation_allowed "$session" "$guard_tab"; then + printf 'refuse\n' + return 0 + fi if response=$("$mover" "$socket" "$ws_id" "$len" 2>/dev/null); then move_status=0 else @@ -1128,8 +1321,8 @@ fm_backend_herdr_emptying_close_plan() { # < # line, or empty for a no-op when no move was attempted. # The rollback is verified against the mover's returned order and focus and # warns on any failure, so a lasting reorder is never silent. -fm_backend_herdr_emptying_move_rollback() { # - local record=$1 marker ws index socket focused order mover response +fm_backend_herdr_emptying_move_rollback() { # [session] [guard-tab-id] + local record=$1 session=${2:-} guard_tab=${3:-} marker ws index socket focused order mover response [ -n "$record" ] || return 0 IFS=$'\t' read -r marker ws index socket focused order </dev/null) \ + if { [ -n "$guard_tab" ] && ! fm_backend_herdr_projection_target_tab_mutation_allowed "$session" "$guard_tab"; } \ + || ! response=$("$mover" "$socket" "$ws" "$index" 2>/dev/null) \ || ! printf '%s' "$response" | jq -e --argjson expected "$order" --arg focused "$focused" ' .result.type == "workspace_list" and ([.result.workspaces[].workspace_id] == $expected) @@ -1165,8 +1359,8 @@ FMEOF # unless the same pid is still the pane's strict bare idle shell, so an # exited or reused pid is never signaled. # Returns 0 only when the pane is confirmed gone. -fm_backend_herdr_death_close_pane() { # - local session=$1 pane_id=$2 shell_pid=$3 ps_bin attempt max_attempts presence resampled_pid +fm_backend_herdr_death_close_pane() { # [guard-tab-id] + local session=$1 pane_id=$2 shell_pid=$3 guard_tab=${4:-} ps_bin attempt max_attempts presence resampled_pid ps_bin=${FM_HERDR_PS_BIN:-ps} case "$shell_pid" in ''|*[!0-9]*) return 1 ;; @@ -1174,6 +1368,7 @@ fm_backend_herdr_death_close_pane() { # command -v "$ps_bin" >/dev/null 2>&1 || return 1 max_attempts=${FM_BACKEND_HERDR_DEATH_CLOSE_POLLS:-40} fm_backend_herdr_pid_is_bare_shell "$ps_bin" "$shell_pid" || return 1 + [ -z "$guard_tab" ] || fm_backend_herdr_projection_target_tab_mutation_allowed "$session" "$guard_tab" || return 1 kill -HUP "$shell_pid" 2>/dev/null || true attempt=0 while [ "$attempt" -lt "$max_attempts" ]; do @@ -1188,6 +1383,7 @@ fm_backend_herdr_death_close_pane() { # resampled_pid=$(fm_backend_herdr_pane_idle_shell_sample "$session" "$pane_id") || return 1 [ "$resampled_pid" = "$shell_pid" ] || return 1 fm_backend_herdr_pid_is_bare_shell "$ps_bin" "$shell_pid" || return 1 + [ -z "$guard_tab" ] || fm_backend_herdr_projection_target_tab_mutation_allowed "$session" "$guard_tab" || return 1 kill -KILL "$shell_pid" 2>/dev/null || true attempt=0 while [ "$attempt" -lt "$max_attempts" ]; do @@ -2123,6 +2319,31 @@ fm_backend_herdr_tab_is_husk() { # esac } +# fm_backend_herdr_server_running_state: whether the named session has a running +# server, as running|stopped|unknown, read from `status --json`'s own tri-state +# `.server.running`. `status` is the one command that answers with a +# running=false BODY instead of refusing, so it works on exactly the sessions +# whose operational calls cannot be reached at all. +# +# The verdict rests on that field rather than on the `server_not_running` error +# code an operational call happens to return, because the field is version +# stable across the supported range while the code is not (verified on 0.8.2 +# protocol 20 and 0.9.0 protocol 22 - docs/verification/runtime-backends.md). +fm_backend_herdr_server_running_state() { # + local session=$1 status + command -v jq >/dev/null 2>&1 || { printf 'unknown'; return 0; } + status=$(fm_backend_herdr_cli "$session" status --json 2>/dev/null) || { + printf 'unknown' + return 0 + } + printf '%s' "$status" | jq -r ' + if .server.running == true then "running" + elif .server.running == false then "stopped" + else "unknown" + end + ' 2>/dev/null || printf 'unknown' +} + # fm_backend_herdr_agent_state: recovery-grade state for the same session-start # sweep as the tmux classifier. It reuses the husk classifier rather than # creating a second Herdr state machine: a structurally gone pane is `missing`, @@ -2158,6 +2379,20 @@ fm_backend_herdr_tab_is_husk() { # # and stays `alive`, preserving the husk classifier's # fail-safe-toward-refusal contract: only a positively proven agent-free pane # ever unlocks recovery. + +# One exception to that last case, and it is deliberately made HERE rather than +# in the husk classifier: a read can fail because the recorded session's server +# is not running at all, which is authoritative absence for every pane in that +# session rather than an ambiguous answer about one of them. Treating it as +# `unreadable` stranded tasks with no sanctioned recovery (issue #4091), so a +# positively stopped server reads `missing` instead. +# +# Only this recovery-grade read is widened. fm_backend_herdr_pane_agent_state +# and the presence classifier under it stay strict, so husk detection, duplicate +# prevention, rollback, and teardown - which can DESTROY things - keep refusing +# on exactly the reads they refused on before. A server that is running, or +# whose state cannot itself be read, still yields `unreadable` here too: absence +# is claimed only from positive evidence of it. fm_backend_herdr_agent_state() { # local target=$1 fm_backend_herdr_parse_target "$target" || { printf 'unreadable'; return 0; } @@ -2172,7 +2407,12 @@ fm_backend_herdr_agent_state() { # printf 'alive' fi ;; - *) printf 'unreadable' ;; + *) + case "$(fm_backend_herdr_server_running_state "$FM_BACKEND_HERDR_SESSION")" in + stopped) printf 'missing' ;; + *) printf 'unreadable' ;; + esac + ;; esac } @@ -2302,7 +2542,7 @@ EOF # A missing, failed, or malformed create response stays ambiguous and grants no # cleanup authority. fm_backend_herdr_projection_create_task() { # - local cwd=$1 workspace_label=$2 task_label=$3 session out tabs panes tab_count pane_count focus_before + local cwd=$1 workspace_label=$2 task_label=$3 session out tabs panes tab_count pane_count focus_before active_tab FM_BACKEND_HERDR_PROJECTION_SESSION="" FM_BACKEND_HERDR_PROJECTION_WORKSPACE_ID="" FM_BACKEND_HERDR_PROJECTION_SEEDED_TAB_ID="" @@ -2378,10 +2618,13 @@ fm_backend_herdr_projection_create_task() { # &2 return 1 fi - fm_backend_herdr_projection_focus_restore "$session" "$focus_before" "seeded-tab prune" || { - echo "error: herdr presentation seeded-tab prune did not preserve exact active focus; leaving its journal quarantined" >&2 - return 1 - } + active_tab=${focus_before#*$'\t'} + if [ "$FM_BACKEND_HERDR_PROJECTION_SEEDED_TAB_ID" != "$active_tab" ]; then + fm_backend_herdr_projection_focus_restore "$session" "$focus_before" "seeded-tab prune" || { + echo "error: herdr presentation seeded-tab prune did not preserve exact active focus; leaving its journal quarantined" >&2 + return 1 + } + fi tabs=$(fm_backend_herdr_cli "$session" tab list --workspace "$FM_BACKEND_HERDR_PROJECTION_WORKSPACE_ID" 2>/dev/null) || { echo "error: could not verify the disposable herdr presentation workspace shape" >&2 diff --git a/bin/fm-afk-contract.sh b/bin/fm-afk-contract.sh new file mode 100755 index 00000000000..593b65e3a01 --- /dev/null +++ b/bin/fm-afk-contract.sh @@ -0,0 +1,828 @@ +#!/usr/bin/env bash +# fm-afk-contract.sh - the one owner of the away-posture record: its schema, the +# mandate-clause fields and their structural check, refusal naming the missing +# part, the read-back rendering, the entry announcement, and the archive at return. +# +# POSTURE. Away mode is a posture of the one supervision session, recorded in +# state/.afk-contract and never inferred from chat. While the record exists the +# home is afk; the captain's first unmarked message archives it (the return path +# in bin/fm-afk-return.sh calls `archive` through bin/fm-afk-launch.sh stop). +# Being away changes how the captain is informed and what happens at a +# captain-owned decision point, never the authority set. Hold-for-return is the +# only reach profile this release records: there is no phone channel, and the +# entry announcement says so every time. +# +# RECORD (state/.afk-contract; written only by this script; YAML-shaped so a +# human can read it, but parsed only here - consumers use the read subcommands): +# version: 1 +# entered: +# entered_epoch: +# expected_return: | - +# reach_channels: none +# reach_announced: +# spend_max_concurrent_workers: +# confirmed: +# confirmed_epoch: +# words: | or |- the captain's words, verbatim, never edited, +# one record line per input line (or `words: -` +# ... when /afk carried no words); `|` retains a +# final newline and `|-` records its absence +# clauses: accepted clauses, recorded from the fields given +# - id: +# action: +# object: e: +# when: e: +# stop: e: | - +# flag: | - +# refused: clauses missing a part, with the part named +# - id: +# text: e: +# missing: +# A proposal (state/.afk-contract.proposed) has the same shape without the +# confirmed fields; confirmation stamps the first entry time. Archived final +# records live under state/afk-contracts/ as .afk-contract, and +# replaced mandates use -superseded-.afk-contract. +# A replacement carries the original session entry forward as the phase-1 +# fail-safe. Durable archive-chain identity and same-second session identity are +# deferred to phase 4 (fm-afk-clauses-execute-r1). +# +# CLAUSE FIELDS. A clause is given as explicit fields, one clause per --action: +# --action --object --when [--stop ] +# action one of: merge land prerelease install rerun dispatch abort-run answer +# discard wake-me. A new verb is a code change here, never a prompt change. +# object the thing the clause acts on, in the captain's words, verbatim. +# when the stated precondition, in the captain's words, verbatim. +# stop optional: what ends the clause early, verbatim. +# NO STATIC NATURAL-LANGUAGE PARSER EXISTS HERE, BY THE CAPTAIN'S MANDATE. The +# object and precondition text are recorded exactly as given and are never +# tokenized, classified, or semantically validated by this script; whether a +# precondition holds is the supervision session's judgment at execution time +# in a later phase. The structural check asserts only that the action, object, +# and precondition fields are present, and that the action is a listed verb. +# THE NEVER-SET SCAN is only a coarse best-effort structural FLAG, never a +# refusal and never the authoritative gate: a clause whose fields mention a +# listed never-set concept is still recorded, with `flag:` naming the concept +# so the read-back and the return brief show it. The scan matches a listed term +# exactly or with a plain inflection (s, es, d, ed, ing, er, ers) at +# punctuation-delimited token boundaries, so an unrelated name such as +# ping-service or tokenize-worker is never flagged, and it can miss spellings, +# with joined compounds such as oneTimeCode a known limitation. Authoritative +# never-set and forbidden-action enforcement is the supervision session's +# judgment at execution time in phase 4. +# A clause missing a required field is refused with that field named, recorded +# under refused:, read back beside the accepted list, and never executes. Ids +# are the input ordinals across accepted and refused clauses. +# THIS RELEASE RECORDS CLAUSES AND DOES NOT EXECUTE THEM: the guarded gates learn +# to cite a clause in a later phase, and the announcement and return brief both +# say so, so a recorded clause is never mistaken for a promise. +# HARD RULE: forbidden, destructive, irreversible, and security-sensitive actions +# are never pre-authorizable regardless of clause text, and no recorded clause is +# authority by itself. +# +# Usage: +# fm-afk-contract.sh propose [--words-file | --words ] +# [--action --object --when [--stop ]]... +# [--expected-return ] [--spend ] +# Compile and write the proposal, then print the read-back. Exit 0 with every +# clause accepted, 3 when at least one clause was refused (the read-back names +# the missing part), and 2 on a usage error. --words-file keeps the file's +# bytes verbatim, trailing newlines included. A refused clause remains in the +# proposal so the captain can restate it before saying go. +# fm-afk-contract.sh confirm +# Promote the proposal into the record with the confirmed timestamp and +# print the entry announcement. A proposal is required when no confirmed +# record exists; an existing record with no proposal is a no-op refresh. +# A replacement is staged before the prior record is archived and replaced. +# fm-afk-contract.sh readback [--proposal] +# fm-afk-contract.sh field [--proposal] +# fm-afk-contract.sh words [--proposal | --path ] +# fm-afk-contract.sh clauses [--proposal | --path ] TSV: id action object when stop +# fm-afk-contract.sh flags [--proposal | --path ] TSV: id concept (flagged clauses only) +# fm-afk-contract.sh validate [--proposal | --path ] exit 0 when the record is readable and, for a record, confirmed +# Backslashes and control whitespace in TSV fields use reversible escapes +# (`\\`, `\t`, `\r`, and `\n`) so every record remains one row per clause; +# a literal `-` is `\x2d` to distinguish it from the empty-stop marker. +# fm-afk-contract.sh refused [--proposal | --path ] TSV: id text missing +# fm-afk-contract.sh archive move the record aside; print its path +# fm-afk-contract.sh archived print that archived record's path +# +# Sourceable: with the BASH_SOURCE guard, other scripts get the path and +# presence helpers (fm_afk_contract_path, fm_afk_contract_present, +# fm_afk_contract_proposal_path, fm_afk_contract_archive_dir) without running main. +set -u + +FM_AFK_CONTRACT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$FM_AFK_CONTRACT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +FM_AFK_CONTRACT_STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" + +# shellcheck source=bin/fm-classify-lib.sh +. "$FM_AFK_CONTRACT_DIR/fm-classify-lib.sh" + +FM_AFK_CONTRACT_VERSION=1 +FM_AFK_CONTRACT_VERBS="merge land prerelease install rerun dispatch abort-run answer discard wake-me" +FM_AFK_CONTRACT_REACH_ANNOUNCED='No phone channel is configured; anything that needs you waits for your return.' +FM_AFK_CONTRACT_SPEND_DEFAULT=4 + +fm_afk_contract_path() { # [state-dir] + printf '%s/.afk-contract' "${1:-$FM_AFK_CONTRACT_STATE}" +} + +fm_afk_contract_proposal_path() { # [state-dir] + printf '%s/.afk-contract.proposed' "${1:-$FM_AFK_CONTRACT_STATE}" +} + +fm_afk_contract_archive_dir() { # [state-dir] + printf '%s/afk-contracts' "${1:-$FM_AFK_CONTRACT_STATE}" +} + +fm_afk_contract_present() { # [state-dir] + [ -f "$(fm_afk_contract_path "${1:-$FM_AFK_CONTRACT_STATE}")" ] +} + +fm_afk_contract_log() { printf 'fm-afk-contract: %s\n' "$*" >&2; } + +fm_afk_contract_usage() { + sed -n '/^# Usage:/,/^# Sourceable:/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//' +} + +fm_afk_contract_now_iso() { + date -u +%Y-%m-%dT%H:%M:%SZ +} + +fm_afk_contract_lower() { # + printf '%s' "$1" | tr '[:upper:]' '[:lower:]' +} + +fm_afk_contract_action() { # + fm_afk_contract_lower "$1" | tr '\t\r\n' ' ' | sed 's/^ *//; s/ *$//; s/ */ /g' +} + +fm_afk_contract_blank() { # + [ -z "$(printf '%s' "$1" | tr -d '[:space:]')" ] +} + +fm_afk_contract_escape() { # + local value=$1 + value=${value//\\/\\\\} + value=${value//$'\t'/\\t} + value=${value//$'\r'/\\r} + value=${value//$'\n'/\\n} + [ "$value" != - ] || value='\x2d' + printf '%s' "$value" +} + +fm_afk_contract_unescape() { # + printf '%b' "$1" +} + +# --- clause structural check and never-set scan ------------------------------ + +# fm_afk_contract_never_set_hit : prints the protected concept the +# text mentions, or nothing. This coarse best-effort structural flag lowercases +# and splits punctuation before checking fixed token stems. It is not authoritative, +# can miss joined compounds such as oneTimeCode, and does not understand language; +# phase-4 supervision judgment owns never-set and forbidden-action enforcement. +fm_afk_contract_never_set_hit() { # + local normalized concept matched i j + local -a tokens stems concepts=( + credential password passcode login signin otp totp hotp 2fa mfa token secret + passphrase apikey legal financial payment invoice pin + 'log in' 'sign in' 'attended prompt' 'one time code' 'one time password' + 'one time passcode' 'verification code' 'security code' 'auth code' + 'authentication code' 'recovery code' 'backup code' 'api key' 'access token' + 'secret key' 'private key' + ) + normalized=$(printf '%s ' "$@" | tr '[:upper:]' '[:lower:]' | sed 's/[^[:alnum:]]/ /g; s/ */ /g') + read -r -a tokens <<< "$normalized" + for concept in "${concepts[@]}"; do + read -r -a stems <<< "$concept" + for ((i = 0; i + ${#stems[@]} <= ${#tokens[@]}; i++)); do + matched=1 + for ((j = 0; j < ${#stems[@]}; j++)); do + case "${tokens[$((i + j))]}" in + "${stems[$j]}"|"${stems[$j]}s"|"${stems[$j]}es"|"${stems[$j]}d"|"${stems[$j]}ed"|"${stems[$j]}ing"|"${stems[$j]}er"|"${stems[$j]}ers") ;; + *) matched=0; break ;; + esac + done + if [ "$matched" -eq 1 ]; then + printf '%s' "$concept" + return 0 + fi + done + done + return 1 +} + +# Check one clause's fields. Sets C_ACTION C_OBJECT C_WHEN C_STOP; on refusal +# C_MISSING names the missing field and the reason. The fields are never parsed: +# presence, the listed verb, and the coarse best-effort flag are the whole check. +fm_afk_contract_clause_check() { # + C_ACTION=$(fm_afk_contract_action "$1") + C_OBJECT=$2 + C_WHEN=$3 + C_STOP=$4 + C_MISSING= + C_FLAG=$(fm_afk_contract_never_set_hit "$C_ACTION" "$C_OBJECT" "$C_WHEN" "$C_STOP") || C_FLAG= + if [ -z "$C_ACTION" ]; then + C_MISSING='action - the clause names no action' + return 1 + fi + case " $FM_AFK_CONTRACT_VERBS " in + *" $C_ACTION "*) ;; + *) + C_MISSING="action - '$C_ACTION' is not a mandate verb (one of: ${FM_AFK_CONTRACT_VERBS// /, })" + return 1 ;; + esac + if fm_afk_contract_blank "$C_OBJECT"; then + C_MISSING='object - the clause names no thing to act on' + return 1 + fi + if fm_afk_contract_blank "$C_WHEN"; then + C_MISSING='when - the clause states no precondition' + return 1 + fi + if [ "$5" -eq 1 ] && fm_afk_contract_blank "$C_STOP"; then + C_MISSING='stop - --stop was given with no text' + return 1 + fi + return 0 +} + +# The refused list keeps the fields exactly as given, so the captain sees what +# was refused; an absent field reads as "(none)". +fm_afk_contract_clause_as_given() { # + local text + text="action=${1:-(none)} object=${2:-(none)} when=${3:-(none)}" + [ "$5" -eq 0 ] || text="$text stop=${4:-(none)}" + printf '%s' "$text" +} + +# --- record writing --------------------------------------------------------- + +fm_afk_contract_validate_iso() { # + fm_utc_iso_to_epoch "$1" >/dev/null 2>&1 +} + +# Compile every input into a record body on stdout (everything except the +# confirmed fields). Inputs: WORDS (verbatim), the parallel clause field arrays +# CLAUSE_ACTIONS CLAUSE_OBJECTS CLAUSE_WHENS CLAUSE_STOPS, EXPECTED_RETURN, SPEND. +fm_afk_contract_render_body() { # + local entered=$1 entered_epoch=$2 ordinal=0 i as_given + local accepted_block="" refused_block="" + i=0 + while [ "$i" -lt "${#CLAUSE_ACTIONS[@]}" ]; do + ordinal=$((ordinal + 1)) + if fm_afk_contract_clause_check "${CLAUSE_ACTIONS[$i]}" "${CLAUSE_OBJECTS[$i]}" "${CLAUSE_WHENS[$i]}" "${CLAUSE_STOPS[$i]}" "${CLAUSE_STOP_GIVENS[$i]}"; then + accepted_block="$accepted_block$(printf ' - id: %s\n action: %s\n object: e:%s\n when: e:%s\n' \ + "$ordinal" "$C_ACTION" "$(fm_afk_contract_escape "$C_OBJECT")" "$(fm_afk_contract_escape "$C_WHEN")" + if [ -n "$C_STOP" ]; then + printf ' stop: e:%s\n' "$(fm_afk_contract_escape "$C_STOP")" + else + printf ' stop: -\n' + fi + if [ -n "$C_FLAG" ]; then + printf ' flag: %s' "$C_FLAG" + else + printf ' flag: -' + fi) +" + else + as_given=$(fm_afk_contract_clause_as_given "${CLAUSE_ACTIONS[$i]}" "${CLAUSE_OBJECTS[$i]}" "${CLAUSE_WHENS[$i]}" "${CLAUSE_STOPS[$i]}" "${CLAUSE_STOP_GIVENS[$i]}"; printf x) + as_given=${as_given%x} + refused_block="$refused_block$(printf ' - id: %s\n text: e:%s\n missing: %s' \ + "$ordinal" "$(fm_afk_contract_escape "$as_given")" "$C_MISSING") +" + fi + i=$((i + 1)) + done + printf 'version: %s\n' "$FM_AFK_CONTRACT_VERSION" + printf 'entered: %s\n' "$entered" + printf 'entered_epoch: %s\n' "$entered_epoch" + printf 'expected_return: %s\n' "${EXPECTED_RETURN:--}" + printf 'reach_channels: none\n' + printf 'reach_announced: %s\n' "$FM_AFK_CONTRACT_REACH_ANNOUNCED" + printf 'spend_max_concurrent_workers: %s\n' "${SPEND:-$FM_AFK_CONTRACT_SPEND_DEFAULT}" + if [ -n "$WORDS" ]; then + local words_body=$WORDS words_indicator='|-' + case "$words_body" in + *$'\n') words_indicator='|'; words_body=${words_body%$'\n'} ;; + esac + printf 'words: %s\n' "$words_indicator" + printf '%s\n' "$words_body" | sed 's/^/ /' + else + printf 'words: -\n' + fi + printf 'clauses:\n' + [ -z "$accepted_block" ] || printf '%s' "$accepted_block" + printf 'refused:\n' + [ -z "$refused_block" ] || printf '%s' "$refused_block" +} + +fm_afk_contract_write_atomic() { # (content on stdin) + local path=$1 pending + mkdir -p "$(dirname "$path")" || return 1 + pending=$(mktemp "$(dirname "$path")/.afk-contract.pending.XXXXXX") || return 1 + if ! cat > "$pending"; then + rm -f "$pending" + return 1 + fi + mv "$pending" "$path" || { rm -f "$pending"; return 1; } +} + +# --- record reading (the only parser) -------------------------------------- + +fm_afk_contract_read_field() { # + local path=$1 name=$2 + [ -f "$path" ] || return 1 + sed -n "s/^${name}: //p" "$path" | head -1 +} + +fm_afk_contract_read_words() { # + local path=$1 + [ -f "$path" ] || return 1 + awk -v record="$path" ' + function die(reason) { + printf "fm-afk-contract: record %s has an invalid words block: %s\n", record, reason > "/dev/stderr" + bad = 1 + exit 2 + } + /^words: \|$/ && !found { found = inwords = 1; keep_final = 1; next } + /^words: \|-$/ && !found { found = inwords = 1; keep_final = 0; next } + /^words: -$/ && !found { found = scalar = 1; next } + !found { next } + $0 == "clauses:" { + if (inwords && count == 0) die("the block indicator has no stored lines") + done = 1 + exit + } + inwords && /^ / { lines[++count] = substr($0, 3); next } + { die("a stored line lacks its two-space record prefix") } + END { + if (bad) exit 2 + if (!found) die("the words field is missing") + if (!done) die("the clauses section does not follow the words field") + for (i = 1; i <= count; i++) { + printf "%s", lines[i] + if (i < count || keep_final) printf "\n" + } + } + ' "$path" +} + +# TSV rows for a list section:
is clauses or refused. +fm_afk_contract_read_list() { #
+ local path=$1 section=$2 + [ -f "$path" ] || return 1 + awk -v want="$section" -v verbs="$FM_AFK_CONTRACT_VERBS" -v record="$path" ' + function row_name() { return (id != "" ? id : ordinal + 1) } + function die(part) { + printf "fm-afk-contract: record %s has malformed %s row %s: missing or invalid %s\n", record, section, row_name(), part > "/dev/stderr" + bad = 1 + exit 2 + } + function valid_action(value, values, count, i) { + count = split(verbs, values, " ") + for (i = 1; i <= count; i++) if (value == values[i]) return 1 + return 0 + } + function flush() { + if (!active) return + if (section == "clauses") { + if (state < 1 || id !~ /^[0-9]+$/) die("id") + if (state < 2 || !valid_action(action)) die("action") + if (state < 3) die("object") + if (state < 4) die("when") + if (state < 5) die("stop") + if (state < 6 || flag == "") die("flag") + if (want == "clauses") printf "%s\t%s\t%s\t%s\t%s\n", id, action, object, when, stop + else if (flag != "-") printf "%s\t%s\n", id, flag + } else { + if (state < 1 || id !~ /^[0-9]+$/) die("id") + if (state < 2) die("text") + if (state < 3 || missing == "") die("missing") + printf "%s\t%s\t%s\n", id, text, missing + } + ordinal++ + active = 0 + state = 0 + id = action = object = when = stop = text = missing = flag = "" + } + BEGIN { section = (want == "flags") ? "clauses" : want } + $0 == section ":" && !found { found = insection = 1; next } + insection && /^[^ ]/ { flush(); done = 1; exit } + !insection { next } + /^ - id: / { + flush() + active = 1 + id = substr($0, 9) + state = 1 + next + } + section == "clauses" && state == 1 && /^ action: / { action = substr($0, 13); state = 2; next } + section == "clauses" && state == 2 && /^ object: e:/ { object = substr($0, 15); state = 3; next } + section == "clauses" && state == 3 && /^ when: e:/ { when = substr($0, 13); state = 4; next } + section == "clauses" && state == 4 && /^ stop: e:/ { stop = substr($0, 13); state = 5; next } + section == "clauses" && state == 4 && /^ stop: -$/ { stop = "-"; state = 5; next } + section == "clauses" && state == 5 && /^ flag: / { flag = substr($0, 11); state = 6; next } + section == "refused" && state == 1 && /^ text: e:/ { text = substr($0, 13); state = 2; next } + section == "refused" && state == 2 && /^ missing: / { missing = substr($0, 14); state = 3; next } + { die(section == "clauses" ? (state == 1 ? "action" : state == 2 ? "object" : state == 3 ? "when" : state == 4 ? "stop" : state == 5 ? "flag" : "row") : (state == 1 ? "text" : state == 2 ? "missing" : "row")) } + END { + if (bad) exit 2 + if (!done) flush() + if (!found) { + printf "fm-afk-contract: record %s lacks its %s section\n", record, section > "/dev/stderr" + exit 2 + } + } + ' "$path" +} + +# A record is valid when its version is the one this script writes and the +# required scalar fields are present. Refuses rather than guessing at a foreign +# schema. +fm_afk_contract_validate() { # + local path=$1 require_confirmed=$2 version entered entered_epoch expected reach announced spend words_header confirmed + local clause_rows refused_rows clause refused id object when stop text decoded + [ -f "$path" ] || return 1 + version=$(fm_afk_contract_read_field "$path" version) + [ "$version" = "$FM_AFK_CONTRACT_VERSION" ] || { + fm_afk_contract_log "record $path carries version '${version:-none}', expected $FM_AFK_CONTRACT_VERSION; refusing to read it" + return 1 + } + entered=$(fm_afk_contract_read_field "$path" entered) + fm_afk_contract_validate_iso "$entered" || { fm_afk_contract_log "record $path has no valid entered time"; return 1; } + entered_epoch=$(fm_afk_contract_read_field "$path" entered_epoch) + case "$entered_epoch" in ''|*[!0-9]*) fm_afk_contract_log "record $path has no entered_epoch"; return 1 ;; esac + expected=$(fm_afk_contract_read_field "$path" expected_return) + [ "$expected" = - ] || fm_afk_contract_validate_iso "$expected" || { fm_afk_contract_log "record $path has no valid expected_return"; return 1; } + reach=$(fm_afk_contract_read_field "$path" reach_channels) + [ "$reach" = none ] || { fm_afk_contract_log "record $path has no valid reach_channels"; return 1; } + announced=$(fm_afk_contract_read_field "$path" reach_announced) + [ -n "$announced" ] || { fm_afk_contract_log "record $path has no reach announcement"; return 1; } + spend=$(fm_afk_contract_read_field "$path" spend_max_concurrent_workers) + case "$spend" in ''|*[!0-9]*|0) fm_afk_contract_log "record $path has no valid spend cap"; return 1 ;; esac + words_header=$(sed -n '/^words: /{p;q;}' "$path") + case "$words_header" in 'words: -'|'words: |'|'words: |-') ;; *) fm_afk_contract_log "record $path has no valid words field"; return 1 ;; esac + fm_afk_contract_read_words "$path" >/dev/null || return 1 + if [ "$require_confirmed" -eq 1 ]; then + confirmed=$(fm_afk_contract_read_field "$path" confirmed) + fm_afk_contract_validate_iso "$confirmed" || { fm_afk_contract_log "record $path has no valid confirmed time"; return 1; } + case "$(fm_afk_contract_read_field "$path" confirmed_epoch)" in + ''|*[!0-9]*) fm_afk_contract_log "record $path was never confirmed"; return 1 ;; + esac + fi + if ! clause_rows=$(fm_afk_contract_read_list "$path" clauses); then + return 1 + fi + while IFS= read -r clause; do + [ -n "$clause" ] || continue + id=$(printf '%s' "$clause" | cut -f1) + object=$(printf '%s' "$clause" | cut -f3) + when=$(printf '%s' "$clause" | cut -f4) + stop=$(printf '%s' "$clause" | cut -f5) + decoded=$(fm_afk_contract_unescape "$object"; printf x) + decoded=${decoded%x} + if fm_afk_contract_blank "$decoded"; then + fm_afk_contract_log "record $path has malformed clauses row $id: missing or invalid object" + return 1 + fi + decoded=$(fm_afk_contract_unescape "$when"; printf x) + decoded=${decoded%x} + if fm_afk_contract_blank "$decoded"; then + fm_afk_contract_log "record $path has malformed clauses row $id: missing or invalid when" + return 1 + fi + if [ "$stop" != - ]; then + decoded=$(fm_afk_contract_unescape "$stop"; printf x) + decoded=${decoded%x} + if fm_afk_contract_blank "$decoded"; then + fm_afk_contract_log "record $path has malformed clauses row $id: missing or invalid stop" + return 1 + fi + fi + done < + local path=$1 title=$2 words count id action object when stop text missing expected spend flag + expected=$(fm_afk_contract_read_field "$path" expected_return) + spend=$(fm_afk_contract_read_field "$path" spend_max_concurrent_workers) + printf '%s\n' "$title" + printf ' entered: %s\n' "$(fm_afk_contract_read_field "$path" entered)" + printf ' expected return: %s\n' "$( [ "$expected" = - ] && printf 'not given' || printf '%s' "$expected")" + printf ' spend cap: %s concurrent workers\n' "$spend" + printf ' reach: hold-for-return only. %s\n' "$(fm_afk_contract_read_field "$path" reach_announced)" + words=$(fm_afk_contract_read_words "$path"; printf x) + words=${words%x} + if [ -n "$words" ]; then + printf ' your words (verbatim):\n' + printf '%s' "$words" | sed 's/^/ /' + case "$words" in *$'\n') ;; *) printf '\n' ;; esac + else + printf ' your words: (none)\n' + fi + printf ' accepted clauses:\n' + count=0 + while IFS="$(printf '\t')" read -r id action object when stop; do + [ -n "$id" ] || continue + count=$((count + 1)) + printf ' %s. %s ' "$id" "$action" + fm_afk_contract_unescape "$object" + printf ' when ' + fm_afk_contract_unescape "$when" + if [ "$stop" != - ]; then + printf ' stop ' + fm_afk_contract_unescape "$stop" + fi + flag=$(fm_afk_contract_read_list "$path" flags | awk -F '\t' -v id="$id" '$1 == id { print $2 }') + [ -z "$flag" ] || printf " - flagged: names '%s', a never-set concept that is never pre-authorizable; recorded, judged at execution" "$flag" + printf '\n' + done <<EOF +$(fm_afk_contract_read_list "$path" clauses) +EOF + [ "$count" -gt 0 ] || printf ' (none)\n' + printf ' refused clauses:\n' + count=0 + while IFS="$(printf '\t')" read -r id text missing; do + [ -n "$id" ] || continue + count=$((count + 1)) + printf ' %s. "' "$id" + fm_afk_contract_unescape "$text" + printf '" - refused: missing %s\n' "$missing" + done <<EOF +$(fm_afk_contract_read_list "$path" refused) +EOF + [ "$count" -gt 0 ] || printf ' (none)\n' + printf ' everything else waits for your return: no red merge without its named check, no discard without a named object and condition, never credentials, legal, financial, or attended prompts, nothing by analogy, and every clause expires at return.\n' + printf ' hard rule: forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text; no recorded clause is authority by itself.\n' + printf ' recorded clauses are held for the return brief and are not executed by this release.\n' +} + +fm_afk_contract_render_announcement() { # <path> + local path=$1 accepted refused flagged expected clause_text + accepted=$(fm_afk_contract_read_list "$path" clauses | grep -c . || true) + refused=$(fm_afk_contract_read_list "$path" refused | grep -c . || true) + flagged=$(fm_afk_contract_read_list "$path" flags | grep -c . || true) + expected=$(fm_afk_contract_read_field "$path" expected_return) + if [ "$accepted" -eq 0 ] && [ "$refused" -eq 0 ]; then + clause_text='No mandate clauses recorded. Forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, and no recorded clause is authority by itself.' + else + clause_text="$accepted mandate clause(s) recorded, $refused refused, and $flagged flagged as naming a never-set concept; recorded clauses are held for the return brief and are not executed by this release; forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, and no recorded clause is authority by itself." + fi + printf 'Away posture confirmed at %s: hold-for-return only. %s %s Expected return: %s. Spend cap: %s concurrent workers.\n' \ + "$(fm_afk_contract_read_field "$path" confirmed)" \ + "$(fm_afk_contract_read_field "$path" reach_announced)" \ + "$clause_text" \ + "$( [ "$expected" = - ] && printf 'not given' || printf '%s' "$expected")" \ + "$(fm_afk_contract_read_field "$path" spend_max_concurrent_workers)" +} + +# --- subcommands ------------------------------------------------------------ + +fm_afk_contract_parse_inputs() { # <args...>; sets WORDS, the CLAUSE_* arrays, EXPECTED_RETURN, SPEND + local words_file='' open=-1 + WORDS=; EXPECTED_RETURN=-; SPEND=$FM_AFK_CONTRACT_SPEND_DEFAULT + CLAUSE_ACTIONS=(); CLAUSE_OBJECTS=(); CLAUSE_WHENS=(); CLAUSE_STOPS=(); CLAUSE_STOP_GIVENS=() + while [ "$#" -gt 0 ]; do + case "$1" in + --words-file) + [ "$#" -gt 1 ] || { fm_afk_contract_log '--words-file requires a path'; return 2; } + words_file=$2 + shift 2 ;; + --words) + [ "$#" -gt 1 ] || { fm_afk_contract_log '--words requires text'; return 2; } + WORDS=$2 + shift 2 ;; + --action) + [ "$#" -gt 1 ] || { fm_afk_contract_log '--action requires a verb; it opens a clause for the --object, --when, and --stop that follow it'; return 2; } + CLAUSE_ACTIONS+=("$2"); CLAUSE_OBJECTS+=(''); CLAUSE_WHENS+=(''); CLAUSE_STOPS+=(''); CLAUSE_STOP_GIVENS+=(0) + open=$(( ${#CLAUSE_ACTIONS[@]} - 1 )) + shift 2 ;; + --object|--when|--stop) + [ "$#" -gt 1 ] || { fm_afk_contract_log "$1 requires text"; return 2; } + [ "$open" -ge 0 ] || { fm_afk_contract_log "$1 must follow the --action that opens its clause"; return 2; } + case "$1" in + --object) CLAUSE_OBJECTS[open]=$2 ;; + --when) CLAUSE_WHENS[open]=$2 ;; + --stop) CLAUSE_STOPS[open]=$2; CLAUSE_STOP_GIVENS[open]=1 ;; + esac + shift 2 ;; + --expected-return) + [ "$#" -gt 1 ] || { fm_afk_contract_log '--expected-return requires a UTC ISO 8601 time'; return 2; } + if ! fm_afk_contract_validate_iso "$2"; then + fm_afk_contract_log "--expected-return must be UTC ISO 8601 (YYYY-MM-DDTHH:MM[:SS]Z), got '$2'" + return 2 + fi + EXPECTED_RETURN=$2 + shift 2 ;; + --spend) + [ "$#" -gt 1 ] || { fm_afk_contract_log '--spend requires a positive integer'; return 2; } + case "$2" in ''|*[!0-9]*|0) fm_afk_contract_log "--spend must be a positive integer, got '$2'"; return 2 ;; esac + SPEND=$2 + shift 2 ;; + *) + fm_afk_contract_log "unknown option '$1'" + return 2 ;; + esac + done + if [ -n "$words_file" ]; then + [ -f "$words_file" ] || { fm_afk_contract_log "words file not found: $words_file"; return 2; } + # Command substitution strips trailing newlines; the sentinel keeps the + # file's bytes verbatim, trailing newlines included. + WORDS=$(cat "$words_file"; printf x) || return 1 + WORDS=${WORDS%x} + fi + return 0 +} + +fm_afk_contract_cmd_propose() { + local entered entered_epoch proposal rc=0 refused + fm_afk_contract_parse_inputs "$@" || return 2 + entered=$(fm_afk_contract_now_iso) + entered_epoch=$(date +%s) + proposal=$(fm_afk_contract_proposal_path) + fm_afk_contract_render_body "$entered" "$entered_epoch" | fm_afk_contract_write_atomic "$proposal" || { + fm_afk_contract_log "failed to write the proposal at $proposal" + return 1 + } + refused=$(fm_afk_contract_read_list "$proposal" refused | grep -c . || true) + [ "$refused" -eq 0 ] || rc=3 + fm_afk_contract_render_readback "$proposal" 'Away posture read-back (proposed, not yet confirmed):' + printf 'Say go to confirm; restate any refused clause first if you want it recorded.\n' + return "$rc" +} + +fm_afk_contract_archive_target() { # <record> [superseded-stamp] + local record=$1 stamp=${2:-} dir entered_epoch target + dir=$(fm_afk_contract_archive_dir) + mkdir -p "$dir" || return 1 + entered_epoch=$(fm_afk_contract_read_field "$record" entered_epoch) + case "$entered_epoch" in ''|*[!0-9]*) entered_epoch=$(date +%s) ;; esac + if [ -n "$stamp" ]; then + target="$dir/$entered_epoch-superseded-$stamp.afk-contract" + [ ! -e "$target" ] || target="$dir/$entered_epoch-superseded-$stamp-$$.afk-contract" + else + target="$dir/$entered_epoch.afk-contract" + fi + printf '%s\n' "$target" +} + +fm_afk_contract_cmd_confirm() { + local record proposal body confirmed confirmed_epoch archived archived_tmp staged session_entered session_entered_epoch + record=$(fm_afk_contract_path) + proposal=$(fm_afk_contract_proposal_path) + confirmed=$(fm_afk_contract_now_iso) + confirmed_epoch=$(date +%s) + if [ -f "$proposal" ]; then + fm_afk_contract_validate "$proposal" 0 || return 1 + body=$(cat "$proposal") + elif [ -f "$record" ]; then + fm_afk_contract_validate "$record" 1 || return 1 + fm_afk_contract_log "away posture already recorded at $(fm_afk_contract_read_field "$record" entered); nothing to confirm" + fm_afk_contract_render_announcement "$record" + return 0 + else + fm_afk_contract_log "no away-posture proposal exists; run propose before confirm" + return 1 + fi + session_entered=$confirmed + session_entered_epoch=$confirmed_epoch + if [ -f "$record" ]; then + session_entered=$(fm_afk_contract_read_field "$record" entered) + session_entered_epoch=$(fm_afk_contract_read_field "$record" entered_epoch) + fi + staged=$(mktemp "$(dirname "$record")/.afk-contract.confirming.XXXXXX") || return 1 + { + printf '%s\n' "$body" | awk -v entered="$session_entered" -v epoch="$session_entered_epoch" ' + /^entered: / { print "entered: " entered; next } + /^entered_epoch: / { print "entered_epoch: " epoch; next } + /^words: / { exit } + { print } + ' + printf 'confirmed: %s\nconfirmed_epoch: %s\n' "$confirmed" "$confirmed_epoch" + printf '%s\n' "$body" | awk 'p{print} /^words: /{p=1; print}' + } > "$staged" || { rm -f "$staged"; return 1; } + fm_afk_contract_validate "$staged" 1 || { rm -f "$staged"; return 1; } + if [ -f "$record" ]; then + archived=$(fm_afk_contract_archive_target "$record" "$confirmed_epoch") || { rm -f "$staged"; return 1; } + # Copy into a temporary name first and rename atomically, so a failed copy + # never leaves a partial archive at a glob-visible name. + archived_tmp=$(mktemp "$(dirname "$archived")/.afk-contract.archiving.XXXXXX") || { rm -f "$staged"; return 1; } + if ! cp -p "$record" "$archived_tmp" || ! mv "$archived_tmp" "$archived"; then + rm -f "$staged" "$archived_tmp" + return 1 + fi + fi + mv "$staged" "$record" || { + rm -f "$staged" + [ -z "${archived:-}" ] || rm -f "$archived" + return 1 + } + if [ -n "${archived:-}" ]; then + fm_afk_contract_log "replaced the earlier away posture; its record is archived at $archived" + fi + rm -f "$proposal" + fm_afk_contract_render_announcement "$record" +} + +fm_afk_contract_cmd_archive() { + local record target + record=$(fm_afk_contract_path) + [ -f "$record" ] || return 0 + if ! fm_afk_contract_validate "$record" 1; then + fm_afk_contract_log "confirmed away-posture record at $record is invalid; refusing to archive" + return 1 + fi + target=$(fm_afk_contract_archive_target "$record") || return 1 + mv "$record" "$target" || return 1 + printf '%s\n' "$target" +} + +fm_afk_contract_select_path() { # <args...> -> prints the record path chosen by --proposal/--path + local path + path=$(fm_afk_contract_path) + while [ "$#" -gt 0 ]; do + case "$1" in + --proposal) path=$(fm_afk_contract_proposal_path); shift ;; + --path) [ "$#" -gt 1 ] || return 2; path=$2; shift 2 ;; + *) return 2 ;; + esac + done + printf '%s' "$path" +} + +fm_afk_contract_main() { + local cmd=${1:-} path + [ -n "$cmd" ] || { fm_afk_contract_usage >&2; return 2; } + shift + case "$cmd" in + propose) fm_afk_contract_cmd_propose "$@" ;; + confirm) [ "$#" -eq 0 ] || { fm_afk_contract_usage >&2; return 2; }; fm_afk_contract_cmd_confirm ;; + readback) + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + [ -f "$path" ] || { fm_afk_contract_log "no record at $path"; return 1; } + if [ "$path" = "$(fm_afk_contract_proposal_path)" ]; then + fm_afk_contract_render_readback "$path" 'Away posture read-back (proposed, not yet confirmed):' + else + fm_afk_contract_render_readback "$path" 'Away posture (confirmed):' + fi ;; + field) + [ "$#" -ge 1 ] || { fm_afk_contract_usage >&2; return 2; } + local name=$1; shift + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + fm_afk_contract_read_field "$path" "$name" ;; + words) + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + fm_afk_contract_read_words "$path" ;; + clauses) + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + fm_afk_contract_read_list "$path" clauses ;; + flags) + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + fm_afk_contract_read_list "$path" flags ;; + validate) + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + if [ "$path" = "$(fm_afk_contract_proposal_path)" ]; then + fm_afk_contract_validate "$path" 0 + else + fm_afk_contract_validate "$path" 1 + fi ;; + refused) + path=$(fm_afk_contract_select_path "$@") || { fm_afk_contract_usage >&2; return 2; } + fm_afk_contract_read_list "$path" refused ;; + archive) fm_afk_contract_cmd_archive ;; + archived) + [ "$#" -eq 1 ] || { fm_afk_contract_usage >&2; return 2; } + path="$(fm_afk_contract_archive_dir)/$1.afk-contract" + [ -f "$path" ] || { fm_afk_contract_log "no archived record for entered_epoch $1"; return 1; } + printf '%s\n' "$path" ;; + -h|--help|help) fm_afk_contract_usage ;; + *) fm_afk_contract_usage >&2; return 2 ;; + esac +} + +if [ "${BASH_SOURCE[0]}" = "${0}" ]; then + fm_afk_contract_main "$@" +fi diff --git a/bin/fm-afk-launch.sh b/bin/fm-afk-launch.sh index 5df2a9d9915..dc8a619f169 100755 --- a/bin/fm-afk-launch.sh +++ b/bin/fm-afk-launch.sh @@ -1,9 +1,24 @@ #!/usr/bin/env bash -# fm-afk-launch.sh - the single owner of the away-mode daemon TERMINAL lifecycle: -# launch it in a NON-VISIBLE tracked terminal per backend, record its exact id, -# tear it down by that exact id, and reconcile a leaked one after a crash. +# fm-afk-launch.sh - the single owner of away-mode ENTRY and EXIT: the +# read-back-and-confirm entry that writes the away-posture record through +# bin/fm-afk-contract.sh, and the away-mode daemon TERMINAL lifecycle where a +# daemon still runs: launch it in a NON-VISIBLE tracked terminal per backend, +# record its exact id, tear it down by that exact id, and reconcile a leaked one +# after a crash. # -# Why this exists (docs/herdr-backend.md "Away-mode daemon terminal launch"): +# ENTRY (the posture record). `/afk [words]` is two steps so the captain hears +# the mandate back before it binds: `propose` compiles the words and clauses +# into a proposal and prints the read-back (bin/fm-afk-contract.sh owns the +# clause fields, the never-set, the refusal wording, and the record schema); `confirm` promotes it +# into state/.afk-contract and prints the entry announcement (hold-for-return +# only: no phone channel exists). The record is the posture in every harness. +# Every harness retains daemon-backed away supervision. Pi and OMP extensions +# stand down while state/.afk exists; docs/watcher-continuity.md owns that handoff. +# `start` and `start-native` require the confirmed record before daemon launch. +# `stop` (the return, driven by bin/fm-afk-return.sh) shuts the daemon down, +# clears state/.afk last, and archives the record under state/afk-contracts/. +# +# Why the terminal lifecycle exists (docs/herdr-backend.md "Away-mode daemon terminal launch"): # bin/fm-afk-start.sh execs the supervise daemon in the FOREGROUND of whatever # terminal it is already in. Harnesses with a native in-pane tracked-background # tool (claude, grok) run it there directly and it is fine. A harness with NO @@ -20,6 +35,16 @@ # FM_SUPERVISOR_TARGET/FM_SUPERVISOR_BACKEND explicitly. # # Usage: +# fm-afk-launch.sh propose [--words-file <path> | --words <text>] +# [--action <verb> --object <text> --when <text> [--stop <text>]]... +# [--expected-return <UTC ISO 8601>] [--spend <n>] +# Record the captain's away words and mandate +# clause fields into a proposal and print the +# read-back. Exit 3 when a clause was refused (its +# missing part is named in the read-back); the +# proposal still records it as refused. +# fm-afk-launch.sh confirm Promote the required proposal and print the entry +# announcement; daemon launch follows. # fm-afk-launch.sh start Capture the captain pane, then (unless the daemon # is already running) launch the daemon in a fresh # non-visible terminal for the detected backend and @@ -32,7 +57,7 @@ # fm-afk-launch.sh stop Correct-ordered exit: SIGTERM the daemon so its # cleanup flushes WHILE state/.afk is still present, # wait for it, close the recorded terminal by exact -# id, then clear state/.afk last. +# id, clear state/.afk, then archive the record last. # fm-afk-launch.sh reconcile Close a recorded-but-dead daemon terminal by exact # id and drop the record (recovery after a crash). # @@ -86,6 +111,11 @@ FM_AFK_LAUNCH_WS_LABEL="firstmate-afk-daemon" # shellcheck source=bin/fm-afk-start.sh . "$FM_AFK_LAUNCH_DIR/fm-afk-start.sh" set +e +# The away-posture record owner; sourced for its path helpers, driven as a +# command for every record mutation so its output reaches the captain. +# shellcheck source=bin/fm-afk-contract.sh +. "$FM_AFK_LAUNCH_DIR/fm-afk-contract.sh" +FM_AFK_CONTRACT_CMD="$FM_AFK_LAUNCH_DIR/fm-afk-contract.sh" fm_afk_launch_log() { printf 'fm-afk-launch: %s\n' "$*" >&2; } @@ -146,7 +176,43 @@ fm_afk_launch_lock_release() { } fm_afk_launch_usage() { - sed -n '2,34p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + sed -n '/^# Usage:/,/^# Supported backends:/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//' +} + +fm_afk_launch_primary_harness() { + "$FM_AFK_LAUNCH_DIR/fm-harness.sh" 2>/dev/null || printf unknown +} + + +fm_afk_launch_catchup_pending() { + if [ -e "$FM_AFK_LAUNCH_STATE/.afk-return-catchup" ]; then + fm_afk_launch_log "return catch-up is still pending; run bin/fm-afk-return.sh check before re-entering away mode" + return 0 + fi + return 1 +} + +fm_afk_launch_record_require() { + local record + record=$(fm_afk_contract_path "$FM_AFK_LAUNCH_STATE") + if ! fm_afk_contract_present "$FM_AFK_LAUNCH_STATE"; then + fm_afk_launch_log "a confirmed away-posture record is required; run propose and confirm before starting the daemon" + return 1 + fi + fm_afk_contract_validate "$record" 1 || { + fm_afk_launch_log "the away-posture record is not confirmed; run confirm before starting the daemon" + return 1 + } +} + +fm_afk_launch_propose() { + fm_afk_launch_catchup_pending && return 1 + "$FM_AFK_CONTRACT_CMD" propose "$@" +} + +fm_afk_launch_confirm() { + fm_afk_launch_catchup_pending && return 1 + "$FM_AFK_CONTRACT_CMD" confirm } # The command run inside the created terminal. Real launch runs the shared @@ -460,15 +526,15 @@ fm_afk_launch_create_tmux() { # <captain-target> <captain-backend> fm_afk_launch_start() { local captain_target captain_backend backup artifact had_afk=0 result - if [ -e "$FM_AFK_LAUNCH_STATE/.afk-return-catchup" ]; then - fm_afk_launch_log "return catch-up is still pending; run bin/fm-afk-return.sh check before re-entering away mode" - return 1 - fi + fm_afk_launch_catchup_pending && return 1 + fm_afk_launch_record_require || return 1 # Capture the captain pane FIRST, before creating anything. captain_target=$(discover_supervisor_target) || { - fm_afk_launch_log "could not resolve the captain supervisor pane (set FM_SUPERVISOR_TARGET)"; return 1; } + fm_afk_launch_log "could not resolve the captain supervisor pane (set FM_SUPERVISOR_TARGET)" + return 1; } captain_backend=$(discover_supervisor_backend) || { - fm_afk_launch_log "could not resolve the captain supervisor backend (set FM_SUPERVISOR_BACKEND)"; return 1; } + fm_afk_launch_log "could not resolve the captain supervisor backend (set FM_SUPERVISOR_BACKEND)" + return 1; } mkdir -p "$FM_AFK_LAUNCH_STATE" @@ -530,10 +596,8 @@ fm_afk_launch_start() { fm_afk_launch_start_native() { local backup artifact had_afk=0 result=0 mkdir -p "$FM_AFK_LAUNCH_STATE" || return 1 - if [ -e "$FM_AFK_LAUNCH_STATE/.afk-return-catchup" ]; then - fm_afk_launch_log "return catch-up is still pending; run bin/fm-afk-return.sh check before re-entering away mode" - return 1 - fi + fm_afk_launch_catchup_pending && return 1 + fm_afk_launch_record_require || return 1 if daemon_lock_held_by_live_daemon; then fm_afk_launch_record_validate_if_present || return 1 fm_afk_launch_flag_write || return 1 @@ -571,7 +635,7 @@ fm_afk_launch_start_native() { } fm_afk_launch_stop() { - local pid pid_identity current_identity result=0 read_result + local pid pid_identity current_identity result=0 read_result archived fm_afk_launch_record_read read_result=$? if [ "$read_result" -eq 2 ]; then @@ -611,15 +675,24 @@ fm_afk_launch_stop() { if [ "$read_result" -eq 0 ]; then fm_afk_launch_close_recorded || result=1 fi - # (3) Clear the away-mode flag LAST. + # (3) Clear the away-mode flag, then (4) archive the posture record LAST so the + # posture ends only once every daemon-side artifact is down. if ! rm -f "$FM_AFK_LAUNCH_STATE/.afk"; then fm_afk_launch_log "failed to clear away-mode flag" result=1 fi + if [ "$result" -eq 0 ] && fm_afk_contract_present "$FM_AFK_LAUNCH_STATE"; then + if archived=$("$FM_AFK_CONTRACT_CMD" archive); then + fm_afk_launch_log "away-posture record archived at $archived" + else + fm_afk_launch_log "failed to archive the away-posture record; it still stands" + result=1 + fi + fi if [ "$result" -eq 0 ]; then - fm_afk_launch_log "away mode stopped; daemon terminal torn down and .afk cleared" + fm_afk_launch_log "away mode stopped; daemon terminal torn down, .afk cleared, and the posture record archived" else - fm_afk_launch_log "away mode stopped; terminal teardown remains recorded for retry" + fm_afk_launch_log "away mode stopped; terminal teardown or the record archive remains recorded for retry" fi return "$result" } @@ -636,6 +709,8 @@ fm_afk_launch_main() { trap 'exit 143' TERM fm_afk_launch_lock_acquire || return 1 case "${1:-start}" in + propose) shift; fm_afk_launch_propose "$@" ;; + confirm) fm_afk_launch_confirm ;; start) fm_afk_launch_start ;; start-native) fm_afk_launch_start_native ;; stop) fm_afk_launch_stop ;; diff --git a/bin/fm-afk-return.sh b/bin/fm-afk-return.sh index cf5addb24cf..923f1259eee 100755 --- a/bin/fm-afk-return.sh +++ b/bin/fm-afk-return.sh @@ -1,35 +1,65 @@ #!/usr/bin/env bash -# fm-afk-return.sh - deterministic away-mode return catch-up gate. +# fm-afk-return.sh - deterministic away-mode return: the return brief and the +# catch-up gate. # # Usage: -# fm-afk-return.sh Stop away mode, present catch-up, and open/check gate. +# fm-afk-return.sh Stop away mode, render the return brief, and open/check the gate. # fm-afk-return.sh begin Same as the default command. -# fm-afk-return.sh check Re-present and close the gate only after blockers resolve. +# fm-afk-return.sh check Re-render the brief and close the gate only after blockers resolve. # fm-afk-return.sh guard Read-only refusal while away or catch-up is pending. # -# `blocked:` is the crewmate protocol's firstmate-actionable verb. A live task's -# open blocked event must be remediated and closed with `resolved [key=...]`, or -# explicitly reclassified in the status stream with a durable reason, before an -# ordinary captain request may proceed. `needs-decision:` is deliberately not -# part of this blocker gate. +# THE RETURN BRIEF (stdout, on begin and on every check) is rendered from durable +# records, never from conversation memory: the archived away-posture record +# (bin/fm-afk-contract.sh), the supervision outcome store +# (bin/fm-branch-outcome.sh), the held set in the backlog (tasks-axi), and the +# status logs. Its order is fixed: supervisor health across the away window +# first, then every mandate clause the captain recorded, including superseded +# in-session read-backs (this release records clauses and does not execute them, +# and the brief says so), then what is +# waiting on the captain, then what was tried and failed or could not be fixed, +# then what the away session handled, then cost. The health snapshot is taken +# BEFORE the daemon shutdown so the shutdown itself cannot read as a gap. +# +# THE GATE. `blocked:` is the crewmate protocol's firstmate-actionable verb. A +# live task's open blocked event must be remediated and closed with +# `resolved [key=...]`, or explicitly reclassified in the status stream with a +# durable reason, before an ordinary captain request may proceed. +# `needs-decision:` is deliberately not part of this blocker gate. The gate +# keeps every open blocker until that blocker's own resolution is proven. +# Captain-verdict outcomes are listed under "waiting on you", but cannot exempt +# a blocker because decision-key provenance is deferred to phase 4 +# (fm-afk-clauses-execute-r1). Away-window attribution uses second-resolution +# epochs; a durable sequence boundary and archive-chain identity are deferred to +# that phase as well. Replacement records carry the original entry boundary and +# superseded mandates are included as the phase-1 fail-safe. # # The durable state/.afk-return-catchup file is written BEFORE daemon shutdown, -# so a crash between stopping, wake presentation, and blocker handling fails closed. -# It retains the presented wake, buffered-escalation, and wedge-marker evidence -# until every live open blocker is closed and `check` succeeds. Repeated begin/check -# calls are idempotent. `guard` never mutates state and is suitable for ordinary -# read entrypoints such as fm-bearings-snapshot.sh. +# so a crash between stopping, wake presentation, and blocker handling fails +# closed. It retains the presented wake, buffered-escalation, wedge-marker, +# health, and posture-record evidence until every live open blocker is closed +# and `check` succeeds. Repeated begin/check calls are idempotent. `guard` +# never mutates state and is suitable for ordinary read entrypoints such as +# fm-bearings-snapshot.sh. set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" GATE="$STATE/.afk-return-catchup" LOCK="$STATE/.afk-return-catchup.lock" +RETURN_GRACE=${FM_GUARD_GRACE:-300} + +# The posture-record owner: path helpers only; every read goes through its +# subcommands. It sources fm-classify-lib.sh, which has no side effects, so the +# advertised read-only guard stays literal. +# shellcheck source=bin/fm-afk-contract.sh +. "$SCRIPT_DIR/fm-afk-contract.sh" +CONTRACT="$SCRIPT_DIR/fm-afk-contract.sh" usage() { - sed -n '2,7p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + sed -n '2,9p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' } clean_field() { @@ -49,32 +79,115 @@ $text EOF } +remove_evidence() { # <kind> <text> <file> + local kind=$1 text=$2 file=$3 record pending + record=$(printf 'evidence\t%s\t%s' "$kind" "$text") + pending=$(mktemp "$(dirname "$file")/.afk-return-evidence-filter.XXXXXX") || return 1 + grep -Fvx "$record" "$file" > "$pending" 2>/dev/null || true + mv "$pending" "$file" +} + +remove_evidence_prefix() { # <kind> <text-prefix> <file> + local kind=$1 text=$2 file=$3 prefix pending + prefix=$(printf 'evidence\t%s\t%s' "$kind" "$text") + pending=$(mktemp "$(dirname "$file")/.afk-return-evidence-filter.XXXXXX") || return 1 + awk -v prefix="$prefix" 'index($0, prefix) != 1 { print }' "$file" > "$pending" 2>/dev/null || true + mv "$pending" "$file" +} + preserve_evidence() { # <destination> local destination=$1 [ -f "$GATE" ] || return 0 - grep '^evidence'"$(printf '\t')" "$GATE" >> "$destination" 2>/dev/null || true + grep -E '^(evidence|window|contract|superseded)'"$(printf '\t')" "$GATE" >> "$destination" 2>/dev/null || true +} + +append_superseded_record() { # <path> <file> + local path=$1 file=$2 row + row=$(printf 'superseded\t%s' "$path") + grep -Fqx "$row" "$file" 2>/dev/null || printf '%s\n' "$row" >> "$file" +} + +remove_superseded_record() { # <path> <file> + local path=$1 file=$2 row pending + row=$(printf 'superseded\t%s' "$path") + pending=$(mktemp "$(dirname "$file")/.afk-return-superseded-filter.XXXXXX") || return 1 + grep -Fvx "$row" "$file" > "$pending" 2>/dev/null || true + mv "$pending" "$file" +} + +# The epoch the away window started at, from the gate's retained contract row, +# else from the live record (before it is archived), else from the legacy away +# flag's own timestamp, else unknown (empty). +gate_contract_epoch() { + awk -F '\t' '$1 == "contract" { print $2; exit }' "$GATE" 2>/dev/null || true +} + +gate_window_epoch() { + awk -F '\t' '$1 == "window" || $1 == "contract" { print $2; exit }' "$GATE" 2>/dev/null || true +} + +window_start_epoch() { + local epoch flag + epoch=$(gate_window_epoch) + case "$epoch" in ''|*[!0-9]*) epoch= ;; esac + if [ -z "$epoch" ] && fm_afk_contract_present "$STATE"; then + epoch=$("$CONTRACT" field entered_epoch 2>/dev/null || true) + fi + if [ -z "$epoch" ] && [ -f "$STATE/.afk" ]; then + flag=$(head -1 "$STATE/.afk" 2>/dev/null || true) + case "$flag" in ''|*[!0-9]*) ;; *) epoch=$flag ;; esac + fi + case "$epoch" in ''|*[!0-9]*) printf '' ;; *) printf '%s' "$epoch" ;; esac +} + +# Reads the store through its owner so a malformed store refuses rather than +# misleads. +STORE_ROWS= +store_rows_load() { # <since-epoch> + local since=$1 raw + STORE_ROWS= + [ -s "$STATE/branch-outcomes.jsonl" ] || return 0 + case "$since" in ''|*[!0-9]*) since=0 ;; esac + raw=$("$SCRIPT_DIR/fm-branch-outcome.sh" list --recent 1000000 2>/dev/null) \ + || return 1 + STORE_ROWS=$(printf '%s\n' "$raw" | jq -r --argjson since "$since" \ + 'select(.epoch >= $since) | [.seq, .task, .verdict, (.statusEndpoint // 0), (.summary // "")] | @tsv' 2>/dev/null) \ + || { STORE_ROWS=; return 1; } +} + +STATUS_SCAN_ERROR= +status_path_readable() { + [ -f "$1" ] && [ -r "$1" ] && [ ! -L "$1" ] } scan_open_blockers() { # -> tab-separated blocker rows - local meta id status key verb summary clean_summary + local meta id status key verb summary clean_summary open + STATUS_SCAN_ERROR= for meta in "$STATE"/*.meta; do [ -f "$meta" ] || continue id=$(basename "$meta") id=${id%.meta} status="$STATE/$id.status" - [ -f "$status" ] || continue + if ! status_path_readable "$status"; then + STATUS_SCAN_ERROR=$status + return 1 + fi + if ! open=$(status_open_decisions "$status"); then + STATUS_SCAN_ERROR=$status + return 1 + fi while IFS="$(printf '\t')" read -r key verb summary; do [ "$verb" = blocked ] || continue clean_summary=$(printf '%s' "$summary" | clean_field) printf 'blocker\t%s\t%s\t%s\n' "$id" "$key" "$clean_summary" done <<EOF -$(status_open_decisions "$status") +$open EOF done } -write_pending_seed() { # Fail-closed marker before any lifecycle mutation. - local pending started +write_pending_seed() { # <window-epoch> <contract-epoch> Fail-closed marker before any lifecycle mutation. + local window_epoch=$1 contract_epoch=$2 pending started mkdir -p "$STATE" || return 1 started=$(awk -F '\t' '$1 == "started" { print $2; exit }' "$GATE" 2>/dev/null || true) [ -n "$started" ] || started=$(date +%s) @@ -83,21 +196,27 @@ write_pending_seed() { # Fail-closed marker before any lifecycle mutation. printf 'schema\tfm-afk-return.v1\n' printf 'started\t%s\n' "$started" printf 'phase\tstopping-and-draining\n' - preserve_evidence /dev/stdout + [ -z "$window_epoch" ] || printf 'window\t%s\n' "$window_epoch" + [ -z "$contract_epoch" ] || printf 'contract\t%s\n' "$contract_epoch" + preserve_evidence /dev/stdout | grep -Ev "^(window|contract)$(printf '\t')" || true } > "$pending" || { rm -f "$pending"; return 1; } mv "$pending" "$GATE" } write_gate() { # <evidence-file> <blockers-file> - local evidence=$1 blockers=$2 pending started + local evidence=$1 blockers=$2 pending started window_epoch contract_epoch pending=$(mktemp "$STATE/.afk-return-catchup.pending.XXXXXX") || return 1 started=$(awk -F '\t' '$1 == "started" { print $2; exit }' "$GATE" 2>/dev/null || true) [ -n "$started" ] || started=$(date +%s) + window_epoch=$(gate_window_epoch) + contract_epoch=$(gate_contract_epoch) { printf 'schema\tfm-afk-return.v1\n' printf 'started\t%s\n' "$started" printf 'phase\tblocked\n' - cat "$evidence" 2>/dev/null || true + [ -z "$window_epoch" ] || printf 'window\t%s\n' "$window_epoch" + [ -z "$contract_epoch" ] || printf 'contract\t%s\n' "$contract_epoch" + grep -Ev "^(window|contract)$(printf '\t')" "$evidence" 2>/dev/null || true cat "$blockers" 2>/dev/null || true } > "$pending" || { rm -f "$pending"; return 1; } mv "$pending" "$GATE" @@ -127,7 +246,7 @@ clear_delivery_artifacts() { } return_guard() { - if [ -e "$STATE/.afk" ]; then + if [ -e "$STATE/.afk" ] || fm_afk_contract_present "$STATE"; then printf 'fm-afk-return: away mode is still active; run bin/fm-afk-return.sh before ordinary captain work\n' >&2 return 3 fi @@ -139,14 +258,262 @@ return_guard() { return 0 } +# --- supervisor health, snapshotted before anything is shut down ------------ + +health_snapshot() { # <evidence-file> + local evidence=$1 beat_age lines="" + beat_age=$(fm_path_age "$STATE/.last-watcher-beat") + if [ -e "$STATE/.watcher-down" ]; then + lines="GAP: watcher downtime was detected during the away window (recovery marker present)" + fi + if [ -e "$STATE/.afk" ] && ! fm_afk_daemon_owns_supervision "$STATE"; then + lines="$lines +GAP: the away daemon was not running at return (the away flag stood with no live daemon)" + fi + if [ "$beat_age" -ge "$RETURN_GRACE" ]; then + lines="$lines +GAP: the watcher beat was ${beat_age}s old at return (grace ${RETURN_GRACE}s)" + fi + if [ -s "$STATE/.subsuper-inject-wedged" ]; then + lines="$lines +delivery wedged: $(head -1 "$STATE/.subsuper-inject-wedged" 2>/dev/null || true)" + fi + if [ -z "$(printf '%s' "$lines" | tr -d '[:space:]')" ]; then + lines="supervision ran through the away window with no detected gap (watcher beat ${beat_age}s old at return)" + fi + append_evidence health "$lines" "$evidence" +} + +# --- the return brief ------------------------------------------------------- + +format_duration() { # <seconds> + local s=$1 + case "$s" in ''|*[!0-9]*) printf 'unknown'; return ;; esac + if [ "$s" -ge 3600 ]; then printf '%dh%02dm' $((s / 3600)) $(((s % 3600) / 60)) + elif [ "$s" -ge 60 ]; then printf '%dm' $((s / 60)) + else printf '%ds' "$s"; fi +} + +epoch_to_iso() { # <epoch> + date -u -r "$1" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d "@$1" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || printf '%s' "$1" +} + +strip_axi_help() { + awk '/^help\[/ { skip = 1; next } skip && /^ / { next } { skip = 0; print }' +} + +MANDATE_COUNT=0 +HELD_READ_FAILED=0 +HELD_READ_PATH= +render_mandate_record() { # <record> [superseded-time] + local record=$1 superseded=${2:-} id action object when stop text missing suffix="" words flag + [ -z "$superseded" ] || suffix=" - superseded at $superseded" + while IFS="$(printf '\t')" read -r id action object when stop; do + [ -n "$id" ] || continue + MANDATE_COUNT=$((MANDATE_COUNT + 1)) + printf ' - %s. %s ' "$id" "$action" + fm_afk_contract_unescape "$object" + printf ' when ' + fm_afk_contract_unescape "$when" + if [ "$stop" != - ]; then + printf ' stop ' + fm_afk_contract_unescape "$stop" + fi + flag=$("$CONTRACT" flags --path "$record" | awk -F '\t' -v id="$id" '$1 == id { print $2 }') + [ -z "$flag" ] || printf " - flagged: names '%s', a never-set concept that is never pre-authorizable" "$flag" + printf '%s - recorded, not executed by this release\n' "$suffix" + done <<EOF +$("$CONTRACT" clauses --path "$record") +EOF + while IFS="$(printf '\t')" read -r id text missing; do + [ -n "$id" ] || continue + MANDATE_COUNT=$((MANDATE_COUNT + 1)) + printf ' - %s. "' "$id" + fm_afk_contract_unescape "$text" + printf '"%s - refused at entry: missing %s\n' "$suffix" "$missing" + done <<EOF +$("$CONTRACT" refused --path "$record") +EOF + words=$("$CONTRACT" words --path "$record"; printf x) + words=${words%x} + if [ -n "$words" ]; then + if [ -n "$superseded" ]; then + printf ' your words superseded at %s:\n' "$superseded" + else + printf ' your words at entry:\n' + fi + printf '%s' "$words" | sed 's/^/ /' + case "$words" in *$'\n') ;; *) printf '\n' ;; esac + fi +} + +render_return_brief() { # <evidence-file> <blockers-file> <since-epoch> + local evidence=$1 blockers=$2 since=$3 now record superseded superseded_at archive_dir stamp + local tag task key summary count routine captain live held_err last verb rows status + now=$(date +%s) + printf '=== Return brief' + if [ -n "$since" ]; then + printf ' (away %s -> %s, %s)' "$(epoch_to_iso "$since")" "$(epoch_to_iso "$now")" "$(format_duration $((now - since)))" + fi + printf ' ===\n' + + # 1. health, first, always. + printf 'Supervisor health:\n' + awk -F '\t' '$1 == "evidence" && ($2 == "health" || ($2 == "lifecycle" && ($3 ~ /^outcome store unreadable/ || $3 ~ /^status file unreadable:/ || $3 ~ /^away-posture record (unreadable|missing):/ || $3 ~ /^archived away-posture record/ || $3 ~ /^superseded away-posture record/))) { print " - " $3 }' "$evidence" + + # 2. the mandate. + printf 'Mandate clauses:\n' + record="" + MANDATE_COUNT=0 + [ -z "$since" ] || record=$("$CONTRACT" archived "$since" 2>/dev/null || true) + if [ -n "$record" ]; then + archive_dir=$(fm_afk_contract_archive_dir "$STATE") + for superseded in "$archive_dir/$since-superseded-"*.afk-contract; do + [ -f "$superseded" ] || continue + stamp=${superseded##*/"$since"-superseded-} + stamp=${stamp%%-*} + stamp=${stamp%.afk-contract} + case "$stamp" in ''|*[!0-9]*) superseded_at=unknown ;; *) superseded_at=$(epoch_to_iso "$stamp") ;; esac + render_mandate_record "$superseded" "$superseded_at" + done + render_mandate_record "$record" + [ "$MANDATE_COUNT" -gt 0 ] || printf ' (none recorded)\n' + else + printf ' (no away-posture record for this window; legacy away flag only)\n' + fi + + # 3. waiting on the captain. + printf 'Waiting on you:\n' + count=0 + HELD_READ_FAILED=0 + HELD_READ_PATH=$(fm_backlog_file "$DATA" 2>/dev/null || printf '%s/backlog.md' "$DATA") + if held=$(fm_backlog_row_list "$DATA" --state held --fields hold_kind,hold_reason,hold_until 2>&1); then + rows=$(printf '%s\n' "$held" | strip_axi_help | grep -v '^count: ' | grep -v '^tasks\[0\]' || true) + if printf '%s\n' "$held" | grep -q '^count: 0'; then + : + elif [ -n "$rows" ]; then + count=$((count + 1)) + printf ' held in the backlog:\n' + printf '%s\n' "$rows" | sed 's/^/ /' + fi + else + held_err=$(printf '%s' "$held" | head -1 | clean_field) + count=$((count + 1)) + HELD_READ_FAILED=1 + printf ' held listing unavailable: %s: %s; catch-up stays gated\n' "$HELD_READ_PATH" "$held_err" + fi + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] || continue + task=$(basename "$meta"); task=${task%.meta} + status="$STATE/$task.status" + status_path_readable "$status" || continue + while IFS="$(printf '\t')" read -r key verb summary; do + [ "$verb" = needs-decision ] || continue + count=$((count + 1)) + printf ' - %s [key=%s] needs your decision: %s\n' "$task" "$key" "$(printf '%s' "$summary" | clean_field)" + done <<EOF +$(status_open_decisions "$status") +EOF + done + rows=$(printf '%s\n' "$STORE_ROWS" | awk -F '\t' '$3 == "captain" { printf " - %s: %s\n", $2, $5 }') + if [ -n "$rows" ]; then + count=$((count + 1)) + printf ' escalated by the away session:\n' + printf '%s\n' "$rows" | sed 's/^/ /' + fi + [ "$count" -gt 0 ] || printf ' (nothing)\n' + + # 4. tried and failed, or could not be fixed. + printf 'Tried and failed, or could not be fixed:\n' + count=0 + while IFS="$(printf '\t')" read -r tag task key summary; do + [ "$tag" = blocker ] || continue + count=$((count + 1)) + printf ' - %s [key=%s] still blocked, firstmate remediates before ordinary work: %s\n' "$task" "$key" "$summary" + done < "$blockers" + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] || continue + task=$(basename "$meta"); task=${task%.meta} + status="$STATE/$task.status" + status_path_readable "$status" || continue + last=$(last_status_line "$status") + [ "$(status_line_verb "$last")" = failed ] || continue + count=$((count + 1)) + printf ' - %s: %s\n' "$task" "$(printf '%s' "$last" | clean_field)" + done + [ "$count" -gt 0 ] || printf ' (nothing)\n' + + # 5. handled while away. + printf 'Handled while away:\n' + routine=$(printf '%s\n' "$STORE_ROWS" | awk -F '\t' '$3 == "routine" { n++ } END { print n + 0 }') + captain=$(printf '%s\n' "$STORE_ROWS" | awk -F '\t' '$3 == "captain" { n++ } END { print n + 0 }') + if [ "$routine" -gt 0 ]; then + printf ' %s routine outcome(s) recorded; the latest:\n' "$routine" + printf '%s\n' "$STORE_ROWS" | awk -F '\t' '$3 == "routine" { printf " - %s: %s\n", $2, $5 }' | tail -5 + else + printf ' (no routine outcomes recorded in the store for this window)\n' + fi + + # 6. cost. + live=0 + for meta in "$STATE"/*.meta; do [ -f "$meta" ] && live=$((live + 1)); done + printf 'Cost: %s supervision outcome(s) recorded (%s routine, %s captain); %s task(s) live at return.\n' \ + "$((routine + captain))" "$routine" "$captain" "$live" +} + return_reconcile() { - local evidence blockers drain_err drained wake_ack_line wake_ack_through wake_ack_generation wedge escalations lifecycle_ok=1 + local evidence blockers drain_err drained wake_ack_line wake_ack_through wake_ack_generation wedge escalations lifecycle_ok=1 since contract_since superseded_record retained_record + local archived_contract tag kind text retained_live restored_epoch evidence=$(mktemp "$STATE/.afk-return-evidence.XXXXXX") || return 1 blockers=$(mktemp "$STATE/.afk-return-blockers.XXXXXX") || { rm -f "$evidence"; return 1; } drain_err=$(mktemp "$STATE/.afk-return-drain.XXXXXX") || { rm -f "$evidence" "$blockers"; return 1; } preserve_evidence "$evidence" + since=$(gate_window_epoch) + contract_since=$(gate_contract_epoch) + + # Health is read before the shutdown below so the shutdown cannot read as a gap; + # a repeated begin/check keeps the first snapshot. + grep -q "^evidence$(printf '\t')health$(printf '\t')" "$evidence" 2>/dev/null || health_snapshot "$evidence" - if [ -e "$STATE/.afk" ] || [ -e "$STATE/.afk-daemon-terminal" ]; then + while IFS="$(printf '\t')" read -r tag kind text; do + [ "$tag" = evidence ] && [ "$kind" = lifecycle ] || continue + case "$text" in + 'away-posture record unreadable: '*'; catch-up stays gated') + retained_live=${text#away-posture record unreadable: } + retained_live=${retained_live%; catch-up stays gated} ;; + 'away-posture record missing: '*'; catch-up stays gated') + retained_live=${text#away-posture record missing: } + retained_live=${retained_live%; catch-up stays gated} ;; + *) continue ;; + esac + if [ ! -f "$retained_live" ]; then + remove_evidence lifecycle "away-posture record unreadable: $retained_live; catch-up stays gated" "$evidence" || lifecycle_ok=0 + append_evidence lifecycle "away-posture record missing: $retained_live; catch-up stays gated" "$evidence" + lifecycle_ok=0 + elif ! fm_afk_contract_validate "$retained_live" 1; then + remove_evidence lifecycle "away-posture record missing: $retained_live; catch-up stays gated" "$evidence" || lifecycle_ok=0 + append_evidence lifecycle "away-posture record unreadable: $retained_live; catch-up stays gated" "$evidence" + lifecycle_ok=0 + else + restored_epoch=$("$CONTRACT" field entered_epoch --path "$retained_live" 2>/dev/null || true) + case "$restored_epoch" in + ''|*[!0-9]*) lifecycle_ok=0 ;; + *) + if write_pending_seed "$restored_epoch" "$restored_epoch"; then + since=$restored_epoch + contract_since=$restored_epoch + remove_evidence lifecycle "away-posture record missing: $retained_live; catch-up stays gated" "$evidence" || lifecycle_ok=0 + remove_evidence lifecycle "away-posture record unreadable: $retained_live; catch-up stays gated" "$evidence" || lifecycle_ok=0 + else + lifecycle_ok=0 + fi ;; + esac + fi + done <<EOF +$(cat "$evidence") +EOF + + if [ -e "$STATE/.afk" ] || [ -e "$STATE/.afk-daemon-terminal" ] || fm_afk_contract_present "$STATE"; then if ! "$SCRIPT_DIR/fm-afk-launch.sh" stop; then lifecycle_ok=0 append_evidence lifecycle 'away-mode shutdown failed; lifecycle state preserved for retry' "$evidence" @@ -168,6 +535,56 @@ return_reconcile() { fi append_evidence wake "$drained" "$evidence" + if fm_afk_contract_present "$STATE"; then + if ! fm_afk_contract_validate "$(fm_afk_contract_path "$STATE")" 1; then + append_evidence lifecycle "away-posture record unreadable: $(fm_afk_contract_path "$STATE"); catch-up stays gated" "$evidence" + lifecycle_ok=0 + else + remove_evidence_prefix lifecycle 'away-posture record unreadable:' "$evidence" || lifecycle_ok=0 + fi + elif [ -n "$contract_since" ]; then + archived_contract=$("$CONTRACT" archived "$contract_since" 2>/dev/null || true) + if [ -z "$archived_contract" ]; then + append_evidence lifecycle "archived away-posture record missing for entered_epoch $contract_since; catch-up stays gated" "$evidence" + lifecycle_ok=0 + elif ! fm_afk_contract_validate "$archived_contract" 1; then + append_evidence lifecycle "archived away-posture record unreadable for entered_epoch $contract_since; catch-up stays gated" "$evidence" + lifecycle_ok=0 + else + remove_evidence_prefix lifecycle 'away-posture record unreadable:' "$evidence" || lifecycle_ok=0 + remove_evidence_prefix lifecycle 'archived away-posture record missing' "$evidence" || lifecycle_ok=0 + remove_evidence_prefix lifecycle 'archived away-posture record unreadable' "$evidence" || lifecycle_ok=0 + fi + + while IFS="$(printf '\t')" read -r tag retained_record; do + [ "$tag" = superseded ] || continue + if [ ! -f "$retained_record" ]; then + remove_evidence lifecycle "superseded away-posture record unreadable: $retained_record; catch-up stays gated" "$evidence" || lifecycle_ok=0 + append_evidence lifecycle "superseded away-posture record missing: $retained_record; catch-up stays gated" "$evidence" + lifecycle_ok=0 + elif ! fm_afk_contract_validate "$retained_record" 1; then + remove_evidence lifecycle "superseded away-posture record missing: $retained_record; catch-up stays gated" "$evidence" || lifecycle_ok=0 + append_evidence lifecycle "superseded away-posture record unreadable: $retained_record; catch-up stays gated" "$evidence" + lifecycle_ok=0 + else + remove_evidence lifecycle "superseded away-posture record missing: $retained_record; catch-up stays gated" "$evidence" || lifecycle_ok=0 + remove_evidence lifecycle "superseded away-posture record unreadable: $retained_record; catch-up stays gated" "$evidence" || lifecycle_ok=0 + remove_superseded_record "$retained_record" "$evidence" || lifecycle_ok=0 + fi + done <<EOF +$(cat "$evidence") +EOF + + for superseded_record in "$(fm_afk_contract_archive_dir "$STATE")/$contract_since-superseded-"*.afk-contract; do + [ -f "$superseded_record" ] || continue + if ! fm_afk_contract_validate "$superseded_record" 1; then + append_superseded_record "$superseded_record" "$evidence" + append_evidence lifecycle "superseded away-posture record unreadable: $superseded_record; catch-up stays gated" "$evidence" + lifecycle_ok=0 + fi + done + fi + if [ -s "$STATE/.subsuper-inject-wedged" ]; then wedge=$(head -1 "$STATE/.subsuper-inject-wedged" 2>/dev/null || true) append_evidence wedge "$wedge" "$evidence" @@ -177,8 +594,26 @@ return_reconcile() { append_evidence escalation "$escalations" "$evidence" fi - scan_open_blockers > "$blockers" - if [ "$lifecycle_ok" -ne 1 ] || [ -s "$blockers" ]; then + if store_rows_load "$since"; then + remove_evidence lifecycle 'outcome store unreadable, catch-up stays gated' "$evidence" || lifecycle_ok=0 + else + append_evidence lifecycle 'outcome store unreadable, catch-up stays gated' "$evidence" + lifecycle_ok=0 + fi + if scan_open_blockers > "$blockers"; then + remove_evidence_prefix lifecycle 'status file unreadable:' "$evidence" || lifecycle_ok=0 + else + append_evidence lifecycle "status file unreadable: $STATUS_SCAN_ERROR; catch-up stays gated" "$evidence" + lifecycle_ok=0 + fi + render_return_brief "$evidence" "$blockers" "$since" + if [ "$HELD_READ_FAILED" -eq 1 ]; then + append_evidence lifecycle "held set unreadable: $HELD_READ_PATH; catch-up stays gated" "$evidence" + lifecycle_ok=0 + else + remove_evidence_prefix lifecycle 'held set unreadable:' "$evidence" || lifecycle_ok=0 + fi + if [ "$lifecycle_ok" -ne 1 ] || grep -q "^blocker$(printf '\t')" "$blockers"; then write_gate "$evidence" "$blockers" || { rm -f "$evidence" "$blockers" "$drain_err"; return 1; } printf 'fm-afk-return: catch-up must finish before the captain request\n' >&2 print_evidence "$GATE" >&2 @@ -211,7 +646,7 @@ return_reconcile() { } main() { - local mode=${1:-begin} rc + local mode=${1:-begin} rc window_epoch contract_epoch case "$mode" in begin|check) ;; guard) return_guard; return ;; @@ -219,18 +654,27 @@ main() { *) usage >&2; return 2 ;; esac - # The mutating begin/check paths need locks and the keyed status fold. - # `guard` returned above without sourcing fm-wake-lib.sh, whose initialization - # creates the state directory, so the advertised read-only guard is literal. + # The mutating begin/check paths need locks, the keyed status fold, and the + # backlog reader. `guard` returned above without sourcing fm-wake-lib.sh, + # whose initialization creates the state directory, so the advertised + # read-only guard is literal. # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" - # shellcheck source=bin/fm-classify-lib.sh - . "$SCRIPT_DIR/fm-classify-lib.sh" + # shellcheck source=bin/fm-tasks-axi-lib.sh + . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" + # shellcheck source=bin/fm-backlog-transition-lib.sh + . "$SCRIPT_DIR/fm-backlog-transition-lib.sh" mkdir -p "$STATE" || return 1 fm_lock_acquire_wait "$LOCK" trap 'fm_lock_release "$LOCK"' EXIT - write_pending_seed || { fm_lock_release "$LOCK"; trap - EXIT; return 1; } + window_epoch=$(window_start_epoch) + contract_epoch=$(gate_contract_epoch) + if [ -z "$contract_epoch" ] && fm_afk_contract_present "$STATE"; then + contract_epoch=$("$CONTRACT" field entered_epoch 2>/dev/null || true) + case "$contract_epoch" in ''|*[!0-9]*) contract_epoch= ;; esac + fi + write_pending_seed "$window_epoch" "$contract_epoch" || { fm_lock_release "$LOCK"; trap - EXIT; return 1; } return_reconcile rc=$? fm_lock_release "$LOCK" diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index 361be7d2610..f6eada330b7 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -931,7 +931,8 @@ fm_backend_target_exists() { # <backend> <target> [expected-label] [expected-ta # successful session inventory and returns `missing` only when it omits the # exact window; the Herdr adapter reuses its husk classifier plus a # stale-registration cross-check (fm_backend_herdr_agent_state owns that -# contract). Zellij remains unverified because its secondmate ghost-tab and +# contract), and maps a positively stopped session server to `missing` only +# in this recovery-grade view. Zellij remains unverified because its secondmate ghost-tab and # agent-process recovery path has not been empirically validated. Orca and cmux # do not support secondmate spawns. fm_backend_agent_state() { # <backend> <target> diff --git a/bin/fm-backlog-handoff.sh b/bin/fm-backlog-handoff.sh index 879de6053db..b40aae28758 100755 --- a/bin/fm-backlog-handoff.sh +++ b/bin/fm-backlog-handoff.sh @@ -50,16 +50,29 @@ # Remote routes use an outbox handoff: one atomic local tasks-axi mv removes the # selected set from the dispatchable backlog into data/handoff/<id>.outbox.md, # then an idempotent confined transfer and fm-backlog-receive.sh deliver it. -# A present outbox remains the remote retry trigger until backlog receipt and -# receiver wake are both confirmed; a companion pending-reply correlation makes -# crash recovery reconcile an attempted or confirmed wake instead of blindly -# resending it. A prepared local wake is bound to the exact sorted +# A present outbox remains the remote retry trigger only until backlog receipt +# is confirmed, then it is released independently of the best-effort receiver +# wake. The wake remains separately tracked by one pending-reply correlation and +# is retried by later resumes and handoffs without blocking new backlog work. +# An undelivered wake stays retryable under that same correlation even after the +# watcher escalates its unknown delivery; only confirmed delivery prevents a +# resend. A prepared local wake is bound to the exact sorted # requested-key batch; an unrelated handoff to that mate refuses until the # original batch is retried, so it cannot discard wake intent for work that # already moved. No two-phase journal exists. -# Every newly durable backlog delivery also sends one marked wake to the -# receiving endpoint. A missing endpoint or a live endpoint that rejects the -# wake makes the handoff fail with the delivered backlog intact. +# Every newly durable backlog delivery attempts one marked wake to the receiving +# endpoint. A local route moves directly into the destination backlog, and a +# missing or rejected local wake makes that command fail with the move intact so +# rerunning the same handoff retries its prepared wake intent. After a durable +# remote receipt, the outbox is released and the handoff succeeds regardless of +# the best-effort wake outcome; an undelivered remote wake remains separately +# tracked in wake-pending state and is retried under the same correlation by +# later resumes and handoffs. If wake-pending state cannot be recorded, the wake +# is reported as DROPPED and marker removal is attempted while the mate still +# owns reconciliation from its durable backlog. Any unsafe, invalid, delivered, +# or undeletable stale marker is reported and ignored by later resumes and +# handoffs, so wake-state cleanup neither suppresses a new wake nor fails a +# completed remote handoff. # Usage: fm-backlog-handoff.sh <secondmate-id> <item-key>... # fm-backlog-handoff.sh --resume-pending set -eu @@ -86,6 +99,7 @@ RECEIVER_WAKE_MESSAGE='New routed work is in your backlog. Run bin/fm-session-st ACTIVE_HANDOFF_LOCK= ACTIVE_REGISTRY_LOCK= +RECEIVER_WAKE_IGNORE_ID= release_remote_locks() { if [ -n "$ACTIVE_HANDOFF_LOCK" ]; then fm_lock_release "$ACTIVE_HANDOFF_LOCK" @@ -315,10 +329,11 @@ warn_stale_public_commitments() { # <secondmate-id> <moved-key>... } # Wake a live receiver after its backlog has become durable. The marked message -# uses the normal endpoint route, so local and remote secondmates share the same -# verified submit and failure semantics. A seeded but not-yet-spawned home is a -# valid handoff destination, but its missing endpoint is reported rather than -# pretending the task was started. +# uses the normal endpoint route and verified submit for both placements. A +# failed local wake fails that local handoff, while a failed remote wake is +# handled as the best-effort state described in the script contract above. A +# seeded but not-yet-spawned home is a valid handoff destination, but its missing +# endpoint is reported rather than pretending the task was started. receiver_wake_batch_id() { # <item-key>... local digest if command -v shasum >/dev/null 2>&1; then @@ -355,7 +370,7 @@ receiver_wake_mark() { # <secondmate-id> <prepared|pending> [batch-id] [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 value=$(cat "$marker" 2>/dev/null || true) case "$value" in - prepared:*|pending:*) + prepared:*) corr=${value#*:} corr=${corr%%:*} rec=$(fm_pending_reply_path "$STATE" "$corr") @@ -363,6 +378,7 @@ receiver_wake_mark() { # <secondmate-id> <prepared|pending> [batch-id] && [ "$(fm_pending_reply_get "$rec" task_id)" = "$id" ] return $? ;; + pending:*) receiver_wake_pending_valid "$id"; return $? ;; pending) ;; *) return 1 ;; esac @@ -432,16 +448,78 @@ receiver_wake_discard_pending() { # <secondmate-id> rm -f -- "$marker" } -receiver_wake_clear_confirmed() { # <secondmate-id> - local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value - [ -e "$marker" ] || [ -L "$marker" ] || return 0 +receiver_wake_pending_valid() { # <secondmate-id> + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec delivered [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 value=$(cat "$marker" 2>/dev/null || true) - case "$value" in - pending|pending:*) return 0 ;; - confirmed|confirmed:*) rm -f -- "$marker" ;; - *) return 1 ;; - esac + case "$value" in pending:*) corr=${value#pending:} ;; *) return 1 ;; esac + printf '%s' "$corr" | grep -Eq '^[a-f0-9]{16}$' || return 1 + rec=$(fm_pending_reply_path "$STATE" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$id" ] || return 1 + delivered=$(fm_pending_reply_get "$rec" delivered_epoch) + [ -z "$delivered" ] || return 1 + fm_pending_reply_corr_reusable "$STATE" "$corr" "$id" +} + +receiver_wake_pending_delivered_valid() { # <secondmate-id> + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + case "$value" in pending:*) corr=${value#pending:} ;; *) return 1 ;; esac + printf '%s' "$corr" | grep -Eq '^[a-f0-9]{16}$' || return 1 + rec=$(fm_pending_reply_path "$STATE" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$id" ] \ + && [ -n "$(fm_pending_reply_get "$rec" delivered_epoch)" ] +} + +receiver_wake_confirmed_valid() { # <secondmate-id> + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" value corr rec + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + value=$(cat "$marker" 2>/dev/null || true) + [ "$value" != confirmed ] || return 0 + case "$value" in confirmed:*) corr=${value#confirmed:} ;; *) return 1 ;; esac + printf '%s' "$corr" | grep -Eq '^[a-f0-9]{16}$' || return 1 + rec=$(fm_pending_reply_path "$STATE" "$corr") + [ -f "$rec" ] && [ ! -L "$rec" ] \ + && [ "$(fm_pending_reply_get "$rec" task_id)" = "$id" ] \ + && [ -n "$(fm_pending_reply_get "$rec" delivered_epoch)" ] +} + +receiver_wake_drop_marker() { # <secondmate-id> <reason> + local id=$1 reason=$2 marker="$STATE/.backlog-handoff-$1.wake-pending" + printf 'receiver wake state: DROPPED marker=%s\n' "$marker" + if [ -e "$marker" ] || [ -L "$marker" ]; then + if rm -f -- "$marker"; then + printf 'warning: best-effort receiver wake for secondmate %s was dropped; removed stale wake marker %s because %s\n' "$id" "$marker" "$reason" >&2 + else + printf 'warning: best-effort receiver wake for secondmate %s was dropped; stale wake marker remains at %s because %s\n' "$id" "$marker" "$reason" >&2 + fi + else + printf 'warning: best-effort receiver wake for secondmate %s was dropped; no wake marker remains at %s because %s\n' "$id" "$marker" "$reason" >&2 + fi + return 0 +} + +receiver_wake_clear_confirmed() { # <secondmate-id> + local id=$1 marker="$STATE/.backlog-handoff-$1.wake-pending" + RECEIVER_WAKE_IGNORE_ID= + [ -e "$marker" ] || [ -L "$marker" ] || return 0 + if receiver_wake_pending_valid "$id"; then + return 0 + fi + if receiver_wake_pending_delivered_valid "$id" || receiver_wake_confirmed_valid "$id"; then + if ! rm -f -- "$marker"; then + RECEIVER_WAKE_IGNORE_ID=$id + printf 'warning: confirmed receiver wake left a stale marker at %s; later handoffs will ignore it\n' "$marker" >&2 + fi + return 0 + fi + receiver_wake_drop_marker "$id" 'wake-pending state is unsafe or invalid' + if [ -e "$marker" ] || [ -L "$marker" ]; then + RECEIVER_WAKE_IGNORE_ID=$id + fi } wake_secondmate_receiver() { # <secondmate-id> <correlation-id> @@ -459,7 +537,7 @@ wake_secondmate_receiver() { # <secondmate-id> <correlation-id> "$SCRIPT_DIR/fm-send.sh" "$id" "$RECEIVER_WAKE_MESSAGE" 2>&1) || rc=$? if [ "$rc" -ne 0 ]; then [ -z "$out" ] || printf '%s\n' "$out" >&2 - printf 'error: backlog delivery to secondmate %s succeeded, but its receiver wake failed; rerun this handoff to retry the wake\n' "$id" >&2 + printf 'error: backlog delivery to secondmate %s succeeded, but its receiver wake failed; retry a tracked remote wake with --resume-pending or a later new handoff, and retry a local wake by rerunning its handoff\n' "$id" >&2 return 1 fi [ -z "$out" ] || printf '%s\n' "$out" @@ -519,7 +597,7 @@ outbox_item_count() { # <path> } remote_deliver_outbox() { # <secondmate-id> <outbox-path> - local id=$1 outbox=$2 remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker + local id=$1 outbox=$2 remote_rel receive_out snapshot bytes hash generation counter counter_tmp current marker wake_rc=0 wake_state=pending [ -f "$outbox" ] && [ ! -L "$outbox" ] || { echo "error: pending outbox is unavailable or unsafe: $outbox" >&2 return 1 @@ -563,25 +641,33 @@ remote_deliver_outbox() { # <secondmate-id> <outbox-path> return 1 fi marker="$STATE/.backlog-handoff-$id.wake-pending" - case "$(cat "$marker" 2>/dev/null || true)" in - pending:*|confirmed|confirmed:*) ;; - *) receiver_wake_mark_pending "$id" || { - echo "error: remote backlog is durable at $id, but receiver wake state could not be recorded; outbox preserved at $outbox" >&2 - return 1 - } ;; - esac - if ! wake_pending_secondmate_receiver "$id" 1; then - echo "error: remote backlog is durable at $id; outbox preserved at $outbox for wake retry" >&2 - return 1 + if [ "$RECEIVER_WAKE_IGNORE_ID" = "$id" ]; then + wake_state=dropped + wake_rc=1 + elif ! receiver_wake_pending_valid "$id" && ! receiver_wake_confirmed_valid "$id"; then + receiver_wake_mark_pending "$id" || { + wake_state=dropped + wake_rc=1 + } + fi + if [ "$wake_rc" -eq 0 ]; then + wake_pending_secondmate_receiver "$id" 1 || wake_rc=$? fi rm -f -- "$outbox" || { - echo "error: receiver wake was confirmed but local outbox cleanup failed: $outbox" >&2 - return 1 - } - rm -f -- "$marker" || { - echo "error: remote outbox cleanup succeeded but confirmed receiver wake state could not be cleared: $marker" >&2 + echo "error: remote backlog is durable at $id, but local outbox cleanup failed: $outbox" >&2 return 1 } + if [ "$wake_rc" -eq 0 ]; then + if ! rm -f -- "$marker"; then + RECEIVER_WAKE_IGNORE_ID=$id + echo "warning: remote outbox and receiver wake completed, but a stale confirmed wake marker remains at $marker; later handoffs will ignore it" >&2 + fi + elif [ "$wake_state" = dropped ]; then + receiver_wake_drop_marker "$id" 'wake-pending state could not be recorded' + echo "warning: remote backlog is durable at $id and its outbox was released after the best-effort receiver wake was dropped" >&2 + else + echo "warning: remote backlog is durable at $id and its outbox was released; the best-effort receiver wake remains pending for a later resume or handoff" >&2 + fi printf '%s\n' "$receive_out" } @@ -735,9 +821,35 @@ resume_pending_outboxes() { return "$failed" } +resume_remote_wake() { # <secondmate-id> + local id=$1 + [ -e "$DATA/handoff/$id.outbox.md" ] || [ -L "$DATA/handoff/$id.outbox.md" ] || { + receiver_wake_clear_confirmed "$id" + receiver_wake_pending_valid "$id" || return 0 + wake_pending_secondmate_receiver "$id" + } +} + +resume_pending_wakes() { + local marker name id failed=0 + [ -d "$STATE" ] || return 0 + for marker in "$STATE"/.backlog-handoff-*.wake-pending; do + [ -e "$marker" ] || [ -L "$marker" ] || continue + name=$(basename "$marker") + id=${name#.backlog-handoff-} + id=${id%.wake-pending} + case "$id" in ''|*[!A-Za-z0-9._-]*) echo "error: unsafe pending wake id: $id" >&2; failed=1; continue ;; esac + [ "$(secondmate_registry_field "$REG" "$id" remote 2>/dev/null || true)" = 1 ] || continue + with_remote_route_locks "$id" resume_remote_wake "$id" || failed=1 + done + return "$failed" +} + if [ "$RESUME_PENDING" -eq 1 ]; then - resume_pending_outboxes - exit $? + FAILED=0 + resume_pending_wakes || FAILED=1 + resume_pending_outboxes || FAILED=1 + exit "$FAILED" fi ACTIVE_REGISTRY_LOCK=$(secondmate_registry_lock_path "$STATE") diff --git a/bin/fm-backlog-transition-lib.sh b/bin/fm-backlog-transition-lib.sh index 42b6d8e8642..4442ec3c64f 100644 --- a/bin/fm-backlog-transition-lib.sh +++ b/bin/fm-backlog-transition-lib.sh @@ -22,22 +22,21 @@ # # SCOPE. fm_backlog_transition_applies is the single gate. It excludes # secondmates (persistent agents are never backlog items, AGENTS.md section 10), -# and homes whose configured backlog backend is manual. Markdown homes that -# keep no backlog file at all are likewise exempt. Those return-1 exemptions -# are never errors; a home on any other configured backend has no markdown -# file requirement at all. An unresolvable configured data directory or +# homes whose configured backlog backend is manual and markdown homes that keep +# no backlog file. Those return-1 exemptions are never errors; an +# unresolvable configured data directory, a backend resolution error, or # incompatible tasks-axi instead returns 2 so callers refuse before mutation. # -# ADDRESSING. The markdown backend owns the explicit `--file <data>/backlog.md` -# on every mutation and row probe so the change lands in the home that owns -# the task regardless of the caller's working directory. Every other -# configured backend is addressed from that data directory's parent - the -# addressing root - with no markdown file override, so the same home's -# `.tasks.toml` supplies its backend, done_keep, and the archive path and -# backend-owned state remains discoverable. The parent of the data directory -# is the addressing root rather than FM_HOME, so a home whose data directory -# is relocated keeps its backlog and its archive together. A root with no -# `.tasks.toml` gets tasks-axi's built-in defaults. +# ADDRESSING. Every call runs from the configured data directory's parent so +# that home's `.tasks.toml` supplies the adapter selection, done_keep, and the +# archive path. A markdown backlog also passes `--file <data>/backlog.md` so the +# change lands in the home that owns the task regardless of the caller's working +# directory. A configured non-markdown adapter is addressed by that root alone, +# because `--file` would override the adapter's own workspace path. The parent of +# the data directory is the addressing root rather than FM_HOME, so a home whose +# data directory is relocated keeps its backlog and its archive together. A root +# with no `.tasks.toml` gets tasks-axi's built-in defaults. +# bin/fm-tasks-axi-lib.sh owns backend precedence and configuration failures. # # CRASH RECOVERY. Only teardown needs a durable record: it removes the meta and # with it the completion links, so a process killed between the two halves would @@ -53,8 +52,9 @@ # leaves the meta itself as the evidence that the row is owed a start. # A captain-held row uses the same record with a `mode=retain` line: replay then # records the deliverable and reopens the row instead of closing it, and never -# closes a row that reads as an open captain call. An answer that closed the row -# first simply retires the record. +# closes a row that reads as an open captain call. An answer that closes the row +# first applies any supported retained artifact from the validated record, then +# replay simply retires the record. # Set by fm_backlog_transition_applies for a return-1 exemption. # shellcheck disable=SC2034 # Output global, read by the sourcing caller. @@ -176,8 +176,98 @@ fm_backlog_data_relative() { # <data-dir> esac } + +# The parent an authorized data directory was named from, kept in the caller's +# own path shape. fm_backlog_record_parent_authorized only applies its FM_HOME +# containment guard to a root that still spells out `$FM_HOME`, so a root +# already resolved through `pwd -P` would skip that guard whenever the data +# directory is a symlink. +fm_backlog_authorized_root() { # <authorized-data-dir> + local data=$1 parent + while [ "$data" != / ] && [ "${data%/}" != "$data" ]; do + data=${data%/} + done + case "$data" in + /) parent=/ ;; + */*) + parent=${data%/*} + [ -n "$parent" ] || parent=/ + ;; + *) parent=. ;; + esac + printf '%s\n' "$parent" +} + +# Any adapter selection or exemption derived from a home's `.tasks.toml` is only +# as safe as that file, so validate it before reading it. +fm_backlog_config_present() { # <root> <authorized-root> + local root=$1 authorized_root=$2 tasks_config="$1/.tasks.toml" + if [ -e "$tasks_config" ] || [ -L "$tasks_config" ]; then + # Leave a dangling config symlink for the backend resolver to diagnose as + # unreadable; reject an existing special file through the home-bound + # validator before any backend parser can touch it. + if [ -L "$tasks_config" ] && [ ! -e "$tasks_config" ]; then + return 0 + fi + fm_backlog_record_present "$tasks_config" "tasks-axi config" "$authorized_root" || return 1 + fi + return 0 +} + +# Every home is bound to its own data directory, whichever adapter it configures, +# so the boundary is authorized first and unconditionally. Only the markdown +# backlog's regular-file requirement is adapter-specific: another adapter keeps +# its rows in its own workspace and need not carry <data>/backlog.md at all. +fm_backlog_source_present() { # <data-dir> <authorized-data-dir> [root authorized-root] + local data=$1 authorized_data=$2 root=${3:-} authorized_root=${4:-} file backend + if [ -z "$root" ]; then + root=$(fm_backlog_root "$data") || return 1 + authorized_root=$(fm_backlog_authorized_root "$authorized_data") + fi + if [ -z "$authorized_root" ]; then + authorized_root=$(fm_backlog_authorized_root "$authorized_data") + fi + fm_backlog_config_present "$root" "$authorized_root" || return 1 + backend=$(fm_tasks_axi_backend "$root" 2>&1) || { + FM_BACKLOG_TRANSITION_ERROR=$backend + return 2 + } + file=$(fm_backlog_file "$data") || return 1 + if [ "$backend" = markdown ]; then + fm_backlog_record_present "$file" "backlog file" "$authorized_data" + return $? + fi + fm_backlog_record_parent_authorized "$file" "backlog data directory" "$authorized_data" parent-only +} + +# Resolve how the owning home's backlog is addressed, for reads and mutations +# alike: sets FM_BACKLOG_AXI_ROOT to the cd target and FM_BACKLOG_AXI_FILE to +# the markdown --file path, empty for every other backend. This is the single +# place that decision is made. A markdown backlog is addressed as +# <data>/backlog.md so the change lands in the home that owns the task +# regardless of the caller's working directory; any other configured adapter +# is addressed by that root alone, because --file would override the adapter's +# own workspace path. The caller invokes fm_tasks_axi inside its own subshell +# - the bound wrapper execs, so a nested subshell here would add a process +# layer between tasks-axi and the caller, which the lock-holding callers' +# interruption contract counts on not existing. +fm_backlog_tasks_axi_addressing() { # <data-dir> + FM_BACKLOG_AXI_FILE= + local data root backend + data=$(fm_backlog_data_absolute "$1") || return $? + root=$(fm_backlog_root "$data") || return $? + backend=$(fm_tasks_axi_backend "$root" 2>&1) || { + FM_BACKLOG_TRANSITION_ERROR=$backend + return 2 + } + FM_BACKLOG_AXI_ROOT=$root + if [ "$backend" = markdown ]; then + FM_BACKLOG_AXI_FILE=$(fm_backlog_file "$data") || return 1 + fi +} + fm_backlog_transition_applies() { # <config-dir> <data-dir> <kind> - local config=$1 data authorized_data=$2 kind=$3 file root + local config=$1 data authorized_data=$2 kind=$3 file root backend authorized_root FM_BACKLOG_TRANSITION_SKIP= if [ "$kind" = secondmate ]; then FM_BACKLOG_TRANSITION_SKIP="secondmates are not backlog items" @@ -192,18 +282,24 @@ fm_backlog_transition_applies() { # <config-dir> <data-dir> <kind> return 2 fi root=$(fm_backlog_root "$data") || return 2 - if [ "$(fm_tasks_axi_backend "$root")" = markdown ]; then - file=$(fm_backlog_file "$data") + authorized_root=$(fm_backlog_authorized_root "$authorized_data") + fm_backlog_config_present "$root" "$authorized_root" || return 2 + backend=$(fm_tasks_axi_backend "$root" 2>&1) || { + FM_BACKLOG_TRANSITION_ERROR=$backend + return 2 + } + if [ "$backend" = markdown ]; then + file=$(fm_backlog_file "$data") || return 2 if [ ! -e "$file" ] && [ ! -L "$file" ]; then - FM_BACKLOG_TRANSITION_SKIP="this home keeps no backlog at $file" + FM_BACKLOG_TRANSITION_SKIP="this home keeps no markdown backlog at $file" return 1 fi - if ! fm_backlog_record_present "$file" "backlog file" "$authorized_data"; then - return 2 - fi + fi + if ! fm_backlog_source_present "$data" "$authorized_data" "$root" "$authorized_root"; then + return 2 fi if ! fm_tasks_axi_compatible; then - FM_BACKLOG_TRANSITION_ERROR="automatic backlog transitions require tasks-axi $FM_TASKS_AXI_MIN or newer with the required update and mv features" + FM_BACKLOG_TRANSITION_ERROR="automatic backlog transitions require tasks-axi ${FM_TASKS_AXI_MIN:-(unknown minimum)} or newer with the required update and mv features" return 2 fi return 0 @@ -286,35 +382,42 @@ fm_tasks_axi() { exit 127 } -# Print one row's `tasks-axi show` output (plus stderr) from the backlog root, -# with `--file` only for the markdown backend; the exit status is tasks-axi's. -# Extra flags (such as --full) are passed through. +# Print one row's `tasks-axi show` output (plus stderr); the exit status is +# tasks-axi's. Extra flags (such as --full) are passed through. fm_backlog_row_show() { # <resolved-data-dir> <id> [flag...] - local data=$1 id=$2 file root + local data=$1 id=$2 addressing_status shift 2 - file=$(fm_backlog_file "$data") || return 1 - root=$(fm_backlog_root "$data") || return 1 - if [ "$(fm_tasks_axi_backend "$root")" = markdown ]; then - (cd "$root" 2>/dev/null && fm_tasks_axi show "$id" "$@" --file "$file" 2>&1) + fm_backlog_tasks_axi_addressing "$data" + addressing_status=$? + if [ "$addressing_status" -ne 0 ]; then + [ -z "${FM_BACKLOG_TRANSITION_ERROR:-}" ] || printf '%s\n' "$FM_BACKLOG_TRANSITION_ERROR" >&2 + return "$addressing_status" + fi + if [ -n "$FM_BACKLOG_AXI_FILE" ]; then + (cd "$FM_BACKLOG_AXI_ROOT" 2>/dev/null && fm_tasks_axi show "$id" "$@" --file "$FM_BACKLOG_AXI_FILE" 2>&1) else - (cd "$root" 2>/dev/null && fm_tasks_axi show "$id" "$@" 2>&1) + (cd "$FM_BACKLOG_AXI_ROOT" 2>/dev/null && fm_tasks_axi show "$id" "$@" 2>&1) fi } fm_backlog_row_list() { # <resolved-data-dir> [flag...] - local data=$1 file root + local data=$1 addressing_status shift - file=$(fm_backlog_file "$data") || return 1 - root=$(fm_backlog_root "$data") || return 1 - if [ "$(fm_tasks_axi_backend "$root")" = markdown ]; then - (cd "$root" 2>/dev/null && tasks-axi list "$@" --file "$file" 2>&1) + fm_backlog_tasks_axi_addressing "$data" + addressing_status=$? + if [ "$addressing_status" -ne 0 ]; then + [ -z "${FM_BACKLOG_TRANSITION_ERROR:-}" ] || printf '%s\n' "$FM_BACKLOG_TRANSITION_ERROR" >&2 + return "$addressing_status" + fi + if [ -n "$FM_BACKLOG_AXI_FILE" ]; then + (cd "$FM_BACKLOG_AXI_ROOT" 2>/dev/null && fm_tasks_axi list "$@" --file "$FM_BACKLOG_AXI_FILE" 2>&1) else - (cd "$root" 2>/dev/null && tasks-axi list "$@" 2>&1) + (cd "$FM_BACKLOG_AXI_ROOT" 2>/dev/null && fm_tasks_axi list "$@" 2>&1) fi } fm_backlog_row_probe() { # <data-dir> <id> - local data authorized_data=$1 file id=$2 out state held blocked hold_kind command_status root + local data authorized_data=$1 id=$2 out state held blocked hold_kind command_status source_status if ! data=$(fm_backlog_data_absolute "$1"); then FM_BACKLOG_ROW_RESULT=error FM_BACKLOG_ROW_STATE= @@ -325,19 +428,11 @@ fm_backlog_row_probe() { # <data-dir> <id> FM_BACKLOG_ROW_STATE= FM_BACKLOG_ROW_HOLD_KIND= FM_BACKLOG_ROW_ERROR= - root=$(fm_backlog_root "$data") || { + fm_backlog_source_present "$data" "$authorized_data" + source_status=$? + if [ "$source_status" -ne 0 ]; then FM_BACKLOG_ROW_ERROR=$FM_BACKLOG_TRANSITION_ERROR - return 1 - } - if [ "$(fm_tasks_axi_backend "$root")" = markdown ]; then - file=$(fm_backlog_file "$data") || { - FM_BACKLOG_ROW_ERROR=$FM_BACKLOG_TRANSITION_ERROR - return 1 - } - if ! fm_backlog_record_present "$file" "backlog file" "$authorized_data"; then - FM_BACKLOG_ROW_ERROR=$FM_BACKLOG_TRANSITION_ERROR - return 1 - fi + return "$source_status" fi out=$(fm_backlog_row_show "$data" "$id") command_status=$? @@ -374,29 +469,29 @@ fm_backlog_row_probe() { # <data-dir> <id> } # Run one tasks-axi mutation against <home>'s backlog, capturing its first -# output line in FM_BACKLOG_TRANSITION_ERROR on failure. The markdown backend -# keeps its explicit <data>/backlog.md file and presence requirement; every -# other configured backend is addressed by the root's own tasks-axi -# configuration, so passing the markdown-era path would write the wrong store. +# output line in FM_BACKLOG_TRANSITION_ERROR on failure. The home boundary is +# authorized through fm_backlog_source_present first; fm_backlog_tasks_axi owns +# how the selected adapter is addressed (ADDRESSING above). fm_backlog_mutate() { # <data-dir> <verb> <id> [flag...] - local data authorized_data=$1 file verb=$2 id=$3 out command_status root + local data authorized_data=$1 verb=$2 id=$3 out command_status source_status if ! data=$(fm_backlog_data_absolute "$1"); then FM_BACKLOG_TRANSITION_ERROR="data directory cannot be resolved: $1" return 1 fi shift 3 FM_BACKLOG_TRANSITION_ERROR= - root=$(fm_backlog_root "$data") || return 1 - if [ "$(fm_tasks_axi_backend "$root")" != markdown ]; then - out=$(cd "$root" 2>/dev/null && fm_tasks_axi "$verb" "$id" "$@" 2>&1) - command_status=$? + fm_backlog_source_present "$data" "$authorized_data" + source_status=$? + [ "$source_status" -eq 0 ] || return "$source_status" + fm_backlog_tasks_axi_addressing "$data" + source_status=$? + [ "$source_status" -eq 0 ] || return "$source_status" + if [ -n "$FM_BACKLOG_AXI_FILE" ]; then + out=$(cd "$FM_BACKLOG_AXI_ROOT" 2>/dev/null && fm_tasks_axi "$verb" "$id" "$@" --file "$FM_BACKLOG_AXI_FILE" 2>&1) else - file=$(fm_backlog_file "$data") || return 1 - fm_backlog_record_present "$file" "backlog file" "$authorized_data" || return 1 - out=$(cd "$root" 2>/dev/null && fm_tasks_axi "$verb" "$id" \ - --file "$file" "$@" 2>&1) - command_status=$? + out=$(cd "$FM_BACKLOG_AXI_ROOT" 2>/dev/null && fm_tasks_axi "$verb" "$id" "$@" 2>&1) fi + command_status=$? [ "$command_status" -ne 0 ] || return 0 FM_BACKLOG_TRANSITION_ERROR=$(printf '%s\n' "$out" | sed -n '1p') if [ -z "$FM_BACKLOG_TRANSITION_ERROR" ]; then @@ -419,17 +514,26 @@ fm_backlog_done() { # <data-dir> <id> [flag...] fm_backlog_mutate "$data" "done" "$id" "$@" } +fm_backlog_row_artifact_supported() { + local id=$1 flag=${2:-} value=${3:-} + case "$flag" in + --pr) return 0 ;; + --report) [ "$value" = "data/$id/report.md" ] ;; + *) return 1 ;; + esac +} + # Keep a captain-held row open across the removal of the work record that # discovered it: record the finished work's deliverable as one line at the end -# of the task body (a line already present is left alone) and return the row to -# Queued, the conventional post-cleanup shape for an open captain call. +# of the task body (a line already present is left alone), preserve supported +# artifacts on the row, and return it to Queued, the conventional post-cleanup +# shape for an open captain call. # bin/fm-fleet-snapshot.sh classifies that retained hold from its structured -# fields; only bin/fm-captain-hold.sh answer closes the call. The links are -# written into the body rather than through `tasks-axi update --report`, -# because that flag rewrites the title of a row that is not Done. +# fields; only bin/fm-captain-hold.sh answer resolves the call. fm_backlog_retain() { # <data-dir> <id> [flag...] local data authorized_data=$1 id=$2 out command_status previous_arg='' local arg deliverable='' line body new_body tmp + local -a row_args=() if ! data=$(fm_backlog_data_absolute "$1"); then FM_BACKLOG_TRANSITION_ERROR="data directory cannot be resolved: $1" return 1 @@ -438,8 +542,16 @@ fm_backlog_retain() { # <data-dir> <id> [flag...] FM_BACKLOG_TRANSITION_ERROR= for arg in "$@"; do case "$previous_arg" in - --report) deliverable="${deliverable:+$deliverable; }report $arg" ;; - --pr) deliverable="${deliverable:+$deliverable; }PR $arg" ;; + --report) + deliverable="${deliverable:+$deliverable; }report $arg" + if fm_backlog_row_artifact_supported "$id" --report "$arg"; then + row_args=(--report "$arg") + fi + ;; + --pr) + deliverable="${deliverable:+$deliverable; }PR $arg" + row_args=(--pr "$arg") + ;; --note) deliverable="${deliverable:+$deliverable; }$arg" ;; esac previous_arg=$arg @@ -488,6 +600,9 @@ fm_backlog_retain() { # <data-dir> <id> [flag...] ;; esac fi + if [ "${#row_args[@]}" -gt 0 ]; then + fm_backlog_mutate "$authorized_data" update "$id" "${row_args[@]}" || return 1 + fi fm_backlog_mutate "$authorized_data" reopen "$id" } @@ -499,9 +614,9 @@ fm_backlog_canonical_existing() { ' "$1" 2>/dev/null } -fm_backlog_record_parent_authorized() { - local path=$1 label=$2 root=$3 parent base parent_resolved expected_path - local path_resolved root_resolved home_resolved final_matches=1 +fm_backlog_record_parent_authorized() { # <path> <label> <root> [parent-only] + local path=$1 label=$2 root=$3 parent_only=${4:-} parent base parent_resolved expected_path + local path_resolved root_resolved root_prefix home_resolved final_matches=1 parent=${path%/*} [ "$parent" != "$path" ] || parent=. base=${path##*/} @@ -535,7 +650,7 @@ fm_backlog_record_parent_authorized() { return 1 } expected_path=${parent_resolved%/}/$base - if [ -e "$path" ] || [ -L "$path" ]; then + if [ -z "$parent_only" ] && { [ -e "$path" ] || [ -L "$path" ]; }; then path_resolved=$(fm_backlog_canonical_existing "$path") || { FM_BACKLOG_TRANSITION_ERROR="$label cannot be resolved at $path" return 1 @@ -544,8 +659,9 @@ fm_backlog_record_parent_authorized() { else path_resolved=$expected_path fi + root_prefix=${root_resolved%/}/ case "$path_resolved" in - "$root_resolved"/*) ;; + "$root_prefix"*) ;; *) FM_BACKLOG_TRANSITION_ERROR="$label resolves outside its authorized directory at $path" return 1 @@ -620,6 +736,24 @@ fm_backlog_meta_spawn_gen() { FM_BACKLOG_META_SPAWN_GEN=$value } +# The same incarnation, read for a caller that only needs to notice a CHANGE. +# A record predating the field carries no incarnation to compare, so it yields +# an empty value and proceeds instead of refusing; comparing that empty value +# across a wait still catches a record that gained, lost, or altered one. An +# ambiguous or unreadable field is still an error, because a record that cannot +# name one exact incarnation cannot be compared at all. +fm_backlog_meta_spawn_gen_optional() { # <meta> <state> + local meta=$1 state=$2 count + FM_BACKLOG_META_SPAWN_GEN= + fm_backlog_record_present "$meta" "task record" "$state" || return 1 + count=$(LC_ALL=C awk -F= '$1 == "spawn_gen" { count++ } END { print count + 0 }' "$meta" 2>/dev/null) || { + FM_BACKLOG_TRANSITION_ERROR="unreadable spawn generation in task record $meta" + return 1 + } + [ "$count" -ne 0 ] || return 0 + fm_backlog_meta_spawn_gen "$meta" "$state" +} + fm_backlog_row_dispatchable() { case "$1" in in_flight\ no\ no|queued\ no\ no) return 0 ;; diff --git a/bin/fm-bearings-snapshot.sh b/bin/fm-bearings-snapshot.sh index d18bae9c989..f5b565bae85 100755 --- a/bin/fm-bearings-snapshot.sh +++ b/bin/fm-bearings-snapshot.sh @@ -55,7 +55,7 @@ # The default landed baseline is balanced across homes: each home keeps its internal # newest-first ordering, homes iterate in deterministic id order, sparse homes do not # waste capacity, and --all-landed switches back to the complete global newest-first -# order. +# order. Which closed rows either side contributes is bin/fm-landed-lib.sh's rule. # # Flags: # (default) compact projection with bounded remote-ledger collection, TOON @@ -82,6 +82,9 @@ FLEET="$SCRIPT_DIR/fm-fleet-snapshot.sh" # shellcheck source=bin/fm-timeout-lib.sh # shellcheck disable=SC1091 . "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-landed-lib.sh +# shellcheck disable=SC1091 +. "$SCRIPT_DIR/fm-landed-lib.sh" # FM_LANDED_JQ_DEFS: the shared landed selector # Bounds (overridable for tests / large fleets). FM_BEARINGS_LANDED=${FM_BEARINGS_LANDED:-6} @@ -335,7 +338,7 @@ MODEL=$(printf '%s' "$SNAP" | jq \ --argjson pr_repos_shown "$PR_REPOS_SHOWN" \ --argjson pr_rows_capped "$PR_ROWS_CAPPED" \ --argjson pr_rows_min_total "$PR_ROWS_MIN_TOTAL" \ - --slurpfile candidate_prs_doc <(printf '%s' "$CANDIDATE_PRS") ' + --slurpfile candidate_prs_doc <(printf '%s' "$CANDIDATE_PRS") "$FM_LANDED_JQ_DEFS"' def doc($v): ($v[0] // error("fm-bearings-snapshot: empty jq payload")); def trunc($n): if . == null then null else (tostring | gsub("\\s+"; " ") | if (length > $n) then (.[:$n] + "…") else . end) end; @@ -399,8 +402,9 @@ MODEL=$(printf '%s' "$SNAP" | jq \ | (($fl | index("paths")) != null) as $f_paths | (($fl | index("actions")) != null) as $f_actions | (($fl | index("endpoints")) != null) as $f_endpoints - | ([ .backlog.records[] | select(.state == "done" and .structured and .hold_kind != "captain") - | {id, title, pr_url, report_path, local_note, completion, home:"(main)", home_id:"(main)"} ]) as $main_done + | ([ .backlog.records[] | select(landed_record) + | {id, title, kind, hold_kind, pr_url, report_path, local_note, completion, + home:"(main)", home_id:"(main)"} ]) as $main_done | ((.secondmate_landed.records) // []) as $mate_done | ($main_done + $mate_done) as $all_landed_rows | ([ $all_landed_rows | group_by(.home_id)[] @@ -546,7 +550,7 @@ MODEL=$(printf '%s' "$SNAP" | jq \ | {id, spawn_gen:(.spawn_gen // null), host:(.host // null), kind:(.reconcile_inventory.kind // null), ids:((.reconcile_inventory.ids // []) | map(select(type == "string")) | sort)} ], decisions_open: (if $all_decisions == 1 then $decisions_all else $decisions_all[:$decisions_n] end), landed: ($done | map({id, what:(.title | trunc(70)), - artifact:(.pr_url // .report_path // .local_note // "-"),owner:.home_id})), + artifact:(landed_artifact // "-"),owner:.home_id})), gates: (if $all_queued == 1 then $gates_all else $gates_all[:$gates_n] end), reports: (if $all_reports == 1 then $reports_all else $reports_all[:$reports_n] end), recorded_prs: (if $all_recorded_prs == 1 then $recorded_prs_all else $recorded_prs_all[:$recorded_prs_n] end) diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index 2abbea0606d..f318d3031e5 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -160,7 +160,7 @@ # checkout command. Used by # fm-session-start.sh's read-only path when another live session holds # the fleet lock, so a second concurrent session never race-mutates -# secondmate homes, pending handoff outboxes, +# secondmate homes, pending handoff outboxes and receiver wakes, # X-mode artifacts, project clones, the usage store and its per-task # session maps, or repair instructions. # Unset/0 (the default) runs every sweep exactly as before - this flag @@ -1427,9 +1427,10 @@ crew_dispatch_validate() { fi err=$(jq -r ' def verified($h): ["claude","codex","opencode","pi","pi-signed","grok","kimi","cursor","muse","gemini","rovo","omp","agy"] | index($h); - def effort_ok($h; $e): + def effort_ok($h; $m; $e): if $e == null then true elif ($e | type) != "string" then false + elif $e == "ultra" then (($h == "pi" or $h == "pi-signed") and (($m | type) == "string") and ($m | startswith("codex-native/")) and ($m | length) > 13) elif $h == "claude" then (["low","medium","high","xhigh","max"] | index($e)) elif $h == "codex" then (["low","medium","high","xhigh"] | index($e)) elif $h == "grok" then (["low","medium","high"] | index($e)) @@ -1453,10 +1454,10 @@ crew_dispatch_validate() { or ($items | any(has("effort") and (((.effort | type) != "string") or (.effort | length) == 0))); def bad_efforts: configured_profiles - | map({h: .harness, e: .effort}) + | map({h: .harness, m: .model, e: .effort}) | map(select(.e != null)) | map(select((.h | type) == "string" and verified(.h))) - | map(select(. as $p | effort_ok($p.h; $p.e) | not)) + | map(select(. as $p | effort_ok($p.h; $p.m; $p.e) | not)) | map("\(.h):\(.e)") | unique; if type != "object" then "top-level value must be an object" diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index 329153c05e8..eae9dc91806 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -785,7 +785,7 @@ Handle routine work yourself. Report only true captain-relevant outcomes or a declared external wait by appending one line: \`echo "{state}: {one short line}" >> $STATUS_FILE\` States: working, needs-decision, blocked, $PAUSED_VERB, done, failed. -Use \`$PAUSED_VERB: {why}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait you expect to clear on its own; use \`blocked:\` when you are stuck and need firstmate to act. +Use \`$PAUSED_VERB: {why}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait you expect to clear on its own, naming when it clears with \`until <YYYY-MM-DDTHH:MMZ>\` (UTC) when you know; use \`blocked:\` when you are stuck and need firstmate to act. Use this only for material phase changes, a captain decision, a real blocker, a failure, work ready for review, or work you landed. Work you landed includes a merge you performed yourself under standing merge authority and one the captain merged on the forge: under that authority nothing is ever \"ready for review\", so a landed merge that goes unreported reaches the captain as silence. This is also how you return the answer to a marked from-firstmate request above. @@ -992,7 +992,9 @@ The report is your only task-authored deliverable, so anything worth keeping mus Use \`$PAUSED_VERB: {why}\` - distinct from \`blocked:\` - ONLY when you are deliberately idling on a known external wait you expect to clear on its own (an upstream release, a rate-limit reset): firstmate then leaves your idle pane alone and rechecks it on a long cadence instead of - treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. + treating it as a possible wedge. When you know when the wait clears, say so in the line with + \`until <YYYY-MM-DDTHH:MMZ>\` (UTC) and firstmate rechecks at that time instead. + Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs to a human (product choices, destructive actions), append \`needs-decision: {summary of options}\` and stop. Firstmate will reply with the decision. diff --git a/bin/fm-busy-event.sh b/bin/fm-busy-event.sh index 17fde457810..aa4bfee82f3 100755 --- a/bin/fm-busy-event.sh +++ b/bin/fm-busy-event.sh @@ -23,6 +23,12 @@ # paths (fm-recovery) may pass --current-gen to bind to the incarnation # armed right now. # +# progress <state-dir> <id> --gen G +# Refresh state/<id>.progress for observed native-harness activity under +# the incarnation lock. This neither changes busy state nor emits a +# turn-ended notification. Arm and retire clear the marker, and an old +# incarnation can never refresh its replacement's progress. +# # retire <state-dir> <id> (--gen G | --current-gen) # Remove one incarnation's sidecar and record while holding the same # writer lock used by arm and apply. An exact gen prevents teardown for @@ -39,6 +45,7 @@ usage() { usage: fm-busy-event.sh arm <state-dir> <id> [--state busy|idle|unknown] [--source S] [--event E] fm-busy-event.sh apply <state-dir> <id> <busy|idle|unknown> (--gen G | --current-gen) --source S --event E + fm-busy-event.sh progress <state-dir> <id> --gen G fm-busy-event.sh retire <state-dir> <id> (--gen G | --current-gen) See the header comment for the full contract. EOF @@ -51,7 +58,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" CMD=${1:-} case "$CMD" in - arm|apply|retire) shift ;; + arm|apply|progress|retire) shift ;; *) usage ;; esac @@ -85,12 +92,14 @@ while [ $# -gt 0 ]; do *) usage ;; esac done -if [ "$CMD" != retire ]; then +if [ "$CMD" = apply ] || [ "$CMD" = arm ]; then case "$NEW_STATE" in busy|idle|unknown) : ;; *) usage ;; esac fm_busy_token_valid "$SOURCE" || { echo "error: invalid --source" >&2; exit 1; } fm_busy_token_valid "$EVENT" || { echo "error: invalid --event" >&2; exit 1; } fi +[ "$CMD" != progress ] || [ "$USE_CURRENT_GEN" = 0 ] || usage + REC=$(fm_busy_record_path "$STATE" "$ID") GEN_FILE=$(fm_busy_gen_path "$STATE" "$ID") LOCK="$REC.lock" @@ -151,7 +160,7 @@ if [ "$CMD" = arm ]; then lock_acquire || exit 1 { printf '%s\n' "$GEN" > "$GEN_FILE.tmp.$$" && mv -f "$GEN_FILE.tmp.$$" "$GEN_FILE" \ - && write_record "$GEN" 1 + && write_record "$GEN" 1 && rm -f "$STATE/$ID.progress" } || { lock_release; umask "$old_umask"; echo "error: arm failed for $ID" >&2; exit 1; } lock_release umask "$old_umask" @@ -159,7 +168,7 @@ if [ "$CMD" = arm ]; then exit 0 fi -# apply / retire +# apply / progress / retire if [ "$USE_CURRENT_GEN" = 1 ] && [ "$CMD" != retire ]; then GEN=$(fm_busy_current_gen "$STATE" "$ID") || { umask "$old_umask" @@ -174,7 +183,7 @@ fi lock_acquire || { umask "$old_umask"; exit 1; } CURRENT=$(fm_busy_current_gen "$STATE" "$ID") || { if [ "$CMD" = retire ] && [ ! -e "$GEN_FILE" ] && [ ! -L "$GEN_FILE" ]; then - rm -f "$REC" || { + rm -f "$REC" "$STATE/$ID.progress" || { lock_release umask "$old_umask" echo "error: busy-state retirement failed for $ID" >&2 @@ -199,7 +208,7 @@ if [ "$GEN" != "$CURRENT" ]; then exit 1 fi if [ "$CMD" = retire ]; then - rm -f "$GEN_FILE" "$REC" || { + rm -f "$GEN_FILE" "$REC" "$STATE/$ID.progress" || { lock_release umask "$old_umask" echo "error: busy-state retirement failed for $ID" >&2 @@ -209,6 +218,12 @@ if [ "$CMD" = retire ]; then umask "$old_umask" exit 0 fi +if [ "$CMD" = progress ]; then + touch "$STATE/$ID.progress" || { lock_release; umask "$old_umask"; exit 1; } + lock_release + umask "$old_umask" + exit 0 +fi OLD_SEQ=0 if [ -f "$REC" ]; then old_line=$(head -n 1 "$REC" 2>/dev/null || true) diff --git a/bin/fm-captain-hold.sh b/bin/fm-captain-hold.sh index 3ebc13cb923..a9ef07b132a 100755 --- a/bin/fm-captain-hold.sh +++ b/bin/fm-captain-hold.sh @@ -47,13 +47,13 @@ # captain's own deferral date through `tasks-axi hold --until`, so a "revisit # later" answer is stored as a date instead of a live card. # -# `answer` records the captain's exact words and closes the call in the same +# `answer` records the captain's exact words and resolves the call in the same # act. It requires a non-empty captain decision file of at most 8192 bytes and # writes a resolution block while preserving the leading hold-set stamp until # the close succeeds (the previous body is preserved and archived through -# tasks-axi --archive-body). It then closes the task with `tasks-axi done` - or, +# tasks-axi --archive-body). It closes a question with `tasks-axi done` - or, # with `--release`, lifts the hold with `tasks-axi unhold` so a captain-gated -# WORK item resumes instead of closing - and restores resolution-first body +# WORK item resumes without closing - and restores resolution-first body # ordering. An exact retry also completes unfinished ordering normalization and # is idempotent only when its requested close mode # matches the newest record; a changed decision or a mode mismatch is rejected. @@ -66,9 +66,9 @@ # `held:` bit, prove the captain owned it. # # ONE KEYED-ANSWER INTAKE, FED BY EVERY CHANNEL. -# "A keyed answer closes its matching captain-held task" is a single +# "A keyed answer resolves its matching captain-held task" is a single # capability, owned here and nowhere else. `answers` reads -# `<task-id>\t<answer>\t<label>[\t<mode>]` lines on stdin and closes each named +# `<task-id>\t<answer>\t<label>[\t<mode>]` lines on stdin and resolves each named # task through the very same `answer` path above, so every guard applies # identically no matter which channel the answer arrived on. The key IS the # task id - no identity arithmetic. The optional fourth field selects the close: @@ -157,7 +157,8 @@ # is (not Done, hold kind captain), 1 means it is not, and 2 means the answer # could not be established, so a caller that must never close a live call can # treat "cannot tell" as its own case instead of as a no. With -# `--distinguish-absent`, an absent local task returns 3 instead of 1. +# `--distinguish-absent`, an absent local task returns 3 instead of 1; a home +# with no backlog file counts as absent, because it records no captain calls. # It prints nothing on these predicate results and mutates nothing, unless # `--identity` asks it to print this call's # LIFECYCLE identity, which it does on an exit 0 only. That identity - the @@ -331,10 +332,11 @@ load_decision() { # <path>; sets DECISION_TEXT and DECISION_DIGEST # the root's own tasks-axi configuration, exactly like the transition library's # mutate path. tasks_axi() { - local data file root + local data file root backend data=$(fm_backlog_data_absolute "$DATA") || fail "data directory cannot be resolved: $DATA" root=$(fm_backlog_root "$data") || fail "$FM_BACKLOG_TRANSITION_ERROR" - if [ "$(fm_tasks_axi_backend "$root")" = markdown ]; then + backend=$(fm_tasks_axi_backend "$root") || return 2 + if [ "$backend" = markdown ]; then file=$(fm_backlog_file "$data") || fail "$FM_BACKLOG_TRANSITION_ERROR" (cd "$root" && tasks-axi "$@" --file "$file") else @@ -558,13 +560,14 @@ captain_beads_setting() { # <entries-output> <setting> # report's handful of attested ids. Returns 0 when the listing loads, and 2 # with the reason on stderr when the graph cannot be read. captain_migration_scan_load() { # <resolved-data-dir> - local data=$1 root entries bd_bin bd_path + local data=$1 root entries bd_bin bd_path backend [ "$CAPTAIN_MIGRATION_SCAN_LOADED" = 1 ] && return 0 root=$(fm_backlog_root "$data") || { printf 'fm-captain-hold: the configured data directory cannot be resolved for a migration scan: %s\n' "$FM_BACKLOG_TRANSITION_ERROR" >&2 return 2 } - if [ "$(fm_tasks_axi_backend "$root")" != beads ]; then + backend=$(fm_tasks_axi_backend "$root") || return 2 + if [ "$backend" != beads ]; then CAPTAIN_MIGRATION_SCAN_LOADED=1 return 0 fi @@ -614,7 +617,7 @@ captain_migration_scan_load() { # <resolved-data-dir> # guess, so it only runs when no marker line matches any identity and it accepts # a row solely when that row is itself still held for the captain. resolve_migrated_entry() { # <origin-or-empty> <entry> - local origin=$1 entry=$2 data root entries prefix derived show + local origin=$1 entry=$2 data root entries prefix derived show backend local candidate candidate_matches prefixed matches count prefixed_matches prefixed_count data=$(fm_backlog_data_absolute "$DATA") || { printf 'fm-captain-hold: the migrated hold of %s cannot be resolved: %s\n' \ @@ -626,7 +629,8 @@ resolve_migrated_entry() { # <origin-or-empty> <entry> "$entry" "${FM_BACKLOG_TRANSITION_ERROR:-the configured data directory $DATA cannot be resolved}" >&2 return 2 } - [ "$(fm_tasks_axi_backend "$root")" = beads ] || return 1 + backend=$(fm_tasks_axi_backend "$root") || return 2 + [ "$backend" = beads ] || return 1 # Every identity this entry could have been migrated under: the raw entry, # and - for a pre-collapse channel key - the derived legacy identity its # origin would have minted, because fm-hold-migration recorded the DERIVED @@ -819,10 +823,10 @@ command_hold() { validate_one_line repo "$repo" [ -z "$origin" ] || body=$(printf 'Origin: %s' "$origin") if [ -n "$body" ]; then - tasks_axi add "$id" "$title" --repo "$repo" --body "$body" >/dev/null \ + tasks_axi add "$id" "$title" --kind captain --repo "$repo" --body "$body" >/dev/null \ || fail "could not create task $id" else - tasks_axi add "$id" "$title" --repo "$repo" >/dev/null \ + tasks_axi add "$id" "$title" --kind captain --repo "$repo" >/dev/null \ || fail "could not create task $id" fi fi @@ -884,10 +888,34 @@ write_resolution_record() { # <task-id> <mode> <shown-body> rm -f -- "$tmp" } +report_retained_artifact_failure() { # <task-id> <marker-path> + printf 'fm-captain-hold: cannot apply the artifact recorded for %s in %s: %s\n' \ + "$1" "$2" "${FM_BACKLOG_TRANSITION_ERROR:-no reason reported}" >&2 +} + +apply_pending_retained_artifact() { # <task-id> + local id=$1 marker + local -a args=() + marker=$(fm_backlog_close_marker_path "$STATE" "$id") || return 1 + [ -e "$marker" ] || [ -L "$marker" ] || return 0 + fm_backlog_close_marker_validate "$marker" "$DATA" "$id" "$STATE" \ + || { report_retained_artifact_failure "$id" "$marker"; return 1; } + [ "$FM_BACKLOG_CLOSE_VALIDATED_MODE" = retain ] || return 0 + args=("${FM_BACKLOG_CLOSE_VALIDATED_ARGS[@]+"${FM_BACKLOG_CLOSE_VALIDATED_ARGS[@]}"}") + case "${args[0]-}" in + --pr|--report) + fm_backlog_row_artifact_supported "$id" "${args[@]}" || return 0 + fm_backlog_mutate "$DATA" update "$id" "${args[@]}" \ + || { report_retained_artifact_failure "$id" "$marker"; return 1; } + ;; + esac +} + close_answered() { # <task-id> <release-0-or-1> if [ "$2" = 1 ]; then tasks_axi unhold "$1" >/dev/null else + apply_pending_retained_artifact "$1" || return 1 tasks_axi "done" "$1" >/dev/null fi } @@ -1761,11 +1789,13 @@ EOF } # Still an open captain call? Exit 0 yes, 1 no, 2 cannot tell (see the header). -# A row this home does not carry is 3 when the caller requests the distinction; -# every other read failure is a 2, printed to stderr, because a mechanical -# closer must never read "cannot tell" as permission to close. +# A row this home does not carry is 3 when the caller requests the distinction, +# and so is a home with no backlog file at all, because a backlog that does not +# exist holds nothing. Every read failure over a record that DOES exist is a 2, +# printed to stderr, because a mechanical closer must never read "cannot tell" +# as permission to close. command_open() { # <task-id> [--identity] [--distinguish-absent] - local id='' identity=0 distinguish_absent=0 data state show shown_body + local id='' identity=0 distinguish_absent=0 data state root file backend show shown_body while [ "$#" -gt 0 ]; do case "$1" in --identity) identity=1 ;; @@ -1784,9 +1814,26 @@ command_open() { # <task-id> [--identity] [--distinguish-absent] exit 2 ;; esac - fm_tasks_axi_compatible || { printf 'fm-captain-hold: compatible tasks-axi is required\n' >&2; exit 2; } data=$(fm_backlog_data_absolute "$DATA") \ || { printf 'fm-captain-hold: data directory cannot be resolved: %s\n' "$DATA" >&2; exit 2; } + root=$(fm_backlog_root "$data") \ + || { printf 'fm-captain-hold: %s\n' "$FM_BACKLOG_TRANSITION_ERROR" >&2; exit 2; } + if ! backend=$(fm_tasks_axi_backend_resolve "$root"); then + exit 2 + fi + if [ "$backend" = markdown ]; then + file=$(fm_backlog_file "$data") \ + || { printf 'fm-captain-hold: %s\n' "$FM_BACKLOG_TRANSITION_ERROR" >&2; exit 2; } + if [ ! -e "$file" ] && [ ! -L "$file" ]; then + # No backlog file at all: this home records no captain calls, so the task + # is absent from it rather than held. A record that EXISTS but cannot be + # read is a different state and still leaves by the exit 2 paths below, + # because that one may hide a live hold. + [ "$distinguish_absent" = 0 ] || return 3 + return 1 + fi + fi + fm_tasks_axi_compatible || { printf 'fm-captain-hold: compatible tasks-axi is required\n' >&2; exit 2; } if fm_backlog_row_probe "$data" "$id"; then state=${FM_BACKLOG_ROW_STATE%% *} if [ "$state" != "done" ] && [ "$FM_BACKLOG_ROW_HOLD_KIND" = captain ]; then diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index 5ac81f00398..5030356c4f7 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -95,15 +95,33 @@ FM_CLASSIFY_CAPTAIN_RE_DEFAULT='done:|needs-decision:|blocked:|failed:|PR ready| # drift between the two consumers. FM_CLASSIFY_PAUSED_VERB overrides it. FM_CLASSIFY_PAUSED_VERB_DEFAULT='paused' -# Bounded re-surface cadence for a declared pause or a verified captain hold. +# Bounded re-surface cadence for a declared external-wait pause. # Far longer than the wedge threshold (FM_STALE_ESCALATE_SECS, default 240s), it -# avoids nagging a deliberate wait while ensuring a forgotten hold cannot rot -# invisibly - it re-surfaces once for a recheck every window. One hour by default; -# both consumers read FM_PAUSE_RESURFACE_SECS with this default so the cadence has -# one owner. +# avoids nagging a deliberate wait while ensuring a forgotten wait cannot rot +# invisibly - it re-surfaces once for a recheck every window. The shared +# pause_resurface_window owner widens the one-hour base for an unchanged wait. +# A worker may name its clearing time with `until` (status_paused_until below), +# which can bring a recheck forward without extending the current window. +# Verified held transfers retain the same bounded rechecks while away. # shellcheck disable=SC2034 # Read by the watcher and daemon (fm-watch.sh, fm-supervise-daemon.sh), not this lib. FM_PAUSE_RESURFACE_SECS_DEFAULT=3600 +# fm_utc_iso_to_epoch <YYYY-MM-DDTHH:MM[:SS]Z>: the one portable UTC ISO 8601 +# reader shared by the declared-wait vocabulary and the away-posture record +# (bin/fm-afk-contract.sh). Prints epoch seconds; returns 1 on any other shape +# so a malformed time is refused rather than read as "now". +fm_utc_iso_to_epoch() { # <timestamp> + local ts=$1 + case "$ts" in + [0-9][0-9][0-9][0-9]-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]Z) ts="${ts%Z}:00Z" ;; + [0-9][0-9][0-9][0-9]-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-5][0-9]Z) ;; + *) return 1 ;; + esac + date -u -j -f '%Y-%m-%dT%H:%M:%SZ' "$ts" +%s 2>/dev/null \ + || date -u -d "$ts" +%s 2>/dev/null \ + || return 1 +} + # How many times an UNCHANGED declared wait may be re-surfaced before its recheck # cadence stops widening. A fixed window re-surfaces a wait that has not changed # at the same rate forever: three tasks correctly parked on one external decision @@ -295,6 +313,23 @@ status_is_paused_or_captain_held() { # <status-line> status_is_paused "$line" || status_is_captain_held "$line" } +# A condition-aware declared wait: a `paused:` line may say WHEN it expects to +# clear with `until <YYYY-MM-DDTHH:MM[:SS]Z>` anywhere in its text (UTC only, so +# no local-zone guess is ever recorded). Prints that time as epoch seconds so a +# supervisor rechecks the wait when the worker said it would clear instead of on +# the flat cadence; returns 1 when the line is not a pause or declares no time, +# or the time is malformed, so a bad token falls back to the cadence rather than +# silencing the wait. +status_paused_until() { # <status-line> -> epoch on stdout + local line=$1 token + status_is_paused "$line" || return 1 + token=$(printf '%s' "$line" \ + | sed -n 's/.*[[:space:]][Uu][Nn][Tt][Ii][Ll][[:space:]]\{1,\}\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]Z\).*/\1/p; s/.*[[:space:]][Uu][Nn][Tt][Ii][Ll][[:space:]]\{1,\}\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]Z\).*/\1/p' \ + | head -1) + [ -n "$token" ] || return 1 + fm_utc_iso_to_epoch "$token" +} + # --- durable keyed decisions ------------------------------------------------ # # The status stream is an append-only EVENT log. Reading it last-event-wins diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index 5f8b2d14a75..282866ba160 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -53,7 +53,8 @@ # until the synchronous guard has consumed its attended fail-open. # # The epoch ledger state/.claude-autoarm-epoch records the latest claim -# generation and outcome so the synchronous Stop guard +# generation and outcome, and binds rewake outcomes to the session-lock pid and +# watcher recovery generation, so the synchronous Stop guard # (bin/fm-turnend-guard.sh --claude) can allow a stop whose recovery this hook # already owns, instead of forcing a duplicate continuation for the same event # epoch. The failure marker @@ -183,10 +184,20 @@ MY_GEN=$FM_AUTOARM_MY_GEN # (cleanup, exit 0) - the harness discards the collected stderr on exit 0, so # even an already-printed banner is never delivered by a losing generation. autoarm_commit() { # <outcome> [marker-file] - if [ -n "${2:-}" ]; then - fm_autoarm_write_owned "$STATE" "$MY_GEN" "$1" "$2" + local outcome=$1 marker=${2:-} session_pid recovery + if [ "$outcome" = rewake ]; then + fm_session_lock_owned_by_self "$STATE" || return 2 + session_pid=$(sed -n '1p' "$STATE/.lock" 2>/dev/null || true) + fm_recovery_marker_snapshot "$STATE/.watcher-down" || return 2 + case "$FM_RECOVERY_MARKER_TOKEN" in + pending:downtime:*|announced:downtime:*) recovery=${FM_RECOVERY_MARKER_TOKEN##*:} ;; + *) return 2 ;; + esac + fm_autoarm_write_owned "$STATE" "$MY_GEN" "$outcome" "$marker" "$session_pid" "$recovery" + elif [ -n "$marker" ]; then + fm_autoarm_write_owned "$STATE" "$MY_GEN" "$outcome" "$marker" else - fm_autoarm_write_owned "$STATE" "$MY_GEN" "$1" + fm_autoarm_write_owned "$STATE" "$MY_GEN" "$outcome" fi } diff --git a/bin/fm-control.sh b/bin/fm-control.sh index a1e435b3b31..9f0998bf050 100755 --- a/bin/fm-control.sh +++ b/bin/fm-control.sh @@ -248,8 +248,8 @@ fi [ "$MODEL_SET" = 0 ] || [ -n "$NEW_MODEL" ] || die "--model requires a non-empty value" [ "$EFFORT_SET" = 0 ] || [ -n "$NEW_EFFORT" ] || die "--effort requires a non-empty value" case "$NEW_EFFORT" in - ''|default|low|medium|high|xhigh|max) ;; - *) die "--effort must be one of default, low, medium, high, xhigh, max" ;; + ''|default|low|medium|high|xhigh|max|ultra) ;; + *) die "--effort must be one of default, low, medium, high, xhigh, max, ultra" ;; esac # --- exact task-id resolution ---------------------------------------------- @@ -652,9 +652,9 @@ resolve_relaunch_profile() { CONFIG_MODEL=$("$SCRIPT_DIR/fm-harness.sh" secondmate-model 2>/dev/null || true) CONFIG_EFFORT=$("$SCRIPT_DIR/fm-harness.sh" secondmate-effort 2>/dev/null || true) case "$CONFIG_EFFORT" in - ''|low|medium|high|xhigh|max) ;; + ''|low|medium|high|xhigh|max|ultra) ;; *) - echo "warning: config/secondmate-harness effort token '$CONFIG_EFFORT' is not one of low, medium, high, xhigh, max; ignoring" >&2 + echo "warning: config/secondmate-harness effort token '$CONFIG_EFFORT' is not one of low, medium, high, xhigh, max, ultra; ignoring" >&2 CONFIG_EFFORT= ;; esac @@ -697,6 +697,9 @@ resolve_relaunch_profile() { else TARGET_EFFORT=default fi + if [ "$TARGET_EFFORT" = ultra ]; then + "$SCRIPT_DIR/fm-harness.sh" validate-native-effort "$TARGET_HARNESS" "$TARGET_MODEL" "$TARGET_EFFORT" || return 1 + fi } # safe_checkpoint: prove, before anything is stopped, that the work a relaunch diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index 6aa3736ef79..df12bed0b10 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -134,10 +134,12 @@ META=${FM_CREW_STATE_META_OVERRIDE:-"$STATE/$ID.meta"} LOG=${FM_CREW_STATE_STATUS_OVERRIDE:-"$STATE/$ID.status"} NM_TIMEOUT=${FM_CREW_STATE_NM_TIMEOUT:-10} case "$NM_TIMEOUT" in ''|*[!0-9]*) NM_TIMEOUT=10 ;; esac -# How many of the most recent `no-mistakes runs` rows the cross-branch fallback -# (fm_nm_runs_status_for_worktree in bin/fm-nm-run-lib.sh) scans. Generous -# enough to still find a branch's own run on a busy multi-crew fleet without -# listing the entire history every call. +# How many of the most recent `no-mistakes runs` rows each ledger read +# (fm_nm_runs_status_for_worktree in bin/fm-nm-run-lib.sh) scans, whether it is +# the cross-branch fallback or the live-sibling probe behind a terminal `axi +# status` answer (docs/configuration.md owns the setting). Generous enough to +# still find a branch's own run on a busy multi-crew fleet without listing the +# entire history every call. FM_CREW_STATE_RUNS_LIMIT=${FM_CREW_STATE_RUNS_LIMIT:-200} case "$FM_CREW_STATE_RUNS_LIMIT" in ''|*[!0-9]*) FM_CREW_STATE_RUNS_LIMIT=200 ;; esac # How long a recorded run-step stays usable as the degraded answer after the run @@ -1046,6 +1048,22 @@ if tracked_output_kind && [ -n "$WORKTREE_BRANCH" ] && [ -n "$LOOKUP_BRANCH" ] \ LOOKUP_COMPLETED=1 if [ -n "$TASK_BRANCH" ]; then HAVE_RUN=1 + # A terminal AXI answer is provisional until the ledger excludes a + # live sibling. Failed or inconclusive probes retain full AXI detail. + if ! fm_nm_run_is_active "$RUN_OUT"; then + live_row= + if live_runs=$(nm_run runs --limit "$FM_CREW_STATE_RUNS_LIMIT"); then + live_row=$(fm_nm_runs_row_for_worktree "$WT" "$LOOKUP_BRANCH" "$live_runs") + fi + IFS=$'\t' read -r live_evidence live_status live_head <<< "$live_row" + if [ "$live_evidence" = attributable ] \ + && [ "$(fm_nm_run_status_class "$live_status")" = live ]; then + COARSE_STATUS=$live_status + COARSE_HEAD=$live_head + COARSE_EVIDENCE=$live_evidence + RUN_SOURCE=coarse + fi + fi else RUN_ATTRIBUTION_FAULT="run on $run_branch is unattributable: task branch not recorded" fi diff --git a/bin/fm-fleet-snapshot.sh b/bin/fm-fleet-snapshot.sh index 7771338034c..5a32c500b32 100755 --- a/bin/fm-fleet-snapshot.sh +++ b/bin/fm-fleet-snapshot.sh @@ -145,9 +145,10 @@ # reconcile_inventory independently of projection trust. # Actionable captain holds appear in decisions_open; every captain hold remains # in the bounded queued inventory with its structured classification metadata. -# Structured-home input must declare the current hold-classifier schema; an -# older live ledger or cached copy is invalid even when it contains no captain -# holds, and leaves the home explicitly unreadable until its producer refreshes it. +# Structured-home input must declare the current home-summary and hold-classifier +# schemas; a live ledger or cached copy missing either declaration or declaring +# an unsupported version is unavailable even when it contains no captain holds. +# These schemas also accept v1 summaries from older producers. # secondmate_landed: {records[],truncated[],unreadable[],partial[]} - the # compatibility landed-work roll-up derived from secondmate_current. Readable # structured homes are partial, not unreadable, when an unavailable child state @@ -155,6 +156,8 @@ # they retain independently trustworthy structured surfaces. An inventory # mismatch also keeps the home's own current classification, which only an # unavailable child state or an untrustworthy backlog collapses to unknown. +# Which closed rows a home contributes is bin/fm-landed-lib.sh's rule, shared +# with the bearings projection so one Recently Landed section has one owner. # secondmate_guidance: return-channel action note for renderers and bearings. # card_precedence: the ordered column ladder every tasks[].card is resolved # against, highest-priority first. Exactly one column wins per task, and the @@ -327,6 +330,9 @@ esac # shellcheck source=bin/fm-timeout-lib.sh # shellcheck disable=SC1091 . "$SCRIPT_DIR/fm-timeout-lib.sh" # fm_run_timed: the owner of bounded external execution +# shellcheck source=bin/fm-landed-lib.sh +# shellcheck disable=SC1091 +. "$SCRIPT_DIR/fm-landed-lib.sh" # FM_LANDED_JQ_DEFS: the shared landed selector usage() { cat <<'EOF' @@ -715,6 +721,21 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG | if $v == null then null else ($v | trim) end; def metadata($rest; $key): cap($rest; ".*(?:\\(|,[[:space:]]*)" + $key + ":[[:space:]]*(?<v>[^,)]*)"); + # LOAD-BEARING, do not remove as a duplicate definition of the kind field. + # tasks-axi 0.2.5 omits the (kind: ...) metadata when a title starts with + # uppercase SCOUT or SHIP at a JavaScript word boundary (ASCII letters, + # digits, and underscore are word characters), so those rows carry no + # explicit kind to read. Without this fallback a scout whose title starts + # with SCOUT reports kind null, its + # recorded report stops counting as a delivery, and it drops out of Recently + # Landed - the defect this selector exists to fix. Pinned by + # the producer word-boundary regression in tests/fm-bearings-snapshot.test.sh. + def kind_of($rest): + metadata($rest; "kind") as $kind + | if $kind != null then $kind + elif ($rest | test("^SCOUT(?![A-Za-z0-9_])")) then "scout" + elif ($rest | test("^SHIP(?![A-Za-z0-9_])")) then "ship" + else null end; def hold_metadata($rest): # Canonical holds have their own parentheses and may contain commas. # Retain legacy combined metadata blocks without reading title prose. @@ -764,7 +785,7 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG checked:($m.check | test("[xX]")), title:title_of($rest), repo:metadata($rest; "repo"), - kind:metadata($rest; "kind"), + kind:kind_of($rest), priority:metadata($rest; "priority"), hold_reason:hold_metadata($rest), hold_kind:metadata($rest; "hold-kind"), @@ -816,6 +837,12 @@ backlog_json() { # [<backlog-path>] - defaults to this home's $BACKLOG | .records |= map( if (.body_lines | length) > 0 then .hold_set = cap(.body_lines[0]; "^Captain hold set:[[:space:]]*(?<v>[0-9]{4}-[0-9]{2}-[0-9]{2}(?:T[0-9]{2}:[0-9]{2}:[0-9]{2}Z)?)$") + | .local_note = (.local_note + // (if any(.body_lines[]; + test("^Resolution recorded by fm-(captain|decision)-hold\\.$")) + then null + else cap(.body_lines[-1]; "^(?<v>local main)$") + end)) | .body_excerpt = ((.body_lines | join(" "))[:240]) else . end) | .records as $records @@ -1456,7 +1483,7 @@ secondmate_home_summary_json() { # <backlog-json-file> <tasks-json-file> --argjson decisions_n "$FM_SNAPSHOT_SECONDMATE_DECISIONS" \ --argjson landed_n "$FM_SNAPSHOT_SECONDMATE_LANDED_PER_HOME" \ --slurpfile backlog "$1" \ - --slurpfile tasks "$2" ' + --slurpfile tasks "$2" "$FM_LANDED_JQ_DEFS"' ($backlog[0] // error("fm-fleet-snapshot: empty JSON payload")) as $backlog | ($tasks[0] // error("fm-fleet-snapshot: empty JSON payload")) as $tasks | def trunc($n): @@ -1478,8 +1505,10 @@ secondmate_home_summary_json() { # <backlog-json-file> <tasks-json-file> hold_until:(.hold_until // null), hold_bucket:(.hold_bucket // null), hold_age_days:(.hold_age_days // null),source:"backlog"} ]) as $captain_holds_all - | ([ $backlog.records[]? | select(.state == "done" and .structured and .hold_kind != "captain") + | ([ $backlog.records[]? | select(landed_record) | {id:(.id | trunc(120)),title:(.title | trunc(120)), + kind:((.kind // null) | if . == null then null else trunc(40) end), + hold_kind:((.hold_kind // null) | if . == null then null else trunc(40) end), pr_url:((.pr_url // null) | if . == null then null else trunc(500) end), report_path:((.report_path // null) | if . == null then null else trunc(500) end), local_note:((.local_note // null) | if . == null then null else trunc(120) end),completion} ] @@ -1787,6 +1816,7 @@ summary_file_read() { # <file> <expected-home> <output-file> return 0 } + summary_file_oversized() { # <file> local bytes [ -f "$1" ] && [ ! -L "$1" ] || return 1 diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index 7e897cd7836..6abc517c12e 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -11,8 +11,10 @@ # it in the tool output of whatever it was doing - the one channel every harness # has. Supervision health is MODEL-AWARE (fm_watcher_supervision_verdict in # bin/fm-wake-lib.sh): under the Claude Stop auto-arm model the watcher runs only -# between turns, so mid-turn a fresh beacon with no live watcher is healthy and -# only a stale beacon (beyond FM_GUARD_GRACE) is a genuine lapse; under the Pi +# between turns, so mid-turn a fresh beacon with no live watcher is healthy, and +# a stale beacon is still healthy while fm_autoarm_midturn_healthy proves a +# Claude auto-arm generation explains the gap; only a stale beacon with no such +# generation is a genuine lapse; under the Pi # extension model the extension tears the watcher down and respawns it on every # actionable wake, so a fresh beacon with a genuinely unheld lock is healthy # while that live Pi session provably owns continuity; any held but unhealthy diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh index b62c4fd073f..96443cf60c9 100755 --- a/bin/fm-harness.sh +++ b/bin/fm-harness.sh @@ -13,6 +13,12 @@ # config/secondmate-harness, or empty when absent. # fm-harness.sh secondmate-effort print the optional EFFORT token from # config/secondmate-harness, or empty when absent. +# fm-harness.sh validate-native-effort <harness> <model> <effort> +# Refuse ultra unless the harness is pi or +# pi-signed and the model explicitly names +# codex-native/<id>. Other efforts retain +# their adapter's existing policy. Native +# Codex validates model support at startup. # config/secondmate-harness format: a single line "<harness> [<model>] [<effort>]", # whitespace-separated. A bare "<harness>" (today's format) behaves exactly as before: # harness only, no model/effort. Only the first non-empty, non-comment line is parsed. @@ -270,7 +276,20 @@ resolve_secondmate_effort() { secondmate_field 3 } +validate_native_effort() { + local harness=${1:-} model=${2:-} effort=${3:-} + [ "$effort" = ultra ] || return 0 + case "$harness" in + pi|pi-signed) + case "$model" in codex-native/?*) return 0 ;; esac + ;; + esac + echo "error: ultra effort requires pi or pi-signed with an explicit codex-native/<model> model" >&2 + return 1 +} + case "${1:-}" in + validate-native-effort) shift; validate_native_effort "$@" ;; crew) resolve_crew ;; secondmate) resolve_secondmate ;; secondmate-model) resolve_secondmate_model ;; diff --git a/bin/fm-landed-lib.sh b/bin/fm-landed-lib.sh new file mode 100644 index 00000000000..875d4acd430 --- /dev/null +++ b/bin/fm-landed-lib.sh @@ -0,0 +1,79 @@ +# shellcheck shell=bash +# Shared "what belongs in Recently Landed" rule. +# Usage: . bin/fm-landed-lib.sh; splice "$FM_LANDED_JQ_DEFS" ahead of a jq +# program, then select backlog rows with `landed_record`. +# +# ONE OWNER for the landed selector. Recently Landed is assembled from two +# separate jq programs - bin/fm-bearings-snapshot.sh projects this home's own +# Done rows, and bin/fm-fleet-snapshot.sh projects each secondmate home's Done +# rows into the roll-up that the same section merges in. Both answer the one +# question "is this closed row a delivery the captain should see", so the rule +# lives here and neither program restates it. +# +# A closed row is never actively held: tasks-axi clears the held flag when a +# task closes, but a non-release answer keeps hold-kind and the hold reason. +# Merge approval removes those annotations through the release contract before +# cleanup records the merged PR or local-only landing, so either artifact on a +# Done captain-hold row is not a delivery. A scout's recorded report is its +# delivery regardless of release state or other links in its title. +# +# The distinction that decides the section is delivery: Recently Landed is +# merged PRs, completed scouts, and finished local-only merges. A closed row +# whose artifact matches its merged or done completion verb is a delivery only +# when it retains no captain-question provenance. A retained scout is identified +# by its kind and recorded report because its title links do not change what it +# delivers. A captain question remains kind captain when it closes, so it is +# never rendered as shipped work even when its text names an artifact. +# A local-only completion is a delivery whether or not its row carries a kind. +# The retained hold-kind alone keeps answered calls out of the section. +# Merged PRs and reported scouts remain distinct. +# +# The backlog-selection compatibility fallback keeps a structured Done row +# whose three parsed artifact fields are absent when it does not retain +# hold-kind captain. +# That preserves kindless rows closed before artifact-aware selection without +# admitting answered captain calls or explicit reportless scouts. +# Already-selected v1 secondmate landed rows may omit kind. For those rows, +# landed_artifact preserves a report_path with a reported completion; this +# display compatibility does not admit kindless reports from raw backlog rows. +# tests/fm-bearings-snapshot.test.sh covers both fresh and cached v1 summaries. + +# shellcheck disable=SC2034 # Output global, read by the sourcing caller. +FM_LANDED_JQ_DEFS=' + def scout_report: + .kind == "scout" + and (.report_path // null) != null; + def landed_artifact: + if scout_report or (.kind == null and .completion.verb == "reported") then (.report_path // null) + elif .completion.verb == "merged" then (.pr_url // null) + elif .completion.verb == "done" then (.local_note // null) + else null + end; + # The kind-is-not-scout guards below and in the fallback are LOAD-BEARING: + # they keep an explicit scout that recorded no report out of Recently Landed. + # Without them such a row has none of the three artifacts, satisfies the + # compatibility fallback and renders as shipped work with an empty artifact. + # Pinned by tests/fm-captain-hold-lifecycle.test.sh on "released, retained, or + # rejected deliveries were misclassified". + def landed_delivery: + scout_report + or (.kind != "scout" + and .kind != "captain" + and .hold_kind != "captain" + and .completion.verb == "merged" + and (.pr_url // null) != null) + or (.kind != "scout" + and .kind != "captain" + and .hold_kind != "captain" + and .completion.verb == "done" + and (.local_note // null) != null); + def landed_record: + .state == "done" and .structured + and (landed_delivery + or (.kind != "scout" + and .kind != "captain" + and .hold_kind != "captain" + and (.pr_url // null) == null + and (.report_path // null) == null + and (.local_note // null) == null)); +' diff --git a/bin/fm-launch-lib.sh b/bin/fm-launch-lib.sh index 596752e6676..95f8afd96f6 100644 --- a/bin/fm-launch-lib.sh +++ b/bin/fm-launch-lib.sh @@ -440,10 +440,11 @@ fm_launch_model_flag() { } # fm_launch_effort_flag: render the per-harness effort flag for <harness> given -# <effort>, or nothing when the effort is empty/default, the harness has no -# effort flag, or the level is outside that harness's verified vocabulary. +# <effort> and optional <model>, or nothing when effort is empty/default or +# unsupported. Pi native ultra requires an explicit codex-native model and +# renders its provider flag rather than Pi's ordinary thinking flag. fm_launch_effort_flag() { - local harness=$1 effort=$2 + local harness=$1 effort=$2 model=${3:-} [ -n "$effort" ] && [ "$effort" != default ] || return 0 case "$harness" in claude) @@ -468,9 +469,18 @@ fm_launch_effort_flag() { low|medium|high) printf -- '--reasoning-effort %s ' "$(fm_launch_shell_quote "$effort")" ;; esac ;; - pi|pi-signed|omp) - # Pi and pi-signed 0.82.0 both accept the full shared effort vocabulary, - # including max, through their --thinking flag. + pi|pi-signed) + if [ "$effort" = ultra ]; then + "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/fm-harness.sh" validate-native-effort "$harness" "$model" "$effort" || return 1 + printf -- '--codex-effort %s ' "$(fm_launch_shell_quote ultra)" + return 0 + fi + case "$effort" in + low|medium|high|xhigh|max) printf -- '--thinking %s ' "$(fm_launch_shell_quote "$effort")" ;; + esac + ;; + omp) + # OMP retains the shared effort vocabulary through its --thinking flag. case "$effort" in low|medium|high|xhigh|max) printf -- '--thinking %s ' "$(fm_launch_shell_quote "$effort")" ;; esac diff --git a/bin/fm-lint.sh b/bin/fm-lint.sh index 3ca44f802d2..9408508aff9 100755 --- a/bin/fm-lint.sh +++ b/bin/fm-lint.sh @@ -33,9 +33,13 @@ # `gh`). That local pass drops --external-sources and excludes SC1091, # SC2034, SC2153, and SC2329. A branch with zero matching changed files # skips ShellCheck and prints a "no changed lint targets" note, then -# still validates workflows. +# still runs the backend-purity check and validates workflows. # Explicit paths always bypass this file-set selection and lint exactly the # given paths, matching the same config, without the workflow YAML check. +# Explicit core bin/ and bin/backends/ scripts still receive the +# backend-purity check. The backend-purity check rejects direct Beads CLI +# invocations in the core bin/ and bin/backends/ scripts so every configured +# backlog backend follows the same tasks-axi lifecycle path. # # Canonical lint defaults to two bounded workers over two stable logical shards. # Each shard writes separate diagnostics, and the parent replays those outputs in @@ -60,9 +64,9 @@ REQUIRED_SHELLCHECK=0.11.0 # Cross-file codes that need --external-sources. Local changed-file mode # cannot judge them, so they stay CI-only. LOCAL_NOX_EXCLUDE=SC1091,SC2034,SC2153,SC2329 -SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" SELF="$SELF_DIR/fm-lint.sh" -ROOT="$(cd "$SELF_DIR/.." && pwd)" +ROOT="$(cd "$SELF_DIR/.." && pwd -P)" cd "$ROOT" || exit 1 FM_LINT_WORKER_SHELLCHECK_PID= @@ -155,6 +159,239 @@ fm_lint_run_workflows() { "$SELF_DIR/fm-lint-workflows.sh" } +# Backend adapters belong behind tasks-axi. Keep direct Beads CLI invocations +# out of firstmate's core scripts so every configured backend follows the same +# lifecycle path. +fm_lint_run_backend_purity() { + local findings path canonical + local -a purity_roots + purity_roots=() + if [ "$EXPLICIT_PATHS" -eq 0 ]; then + purity_roots=(bin/*.sh bin/backends/*.sh) + else + for path in "${ROOTS[@]}"; do + [ -f "$path" ] || continue + # shellcheck disable=SC2016 # Perl, not the shell, expands $ARGV. + canonical=$("$PERL_BIN" -MCwd=realpath -e ' + my $resolved = realpath($ARGV[0]); + exit 1 unless defined $resolved; + print $resolved; + ' "$path" 2>/dev/null) || continue + case "$canonical" in + "$ROOT"/bin/*.sh|"$ROOT"/bin/backends/*.sh) + purity_roots+=("$canonical") + ;; + esac + done + fi + [ "${#purity_roots[@]}" -gt 0 ] || return 0 + findings=$(LC_ALL=C awk ' + function hex_value(character) { + return index("0123456789abcdef", tolower(character)) - 1 + } + function ansi_number(digits, base, i, value) { + value=0 + for (i=1; i <= length(digits); i++) value=value * base + hex_value(substr(digits, i, 1)) + return value + } + # Non-printable and non-ASCII bytes can never spell the bd command, so a + # placeholder keeps them from colliding into it. + function ansi_character(value) { + if (value < 32 || value > 126) return "?" + return sprintf("%c", value) + } + function invokes_bd(segment) { + sub(/^[[:space:]]+/, "", segment) + while (1) { + previous=segment + sub(/^(if|then|elif|else|while|until|do)[[:space:]]+/, "", segment) + sub(/^![[:space:]]+/, "", segment) + sub(/^(command|exec)[[:space:]]+/, "", segment) + sub(/^[[:alpha:]_][[:alnum:]_]*=[^[:space:]]+[[:space:]]+/, "", segment) + if (segment ~ /^env[[:space:]]+/) { + sub(/^env[[:space:]]+/, "", segment) + while (1) { + if (segment ~ /^--[[:space:]]+/) { + sub(/^--[[:space:]]+/, "", segment) + break + } + if (segment ~ /^(-u|--unset|-C|--chdir|-S|--split-string|--argv0)[[:space:]]+[^[:space:]]+[[:space:]]+/) { + sub(/^(-u|--unset|-C|--chdir|-S|--split-string|--argv0)[[:space:]]+[^[:space:]]+[[:space:]]+/, "", segment) + continue + } + if (segment ~ /^--(unset|chdir|split-string|argv0)=[^[:space:]]+[[:space:]]+/) { + sub(/^--(unset|chdir|split-string|argv0)=[^[:space:]]+[[:space:]]+/, "", segment) + continue + } + if (segment ~ /^(-i|--ignore-environment|-0|--null|-v|--debug)[[:space:]]+/) { + sub(/^(-i|--ignore-environment|-0|--null|-v|--debug)[[:space:]]+/, "", segment) + continue + } + if (segment ~ /^[[:alpha:]_][[:alnum:]_]*=[^[:space:]]+[[:space:]]+/) { + sub(/^[[:alpha:]_][[:alnum:]_]*=[^[:space:]]+[[:space:]]+/, "", segment) + continue + } + break + } + } + if (segment == previous) break + } + command_word="" + quote="" + ansi=0 + for (position=1; position <= length(segment); position++) { + character=substr(segment, position, 1) + if (quote == "") { + if (character ~ /[[:space:]]/) break + if (character == "$" && position < length(segment)) { + next_character=substr(segment, position + 1, 1) + if (next_character == "\"" || next_character == sprintf("%c", 39)) { + position++ + quote=next_character + ansi=(next_character == sprintf("%c", 39)) ? 1 : 0 + continue + } + } + if (character == "\"" || character == sprintf("%c", 39)) { + quote=character + ansi=0 + continue + } + if (character == "\\") { + position++ + if (position > length(segment)) return 0 + character=substr(segment, position, 1) + } + command_word=command_word character + continue + } + if (character == quote) { + quote="" + ansi=0 + continue + } + if (character == "\\" && (quote == "\"" || ansi)) { + position++ + if (position > length(segment)) return 0 + escape=substr(segment, position, 1) + if (ansi) { + # ANSI-C quoting decodes escapes, so an encoded spelling of the + # command still runs bd and must be decoded here to be caught. + value=-1 + if (escape == "x" || escape == "u" || escape == "U") { + max_digits=2 + if (escape == "u") max_digits=4 + if (escape == "U") max_digits=8 + digits="" + while (length(digits) < max_digits && position < length(segment)) { + digit=substr(segment, position + 1, 1) + if (digit !~ /[0-9A-Fa-f]/) break + digits=digits digit + position++ + } + if (digits == "") { + # An escape prefix with no digits yields the prefix character. + command_word=command_word escape + continue + } + value=ansi_number(digits, 16) + } else if (escape ~ /[0-7]/) { + digits=escape + while (length(digits) < 3 && position < length(segment)) { + digit=substr(segment, position + 1, 1) + if (digit !~ /[0-7]/) break + digits=digits digit + position++ + } + value=ansi_number(digits, 8) + } + if (value >= 0) { + if (value == 0) { + # NUL truncates the bash word. + quote="" + break + } + command_word=command_word ansi_character(value) + continue + } + if (escape == "c") { + # Control characters can never spell the bd command. + if (position < length(segment)) position++ + command_word=command_word "?" + continue + } + if (escape ~ /^[abeEfnrtv]$/) { + command_word=command_word "?" + continue + } + # Remaining ANSI-C escapes keep their character, and bash drops + # the backslash before any other character. + command_word=command_word escape + continue + } + character=escape + } + command_word=command_word character + } + if (quote != "") return 0 + return command_word ~ /(^|\/)bd$/ + } + function split_commands(line, segments, position, character, quote, current, count) { + delete segments + count=0 + current="" + quote="" + for (position=1; position <= length(line); position++) { + character=substr(line, position, 1) + if (quote != "") { + current=current character + if (character == quote) { + quote="" + } else if (quote == "\"" && character == "\\") { + position++ + if (position <= length(line)) current=current substr(line, position, 1) + } + continue + } + if (character == "\\") { + current=current character + position++ + if (position <= length(line)) current=current substr(line, position, 1) + continue + } + if (character == "\"" || character == sprintf("%c", 39)) { + quote=character + current=current character + continue + } + if (character ~ /[();|&{}]/) { + segments[++count]=current + current="" + continue + } + current=current character + } + if (quote != "") return split(line, segments, /[();|&{}]+/) + segments[++count]=current + return count + } + /^[[:space:]]*#/ { next } + { + count=split_commands($0, segments) + for (i=1; i<=count; i++) { + if (invokes_bd(segments[i])) { + print FILENAME ":" FNR ": direct Beads CLI invocation bypasses tasks-axi" + break + } + } + } + ' "${purity_roots[@]}") + [ -z "$findings" ] || { + printf '%s\n' "$findings" >&2 + return 1 + } +} + JOBS=${FM_LINT_JOBS:-2} TELEMETRY=${FM_LINT_TELEMETRY:-} FAST=0 @@ -322,6 +559,7 @@ fi if [ "$CHANGED_MODE" -eq 1 ] && [ "$ROOT_COUNT" -eq 0 ]; then printf 'fm-lint.sh: no changed lint targets\n' overall_rc=0 + fm_lint_run_backend_purity || overall_rc=$? fm_lint_run_workflows || overall_rc=$? exit "$overall_rc" fi @@ -634,6 +872,12 @@ EOF fi fi +purity_rc=0 +fm_lint_run_backend_purity || purity_rc=$? +if [ "$overall_rc" -eq 0 ] && [ "$purity_rc" -ne 0 ]; then + overall_rc=$purity_rc +fi + if [ "$overall_rc" -eq 0 ]; then fm_lint_run_workflows || overall_rc=$? else diff --git a/bin/fm-mail-check.sh b/bin/fm-mail-check.sh new file mode 100755 index 00000000000..6d73102594c --- /dev/null +++ b/bin/fm-mail-check.sh @@ -0,0 +1,394 @@ +#!/usr/bin/env bash +# fm-mail-check.sh - recurring received-mail poll as a standing watcher check. +# +# Usage: +# fm-mail-check.sh [check] +# fm-mail-check.sh arm +# fm-mail-check.sh disarm +# fm-mail-check.sh --help +# +# `check` runs the mail poll from this home (sourcing the same .env and using +# the same inbox state as fm-mail.sh itself). It composes with the existing +# watcher state-check contract instead of needing a schedule of its own: a +# printed line becomes a `check:` wake so firstmate can drain durable +# `check: mail <uid>` rows the poll already queued. +# +# `arm` writes state/mail.check.sh and binds its bytes with +# fm-check-register.sh, so the watcher dispatches it on its normal +# FM_CHECK_INTERVAL cadence and turns its one line into a `check:` wake. +# `disarm` removes the shim, its trust binding, and the report record. +# +# Mail configuration is read from the home's own .env by the poll, so arming +# needs no configuration of its own. A home that is armed before its .env has +# FM_MAIL_USER, FM_MAIL_PASS, FM_IMAP_HOST, and FM_SMTP_HOST is reported once +# for the missing value until the .env is fixed, which makes a partially +# configured channel a wake instead of a silent gap. +# +# Reporting keeps state/.mail-check as the news key, but prints whenever +# the poll is not a proven no-op. A proven no-op is a repeated identical +# line, not a timeout, with no publication evidence. Publication evidence +# is a timeout, fail-closed-after-queue diagnostics, a queued mail: check +# key, or growth of state/.mail-woken. Same-line silence is only for a +# proven no-op: successful poll with no new mail, or a repeated pre-wake +# failure (missing env, connection refused before wake_for, missing +# python3, missing fm-mail.sh, heal could not record a uid) that cannot +# have queued mail. Fail-closed after a queued wake and timeout always +# doorbell. +# +# The poll must finish inside the watcher's per-check bound +# (FM_CHECK_TIMEOUT, default 30, read from this check's own environment +# because the watcher runs it as a direct child). The internal budget +# FM_MAIL_CHECK_BUDGET (default 15, valid 5..25) is cut down to whatever fits +# inside that bound before the poll starts. A poll that does not finish is a +# real condition, so the budget is enforced rather than assumed: a timed-out +# poll reports one line naming the budget instead of leaving the check silent. +set -u +export LC_ALL=C + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +RECORD="$STATE/.mail-check" +CHECK_ID=mail +CHECK_SHIM="$STATE/$CHECK_ID.check.sh" +CHECK_TRUST="$STATE/$CHECK_ID.check-trust" +MAIL_BIN="$SCRIPT_DIR/fm-mail.sh" +REGISTER_BIN="$SCRIPT_DIR/fm-check-register.sh" +RECORD_SCHEMA=fm-mail-check-v1 +MAX_LINE=240 + +# shellcheck source=bin/fm-timeout-lib.sh +. "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-pr-lib.sh +. "$SCRIPT_DIR/fm-pr-lib.sh" +# shellcheck source=bin/fm-line-cap-lib.sh +. "$SCRIPT_DIR/fm-line-cap-lib.sh" +# shellcheck source=bin/fm-check-lib.sh +. "$SCRIPT_DIR/fm-check-lib.sh" + +usage() { + cat <<'EOF' +Usage: + fm-mail-check.sh [check] run the received-mail poll; wake line unless the poll is a proven no-op + fm-mail-check.sh arm write and register state/mail.check.sh + fm-mail-check.sh disarm remove the check shim, its trust binding, and the record + fm-mail-check.sh --help print this help + +Mail configuration (FM_MAIL_USER, FM_MAIL_PASS, FM_IMAP_HOST, FM_SMTP_HOST, +FM_IMAP_PORT, FM_SMTP_PORT) is read from <FM_HOME>/.env by fm-mail.sh. +See docs/configuration.md "Mail plane" for the schema. +EOF +} + +die_usage() { + printf 'fm-mail-check: %s\n' "$1" >&2 + usage >&2 + exit 2 +} + +record_epoch_now() { + case "${FM_MAIL_CHECK_NOW:-}" in + ''|*[!0-9]*) date +%s ;; + *) printf '%s\n' "$FM_MAIL_CHECK_NOW" ;; + esac +} + +CHECK_TIMEOUT=${FM_CHECK_TIMEOUT:-30} +case "$CHECK_TIMEOUT" in + ''|*[!0-9]*|0) CHECK_TIMEOUT=30 ;; +esac + +BUDGET_SECS=${FM_MAIL_CHECK_BUDGET:-15} +case "$BUDGET_SECS" in + ''|*[!0-9]*|0) + printf 'fm-mail-check: FM_MAIL_CHECK_BUDGET must be a whole number from 5 to 25\n' >&2 + exit 2 + ;; +esac +if [ "$BUDGET_SECS" -lt 5 ] || [ "$BUDGET_SECS" -gt 25 ]; then + printf 'fm-mail-check: FM_MAIL_CHECK_BUDGET must be a whole number from 5 to 25\n' >&2 + exit 2 +fi + +# fm_run_timed counts a whole second before it alarms, so the budget has to fit +# inside the watcher's own bound with the alarm and kill margins left over. +BUDGET_MAX=$((CHECK_TIMEOUT - 3)) +[ "$BUDGET_MAX" -ge 1 ] || BUDGET_MAX=1 +if [ "$BUDGET_SECS" -gt "$BUDGET_MAX" ]; then + BUDGET_SECS=$BUDGET_MAX +fi + +# One poll summary, built only from the poll's own combined output. The +# poll's own "fm-mail: ..." diagnostics name the missing setup value, the +# missing python3, or the failure precisely, so they are preferred to a raw +# python backtrace; success-wake lines are skipped because a fail-closed poll +# may already have printed them; anything else is summarized rather than +# dropped, and an empty failure gets a truth-stating fallback. +poll_summary() { + local rc=$1 out=$2 line + line=$(printf '%s\n' "$out" | sed -n '/^fm-mail: woke for /d; s/^fm-mail: //p' | head -n 1) + if [ -z "$line" ]; then + line=$(printf '%s\n' "$out" | sed -n '/^fm-mail: woke for /d; /^$/d; p' | head -n 1) + fi + if [ -z "$line" ]; then + line="poll failed (rc=$rc)" + fi + printf '%s\n' "$line" +} + +record_read() { + local line first=1 + RECORD_REPORTED= + [ -f "$RECORD" ] || return 0 + while IFS= read -r line; do + if [ "$first" = 1 ]; then + first=0 + [ "$line" = "$RECORD_SCHEMA" ] || return 0 + continue + fi + case "$line" in + reported=*) RECORD_REPORTED=${line#reported=} ;; + esac + done < "$RECORD" + return 0 +} + +record_write() { + local reported=$1 tmp + tmp=$(mktemp "$RECORD.XXXXXX" 2>/dev/null) || return 1 + chmod 0600 "$tmp" 2>/dev/null || { rm -f -- "$tmp"; return 1; } + { + printf '%s\n' "$RECORD_SCHEMA" + printf 'epoch=%s\n' "$(record_epoch_now)" + printf 'reported=%s\n' "$reported" + } > "$tmp" || { rm -f -- "$tmp"; return 1; } + mv -f -- "$tmp" "$RECORD" || { rm -f -- "$tmp"; return 1; } + return 0 +} + +# True when this poll has publication evidence, so a repeated diagnostic is +# not a proven no-op. Stdout is a side channel; the durable ledger (queued +# mail: check keys, or growth of .mail-woken) is the same record the poll +# trusts. Fail-closed statuses 2 and 4 queue a wake without printing +# "woke for". +poll_has_publication_evidence() { + local rc=${1:-0} out=$2 woken_before=$3 + [ "$rc" -eq 124 ] && return 0 + if [ -n "$out" ] && printf '%s\n' "$out" | grep -qE \ + '^fm-mail: woke for |the wake stays queued|could not clear retry for recovered' + then + return 0 + fi + if [ -s "$STATE/.wake-queue" ] && grep -q $'\tcheck\tmail:' "$STATE/.wake-queue"; then + return 0 + fi + if [ -f "$STATE/.mail-woken" ]; then + if [ -z "$woken_before" ] || [ ! -f "$woken_before" ] \ + || ! cmp -s "$woken_before" "$STATE/.mail-woken"; then + return 0 + fi + fi + return 1 +} + +action_check() { + local out rc line woken_before queued=0 + mkdir -p "$STATE" || return 1 + woken_before=$(mktemp) || woken_before= + if [ -n "$woken_before" ]; then + if [ -f "$STATE/.mail-woken" ]; then + cp "$STATE/.mail-woken" "$woken_before" 2>/dev/null || : > "$woken_before" + else + : > "$woken_before" + fi + fi + if [ ! -x "$MAIL_BIN" ]; then + line="fm-mail.sh is missing next to this check ($MAIL_BIN)" + else + out=$(fm_run_timed "$BUDGET_SECS" "$MAIL_BIN" poll 2>&1) || rc=$? + if [ "${rc:-0}" -eq 124 ]; then + line="poll did not finish within the ${BUDGET_SECS}s budget" + elif [ "${rc:-0}" -ne 0 ]; then + line=$(poll_summary "$rc" "$out") + elif printf '%s\n' "$out" | grep -q '^fm-mail: woke for '; then + # A successful poll can still surface new mail: the poll itself already + # appended the durable mail wake rows, but the watcher only calls wake() + # when THIS check's output is non-empty. Emit one line naming a surfaced + # uid (the last woke-for in this poll) so the watcher wakes the agent to + # drain the queued mail rows; without it, new mail sits queued and silent. + line=$(printf '%s\n' "$out" | grep '^fm-mail: woke for ' | tail -n 1 | sed 's/^fm-mail: /new mail: /') + else + line= + fi + fi + record_read + # Report before recording, so a record that cannot be written costs a + # repeated report rather than a lost one. The record keeps the whole line so + # the news key and the printed report never diverge. Invert the print gate: + # emit unless this poll is a proven no-op (same line, not a timeout, and no + # publication evidence). + if poll_has_publication_evidence "${rc:-0}" "${out:-}" "$woken_before"; then + queued=1 + fi + [ -n "$woken_before" ] && rm -f -- "$woken_before" + if [ -n "$line" ] && { [ "$line" != "$RECORD_REPORTED" ] || [ "$queued" -eq 1 ]; }; then + fm_cap_line_var "mail: $line" "$MAX_LINE" + printf '%s\n' "$FM_LINE_CAP_LINE" + fi + record_write "$line" || true + return 0 +} + +# The home is embedded already resolved, because the watcher runs the shim from +# its own working directory and a relative spelling would send the check to a +# different home, or to none at all. +shim_content() { + local home=$1 + printf '%s\n' \ + '#!/usr/bin/env bash' \ + '# Auto-generated by fm-mail-check.sh - received-mail poll shim.' \ + '# The watcher validates these bytes, then dispatches the trusted check script.' \ + "export FM_HOME=$(printf '%q' "$home")" \ + "exec $(printf '%q' "$SCRIPT_DIR/fm-mail-check.sh") check" +} + +# Write the shim the way this repo writes its other trusted check shim: the +# guards run before anything is written, so a symlink at the shim path is +# refused instead of followed, and the bytes arrive by rename so the watcher +# never reads a half-written shim and rejects it as unauthenticated. +SHIM_WRITE_TMP= + +shim_write() { + local want=$1 device tmp + [ -d "$STATE" ] && [ ! -L "$STATE" ] || return 1 + device=$(fm_pr_file_device "$STATE") || return 1 + [ -n "$device" ] || return 1 + fm_pr_regular_destination_on_device_or_absent "$CHECK_SHIM" "$device" || return 1 + if [ -e "$CHECK_SHIM" ] && [ "$(fm_pr_file_mode "$CHECK_SHIM")" = 700 ] \ + && [ "$(cat "$CHECK_SHIM" 2>/dev/null)" = "$want" ]; then + return 0 + fi + tmp=$(umask 077; mktemp "$STATE/.fm-mail-check.XXXXXX" 2>/dev/null) || return 1 + SHIM_WRITE_TMP=$tmp + if ! printf '%s\n' "$want" > "$tmp" \ + || ! chmod 0700 "$tmp" \ + || ! fm_pr_private_file_valid "$tmp" 700 "$device"; then + rm -f -- "$tmp" + SHIM_WRITE_TMP= + return 1 + fi + if ! fm_pr_regular_destination_on_device_or_absent "$CHECK_SHIM" "$device" \ + || ! mv -f -- "$tmp" "$CHECK_SHIM"; then + rm -f -- "$tmp" + SHIM_WRITE_TMP= + return 1 + fi + SHIM_WRITE_TMP= + fm_pr_private_file_valid "$CHECK_SHIM" 700 "$device" +} + +# Keep a byte copy of a shim that is already in place, so a failed arm can put +# back the shim a working home was already using rather than an equivalent +# rewrite. The trust binding is over the bytes, so a rewrite would satisfy it +# too, but a home that was armed stays armed with what it had. +shim_backup() { + local device tmp + device=$(fm_pr_file_device "$STATE") || return 1 + [ -n "$device" ] || return 1 + tmp=$(umask 077; mktemp "$STATE/.fm-mail-check.XXXXXX" 2>/dev/null) || return 1 + if ! cat "$CHECK_SHIM" > "$tmp" 2>/dev/null \ + || ! chmod 0700 "$tmp" \ + || ! fm_pr_private_file_valid "$tmp" 700 "$device"; then + rm -f -- "$tmp" + return 1 + fi + printf '%s\n' "$tmp" +} + +ARM_BACKUP= + +# An unregistered shim is not inert: the watcher rejects it on every cycle and +# wakes firstmate about unauthenticated state checks. So the one rule after a +# failed or interrupted arm is that the home never holds a shim without a +# matching trust binding. The shim a working home had is put back and kept only +# when it is still bound; otherwise the shim goes, so the home is plainly not +# armed and the failure is the only thing the operator has to act on. +arm_rollback() { + [ -z "$SHIM_WRITE_TMP" ] || rm -f -- "$SHIM_WRITE_TMP" + SHIM_WRITE_TMP= + if [ -n "$ARM_BACKUP" ]; then + mv -f -- "$ARM_BACKUP" "$CHECK_SHIM" 2>/dev/null || rm -f -- "$ARM_BACKUP" + ARM_BACKUP= + if fm_custom_check_registered "$STATE" "$CHECK_ID"; then + return 0 + fi + fi + rm -f -- "$CHECK_SHIM" +} + +# shellcheck disable=SC2329 # Registered by action_arm's signal trap. +arm_interrupted() { + arm_rollback + printf 'fm-mail-check: arming was interrupted, so state/%s.check.sh is not armed\n' "$CHECK_ID" >&2 + exit 1 +} + +action_arm() { + local want home + if [ ! -x "$MAIL_BIN" ]; then + printf 'fm-mail-check: the mail plane is missing at %s; cannot arm\n' "$MAIL_BIN" >&2 + return 1 + fi + mkdir -p "$STATE" || return 1 + case "$FM_HOME" in + /*) home=$FM_HOME ;; + *) + home=$(CDPATH='' cd -- "$FM_HOME" 2>/dev/null && pwd -P) || { + printf 'fm-mail-check: cannot resolve FM_HOME %s\n' "$FM_HOME" >&2 + return 1 + } + ;; + esac + want=$(shim_content "$home") + ARM_BACKUP= + if [ -f "$CHECK_SHIM" ] && [ ! -L "$CHECK_SHIM" ]; then + ARM_BACKUP=$(shim_backup) || { + printf 'fm-mail-check: could not save the existing %s\n' "$CHECK_SHIM" >&2 + return 1 + } + fi + # The shim exists unbound from the rename until the register returns, so a + # signal in that window rolls back the same way a failure does. + trap arm_interrupted HUP INT TERM + if ! shim_write "$want"; then + trap - HUP INT TERM + arm_rollback + printf 'fm-mail-check: could not write %s\n' "$CHECK_SHIM" >&2 + return 1 + fi + if ! FM_HOME="$home" "$REGISTER_BIN" "$CHECK_ID" >/dev/null; then + trap - HUP INT TERM + arm_rollback + printf 'fm-mail-check: could not register %s\n' "$CHECK_SHIM" >&2 + return 1 + fi + trap - HUP INT TERM + [ -z "$ARM_BACKUP" ] || rm -f -- "$ARM_BACKUP" + ARM_BACKUP= + printf 'armed: state/%s.check.sh\n' "$CHECK_ID" + return 0 +} + +action_disarm() { + rm -f -- "$CHECK_SHIM" "$CHECK_TRUST" "$RECORD" + printf 'disarmed: state/%s.check.sh\n' "$CHECK_ID" + return 0 +} + +case "${1:-check}" in + check) action_check ;; + arm) action_arm ;; + disarm) action_disarm ;; + -h|--help) usage ;; + *) die_usage "unknown action: $1" ;; +esac \ No newline at end of file diff --git a/bin/fm-mail.py b/bin/fm-mail.py new file mode 100755 index 00000000000..ae123cbcb5b --- /dev/null +++ b/bin/fm-mail.py @@ -0,0 +1,491 @@ +#!/usr/bin/env python3 +# fm-mail.py - the IMAP/SMTP engine behind bin/fm-mail.sh. +# +# A small mail client used by fm-mail.sh: +# read List unseen INBOX mail as a compact digest. +# send <to> <subj> <body | -> Send one SMTP message; "-" reads stdin. +# poll_list Emit unseen mail as tab-separated rows for the bash +# poll, bounded to uids this home has not surfaced, +# plus a retry-set of previously unfetchable uids; +# persists the retry-scan position and cap-1 turn flag. +# seen <cursor> Print a cursor file (used by `status`). +# +# All configuration arrives through the environment, never through arguments, +# so credentials never appear in argv or logs. read/poll use BODY.PEEK so mail +# is never marked seen before firstmate answers it. +import imaplib +import os +import re +import socket +import ssl +import sys +import email +import smtplib +from email.header import decode_header, make_header +from email.message import EmailMessage +from email.utils import formatdate + +USER = os.environ['FM_MAIL_USER'] +PW = os.environ['FM_MAIL_PASS'] +IMH = os.environ['FM_IMAP_HOST'] +IMP = int(os.environ['FM_IMAP_PORT']) +STH = os.environ['FM_SMTP_HOST'] +STP = int(os.environ['FM_SMTP_PORT']) +CTX = ssl.create_default_context() + + +def mail_timeout(): + """Seconds for IMAP/SMTP sockets. Invalid or non-positive values become 20.""" + raw = os.environ.get('FM_MAIL_TIMEOUT', '20') + try: + value = float(raw) + except (TypeError, ValueError): + value = 20.0 + if value <= 0: + value = 20.0 + return value + + +MAIL_TIMEOUT = mail_timeout() +socket.setdefaulttimeout(MAIL_TIMEOUT) + +MAX_PREVIEW = 200 +READ_LIMIT = 20 + + +def dec(s): + """Decode an RFC-2047 header to display text, tolerating malformed input.""" + if not s: + return '' + try: + return str(make_header(decode_header(s))) + except Exception: + return str(s) + + +def clean(s): + """Collapse tabs/newlines/CR in a header value to single spaces so a + crafted Subject/From can never split the tab-separated poll row or inject + a fake uid line for the bash layer; strip surrounding whitespace too.""" + return re.sub(r'[\t\r\n]+', ' ', s or '').strip() + + +def connect_mailbox(): + m = imaplib.IMAP4_SSL(IMH, IMP, ssl_context=CTX, timeout=MAIL_TIMEOUT) + m.login(USER, PW) + return m + + +def body_preview(msg): + """First non-empty text/plain line, else first non-empty text/html line, + else empty. An empty plain-text alternative falls through to html so a + valid message never loses its promised preview.""" + try: + if msg is None: + return '' + for part in msg.walk(): + if part.get_content_type() == 'text/plain': + text = (part.get_payload(decode=True) or b'').decode('utf-8', 'replace').strip() + if text: + return text + for part in msg.walk(): + if part.get_content_type() == 'text/html': + raw = (part.get_payload(decode=True) or b'').decode('utf-8', 'replace') + raw = re.sub(r'(?is)<(style|script)[^>]*>.*?</\1>', ' ', raw) + preview = re.sub(r'<[^>]+>', ' ', raw) + preview = ' '.join(preview.split()) + if preview: + return preview + except Exception: + return '' + return '' + + +def cmd_read(): + try: + m = connect_mailbox() + m.select('INBOX') + typ, data = m.uid('search', None, 'UNSEEN') + ids = (data[0] or b'').split() + if not ids: + print('(no unseen mail)') + m.logout() + return 0 + for i in ids[-READ_LIMIT:]: + uid = i.decode() if isinstance(i, bytes) else str(i) + typ, msg = m.uid('fetch', i, '(BODY.PEEK[])') + if typ != 'OK' or not msg or not msg[0] or not msg[0][1]: + print('---') + print('Uid:', uid) + print('From:', '(unfetchable)') + print('Date:', '') + print('Subj:', 'unfetchable body - see fm-mail read') + print('Body:', '(body unavailable)') + continue + mi = email.message_from_bytes(msg[0][1]) + print('---') + print('From:', dec(mi.get('From'))) + print('Date:', dec(mi.get('Date'))) + print('Subj:', dec(mi.get('Subject'))) + preview = body_preview(mi) + if preview: + first = preview.splitlines()[0] + print('Body:', (first[:MAX_PREVIEW] if first else '')) + else: + print('Body:', '(body unavailable)') + try: + m.logout() + except Exception: + pass + return 0 + except Exception as e: + print('fm-mail read error:', e) + return 1 + + +def cmd_send(to, subj, body): + try: + if body == '-': + body = sys.stdin.read().rstrip('\n') + m = EmailMessage() + m['From'] = USER + m['To'] = to + m['Subject'] = subj + m['Date'] = formatdate(localtime=True) + m.set_content(body) + with smtplib.SMTP_SSL(STH, STP, context=CTX, timeout=MAIL_TIMEOUT) as s: + s.login(USER, PW) + s.send_message(m) + print('sent to', to) + return 0 + except Exception as e: + print('fm-mail send error:', e) + return 1 + + +def cmd_seen(cursor_path): + line = open(cursor_path).read().strip() if os.path.exists(cursor_path) else '(none)' + print('cursor:', line) + return 0 + + +def load_cursor(cursor_path): + """Return (stored_generation, seen_uids) from the local cursor file.""" + stored_gen = '' + seen = set() + if not os.path.exists(cursor_path): + return stored_gen, seen + with open(cursor_path, encoding='utf-8', errors='replace') as f: + for line in f: + line = line.strip() + if not line: + continue + if line.startswith('uidvalidity='): + stored_gen = line.split('=', 1)[1] + else: + seen.add(line) + return stored_gen, seen + + +def load_retry(retry_path): + """Return (retry_set, retry_order) from the local retry file.""" + retry = set() + ordered = [] + if not retry_path or not os.path.exists(retry_path): + return retry, ordered + with open(retry_path, encoding='utf-8', errors='replace') as f: + for line in f: + uid = line.strip() + if not uid or uid in retry: + continue + retry.add(uid) + ordered.append(uid) + return retry, ordered + + +def load_retry_pos(pos_path, n): + """Return the durable retry-scan start position, clamped into range.""" + if not pos_path: + return 0 + try: + pos = int(open(pos_path).read().strip() or '0') + except (OSError, ValueError): + return 0 + if n <= 0: + return 0 + return pos % n + + +def retry_scan_window(order, pos, window): + """Take the bounded retry scan starting at the durable position, wrapping + around the end of the retry file. cmd_poll_list owns when and by how much + the durable position advances after this window is considered.""" + if not order: + return [] + start = pos % len(order) + rotated = order[start:] + order[:start] + if len(order) <= window: + return rotated + return rotated[:window] + + +def save_retry_pos(pos_path, order_len, window, pos): + """Persist the next retry-scan start position: (pos + window) mod order_len. + cmd_poll_list owns what window means on each persist path. A failed write + propagates so the poll fails closed rather than silently restarting the + retry scan at the same head every poll.""" + if not pos_path: + return + if order_len <= 0: + next_pos = 0 + else: + next_pos = (pos + window) % order_len + with open(pos_path, 'w', encoding='utf-8') as f: + f.write(str(next_pos) + '\n') + + +def load_turn(path): + """Return the durable alternating-turn flag (0=new,1=retry) for a single + contended slot.""" + if not path: + return 0 + try: + return int(open(path).read().strip() or '0') % 2 + except (OSError, ValueError): + return 0 + + +def save_turn(path, turn): + """Persist the alternating-turn flag. A failed write propagates so the + poll fails closed rather than silently selecting the same class forever.""" + if not path: + return + with open(path, 'w', encoding='utf-8') as f: + f.write(str(turn % 2) + '\n') + + +def cmd_poll_list(): + # Bound the expensive header fetches: only uids not already recorded in the + # cursor are considered as new, then previously unfetchable retry-set uids + # (already in the cursor) are fetched again so a transient IMAP failure + # cannot permanently replace real metadata with degraded placeholders. A + # bounded window of candidates is scanned to fill the per-poll cap, new + # uids first so a large retry backlog can never starve new mail. + cap = int(os.environ.get('FM_MAIL_POLL_MAX_WAKES') or '20') + if cap < 1: + cap = 20 + stored_gen, seen = load_cursor(os.environ.get('FM_MAIL_CURSOR', '')) + retry, retry_order = load_retry(os.environ.get('FM_MAIL_RETRY', '')) + retry_pos_path = os.environ.get('FM_MAIL_RETRY_POS', '') + retry_pos = load_retry_pos(retry_pos_path, len(retry_order)) + m = None + try: + m = connect_mailbox() + m.select('INBOX') + ur = m.untagged_responses.get('UIDVALIDITY') + uidv = clean(ur[-1].decode()) if ur else '' + typ, data = m.uid('search', None, 'UNSEEN') + unseen = [] + for x in (data[0] or b'').split(): + uid = x.decode() if isinstance(x, bytes) else str(x) + unseen.append(uid) + if uidv and uidv == stored_gen: + # Same mailbox generation: skip uids this home already surfaced so + # the fetch budget goes to genuinely new mail. Retry-set uids are + # only meaningful for this generation. + new_uids = [u for u in unseen if u not in seen] + else: + # On a generation change the cursor and retry set are stale, so + # list everything as new and ignore retry membership; bash clears + # both files before the wake loop. + new_uids = list(unseen) + retry = set() + retry_order = [] + # Bound the expensive fetch work with a window, applied to each class + # separately so a large new-mail backlog cannot slice retry candidates + # out of the scan. The retry scan starts at a durable position; the + # persist block below owns when that position advances. + window = max(cap * 4, cap + 10) + new_candidates = new_uids[:window] + # Only a retry uid that is already surfaced (in the cursor) is a pure + # retry re-fetch. A retry-set uid that is not yet in the cursor is a + # degraded wake that failed to record - it stays a new candidate so + # the next poll surfaces it again as degraded instead of silently + # dropping it. The window itself (regardless of seen membership) is + # kept so a scan window of only unseen uids can still advance the + # durable cursor past itself, never stalling the march over the whole + # retry set. + retry_window = retry_scan_window(retry_order, retry_pos, window) + retry_candidates = [u for u in retry_window if u in seen] + turn_path = os.environ.get('FM_MAIL_TURN', '') + next_turn = None + if cap == 1 and new_candidates and retry_candidates: + # A single contended slot alternates between new surfacing and + # retry recovery, so a sustained new-mail flood can never starve + # recovered metadata indefinitely, and a retry backlog can never + # delay new mail for more than one poll. + if load_turn(turn_path) == 0: + new_budget, retry_budget = 1, 0 + next_turn = 1 + else: + new_budget, retry_budget = 0, 1 + next_turn = 0 + else: + # Reserve a quarter of the cap (at least one) for retry successes + # so a sustained new-mail flood cannot starve recovered metadata, + # but never let the reservation fully suppress new mail: when both + # classes have candidates, new mail always keeps at least one slot. + retry_budget = max(1, cap // 4) if retry_candidates else 0 + new_budget = cap - retry_budget + out = [] + new_emitted = 0 + retry_emitted = 0 + retry_examined = 0 + retry_idx = -1 + first_retry_emitted_index = -1 + for u in new_candidates + retry_candidates: + is_retry = u in retry and u in seen + if is_retry: + retry_idx += 1 + if is_retry: + if retry_emitted >= retry_budget: + # Past the retry budget: leave this candidate in the scan + # (do not advance past it) so a later poll reaches it once + # budget frees up. Advancing the durable position by the + # full window while emitting only the budgeted prefix would + # revisit the same prefix forever and strand later + # recovered uids (a scan is a cursor over the whole retry + # set, and every uid must be reachable). + continue + retry_examined += 1 + elif new_emitted >= new_budget: + continue + # A raised or empty FETCH is treated as a failure for THIS uid only, + # so one bad message can never abort the bounded scan: a new uid is + # surfaced degraded, a retry uid is left for a later scan step, and + # the scan advances. + try: + typ, msg = m.uid('fetch', u.encode(), '(BODY.PEEK[HEADER])') + if typ != 'OK' or not msg or not msg[0]: + raise ValueError('no header data') + mi = email.message_from_bytes(msg[0][1]) + uid = clean(u) + idate = clean(dec(mi.get('Date'))) + subj = clean(dec(mi.get('Subject'))) + fr = clean(dec(mi.get('From'))) + except Exception: + if is_retry: + continue + out.append((clean(u), '', '(no header)', + 'unfetchable header - see fm-mail read', 'degraded')) + new_emitted += 1 + continue + status = 'retry' if is_retry else 'ok' + out.append((uid, idate, fr, subj, status)) + if is_retry: + retry_emitted += 1 + if first_retry_emitted_index == -1: + first_retry_emitted_index = retry_idx + else: + new_emitted += 1 + # Finish every IMAP round-trip before emit or persist so a hung + # logout cannot run after the retry-scan position advances. Then emit + # the mailbox generation guard and each message row (uid, date, from, + # subject, status) so the bash layer diffs against the cursor and the + # retry set. Flush stdout before persisting: under a pipe CPython + # block-buffers, and a timeout kill would otherwise discard unflushed + # rows after the position had already advanced. An interruption + # between emission and the position write must never advance the + # cursor over rows that never reached the bash wake layer. A failed + # position write still fails the poll loudly, so the same bounded + # window is re-scanned on the next poll rather than silently + # restarting from the old head. The persist block below owns when the + # retry-scan position advances, including under a new-mail flood. + try: + m.logout() + except Exception: + pass + m = None + print('uidvalidity\t%s' % uidv) + for uid, idate, fr, subj, status in out: + print('%s\t%s\t%s\t%s\t%s' % (uid, idate, fr, subj, status)) + sys.stdout.flush() + # The retry-scan cursor must keep marching so every retry uid is + # reachable, but it must never advance past a uid whose wake did not + # durably publish. Rows are handed to the bash wake layer immediately + # below; Python cannot observe whether every wake_for succeeded, so the + # durable position advances only up to (never past) the first emitted + # retry uid. If that uid's wake fails to publish, it stays at the head + # of the scan for the next poll; if the wake succeeds, the bash layer + # removes it from the retry set and the same numeric start scans the + # next remaining uid. Advance is keyed off whether a retry row was + # emitted (first_retry_emitted_index), never off whether `out` is + # empty: new-mail rows filling the poll must not stall the retry + # cursor (Greptile 'Retry window stops progressing'). When no retry + # row was emitted, candidates were examined (unfetchable) or the + # window held only unseen uids, and the position advances so the + # scan does not stall. An emitted retry at index 0 leaves the + # position unchanged, same as landing on that uid. + # Three cases advance it: + # 1. budget > 0 and a retry row was emitted past index 0 -> by the + # number of unfetchable retry candidates before the first emitted + # one, landing the cursor on that uid (never past it). + # 2. budget > 0 but no retry row emitted -> by the candidates actually + # examined within budget (fetched or unfetchable), never the full + # window (Greptile 'Retry cursor skips candidates'), even when + # new-mail rows fill `out`. + # 3. budget == 0 because the window held only unseen uids (none + # qualified as a seen retry) -> by the scanned window itself, so + # a leading stale window cannot stall the march and strand a + # later eligible retry uid (Greptile 'Retry cursor stalls + # permanently'), even when new-mail rows fill `out`. + # A cap=1 new-mail turn (qualifiers exist but yield deliberately, + # retry_budget 0 with retry_candidates non-empty) leaves the position + # unchanged so an unexamined window is never skipped. + if retry_budget > 0 and len(retry_candidates) > 0: + if first_retry_emitted_index > 0: + save_retry_pos(retry_pos_path, len(retry_order), + first_retry_emitted_index, retry_pos) + elif first_retry_emitted_index < 0: + save_retry_pos(retry_pos_path, len(retry_order), + max(1, retry_examined), retry_pos) + elif len(retry_window) > 0 and len(retry_candidates) == 0: + save_retry_pos(retry_pos_path, len(retry_order), + len(retry_window), retry_pos) + # Persist the cap-one alternation turn only after the rows are emitted + # and flushed, so a kill between the decision and the emit can never + # skip an unspent turn. + if next_turn is not None: + save_turn(turn_path, next_turn) + return 0 + except Exception as e: + # stderr, not stdout: the bash poll's command substitution captures + # stdout, so a poll error printed to stdout is swallowed with the list + # and the poll dies rc=1 with nothing left to report. + print('fm-mail poll error:', e, file=sys.stderr) + return 1 + finally: + if m is not None: + try: + m.logout() + except Exception: + pass + + +def main(): + cmd = sys.argv[1] if len(sys.argv) > 1 else '' + if cmd == 'read': + return cmd_read() + if cmd == 'send': + if len(sys.argv) < 5: + return 1 + return cmd_send(sys.argv[2], sys.argv[3], sys.argv[4]) + if cmd == 'seen': + return cmd_seen(sys.argv[2] if len(sys.argv) > 2 else '') + if cmd == 'poll_list': + return cmd_poll_list() + raise SystemExit('unknown command') + + +if __name__ == '__main__': + sys.exit(main()) \ No newline at end of file diff --git a/bin/fm-mail.sh b/bin/fm-mail.sh new file mode 100755 index 00000000000..a7f0ba55f5a --- /dev/null +++ b/bin/fm-mail.sh @@ -0,0 +1,651 @@ +#!/usr/bin/env bash +# fm-mail.sh - general-purpose mail plane for reading and sending mail. +# +# Reads inbound mail over IMAP and sends mail over SMTP on demand. This is an +# ordinary mail client surface, not an escalation of authority: every surfaced +# message is a notification firstmate reads before deciding, and firstmate still +# applies its own judgment exactly as it would for a TUI message (including +# return/away and other rules). +# +# Subcommands: +# read List unseen INBOX mail as a compact digest (From / +# Date / Subject / first line). +# send <to> <subject> <body | -> +# Send one message. A "-" body reads plain text from +# stdin. +# poll Surface UNSEEN mail this home has not yet woken as a +# `check` wake so firstmate answers it concisely. IMAP +# \Seen mail never wakes a poll, no message is ever +# marked read (BODY.PEEK), and every surfaced message +# is keyed by its immutable IMAP UID so expunge +# renumbering never re-wakes or loses mail. A uid whose +# header could not be fetched is woken once degraded +# and later re-woken once with recovered metadata. The +# cursor also records the mailbox generation +# (UIDVALIDITY) so a recreated mailbox cannot reuse a +# numeric uid and suppress a new wake, and overlapping +# polls are serialized on the mail-seen lock. poll +# itself has no scheduler: run it manually, from +# `at`/cron, or via the standing check armed by +# bin/fm-mail-check.sh (docs/configuration.md +# "Mail plane"). +# status Print configuration and the last poll cursor. No +# network, no wake. +# +# Volume: poll surfaces at most FM_MAIL_POLL_MAX_WAKES messages per run +# (default 20, valid 1..200); a larger flood is left unseen so the next poll +# surfaces the next batch, keeping the durable wake queue bounded no matter how +# much inbound mail arrives. A header fetch that fails is still surfaced once +# (degraded placeholders) and retried on later polls until the real metadata +# lands; a persistently unfetchable uid is never skipped and never re-wakes. +# +# Deployment - credentials and endpoints are read from the environment, +# filling missing keys from the gitignored $FM_HOME/.env (same convention +# as the Relay/FMX token; env wins). Add these four required values, plus +# the optional ports and timeout: +# FM_MAIL_USER=<imap/smtp account> +# FM_MAIL_PASS=<password> +# FM_IMAP_HOST=<imap host> +# FM_IMAP_PORT=<imap port> (default 993, implicit TLS) +# FM_SMTP_HOST=<smtp host> +# FM_SMTP_PORT=<smtp port> (default 465, implicit TLS) +# FM_MAIL_TIMEOUT=<seconds> (default 20; IMAP/SMTP socket timeout) +# FM_HOME falls back to the repo root when unset. This script carries no secret +# and no default endpoint that could resolve against a wrong home; FM_MAIL_USER, +# FM_MAIL_PASS, FM_IMAP_HOST, and FM_SMTP_HOST are always required, and +# FM_MAIL_PASS is never logged. The wake library is sourced from next to this +# script, not from $FM_HOME/bin; cursor, journal, retry set, and queue stay +# under $FM_HOME/state. +# +# IMAP/SMTP work is delegated to bin/fm-mail.py (imaplib/smtplib, implicit TLS +# on 993/465). STARTTLS and port 587 are not supported. BODY.PEEK is used on +# read/poll so mail is never marked seen before firstmate actually answers it. + +set -euo pipefail + +# --- resolve home, env, and endpoints ------------------------------------- +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_HOME="${FM_HOME:-}" +if [ -z "$FM_HOME" ]; then + FM_HOME="$(cd "$SCRIPT_DIR/.." && pwd)" +fi +ENV_FILE="$FM_HOME/.env" +# Load the home .env for keys not already set, so a direct invocation's +# environment overrides .env exactly like the Relay/FMX contract (fmx_env_get: +# "env wins over .env"). Tolerates a leading "export ", surrounding whitespace, +# one layer of matching quotes, comments, and blank lines. +if [ -f "$ENV_FILE" ]; then + while IFS= read -r line || [ -n "$line" ]; do + line="${line#"${line%%[![:space:]]*}"}" + case "$line" in + ''|\#*) continue ;; + export\ *) line="${line#export }" ;; + esac + case "$line" in + *=*) ;; + *) continue ;; + esac + key="${line%%=*}" + key="${key#"${key%%[![:space:]]*}"}" + val="${line#*=}" + val="${val#"${val%%[![:space:]]*}"}" + val="${val%"${val##*[![:space:]]}"}" + case "$val" in + \"*\") val=${val#\"}; val=${val%\"} ;; + \'*\') val=${val#\'}; val=${val%\'} ;; + esac + if [ -n "$key" ] && [ -z "${!key:-}" ]; then + export "$key=$val" + fi + done < "$ENV_FILE" +fi + +for r in FM_MAIL_USER FM_MAIL_PASS FM_IMAP_HOST FM_SMTP_HOST; do + if [ -z "${!r:-}" ]; then + echo "fm-mail: missing required \$FM_HOME/.env value: $r" >&2 + echo "fm-mail: add $r (and the other three FM_MAIL_* values) to $ENV_FILE" >&2 + exit 1 + fi +done +IMAP_HOST="$FM_IMAP_HOST" +IMAP_PORT="${FM_IMAP_PORT:-993}" +SMTP_HOST="$FM_SMTP_HOST" +SMTP_PORT="${FM_SMTP_PORT:-465}" +case "$IMAP_PORT" in + ''|*[!0-9]*|0) + echo "fm-mail: FM_IMAP_PORT must be a positive integer, got: ${FM_IMAP_PORT:-}" >&2 + exit 1 + ;; +esac +case "$SMTP_PORT" in + ''|*[!0-9]*|0) + echo "fm-mail: FM_SMTP_PORT must be a positive integer, got: ${FM_SMTP_PORT:-}" >&2 + exit 1 + ;; +esac +MAIL_MAX_WAKES="${FM_MAIL_POLL_MAX_WAKES:-20}" +case "$MAIL_MAX_WAKES" in + ''|*[!0-9]*|0) MAIL_MAX_WAKES=20 ;; +esac +if [ "$MAIL_MAX_WAKES" -gt 200 ]; then + MAIL_MAX_WAKES=200 +fi + +PY="$(command -v python3 || true)" +if [ -z "$PY" ]; then + echo "fm-mail: python3 required" >&2 + exit 1 +fi +PY_BIN="$SCRIPT_DIR/fm-mail.py" +if [ ! -f "$PY_BIN" ]; then + echo "fm-mail: $PY_BIN missing" >&2 + exit 1 +fi + +STATE_DIR="$FM_HOME/state" +mkdir -p "$STATE_DIR" +CURSOR="$STATE_DIR/.mail-seen" +# Durable emission journal: every successfully published poll wake records its +# uid here under the queue lock, immediately after the wake row is appended and +# before the cursor records it. A journal entry therefore always proves a wake +# was published, so a mail is never silently suppressed. The fleet wake drain +# acknowledges and removes consumed wake rows from its own queue, so the queue +# alone cannot prove that a wake was ever emitted after an ack; this journal is +# fm-mail's own record of emission and survives any drain ack, which makes +# recovery exactly-once instead of racing the drain. +WOKEN="$STATE_DIR/.mail-woken" +# Generation-scoped retry set: a uid whose header fetch failed is recorded +# here after its degraded wake so a later poll can fetch the real metadata. +# Cleared with the cursor and journal on a UIDVALIDITY change. The retry-scan +# position (.mail-retry-pos) is a durable cursor over this set so the bounded +# per-poll retry window marches through every uid; it is cleared with the set. +RETRY="$STATE_DIR/.mail-retry" +RETRY_POS="$STATE_DIR/.mail-retry-pos" +# Alternating-turn flag for a single contended wake slot (new surfacing vs +# retry recovery) at cap 1; cleared with the retry machinery on a generation +# change so a new mailbox starts with new mail first. +TURN="$STATE_DIR/.mail-turn" + +# Invoke the python engine with the resolved endpoints, cursor, and cap in the +# environment so credentials never reach argv. +run_py() { + FM_MAIL_USER="$FM_MAIL_USER" FM_MAIL_PASS="$FM_MAIL_PASS" \ + FM_IMAP_HOST="$IMAP_HOST" FM_IMAP_PORT="$IMAP_PORT" \ + FM_SMTP_HOST="$SMTP_HOST" FM_SMTP_PORT="$SMTP_PORT" \ + FM_MAIL_CURSOR="$CURSOR" FM_MAIL_RETRY="$RETRY" \ + FM_MAIL_RETRY_POS="$RETRY_POS" FM_MAIL_TURN="$TURN" \ + FM_MAIL_POLL_MAX_WAKES="$MAIL_MAX_WAKES" \ + "$PY" "$PY_BIN" "$@" +} + +usage() { + cat <<'EOF' +fm-mail.sh read +fm-mail.sh send <to> <subject> <body | -> +fm-mail.sh poll +fm-mail.sh status +EOF +} + +mail_seen() { + # $1 = uid; returns 0 when the cursor already records the uid as surfaced. + grep -Fqx "$1" "$CURSOR" +} + +mail_retry_add() { + # $1 = uid; record that a degraded surfacing should be retried. + local id=$1 + [ -n "$id" ] || return 1 + if [ -f "$RETRY" ] && grep -Fqx "$id" "$RETRY"; then + return 0 + fi + printf '%s\n' "$id" >> "$RETRY" || return 1 + return 0 +} + +mail_retry_remove() { + # $1 = uid; drop a recovered uid from the retry set. + local id=$1 rc=0 + [ -n "$id" ] || return 0 + [ -f "$RETRY" ] || return 0 + grep -vx -e "$id" "$RETRY" > "$RETRY.tmp.$$" 2>/dev/null || rc=$? + if [ "$rc" -eq 0 ] || [ "$rc" -eq 1 ]; then + chmod 0600 "$RETRY.tmp.$$" 2>/dev/null || true + mv -f -- "$RETRY.tmp.$$" "$RETRY" || { + rm -f -- "$RETRY.tmp.$$" + return 1 + } + return 0 + fi + rm -f -- "$RETRY.tmp.$$" + return 1 +} + +mail_retry_published() { + # $1 = generation, $2 = uid; 0 when the journal records a recovery/ok publish. + [ -s "$WOKEN" ] || return 1 + awk -F '\t' -v g="$1" -v i="$2" \ + '$1 == g && $2 == i && $3 == "retry" { found=1 } END { exit found ? 0 : 1 }' \ + "$WOKEN" +} + +mail_prune_journal() { + # Drop heal-only journal lines. Keep retry-tagged lines while the uid is + # still in the retry set (or the set cannot be read), so a post-publish + # retry-clear failure cannot re-append a recovery wake. + local jtmp jgen juid jtag + [ -s "$WOKEN" ] || return 0 + jtmp=$(mktemp "$WOKEN.keep.XXXXXX") || return 1 + while IFS=$'\t' read -r jgen juid jtag || [ -n "$jgen" ]; do + [ "$jtag" = retry ] || continue + if [ -f "$RETRY" ] && [ ! -r "$RETRY" ]; then + printf '%s\t%s\t%s\n' "$jgen" "$juid" "$jtag" + continue + fi + if [ -f "$RETRY" ] && grep -Fqx "$juid" "$RETRY"; then + printf '%s\t%s\t%s\n' "$jgen" "$juid" "$jtag" + fi + done < "$WOKEN" > "$jtmp" + mv -f -- "$jtmp" "$WOKEN" || { + rm -f -- "$jtmp" + return 1 + } + return 0 +} + +mail_record_evidence() { + # Write the journal and cursor records; return 0 only when the journal (the + # proof a wake was published) committed. The journal is written FIRST and is + # mandatory: a mail can never be marked surfaced in the cursor without the + # journal recording its wake, so a crash or write failure can never leave a + # uid cursor-recorded but silently suppressed (cursor-without-journal). If + # the journal write fails, the cursor is NOT written and this returns 1, so + # wake_for rolls back / fails closed and the next poll legitimately re-wakes + # the mail instead of treating it as already surfaced. + # A non-empty $3 tags the journal line (retry) so a later poll can skip + # re-appending. + local generation=$1 id=$2 tag=${3:-} + if [ -n "$tag" ]; then + if ! printf '%s\t%s\t%s\n' "$generation" "$id" "$tag" >> "$WOKEN"; then + return 1 + fi + elif ! printf '%s\t%s\n' "$generation" "$id" >> "$WOKEN"; then + return 1 + fi + if ! printf '%s\n' "$id" >> "$CURSOR"; then + # Journal committed but the cursor did not: the wake is still proven by the + # journal and healed into the cursor on the next poll (journal recovery is + # exactly-once). Returning 0 keeps the durable contract: a journal entry + # always means the wake was published. + return 0 + fi + return 0 +} + +mail_rollback_wake_locked() { + # Remove a just-appended wake row plus any partial journal/cursor evidence. + # Runs under the held FM_WAKE_QUEUE_LOCK, so the rewrite cannot race an + # acknowledgement. The journal entry is removed FIRST and required: deleting + # the wake row while a journal entry survives would let the next heal mark + # the uid surfaced without a wake. If the journal cannot be verified and + # cleaned, fail the rollback so the row stays queued and is delivered - + # never suppressed. + local wake_key=$1 generation=$2 id=$3 clean_key tmp jtmp + clean_key=$(printf '%s' "$wake_key" | fm_wake_clean_field) + # Journal evidence: only when this uid has an entry must it be removed now. + # When the journal is unwritable (the usual reason both records failed) no + # entry exists and there is nothing to clean. + if awk -F '\t' -v g="$generation" -v i="$id" \ + '$1 == g && $2 == i { found=1 } END { exit found ? 0 : 1 }' \ + "$WOKEN" 2>/dev/null; then + jtmp=$(mktemp "$WOKEN.rm.XXXXXX") || return 1 + if ! awk -F '\t' -v g="$generation" -v i="$id" \ + '!($1 == g && $2 == i)' "$WOKEN" > "$jtmp" 2>/dev/null; then + rm -f -- "$jtmp" + return 1 + fi + if ! mv -f -- "$jtmp" "$WOKEN" 2>/dev/null; then + rm -f -- "$jtmp" + return 1 + fi + fi + # Queue row: remove it (required) so nothing ackable survives without a + # durable record. + tmp=$(mktemp "$FM_WAKE_QUEUE.rollback.XXXXXX") || return 1 + if ! awk -F '\t' -v key="$clean_key" ' + NF >= 5 && $3 == "check" && $4 == key { next } + { print } + ' "$FM_WAKE_QUEUE" > "$tmp"; then + rm -f -- "$tmp" + return 1 + fi + chmod 0600 "$tmp" 2>/dev/null || true + if ! mv -f -- "$tmp" "$FM_WAKE_QUEUE"; then + rm -f -- "$tmp" + return 1 + fi + # Cursor record: best-effort; a surviving cursor line only means already + # surfaced, which the heal tolerates. + if grep -vx -e "$id" "$CURSOR" > "$CURSOR.tmp.$$" 2>/dev/null; then + if mv -f -- "$CURSOR.tmp.$$" "$CURSOR" 2>/dev/null; then + : + fi + fi + rm -f -- "$CURSOR.tmp.$$" + return 0 +} + +wake_for() { + # Publish one `check` wake and its durable records under a single held + # FM_WAKE_QUEUE_LOCK. The key is generation-aware when the mailbox reports a + # UIDVALIDITY, so a restored mailbox's reused uid can never collide with a + # stale wake key. The wake row is appended first, then the evidence records; + # the drain acknowledges and deletes consumed rows only under the same lock, + # so it can never remove our wake between the surface and the uid record. + # A journal entry therefore always means the wake was published - a mail is + # never silently suppressed. If no durable record can be written the row is + # rolled back for a clean retry, and only when the journal, the cursor, and + # the queue rewrite all fail does the poll fail closed, accepting a possible + # duplicate over a lost mail. + # + # The one irreducible residual is a kill in the microseconds between the + # queue append and the journal write, followed by the drain acknowledging the + # row before the next poll heals it: neither the journal nor the cursor then + # holds the uid, and the next poll wakes the mail again. A possible duplicate + # (never a missed mail) is the deliberate, bounded tradeoff for keeping the + # durable record write on the same held lock as the publish. + # + # Returns: + # 0 - the wake row was appended and a durable uid record landed. + # 1 - the wake row was appended and rolled back; nothing was delivered. + # 2 - the wake row survived with no durable record (fail-closed); the drain + # delivers it and the next poll's heal records the uid. + # 3 - the wake row was never appended; nothing was delivered. + # 4 - the wake was delivered but the optional retry-id cleanup failed. + local generation=$1 id=$2 summary=$3 retry_id=${4:-} lib="$SCRIPT_DIR/fm-wake-lib.sh" status=0 tag="" + local wake_key="mail:$id" + [ -n "$retry_id" ] && tag=retry + if [ -n "$generation" ]; then + wake_key="mail:$generation/$id" + fi + if [ ! -f "$lib" ]; then + echo "fm-mail: $lib missing; cannot wake" >&2 + return 1 + fi + # shellcheck source=bin/fm-wake-lib.sh + # shellcheck disable=SC1091 + . "$lib" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + if fm_wake_append_locked check "$wake_key" "check: mail $id - $summary"; then + if mail_record_evidence "$generation" "$id" "$tag"; then + : + elif mail_rollback_wake_locked "$wake_key" "$generation" "$id"; then + echo "fm-mail: wake for $id rolled back (journal and cursor writes failed); retried on next poll" >&2 + status=1 + elif mail_record_evidence "$generation" "$id" "$tag"; then + echo "fm-mail: wake for $id durably recorded after the queue rewrite failed" >&2 + else + echo "fm-mail: wake for $id could not be rolled back or durably recorded; the wake stays queued and the next poll heals it - a possible duplicate, never a lost mail" >&2 + status=2 + fi + else + echo "fm-mail: wake append failed for $id; retried on next poll" >&2 + status=3 + fi + # A recovered uid must stay retry-eligible until the wake is durably + # published, so the retry record is cleared only after a successful append. + # This removes the kill-window between "retry removed" and "wake published" + # that could strand recovered metadata: the uid would be cursor-recorded from + # the earlier degraded wake but no longer in the retry set, so later polls + # would never re-fetch it. + if { [ "$status" -eq 0 ] || [ "$status" -eq 2 ]; } && [ -n "$retry_id" ]; then + if ! mail_retry_remove "$retry_id"; then + echo "fm-mail: could not clear retry for recovered $retry_id after publish; retried on next poll" >&2 + status=4 + fi + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return "$status" +} + +mail_stored_generation() { + # Print the mailbox generation the local cursor was last reset to, or "". + [ -f "$CURSOR" ] || : > "$CURSOR" + grep -m1 '^uidvalidity=' "$CURSOR" | cut -d= -f2 || true +} + +mail_heal() { + # Reconcile a poll interrupted between its operations. Emission is a + # three-phase commit: the wake append publishes the surfacing, the journal + # write then proves THIS home emitted it, and the cursor record finally + # declares the uid surfaced. Each phase is healed from durable evidence: + # + # 1. Journal heal - a journal entry is proof a wake was published, written + # immediately after a successful wake append under the same lock. It + # survives the fleet drain's ack (which physically removes consumed wake + # rows from the queue), so a poll killed after appending its wake but + # before recording the uid is recovered even when the drain already + # acknowledged that wake: the uid is recorded without re-waking, never + # duplicate. + # 2. Queue heal - a queued wake whose uid is absent from the cursor (kill in + # the tiny gap between wake append and journal write) is likewise recorded + # without re-waking. + # Both are generation-scoped: only evidence matching the CURRENT mailbox + # generation is healed, so a legacy key or a stale prior-generation wake can + # never mark a reused numeric uid as surfaced in the new mailbox. + local generation=$1 jgen juid jtag keyrest keygen keyuid heal_ok=0 + if [ -s "$WOKEN" ]; then + while IFS=$'\t' read -r jgen juid jtag; do + [ -n "$juid" ] || continue + [ "$jgen" != "$generation" ] && continue + if ! mail_seen "$juid"; then + if printf '%s\n' "$juid" >> "$CURSOR"; then + : + else + heal_ok=1 + fi + fi + done < "$WOKEN" + # Drop heal-only journal lines once every uid is durably recorded. Keep + # retry-tagged lines for uids still in the retry set so a post-publish + # retry-clear failure cannot re-append a recovery wake. If any cursor + # write failed, keep the whole journal so the next poll can retry it. + if [ "$heal_ok" -eq 0 ]; then + mail_prune_journal || true + fi + fi + while IFS= read -r k; do + keyrest="${k#mail:}" + [ "$keyrest" = "$k" ] && continue + keygen="" + keyuid="" + case "$keyrest" in + */*) keygen="${keyrest%%/*}"; keyuid="${keyrest#*/}" ;; + *) keyuid="$keyrest" ;; + esac + [ -z "$keyuid" ] && continue + [ "$keygen" != "$generation" ] && continue + if ! mail_seen "$keyuid"; then + if printf '%s\n' "$keyuid" >> "$CURSOR"; then + : + else + heal_ok=1 + fi + fi + done < <(fm_wake_queued_keys check 2>/dev/null || true) + return "$heal_ok" +} + +mail_poll() { + # List unseen mail (uid,date,from,subj,status) plus the mailbox generation + # guard, then wake each NEW uid and each recovered retry uid. status is + # ok, retry, or degraded; an empty status is treated as ok so a legacy + # four-field row still wakes. Never marks anything read. Overlapping polls + # are serialized on the mail-seen lock; each poll first heals a run + # interrupted between its phases (mail_heal), so an overlapping poll or an + # interrupted run can never lose a mail. wake_for owns the remaining + # kill-window duplicate residual. + local list generation first_line uid fr subj status woke=0 need_wake line wake_rc=0 + if [ ! -f "$SCRIPT_DIR/fm-wake-lib.sh" ]; then + echo "fm-mail: $SCRIPT_DIR/fm-wake-lib.sh missing; cannot poll" >&2 + return 1 + fi + # shellcheck source=bin/fm-wake-lib.sh + # shellcheck disable=SC1091 + . "$SCRIPT_DIR/fm-wake-lib.sh" + fm_lock_acquire_wait "$STATE_DIR/.mail-seen.lock" + if ! list="$(run_py poll_list)"; then + # The poll engine already printed its cause on stderr; just release the + # lock and fail instead of letting set -e abort the whole script with the + # lock still held. + fm_lock_release "$STATE_DIR/.mail-seen.lock" + return 1 + fi + # Split the generation guard without `head`. + # Under `set -o pipefail`, `printf | head -n1` can EPIPE a multi-row list and abort the poll. + first_line="${list%%$'\n'*}" + generation="${first_line#*$'\t'}" + if [ "$list" = "$first_line" ]; then + list="" + else + list="${list#*$'\n'}" + fi + + # A recreated/restored mailbox has a new UIDVALIDITY; a numeric uid can be + # reused, so a stale cursor must not suppress its wake. Journal entries from + # the old mailbox are equally stale: they describe wakes from before the + # mailbox identity changed, so clear them rather than risk healing a reused + # uid into the new generation. The retry set is equally stale. + if [ -n "$generation" ] && [ "$(mail_stored_generation)" != "$generation" ]; then + printf 'uidvalidity=%s\n' "$generation" > "$CURSOR" + : > "$WOKEN" + : > "$RETRY" + : > "$RETRY_POS" + : > "$TURN" + fi + + if ! mail_heal "$generation"; then + echo "fm-mail: heal could not record a uid; journal kept; retried on next poll" >&2 + fm_lock_release "$STATE_DIR/.mail-seen.lock" + return 1 + fi + + # cut -f keeps empty TSV fields; IFS-tab read would collapse the empty + # Date on a degraded row and shift status off the end. + while IFS= read -r line || [ -n "$line" ]; do + [ -z "$line" ] && continue + uid=$(printf '%s\n' "$line" | cut -f1) + fr=$(printf '%s\n' "$line" | cut -f3) + subj=$(printf '%s\n' "$line" | cut -f4) + status=$(printf '%s\n' "$line" | cut -f5) + [ -z "$uid" ] && continue + [ -z "$status" ] && status=ok + need_wake=0 + case "$status" in + retry) + # Already cursor-recorded from the degraded wake. Surface recovered + # metadata once; if a recovery/ok publish is already journaled, retry + # the retry-set clear without appending another wake. + if mail_retry_published "$generation" "$uid"; then + if ! mail_retry_remove "$uid"; then + echo "fm-mail: could not clear retry for recovered $uid after publish; retried on next poll" >&2 + fm_lock_release "$STATE_DIR/.mail-seen.lock" + return 1 + fi + else + need_wake=1 + fi + ;; + *) + if ! mail_seen "$uid"; then + need_wake=1 + fi + ;; + esac + if [ "$need_wake" -eq 1 ]; then + # Wake first, then record, then clear retry eligibility: the wake append, + # journal, cursor commit, and retry-record removal all happen together + # under the wake-queue lock inside wake_for (so no drain ack can split + # them), and a failure stops the poll so the next run retries. A kill + # before the append leaves nothing and the next poll retries; a kill + # after the append is healed above without re-waking. + # Reaching the per-poll wake cap stops the loop: the remaining unseen + # mail stays out of the cursor and surfaces on the next poll, so a flood + # bounds the durable wake queue instead of flooding firstmate. + if [ "$woke" -ge "$MAIL_MAX_WAKES" ]; then + echo "fm-mail: per-poll wake cap ($MAIL_MAX_WAKES) reached; remaining mail surfaces on the next poll" >&2 + break + fi + if [ "$status" = degraded ]; then + # Record the retry BEFORE the wake so a failed retry write can never + # leave the uid cursor-recorded but unrecoverable: the mail stays + # unseen and is retried next poll instead. + if ! mail_retry_add "$uid"; then + echo "fm-mail: could not record retry for $uid; retried on next poll" >&2 + fm_lock_release "$STATE_DIR/.mail-seen.lock" + return 1 + fi + fi + wake_rc=0 + # Clear the retry record as part of the wake publish transaction. For a + # recovered uid this removes the dangerous gap where the retry was cleared + # but the wake had not yet published; a kill in that gap would leave the + # uid cursor-recorded from the degraded wake but no longer retry-eligible, + # so its recovered metadata could never surface. For normal (ok) mail it + # also clears any stale retry entry left by a rolled-back earlier wake. + # Degraded mail keeps its retry entry so the next poll retries the fetch. + retry_arg="" + if [ "$status" = retry ] || [ "$status" = ok ]; then + retry_arg="$uid" + fi + wake_for "$generation" "$uid" "mail from $fr - ${subj:-no subject}" "$retry_arg" || wake_rc=$? + if [ "$wake_rc" -eq 0 ]; then + echo "fm-mail: woke for $uid" + woke=$((woke + 1)) + else + echo "fm-mail: wake failed for $uid; retried on next poll" >&2 + fm_lock_release "$STATE_DIR/.mail-seen.lock" + return 1 + fi + fi + done <<< "$list" + fm_lock_release "$STATE_DIR/.mail-seen.lock" + if [ "$woke" -eq 0 ]; then + echo "fm-mail: no new mail" + fi + return 0 +} + +case "${1:-}" in + read) + run_py read + ;; + send) + to="${2:-}" + subj="${3:-}" + body="${4:--}" + if [ -z "$to" ] || [ -z "$subj" ]; then + usage + exit 1 + fi + if [ "$body" = "-" ]; then + body="$(cat)" + fi + printf '%s' "$body" | run_py send "$to" "$subj" "-" + ;; + status) + echo "mail account: $FM_MAIL_USER" + echo "imap: $IMAP_HOST:$IMAP_PORT smtp: $SMTP_HOST:$SMTP_PORT" + run_py seen "$CURSOR" || true + ;; + poll) + mail_poll + ;; + -h|--help) + usage + ;; + *) + usage + exit 1 + ;; +esac \ No newline at end of file diff --git a/bin/fm-merge-local.sh b/bin/fm-merge-local.sh index b4f0ca37a10..41e0621bb27 100755 --- a/bin/fm-merge-local.sh +++ b/bin/fm-merge-local.sh @@ -10,6 +10,11 @@ # auto-approves), and only as a clean fast-forward - it refuses a diverged branch # and tells you to have the crewmate rebase. See AGENTS.md prime directives, # project management, and task lifecycle. +# The task's existing per-task control lock serializes the captain-hold check +# through that fast-forward. A still-held or unreadable row refuses before the +# merge, so a captain approval must be recorded as an `answer --release` before +# this entrypoint is invoked. The lock ends when the fast-forward returns; +# docs/captain-hold-lifecycle.md owns the accepted merge-to-cleanup residual. # Usage: fm-merge-local.sh <task-id> set -eu @@ -17,16 +22,54 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +# shellcheck source=bin/fm-pr-lib.sh +. "$SCRIPT_DIR/fm-pr-lib.sh" +# shellcheck source=bin/fm-backlog-transition-lib.sh +. "$SCRIPT_DIR/fm-backlog-transition-lib.sh" +if [ "$#" -ne 1 ] || ! fm_pr_task_id_valid "$1"; then + echo "error: invalid local merge request" >&2 + exit 2 +fi +ID=$1 +fm_backlog_directory_present "$STATE" "state directory" || { + echo "error: local merge refused: $FM_BACKLOG_TRANSITION_ERROR" >&2 + exit 1 +} +META="$STATE/$ID.meta" + +# shellcheck source=bin/fm-wake-lib.sh +. "$SCRIPT_DIR/fm-wake-lib.sh" "$FM_ROOT/bin/fm-guard.sh" || true # Role partition: landing local-only work is MAIN-owned; the Pi supervision # branch reports readiness and never lands (contract: bin/fm-lease-lib.sh; -# no-op in homes without a branch actor). +# no-op in homes without a branch actor). This precedes reading the task +# record, because the wrong actor is refused for its role whatever it says. # shellcheck source=bin/fm-lease-lib.sh . "$SCRIPT_DIR/fm-lease-lib.sh" fm_lease_forbid_branch "local-only landing (fm-merge-local)" -ID=${1:?usage: fm-merge-local.sh <task-id>} -META="$STATE/$ID.meta" + [ -f "$META" ] || { echo "error: no meta for task $ID at $META" >&2; exit 1; } +if ! fm_backlog_meta_spawn_gen_optional "$META" "$STATE"; then + echo "error: local merge refused: $FM_BACKLOG_TRANSITION_ERROR" >&2 + exit 1 +fi +MERGE_EXPECTED_SPAWN_GEN=$FM_BACKLOG_META_SPAWN_GEN + +MERGE_CONTROL_LOCK= +merge_control_cleanup() { + [ -z "$MERGE_CONTROL_LOCK" ] || fm_lock_release "$MERGE_CONTROL_LOCK" || true +} +trap merge_control_cleanup EXIT +MERGE_CONTROL_LOCK="$STATE/.control-$ID.lock" +fm_lock_acquire_wait "$MERGE_CONTROL_LOCK" +if ! fm_backlog_meta_spawn_gen_optional "$META" "$STATE"; then + echo "error: task $ID changed while waiting to merge; refusing: $FM_BACKLOG_TRANSITION_ERROR" >&2 + exit 1 +fi +if [ "$FM_BACKLOG_META_SPAWN_GEN" != "$MERGE_EXPECTED_SPAWN_GEN" ]; then + echo "error: task $ID changed incarnation while waiting to merge; refusing" >&2 + exit 1 +fi PROJ=$(grep '^project=' "$META" | cut -d= -f2-) MODE=$(grep '^mode=' "$META" | cut -d= -f2- || true) @@ -74,6 +117,24 @@ if ! git -C "$PROJ" merge-base --is-ancestor "$DEFAULT" "$BRANCH"; then fi before=$(git -C "$PROJ" rev-parse --short "$DEFAULT") -git -C "$PROJ" merge --ff-only "$BRANCH" >/dev/null +hold_status=0 +FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" \ + "$SCRIPT_DIR/fm-captain-hold.sh" open "$ID" --distinguish-absent || hold_status=$? +case "$hold_status" in + 0) + echo "error: task $ID is still held for the captain; release it before merging" >&2 + exit 1 + ;; + 1|3) ;; + *) + echo "error: could not determine whether task $ID is still held for the captain; refusing to merge" >&2 + exit 1 + ;; +esac +merge_status=0 +git -C "$PROJ" merge --ff-only "$BRANCH" >/dev/null || merge_status=$? +fm_lock_release "$MERGE_CONTROL_LOCK" || true +MERGE_CONTROL_LOCK= +[ "$merge_status" -eq 0 ] || exit "$merge_status" after=$(git -C "$PROJ" rev-parse --short "$DEFAULT") echo "merged $BRANCH into local $DEFAULT ($before -> $after) in $PROJ" diff --git a/bin/fm-nm-run-lib.sh b/bin/fm-nm-run-lib.sh index d2f5d8729bd..5410fd1544b 100644 --- a/bin/fm-nm-run-lib.sh +++ b/bin/fm-nm-run-lib.sh @@ -195,10 +195,22 @@ fm_nm_runs_row_fields_valid() { # <status> <branch> <head> <date> <time> <pr> < return 0 } +# Classify a ledger status for live-over-terminal selection. Unknown statuses +# retain their existing newest-row precedence. +fm_nm_run_status_class() { + case "${1:-}" in + completed|failed|cancelled) printf terminal ;; + running) printf live ;; + *) printf unknown ;; + esac +} + # One owner for runs-ledger attribution, returning the newest matching branch # row as <evidence>\t<status>\t<head>. Evidence is attributable, inconclusive, -# or rejected; an absent branch prints nothing. Older rows never answer as the -# current run. The only use of an older row is the approved narrow anchor: +# or rejected; an absent branch prints nothing. A binding terminal row yields +# to an older live row that also binds. An unresolved live sibling requires the +# held terminal row to be exactly local HEAD; no other unknown head is admitted. +# Within a liveness class the newest row wins. The other older-row use is the anchor: # a newest running unresolved head is attributable when the immediately older # same-branch row is completed, failed, or cancelled at exactly local HEAD. # Missing, nonterminal, and mismatched anchors remain inconclusive. This is a @@ -208,18 +220,29 @@ fm_nm_runs_row_fields_valid() { # <status> <branch> <head> <date> <time> <pr> < # Malformed input is inconclusive, never proof that this branch has no run. fm_nm_runs_row_for_worktree() { # <worktree> <branch> <runs-list-output> local wt=$1 branch=$2 list=$3 row st br sha relation pending_head='' - local day clock pr extra + local day clock pr extra decided_status='' decided_head='' decided_relation='' [ -n "$list" ] || return 0 while IFS= read -r row; do row=$(fm_nm_trim "$row") [ -n "$row" ] || continue IFS=$' \t' read -r st br sha day clock pr extra <<< "$row" if ! fm_nm_runs_row_fields_valid "$st" "$br" "$sha" "$day" "$clock" "$pr" "$extra"; then + [ -z "$decided_status" ] || break printf 'inconclusive\tmalformed\t-' return 0 fi [ "$br" = "$branch" ] || continue relation=$(fm_nm_head_relation "$wt" "$sha") + if [ -n "$decided_status" ]; then + [ "$(fm_nm_run_status_class "$st")" = live ] || continue + case "$relation" in + equal|run-ahead) ;; + unresolved) [ "$decided_relation" = equal ] || continue ;; + *) continue ;; + esac + printf 'attributable\trunning\t%s' "$sha" + return 0 + fi if [ -n "$pending_head" ]; then case "$st:$relation" in completed:equal|failed:equal|cancelled:equal) @@ -229,7 +252,16 @@ fm_nm_runs_row_for_worktree() { # <worktree> <branch> <runs-list-output> return 0 fi case "$relation" in - equal|run-ahead) printf 'attributable\t%s\t%s' "$st" "$sha"; return 0 ;; + equal|run-ahead) + if [ "$(fm_nm_run_status_class "$st")" = terminal ]; then + decided_status=$st + decided_head=$sha + decided_relation=$relation + else + printf 'attributable\t%s\t%s' "$st" "$sha" + return 0 + fi + ;; unresolved) case "$st" in running) pending_head=$sha ;; @@ -239,6 +271,10 @@ fm_nm_runs_row_for_worktree() { # <worktree> <branch> <runs-list-output> *) printf 'rejected\t%s\t%s' "$st" "$sha"; return 0 ;; esac done <<< "$list" + if [ -n "$decided_status" ]; then + printf 'attributable\t%s\t%s' "$decided_status" "$decided_head" + return 0 + fi [ -z "$pending_head" ] || printf 'inconclusive\trunning\t%s' "$pending_head" return 0 } diff --git a/bin/fm-parent-channel-lib.sh b/bin/fm-parent-channel-lib.sh index 15a1baaf7fd..8b1feccd80c 100644 --- a/bin/fm-parent-channel-lib.sh +++ b/bin/fm-parent-channel-lib.sh @@ -3,10 +3,9 @@ # # WHY THIS EXISTS. A secondmate is a firstmate in its own home, and nobody reads # its chat: the captain and the main firstmate see only what is appended to the -# parent channel. AGENTS.md tells every firstmate to reach the captain and to -# address the captain in every response, so a mate model reliably "reports" a -# PR-ready result, a finding, a decision, a blocker, or a failure in its own -# chat and skips the one status-file append that would actually deliver it. +# parent channel. A mate can satisfy AGENTS.md's address rule in local chat +# while skipping the charter's return-channel instruction, so a PR-ready result, +# finding, decision, blocker, or failure never reaches the parent. # Four such misses were observed on 2026-09-02 across two mate homes; the # watcher had delivered the parent's request each time and the work was done. # The problem is therefore not one missed PR notice but every captain-facing diff --git a/bin/fm-pending-reply-lib.sh b/bin/fm-pending-reply-lib.sh index 27c9bc9e90d..79283ba0941 100755 --- a/bin/fm-pending-reply-lib.sh +++ b/bin/fm-pending-reply-lib.sh @@ -38,6 +38,12 @@ # phase= awaiting_report | delivery_unknown | recovery_sending | # recovery_sent | recovery_failed | recovery_unknown | # escalated | resolved +# An escalated record with an empty delivered_epoch is +# a delivery-unknown escalation, not a missed report: +# its owner may still resend the same correlation, and +# fm_pending_reply_reset_known_undelivered returns it to +# awaiting_report for that resend (see the retryable +# undelivered escalation note below) # turn_seen_busy= 0|1 after delivery for the original request turn # request_turn_completed_epoch= # recovery_attempted_epoch= @@ -76,6 +82,19 @@ # owned below (fm_pending_reply_resolved_note), because a bare answered: note is # not a reserved-key transition and would leave the decision open. # +# Retryable undelivered escalation: a delivery-unknown escalation reports that +# the request may never have reached the mate, so the request stays the owner's +# to resend under the same correlation (fm-send's FM_PENDING_REPLY_EXISTING_CORR +# contract; the remote enqueue deduplicates onto the same record). The resend +# resets the record to awaiting_report and leaves the published escalation +# decision open: a confirmed delivery does not settle the request, only a +# correlated report does. A later missed-report escalation reuses that key +# rather than opening a duplicate, and only the ordinary resolve close closes +# it. A delivered record, whatever its phase, is never reset. Without this, a +# wake retried only through its owner +# (bin/fm-backlog-handoff.sh's receiver wake) stayed refused forever once the +# watcher escalated between the lost transport and the next resume. +# # Sourced by bin/fm-send.sh, bin/fm-watch.sh, bin/fm-secondmate-report.sh, and # tests. No side effects on source. set -u / set -e safe. # @@ -220,7 +239,9 @@ fm_pending_reply_corr_reusable() { # <state-dir> <corr_id> <task_id> phase=$(fm_pending_reply_get "$rec" phase) case "$phase" in awaiting_report|recovery_sending|recovery_sent) return 0 ;; - delivery_unknown) + delivery_unknown|escalated) + # Undelivered only: a delivery-unknown escalation stays the owner's to + # resend, while an escalation after delivery guards a missed report. delivered=$(fm_pending_reply_get "$rec" delivered_epoch) [ -z "$delivered" ] return $? @@ -489,10 +510,12 @@ fm_pending_reply_delivery_attempt_unresolved() { # <state-dir> <corr_id> return 1 } -# A definitive backend rejection makes the existing correlation retryable again. -# Reconciliation may have aged the same attempted sidecar to delivery_unknown -# while the backend call was in flight, so both undelivered phases converge here -# under the per-correlation lock; a confirmed delivery can never be reset. +# A definitive backend rejection, or an owner's idempotent remote resend, makes +# the existing correlation retryable again. Reconciliation may have aged the +# same attempted sidecar to delivery_unknown while the backend call was in +# flight, and the watcher may then have escalated that unknown delivery, so all +# three undelivered phases converge here under the per-correlation lock; a +# confirmed delivery can never be reset, whatever its phase. fm_pending_reply_reset_known_undelivered() { # <state-dir> <corr_id> local state=$1 corr=$2 lock rc=0 local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK @@ -512,7 +535,7 @@ _fm_pending_reply_reset_known_undelivered_locked() { # <state-dir> <corr_id> delivered=$(fm_pending_reply_get "$rec" delivered_epoch) [ -z "$delivered" ] || return 1 phase=$(fm_pending_reply_get "$rec" phase) - case "$phase" in awaiting_report|delivery_unknown) ;; *) return 1 ;; esac + case "$phase" in awaiting_report|delivery_unknown|escalated) ;; *) return 1 ;; esac marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr") [ -e "$marker" ] || [ -L "$marker" ] || { [ "$phase" = awaiting_report ] @@ -1224,7 +1247,7 @@ _fm_pending_reply_maybe_escalate_locked() { # <state-dir> <corr_id> parent_status=$(fm_pending_reply_get "$rec" parent_status) case "$phase" in delivery_unknown) kind=delivery-unknown ;; - recovery_failed|recovery_unknown) kind=recovery-delivery ;; + recovery_failed|recovery_unknown) kind='recovery-delivery' ;; *) kind=missed ;; esac payload=$(fm_pending_reply_escalation_payload "$rec" "$kind") || return 1 diff --git a/bin/fm-pr-merge.sh b/bin/fm-pr-merge.sh index 3de631243d8..3cf2cac6fad 100755 --- a/bin/fm-pr-merge.sh +++ b/bin/fm-pr-merge.sh @@ -76,6 +76,13 @@ # recorded value stale. Reading that state needs glab and jq, and either one # absent stops the merge before any state is recorded. # +# Before either forge merge, the task's existing per-task control lock +# serializes the captain-hold check through the forge command. A still-held or +# unreadable row refuses before that command, so a captain approval must be +# recorded as an `answer --release` before this entrypoint is invoked. The lock +# ends when the local forge command returns; docs/captain-hold-lifecycle.md owns +# the accepted asynchronous-landing and merge-to-cleanup residuals. +# # Extra args must not include --repo or -R in any form, including a bundled # short-option cluster such as -yR, because the repository comes only from the # URL, nor --sha on GitLab because the head comes only from the live read. @@ -113,18 +120,14 @@ esac # shellcheck source=bin/fm-pr-lib.sh . "$SCRIPT_DIR/fm-pr-lib.sh" +# shellcheck source=bin/fm-backlog-transition-lib.sh +. "$SCRIPT_DIR/fm-backlog-transition-lib.sh" # shellcheck source=bin/fm-merge-outcome-lib.sh . "$SCRIPT_DIR/fm-merge-outcome-lib.sh" # shellcheck source=bin/fm-issue-lib.sh . "$SCRIPT_DIR/fm-issue-lib.sh" # shellcheck source=bin/fm-forge-lib.sh . "$SCRIPT_DIR/fm-forge-lib.sh" -# Role partition: merging is MAIN-owned; the Pi supervision branch reports the -# green PR and never merges (contract: bin/fm-lease-lib.sh; no-op in homes -# without a branch actor). -# shellcheck source=bin/fm-lease-lib.sh -. "$SCRIPT_DIR/fm-lease-lib.sh" -fm_lease_forbid_branch "PR merge (fm-pr-merge)" if [ "$#" -lt 2 ]; then echo "error: invalid PR merge request" >&2 @@ -391,12 +394,47 @@ reject_head_overrides() { reject_repo_overrides "$@" || exit 1 [ "$PROVIDER" != gitlab ] || reject_head_overrides "$@" || exit 1 -# Task-derived paths are constructed only after the canonical ID validation. +fm_backlog_directory_present "$STATE" "state directory" || { + echo "error: PR merge refused: $FM_BACKLOG_TRANSITION_ERROR" >&2 + exit 1 +} META="$STATE/$ID.meta" + +# shellcheck source=bin/fm-wake-lib.sh +. "$SCRIPT_DIR/fm-wake-lib.sh" +# Role partition: merging is MAIN-owned; the Pi supervision branch reports the +# green PR and never merges (contract: bin/fm-lease-lib.sh; no-op in homes +# without a branch actor). This precedes reading the task record, because the +# wrong actor is refused for its role whatever that record says. +# shellcheck source=bin/fm-lease-lib.sh +. "$SCRIPT_DIR/fm-lease-lib.sh" +fm_lease_forbid_branch "PR merge (fm-pr-merge)" + if [ ! -f "$META" ] || [ -L "$META" ]; then echo "error: task metadata is unavailable" >&2 exit 1 fi +if ! fm_backlog_meta_spawn_gen_optional "$META" "$STATE"; then + echo "error: PR merge refused: $FM_BACKLOG_TRANSITION_ERROR" >&2 + exit 1 +fi +MERGE_EXPECTED_SPAWN_GEN=$FM_BACKLOG_META_SPAWN_GEN + +MERGE_CONTROL_LOCK= +merge_control_cleanup() { + [ -z "$MERGE_CONTROL_LOCK" ] || fm_lock_release "$MERGE_CONTROL_LOCK" || true +} +trap merge_control_cleanup EXIT +MERGE_CONTROL_LOCK="$STATE/.control-$ID.lock" +fm_lock_acquire_wait "$MERGE_CONTROL_LOCK" +if ! fm_backlog_meta_spawn_gen_optional "$META" "$STATE"; then + echo "error: task $ID changed while waiting to merge; refusing: $FM_BACKLOG_TRANSITION_ERROR" >&2 + exit 1 +fi +if [ "$FM_BACKLOG_META_SPAWN_GEN" != "$MERGE_EXPECTED_SPAWN_GEN" ]; then + echo "error: task $ID changed incarnation while waiting to merge; refusing" >&2 + exit 1 +fi # Reading the merge request state needs both tools. Report them together and # before anything is recorded, so a missing tool is a named prerequisite rather @@ -919,6 +957,23 @@ record_pr_metadata() { } } +require_released_captain_hold() { + local hold_status=0 + FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" \ + "$SCRIPT_DIR/fm-captain-hold.sh" open "$ID" --distinguish-absent || hold_status=$? + case "$hold_status" in + 0) + echo "error: task $ID is still held for the captain; release it before merging" >&2 + return 1 + ;; + 1|3) return 0 ;; + *) + echo "error: could not determine whether task $ID is still held for the captain; refusing to merge" >&2 + return 1 + ;; + esac +} + FM_PR_GITHUB_AUTO_REQUESTED=false FM_PR_GITHUB_MERGE_ACCEPTED=false FM_PR_GITHUB_CALLER_METHOD= @@ -1261,11 +1316,15 @@ case "$PROVIDER" in exit 1 fi fi - if merge_output=$(gh-axi pr merge "$PR_NUMBER" --repo "$PR_OWNER/$PR_REPO" \ - "${merge_args[@]+"${merge_args[@]}"}" "$@" 2>&1); then + require_released_captain_hold || exit 1 + merge_status=0 + merge_output=$(gh-axi pr merge "$PR_NUMBER" --repo "$PR_OWNER/$PR_REPO" \ + "${merge_args[@]+"${merge_args[@]}"}" "$@" 2>&1) || merge_status=$? + merge_control_cleanup + MERGE_CONTROL_LOCK= + if [ "$merge_status" -eq 0 ]; then FM_PR_GITHUB_MERGE_ACCEPTED=true else - merge_status=$? [ -z "$merge_output" ] || printf '%s\n' "$merge_output" >&2 if [ "$github_landed_observed" = true ] || github_read_outcome; then if [ "$FM_PR_GITHUB_MERGED" = true ]; then @@ -1316,6 +1375,7 @@ case "$PROVIDER" in # in between is refused by GitLab instead of merged unverified. --yes only # skips the interactive confirmation, which no supervised run can answer; # the conditions above are what authorize the merge. + require_released_captain_hold || exit 1 # Capture the command status rather than letting set -e abort here: if the # forge LANDS the merge and a post-execution step or the response transport # then fails, aborting would skip the confirmation read entirely and leave a @@ -1324,6 +1384,8 @@ case "$PROVIDER" in gitlab_merge_rc=0 GITLAB_HOST="$FM_PR_HOST" glab mr merge "$PR_NUMBER" -R "$PROJECT_URL" \ --sha "$FM_PR_MERGE_HEAD" --yes "$@" || gitlab_merge_rc=$? + merge_control_cleanup + MERGE_CONTROL_LOCK= # Before the command status was captured, set -e guaranteed the merge command # had succeeded whenever this ran, so "accepted the merge request" was always # true. It is not any more, and a confirm that claims acceptance on a failed diff --git a/bin/fm-procevent-lib.sh b/bin/fm-procevent-lib.sh index ff8ef62a920..266365dd526 100644 --- a/bin/fm-procevent-lib.sh +++ b/bin/fm-procevent-lib.sh @@ -178,9 +178,7 @@ fm_procevent_owner_lease_seconds() { printf '%s\n' "$value" } -# How often a runner's guard re-reads that lease. One watcher cycle at the -# default poll interval, so the guard costs about as much as the cycle that -# refreshes what it reads. +# Detection-interval semantics: docs/configuration.md, Process-to-event sources. FM_PROCEVENT_OWNER_CHECK_DEFAULT_SECONDS=15 FM_PROCEVENT_OWNER_CHECK_MIN_SECONDS=1 FM_PROCEVENT_OWNER_CHECK_MAX_SECONDS=3600 @@ -321,6 +319,18 @@ fm_procevent_source_lock_acquire() { fm_lock_acquire_wait "$(fm_procevent_source_lock_path "$id")" } +# fm_procevent_source_lock_try_acquire <source-id> +# Non-blocking acquisition for release_start_claim in bin/fm-procevent.sh; +# that caller owns the exit-cleanup lock-order invariant. +fm_procevent_source_lock_try_acquire() { + local id=$1 root + fm_procevent_source_id_valid "$id" || return 1 + root=$(fm_procevent_claim_root) + (umask 077; mkdir -p "$root") || return 1 + [ -d "$root" ] && [ ! -L "$root" ] || return 1 + fm_lock_try_acquire "$(fm_procevent_source_lock_path "$id")" +} + fm_procevent_source_lock_release() { fm_lock_release "$(fm_procevent_source_lock_path "$1")" } diff --git a/bin/fm-procevent.sh b/bin/fm-procevent.sh index aaa7d60c541..93361b64552 100755 --- a/bin/fm-procevent.sh +++ b/bin/fm-procevent.sh @@ -771,10 +771,14 @@ cmd_start() { CLAIM_STATE_DEVICE=$FM_PROCEVENT_CLAIM_STATE_DEVICE CLAIM_STATE_INODE=$FM_PROCEVENT_CLAIM_STATE_INODE STAGED_OUTPUT= + # Exit cleanup must not wait for the source lock: retire and reconcile hold it + # while waiting for this runner, so blocking here creates a circular wait + # broken only by KILL. On contention, leave the generation-bound claim for + # the stopper or subsequent reconciliation to reclaim. release_start_claim() { extension_lifecycle_lock_release 2>/dev/null || true [ -z "$STAGED_OUTPUT" ] || rm -f -- "$STAGED_OUTPUT" - fm_procevent_source_lock_acquire "$CLAIM_ID" 2>/dev/null || return 0 + fm_procevent_source_lock_try_acquire "$CLAIM_ID" 2>/dev/null || return 0 if fm_procevent_claim_load_locked "$CLAIM_ID" 2>/dev/null \ && [ "$FM_PROCEVENT_CLAIM_HOME" = "$CLAIM_HOME" ] \ && [ "$FM_PROCEVENT_CLAIM_PID" = "$CLAIM_PID" ] \ @@ -1097,19 +1101,25 @@ start_owner_guard() { # <source-id> # The runner's owner guard, which bounds an accidentally orphaned detached # runner after its home ends. It revalidates the recorded physical state root -# and its lease on a bounded cadence and, after two consecutive checks cannot prove +# and its lease on a bounded cadence and, after two consecutive reads cannot prove # both, invokes the identity-gated stop for the runner's whole process group - # which is what reaches the blocking child and everything that child spawned, # exactly as retirement does. A failed verified stop stays on the retry cadence; # an absent leader ends the guard without signalling an ambiguous group. # +# Those two reads are spaced HALF a check interval apart, so the pair completes +# within one check interval rather than costing two. That keeps the debounce - +# one unreadable read still cannot end a live runner - while bounding detection +# at the lease plus a single check interval. The spacing is what was tightened; +# the second read is what must not be traded away for it. +# # Scope is the owning state root and this one runner generation. It never # matches on a script name, a command line, or a process name: those are shared # by every home running the same adapter, and a live source in another home # proves its own owner through that home's own lease. cmd_owner_watchdog() { # <source-id> <runner-pid> <runner-identity> <ready-file> <state-device> <state-inode> local id=${1-} pid=${2-} identity=${3-} ready=${4-} state_device=${5-} state_inode=${6-} - local lease tick misses=0 pid_state state_identity current_device current_inode + local lease tick half misses=0 pid_state state_identity current_device current_inode [ "$#" -eq 6 ] || usage fm_procevent_source_id_valid "$id" || die "source id must be path-safe: $id" case "$pid" in ''|*[!0-9]*) die "runner pid must be a positive integer: $pid" ;; esac @@ -1124,6 +1134,16 @@ cmd_owner_watchdog() { # <source-id> <runner-pid> <runner-identity> <ready-file || die "FM_PROCEVENT_OWNER_LEASE_SECONDS must be whole seconds from $FM_PROCEVENT_OWNER_LEASE_MIN_SECONDS to $FM_PROCEVENT_OWNER_LEASE_MAX_SECONDS" tick=$(fm_procevent_owner_check_seconds) \ || die "FM_PROCEVENT_OWNER_CHECK_SECONDS must be whole seconds from $FM_PROCEVENT_OWNER_CHECK_MIN_SECONDS to $FM_PROCEVENT_OWNER_CHECK_MAX_SECONDS" + # Force base ten before any arithmetic. The validator accepts a zero-prefixed + # value and `[` reads it as decimal, but `$(( ))` would read it as octal: 010 + # would halve to 4 rather than 5, and 08 would not be a number at all and + # would end the guard before it reports ready, so the runner would fail closed + # and never listen. Every value the validator accepts must keep working. + tick=$((10#$tick)) + # Half the configured interval, kept exact for an odd interval so the smallest + # configurable interval still yields two reads rather than collapsing to one. + half=$((tick / 2)) + [ $((tick % 2)) -eq 0 ] || half="$half.5" fm_procevent_pid_state "$pid" "$identity" pid_state=$? [ "$pid_state" -eq 0 ] || die "runner identity changed before owner guard initialization" @@ -1137,7 +1157,7 @@ cmd_owner_watchdog() { # <source-id> <runner-pid> <runner-identity> <ready-file printf 'ready\n' > "$ready" || die "cannot confirm owner guard initialization" trap - EXIT while :; do - sleep "$tick" + sleep "$half" fm_procevent_pid_state "$pid" "$identity" pid_state=$? case "$pid_state" in @@ -1157,6 +1177,8 @@ cmd_owner_watchdog() { # <source-id> <runner-pid> <runner-identity> <ready-file continue fi # Two consecutive misses, so one unreadable read cannot end a live runner. + # They are half an interval apart, so requiring the second costs detection + # time inside the interval already budgeted rather than a second interval. misses=$((misses + 1)) [ "$misses" -ge 2 ] || continue if stop_runner_pid "$pid" "$identity"; then @@ -1270,20 +1292,35 @@ cmd_reconcile() { # its own process group leader, so the group signal is what actually reaches the # blocking child - signalling only the runner would leave that child alive and # reparented, which is exactly how a source that never completes leaks. -runner_group_signal() { # <signal> <pid> <identity> - local signal=$1 pid=$2 identity=$3 state pgid - # KNOWN LIMIT: only an alive identity-matched leader proves group ownership. - # Detected reused PIDs and absent leaders are refused before signalling; - # launch pacing, leases, and reconcile cleanup are the backstop. - fm_procevent_pid_state "$pid" "$identity" - state=$? - case "$state" in - 0) ;; - 1) fm_procevent_group_alive "$pid" && return 2; return 1 ;; - *) return 2 ;; - esac - pgid=$(ps -o pgid= -p "$pid" 2>/dev/null | tr -d '[:space:]') || return 2 - [ "$pgid" = "$pid" ] || return 2 +# docs/configuration.md owns the operating contract and unproved-group limits. +# A leaderless group nobody in this call ever proved remains refused for every +# caller, and that untouched refusal is what makes a crashed leader's group +# permanent. Relaxing it is a SEPARATE OPEN QUESTION, not something this path +# assumes: an unresolved question has to be marked unresolved where the decision +# is made, because a reader who does not know it is open will read a bare refusal +# as settled design and eventually relax it. +runner_group_signal() { # <signal> <pid> <identity> [proved] + local signal=$1 pid=$2 identity=$3 proved=${4-} state pgid + if [ -n "$proved" ]; then + # This stop proved ownership before TERM; only its own escalation may reuse + # that same proof within the same stop_runner_pid call. Re-reading the leader + # as our signal ends it would discard that proof, not disprove ownership. + # A group encountered without proof remains refused by the unproved path. + fm_procevent_group_alive "$pid" || return 1 + else + # Before the first signal, require a live identity-matched group leader: + # absent, unreadable, reused, or nonleader PIDs cannot prove ownership. + # Launch pacing, leases, and reconcile cleanup remain the backstop. + fm_procevent_pid_state "$pid" "$identity" + state=$? + case "$state" in + 0) ;; + 1) fm_procevent_group_alive "$pid" && return 2; return 1 ;; + *) return 2 ;; + esac + pgid=$(ps -o pgid= -p "$pid" 2>/dev/null | tr -d '[:space:]') || return 2 + [ "$pgid" = "$pid" ] || return 2 + fi # KNOWN LIMIT: portable shell cannot make this verification and signal atomic, # so the PID and group could be reused in the interval between them. kill -"$signal" -"$pid" 2>/dev/null || return 2 @@ -1301,7 +1338,7 @@ stop_runner_pid() { # <pid> <identity> sleep 0.1 i=$((i + 1)) done - runner_group_signal KILL "$pid" "$identity" + runner_group_signal KILL "$pid" "$identity" proved signal_state=$? [ "$signal_state" -eq 0 ] || return "$signal_state" i=0 diff --git a/bin/fm-remote-doctor.sh b/bin/fm-remote-doctor.sh index aad9ce44aa7..c6bea4de725 100755 --- a/bin/fm-remote-doctor.sh +++ b/bin/fm-remote-doctor.sh @@ -12,10 +12,19 @@ # A remote second mate always runs on the Herdr backend in the dedicated # fm-remote session. Its account therefore needs the Firstmate-owned Aqua Herdr # agent plus the sibling dev.firstmate.remote-job worker that runs normal fm-on -# commands through the Aqua or Linux job-worker path. Doctor remains invokable -# over the plain-SSH bootstrap path to inspect and repair that worker. SSH cannot -# create an Aqua session, so a host with no GUI login is a human gap rather than -# something --fix attempts to bypass. +# commands through the Aqua or Linux job-worker path. On darwin, that Herdr +# agent runs bin/fm-remote-herdr-guard.sh through the remote account's login +# shell (`-l -c`) so the server inherits the account's own environment; the +# gui/<uid> launchd domain it is bootstrapped into, not the shell, is what +# gives the server and its panes the Aqua audit session and login-keychain +# access. The guard execs the server in the foreground under launchd, leaves an +# Aqua-born server alone, and takes the session over from a server born +# outside that session (an SSH remote attach wins the socket at boot), because +# such a server's panes cannot read the login keychain; +# bin/fm-remote-herdr-owner-lib.sh owns that birth test. Doctor remains +# invokable over the plain-SSH bootstrap path to inspect and repair that worker. +# SSH cannot create an Aqua session, so a host with no GUI login is a human +# gap rather than something --fix attempts to bypass. # # Line protocol, one fact per line, stable for script consumers: # mode=check|fix @@ -56,6 +65,8 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(CDPATH='' cd "$SCRIPT_DIR/.." && pwd -P)}" . "$SCRIPT_DIR/fm-remote-job-lib.sh" # shellcheck source=bin/fm-tasks-axi-lib.sh . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" +# shellcheck source=bin/fm-remote-herdr-owner-lib.sh +. "$SCRIPT_DIR/fm-remote-herdr-owner-lib.sh" REQUIRED_TOOLS=(git jq herdr tasks-axi treehouse) HARNESS_TOOLS=(claude codex opencode pi pi-signed grok kimi) OPTIONAL_TOOLS=(tmux no-mistakes gh) @@ -149,14 +160,47 @@ herdr_adapter_load() { FM_REMOTE_DOCTOR_HERDR_LOADED=1 } +herdr_server_status_json() { + herdr_adapter_load || return 1 + fm_backend_herdr_cli "$HERDR_SESSION_NAME" status --json 2>/dev/null +} + herdr_server_running() { local running - herdr_adapter_load || return 1 - running=$(fm_backend_herdr_cli "$HERDR_SESSION_NAME" status --json 2>/dev/null \ - | jq -r '.server.running // false' 2>/dev/null) || return 1 + running=$(herdr_server_status_json | jq -r '.server.running // false' 2>/dev/null) || return 1 [ "$running" = true ] } +# Birth of the process serving the session, as the guard classifies it: +# prints "<birth> <pid>" (launchd, worker, ssh, or unknown), "unproven" when +# no herdr process can be shown to hold the socket, or "nolsof" when lsof does +# not resolve. bin/fm-remote-herdr-owner-lib.sh owns the markers. +herdr_server_birth() { + local socket owner rc birth + socket=$(herdr_server_status_json | jq -r '.server.socket // empty' 2>/dev/null) || socket= + owner=$(fm_remote_herdr_socket_owner "$socket"); rc=$? + if [ "$rc" -eq 2 ]; then + printf 'nolsof\n' + return 0 + fi + if [ -z "$owner" ]; then + printf 'unproven\n' + return 0 + fi + birth=$(fm_remote_herdr_owner_birth "$owner") + printf '%s %s\n' "$birth" "$owner" +} + +# On darwin the session is ready only when its server was born in the Aqua +# login session; elsewhere any running server is. +herdr_server_aqua_owned() { + local birth + herdr_server_running || return 1 + [ "$PLATFORM" = darwin ] || return 0 + birth=$(herdr_server_birth) + fm_remote_herdr_birth_is_aqua "${birth%% *}" +} + launch_agent_is_aqua() { local stripped [ -f "$LAUNCH_AGENT_PLIST" ] && [ ! -L "$LAUNCH_AGENT_PLIST" ] || return 1 @@ -167,8 +211,67 @@ launch_agent_is_aqua() { return 1 } -render_launch_agent() { # <resolved-herdr-path> - local herdr_bin=$1 +launch_agent_shell_quote() { # <value> + printf "'%s'" "$(printf '%s' "$1" | sed "s/'/'\\\\''/g")" +} + +launch_agent_xml_escape() { # <value> + printf '%s' "$1" | sed 's/&/\&/g; s/</\</g; s/>/\>/g' +} + +# Directory Services UserShell is the account's real login shell on darwin +# (bash, fish, zsh, ...). Fall back without failing the render: $SHELL, then +# /bin/sh. Separate -l and -c so fish accepts the flags. +resolve_launch_agent_shell() { + local user raw shell + if [ -n "${FM_LAUNCH_AGENT_SHELL:-}" ] && [ -x "$FM_LAUNCH_AGENT_SHELL" ]; then + printf '%s' "$FM_LAUNCH_AGENT_SHELL" + return 0 + fi + user=$(id -un 2>/dev/null || true) + if [ -n "$user" ] && command -v dscl >/dev/null 2>&1 && command -v perl >/dev/null 2>&1; then + raw=$(perl -e '$SIG{ALRM} = sub { exit 124 }; alarm 2; exec @ARGV' \ + dscl . -read "/Users/$user" UserShell 2>/dev/null || true) + shell=$(printf '%s\n' "$raw" | awk ' + /^UserShell:[[:space:]]+/ { + sub(/^UserShell:[[:space:]]+/, "") + if (length) { print; exit } + } + ') + if [ -n "$shell" ] && [ -x "$shell" ]; then + printf '%s' "$shell" + return 0 + fi + fi + if [ -n "${SHELL:-}" ] && [ -x "$SHELL" ]; then + printf '%s' "$SHELL" + return 0 + fi + printf '%s' /bin/sh +} + +# Login-shell command that execs the Firstmate-owned guard, which in turn execs +# the resolved herdr so launchd keeps one foreground process in the Aqua +# session, or exits 0 when an Aqua-born server already owns the session. +# KeepAlive={SuccessfulExit=false} is load-bearing for that exit: an +# unconditional KeepAlive would respawn the job every throttle interval +# forever while a foreign server holds the socket, exactly the loop this guard +# replaces, and would never let the guard's "nothing to do" verdict rest. +launch_agent_guard_path() { + printf '%s/bin/fm-remote-herdr-guard.sh' "$FM_ROOT" +} + +launch_agent_exec_command() { # <resolved-herdr-path> + printf 'exec %s %s %s' \ + "$(launch_agent_shell_quote "$(launch_agent_guard_path)")" \ + "$(launch_agent_shell_quote "$1")" \ + "$(launch_agent_shell_quote "$HERDR_SESSION_NAME")" +} + +render_launch_agent() { # <resolved-herdr-path> <resolved-login-shell> + local herdr_bin=$1 shell=$2 exec_cmd shell_xml + shell_xml=$(launch_agent_xml_escape "$shell") + exec_cmd=$(launch_agent_exec_command "$herdr_bin") cat <<XML <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> @@ -178,17 +281,22 @@ render_launch_agent() { # <resolved-herdr-path> <string>$LAUNCH_AGENT_LABEL</string> <key>ProgramArguments</key> <array> - <string>$herdr_bin</string> - <string>server</string> - <string>--session</string> - <string>$HERDR_SESSION_NAME</string> + <string>$shell_xml</string> + <string>-l</string> + <string>-c</string> + <string>$exec_cmd</string> </array> <key>LimitLoadToSessionType</key> <string>Aqua</string> <key>RunAtLoad</key> <true/> <key>KeepAlive</key> - <true/> + <dict> + <key>SuccessfulExit</key> + <false/> + </dict> + <key>ThrottleInterval</key> + <integer>10</integer> <key>StandardOutPath</key> <string>$LAUNCH_AGENT_LOG</string> <key>StandardErrorPath</key> @@ -198,30 +306,34 @@ render_launch_agent() { # <resolved-herdr-path> XML } -launch_agent_contract_matches() { - local herdr_bin actual expected +launch_agent_contract_matches() { # <resolved-login-shell> + local shell=$1 herdr_bin actual expected [ -f "$LAUNCH_AGENT_PLIST" ] && [ ! -L "$LAUNCH_AGENT_PLIST" ] || return 1 herdr_bin=$(command -v herdr 2>/dev/null) || return 1 actual=$(tr -d ' \t\r\n' < "$LAUNCH_AGENT_PLIST" 2>/dev/null) || return 1 - expected=$(render_launch_agent "$herdr_bin" | tr -d ' \t\r\n') || return 1 + expected=$(render_launch_agent "$herdr_bin" "$shell" | tr -d ' \t\r\n') || return 1 [ "$actual" = "$expected" ] } -launch_agent_loaded_contract_matches() { - local loaded herdr_bin herdr_compact plist_compact log_compact args +launch_agent_loaded_contract_matches() { # <resolved-login-shell> + local shell=$1 loaded herdr_bin exec_compact shell_compact plist_compact log_compact args herdr_bin=$(command -v herdr 2>/dev/null) || return 1 loaded=$(launchctl print "gui/$UID_NUM/$LAUNCH_AGENT_LABEL" 2>/dev/null) || return 1 loaded=$(printf '%s' "$loaded" | tr -d ' \t\r\n') || return 1 - herdr_compact=$(printf '%s' "$herdr_bin" | tr -d ' \t\r\n') || return 1 + exec_compact=$(launch_agent_exec_command "$herdr_bin" | tr -d ' \t\r\n') || return 1 + shell_compact=$(printf '%s' "$shell" | tr -d ' \t\r\n') || return 1 plist_compact=$(printf '%s' "$LAUNCH_AGENT_PLIST" | tr -d ' \t\r\n') || return 1 log_compact=$(printf '%s' "$LAUNCH_AGENT_LOG" | tr -d ' \t\r\n') || return 1 - args="arguments={$herdr_compact"'server--session'"$HERDR_SESSION_NAME}" + args="arguments={${shell_compact}-l-c${exec_compact}}" [[ "$loaded" == *"path=$plist_compact"* ]] || return 1 - [[ "$loaded" == *"program=$herdr_compact"* ]] || return 1 + [[ "$loaded" == *"program=$shell_compact"* ]] || return 1 [[ "$loaded" == *"$args"* ]] || return 1 [[ "$loaded" == *"stdoutpath=$log_compact"* ]] || return 1 [[ "$loaded" == *"stderrpath=$log_compact"* ]] || return 1 - [[ "$loaded" == *'properties=keepalive|runatload'* ]] || return 1 + # launchd renders KeepAlive={SuccessfulExit=false} as a successful-exit + # semaphore rather than a keepalive property. + [[ "$loaded" == *'successfulexit=>0'* ]] || return 1 + [[ "$loaded" == *'properties=runatload'* ]] || return 1 } # --- remote job and tool checks --------------------------------------------- @@ -451,8 +563,16 @@ fix_remote_job_worker() { # --- checks ----------------------------------------------------------------- check_herdr() { - local resolved + local resolved selected if resolved=$(command -v herdr 2>/dev/null) && [ -x "$resolved" ]; then + if herdr_adapter_load; then + fm_backend_herdr_client_select "$HERDR_SESSION_NAME" + selected=$(fm_backend_herdr_bin) + if [ "$selected" != herdr ] && [ "$selected" != "$resolved" ]; then + record herdr "ok: $selected (bypassing $resolved)" + return 0 + fi + fi record herdr "ok: $resolved" return 0 fi @@ -483,7 +603,8 @@ check_gui_session() { "log that account in once at the console, and enable automatic login in System Settings > Users & Groups if the machine runs headless; SSH cannot create a GUI session, and Firstmate never writes an auto-login password or changes FileVault" } -check_launch_agent() { +check_launch_agent() { # <resolved-login-shell> + local shell=$1 if [ "$PLATFORM" != darwin ]; then record launchagent "skip: launch agents apply only on darwin" record launchagent-scope "skip: launch agents apply only on darwin" @@ -491,7 +612,7 @@ check_launch_agent() { return 0 fi if [ -f "$LAUNCH_AGENT_PLIST" ] && [ ! -L "$LAUNCH_AGENT_PLIST" ]; then - if launch_agent_contract_matches; then + if launch_agent_contract_matches "$shell"; then record launchagent "ok: $LAUNCH_AGENT_PLIST matches the Firstmate-owned contract" else record launchagent "fixable: $LAUNCH_AGENT_PLIST does not match the current Firstmate-owned contract" \ @@ -508,17 +629,18 @@ check_launch_agent() { "rerun this command with --fix to install it" record launchagent-scope "skip: no launch agent is installed yet" fi - check_launch_agent_loaded + check_launch_agent_loaded "$shell" } -check_launch_agent_loaded() { +check_launch_agent_loaded() { # <resolved-login-shell> + local shell=$1 if [ -z "$UID_NUM" ] || ! command -v launchctl >/dev/null 2>&1; then record launchagent-loaded "human: the launch agent domain gui/<uid> cannot be inspected on this account" \ "restore launchctl and a readable account uid, then rerun this command" return 0 fi if launchctl print "gui/$UID_NUM/$LAUNCH_AGENT_LABEL" >/dev/null 2>&1; then - if launch_agent_loaded_contract_matches; then + if launch_agent_loaded_contract_matches "$shell"; then record launchagent-loaded "ok: gui/$UID_NUM/$LAUNCH_AGENT_LABEL matches the effective contract" else record launchagent-loaded "fixable: gui/$UID_NUM/$LAUNCH_AGENT_LABEL does not match the effective Firstmate-owned contract" \ @@ -542,7 +664,29 @@ check_herdr_server() { return 0 fi if herdr_server_running; then - record herdr-server "ok: session $HERDR_SESSION_NAME is running" + if [ "$PLATFORM" != darwin ]; then + record herdr-server "ok: session $HERDR_SESSION_NAME is running" + return 0 + fi + local birth + birth=$(herdr_server_birth) + case "$birth" in + launchd\ *|worker\ *) + record herdr-server "ok: session $HERDR_SESSION_NAME is running in the Aqua login session (pid ${birth#* }, ${birth%% *})" + ;; + nolsof) + record herdr-server "human: session $HERDR_SESSION_NAME is running but lsof does not resolve, so its server's birth cannot be proven" \ + "install lsof on that account so the launch agent and this check can tell an Aqua-born server from one started over SSH" + ;; + unproven) + record herdr-server "fixable: session $HERDR_SESSION_NAME is running but no herdr process can be shown to own its socket, so its birth cannot be proven" \ + "rerun this command with --fix so the launch agent takes the session over (its current panes close and the parent firstmate relaunches its mates)" + ;; + *) + record herdr-server "fixable: session $HERDR_SESSION_NAME is served by pid ${birth#* } born outside the Aqua login session (${birth%% *}), so its panes cannot reach the login keychain" \ + "rerun this command with --fix so the launch agent takes the session over (its current panes close and the parent firstmate relaunches its mates)" + ;; + esac return 0 fi if [ "$PLATFORM" = darwin ] && ! check_is_ok gui-session; then @@ -574,14 +718,15 @@ check_entrypoint_link() { "rerun this command with --fix to create it" } -run_checks() { +run_checks() { # <resolved-login-shell> + local shell=$1 CHECK_NAMES=() CHECK_VALUES=() CHECK_ACTIONS=() check_herdr check_gui_session check_remote_job_worker - check_launch_agent + check_launch_agent "$shell" check_herdr_server check_entrypoint_link } @@ -592,8 +737,8 @@ fix_report() { # <check> applied|failed <text> printf 'fix %s=%s: %s\n' "$1" "$2" "$3" } -write_launch_agent() { - local herdr_bin tmp +write_launch_agent() { # <resolved-login-shell> + local shell=$1 herdr_bin tmp if ! herdr_bin=$(command -v herdr 2>/dev/null); then fix_report launchagent failed "herdr does not resolve, so no launch agent was written" return 1 @@ -610,14 +755,14 @@ write_launch_agent() { fi mkdir -p "$LAUNCH_AGENT_LOG_DIR" 2>/dev/null || true tmp="$LAUNCH_AGENT_DIR/.$LAUNCH_AGENT_LABEL.plist.tmp.$$" - render_launch_agent "$herdr_bin" > "$tmp" + render_launch_agent "$herdr_bin" "$shell" > "$tmp" chmod 0644 "$tmp" 2>/dev/null || true if ! mv -f -- "$tmp" "$LAUNCH_AGENT_PLIST" 2>/dev/null; then rm -f -- "$tmp" fix_report launchagent failed "cannot publish $LAUNCH_AGENT_PLIST" return 1 fi - fix_report launchagent applied "wrote the Aqua-scoped $LAUNCH_AGENT_LABEL launch agent running $herdr_bin server" + fix_report launchagent applied "wrote the Aqua-scoped $LAUNCH_AGENT_LABEL launch agent running $(launch_agent_guard_path) for $herdr_bin via $shell -l -c" } # Reload rather than plain bootstrap so a rewritten plist replaces a stale @@ -643,7 +788,7 @@ reload_launch_agent() { # <check-to-report-under> return 1 fi if ! wait_for_herdr_server; then - fix_report "$report" failed "the herdr server for session $HERDR_SESSION_NAME did not report running within 10s" + fix_report "$report" failed "the herdr server for session $HERDR_SESSION_NAME did not come up inside the Aqua launch agent within 10s" return 1 fi fix_report "$report" applied "bootstrapped and started $LAUNCH_AGENT_LABEL in gui/$UID_NUM" @@ -652,7 +797,7 @@ reload_launch_agent() { # <check-to-report-under> wait_for_herdr_server() { local i=0 while [ "$i" -lt 20 ]; do - herdr_server_running && return 0 + herdr_server_aqua_owned && return 0 i=$((i + 1)) sleep 0.5 done @@ -685,8 +830,8 @@ link_entrypoint() { fix_report entrypoint-link applied "linked $ENTRYPOINT_LINK to $want" } -apply_fixes() { - local i name value launch_agent_written=0 launch_agent_reloaded=0 remote_job_fixed=0 +apply_fixes() { # <resolved-login-shell> + local shell=$1 i name value launch_agent_written=0 launch_agent_reloaded=0 remote_job_fixed=0 repair_required_wrappers i=0 while [ "$i" -lt "${#CHECK_NAMES[@]}" ]; do @@ -703,7 +848,7 @@ apply_fixes() { launchagent|launchagent-scope) [ "$launch_agent_written" -eq 0 ] || continue launch_agent_written=1 - write_launch_agent || continue + write_launch_agent "$shell" || continue # A freshly written plist runs nothing until it is (re)loaded, and only # an existing GUI session can hold it. check_is_ok gui-session || continue @@ -750,12 +895,16 @@ else fi printf 'platform=%s\n' "$PLATFORM" -run_checks +LAUNCH_AGENT_SHELL= +if [ "$PLATFORM" = darwin ]; then + LAUNCH_AGENT_SHELL=$(resolve_launch_agent_shell) +fi +run_checks "$LAUNCH_AGENT_SHELL" if [ "$MODE" = fix ]; then - apply_fixes + apply_fixes "$LAUNCH_AGENT_SHELL" # Re-derive every check from the host itself, so what prints below is the # state after repair rather than the intent of a repair. - run_checks + run_checks "$LAUNCH_AGENT_SHELL" fi if [ "${FM_REMOTE_JOB_ACTIVE:-}" = 1 ] || ! remote_job_identity_ok; then diff --git a/bin/fm-remote-entrypoint.sh b/bin/fm-remote-entrypoint.sh index 77da2bbe171..25d9c0b6e90 100755 --- a/bin/fm-remote-entrypoint.sh +++ b/bin/fm-remote-entrypoint.sh @@ -31,7 +31,7 @@ set -eu PROTOCOL=1 -DOCTOR_SHA256=7bb13d9fad8455978bf109d4681a3aa3cb170565c8a74be4ec7b520427db14c2 +DOCTOR_SHA256=78efccd6cb7a0123400e49fa323292a64c8e3c7ebd3717151be69f87735302fb REAL_SOURCE=$(python3 -c 'import os, sys; print(os.path.realpath(sys.argv[1]))' "${BASH_SOURCE[0]}" 2>/dev/null) || REAL_SOURCE=$(realpath "${BASH_SOURCE[0]}" 2>/dev/null) || REAL_SOURCE=${BASH_SOURCE[0]} diff --git a/bin/fm-remote-herdr-guard.sh b/bin/fm-remote-herdr-guard.sh new file mode 100755 index 00000000000..46919ae49d4 --- /dev/null +++ b/bin/fm-remote-herdr-guard.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# launchd exec target for the Firstmate-owned dev.firstmate.herdr.fm-remote +# launch agent: make the Aqua login session own the fm-remote Herdr server. +# +# Usage: +# fm-remote-herdr-guard.sh <herdr-path> <session> +# +# bin/fm-remote-doctor.sh renders the launch agent as the account's login +# shell running `exec <this script> <herdr> fm-remote` with +# LimitLoadToSessionType=Aqua, RunAtLoad, KeepAlive={SuccessfulExit=false}, +# and ThrottleInterval=10, then bootstraps it into gui/<uid>. That domain, not +# the login shell, is what gives this process and every server it execs the +# Aqua audit session and login-keychain access; the login shell only gives the +# server the account's own environment. +# `herdr server` stays in the foreground under launchd, as verified in +# docs/verification/runtime-backends.md under "fm-remote server birth and login-keychain access", so the final exec provides the complete supervision lifecycle. +# +# Decision, made once per launch (exit codes matter under SuccessfulExit=false: +# 0 tells launchd the job is done until something restarts it, non-zero asks +# for a retry after the throttle interval): +# no server owns the session socket -> exec `herdr server --session <s>` +# (foreground, launchd-supervised) +# the owner was born in the Aqua session (launchd or the Aqua remote-job +# worker) -> exit 0, leave it alone +# the owner was born anywhere else (an SSH remote attach, a shell over +# ssh/mosh, or a birth it cannot prove) -> `herdr server stop`, wait until the +# socket is released, then exec +# `herdr server --session <s>` at once +# so the socket is rebound before a +# reconnecting SSH attach can start +# another foreign server +# the foreign server does not release the socket in time -> exit 1 +# A takeover closes every pane in that session; the parent firstmate's +# secondmate liveness sweep relaunches its mates into the Aqua-born server. +# bin/fm-remote-herdr-owner-lib.sh owns the owner discovery and the birth +# markers; FM_REMOTE_HERDR_GUARD_STOP_WAIT_TENTHS (default 50) bounds the +# release wait in tenths of a second. Every decision prints one line to +# stdout, which launchd routes to the agent's log. +set -u + +SCRIPT_SELF=${BASH_SOURCE[0]} +SCRIPT_DIR=${SCRIPT_SELF%/*} +[ "$SCRIPT_DIR" != "$SCRIPT_SELF" ] || SCRIPT_DIR=. +SCRIPT_DIR=$(CDPATH='' cd -- "$SCRIPT_DIR" && pwd -P) +# shellcheck source=bin/fm-remote-herdr-owner-lib.sh +. "$SCRIPT_DIR/fm-remote-herdr-owner-lib.sh" + +usage() { sed -n '2,6p' "$0" | sed 's/^# \{0,1\}//'; exit 2; } +[ "$#" -eq 2 ] || usage +HERDR_BIN=$1 +SESSION=$2 +[ -n "$HERDR_BIN" ] && [ -x "$HERDR_BIN" ] || { printf 'fm-remote-herdr-guard: herdr is not executable: %s\n' "$HERDR_BIN" >&2; exit 1; } +[ -n "$SESSION" ] || usage +command -v jq >/dev/null 2>&1 || { printf 'fm-remote-herdr-guard: jq does not resolve on the launch agent PATH\n' >&2; exit 1; } +STOP_WAIT_TENTHS=${FM_REMOTE_HERDR_GUARD_STOP_WAIT_TENTHS:-50} + +log() { printf 'fm-remote-herdr-guard: %s\n' "$*"; } + +herdr_status() { # prints the session's status JSON, empty when herdr fails + HERDR_SESSION="$SESSION" "$HERDR_BIN" status --json --session "$SESSION" 2>/dev/null || true +} + +status_running() { # <status-json> + [ "$(printf '%s' "$1" | jq -r '.server.running // false' 2>/dev/null)" = true ] +} + +start_server() { + log "starting the herdr server for session $SESSION inside this launch agent (pid $$)" + exec "$HERDR_BIN" server --session "$SESSION" +} + +STATUS=$(herdr_status) +if ! status_running "$STATUS"; then + log "no server owns session $SESSION" + start_server +fi + +SOCKET=$(printf '%s' "$STATUS" | jq -r '.server.socket // empty' 2>/dev/null) +OWNER=$(fm_remote_herdr_socket_owner "$SOCKET"); OWNER_RC=$? +if [ "$OWNER_RC" -eq 2 ]; then + log "session $SESSION is running but lsof does not resolve, so its server's birth cannot be proven" + BIRTH=unknown +elif [ -z "$OWNER" ]; then + log "session $SESSION is running but no herdr process could be proven to own ${SOCKET:-its socket}" + BIRTH=unknown +else + BIRTH=$(fm_remote_herdr_owner_birth "$OWNER") +fi + +if fm_remote_herdr_birth_is_aqua "$BIRTH"; then + log "session $SESSION is served by pid $OWNER born in the Aqua login session ($BIRTH); nothing to do" + exit 0 +fi + +log "session $SESSION is served by ${OWNER:+pid }${OWNER:-an unproven process} born outside the Aqua login session ($BIRTH); its panes cannot reach the login keychain, taking the session over" +HERDR_SESSION="$SESSION" "$HERDR_BIN" server stop --session "$SESSION" >/dev/null 2>&1 \ + || log "herdr server stop for session $SESSION did not succeed; waiting for the socket anyway" +i=0 +while [ "$i" -lt "$STOP_WAIT_TENTHS" ]; do + if ! status_running "$(herdr_status)"; then + log "session $SESSION released its socket after $i tenths of a second" + start_server + fi + sleep 0.1 + i=$((i + 1)) +done +log "the foreign server for session $SESSION did not release its socket within $STOP_WAIT_TENTHS tenths of a second; exiting 1 so launchd retries" +exit 1 diff --git a/bin/fm-remote-herdr-owner-lib.sh b/bin/fm-remote-herdr-owner-lib.sh new file mode 100755 index 00000000000..dceaf5a051f --- /dev/null +++ b/bin/fm-remote-herdr-owner-lib.sh @@ -0,0 +1,176 @@ +#!/usr/bin/env bash +# Who owns a Herdr session socket, and was that process born in the Aqua +# login session? +# +# Source this file; it defines functions only. It is the single owner of the +# socket-owner discovery and birth classification shared by +# bin/fm-remote-herdr-guard.sh (the launch agent's exec target) and +# bin/fm-remote-doctor.sh (the readiness check for that session). +# +# Why birth matters: a herdr server, and every pane and agent it later spawns, +# keeps the macOS audit session of whatever started it. Only the Aqua login +# session (the gui/<uid> launchd domain) can read the login keychain without a +# UI prompt. A server started over SSH - herdr's own remote attach does this +# when it finds no server, and it wins the socket at boot because sshd accepts +# connections before the login session exists - runs in sshd's audit session, +# where `security find-generic-password -w` exits 36 (interaction not allowed) +# and every claude pane silently falls back to a stale plaintext credentials +# file and reports "Login expired". docs/verification/runtime-backends.md +# ("fm-remote server birth and login-keychain access") holds the dated +# evidence for every marker read here. +# +# Functions: +# fm_remote_herdr_socket_owner <socket-path> +# Prints the pid of the herdr process that holds <socket-path>, or nothing +# when no herdr process does. Reads `lsof -U -a -c herdr -F pn`; on macOS +# `pgrep -f` cannot see the herdr server's argv, so lsof is the owner +# source. When several herdr processes list the path, the one whose argv +# runs `server` wins. Returns 2, printing nothing, when lsof does not +# resolve; the caller decides what an unprovable owner means. +# fm_remote_herdr_process_env <pid> +# Prints the process environment as NAME=VALUE lines: `ps -Eww` on darwin +# (own-uid processes only, and macOS hides the environment of Apple +# platform binaries such as /bin/sleep even from the same user; a herdr +# server is never one), /proc/<pid>/environ elsewhere. +# fm_remote_herdr_process_ancestry <pid> +# Prints "<pid> <command>" for <pid> and each ancestor up to pid 1. +# fm_remote_herdr_owner_birth <pid> +# Prints exactly one word, the strongest marker present: +# ssh SSH_CONNECTION, SSH_CLIENT, or SSH_TTY in the environment, or +# an ancestor that is sshd or herdr's remote-client-bridge +# (matched on argv[0] and whole arguments only) +# launchd XPC_SERVICE_NAME=<label>, with launchctl proving that job is +# the owner in gui/<uid> or is loaded only in that domain +# worker FM_REMOTE_JOB_ACTIVE=1, with launchctl proving that +# dev.firstmate.remote-job is loaded only in gui/<uid> +# unknown none of the above; XPC_SERVICE_NAME alone, including value 0, +# does not prove an Aqua birth +# fm_remote_herdr_birth_is_aqua <birth> +# Succeeds only for launchd and worker. `unknown` is deliberately not +# Aqua: a server that cannot prove its birth is treated like a foreign one, +# because leaving it in place silently reproduces the keychain failure. + +fm_remote_herdr_socket_owner() { # <socket-path> + local socket=$1 real pid='' line candidates='' candidate cmd + [ -n "$socket" ] || return 1 + command -v lsof >/dev/null 2>&1 || return 2 + real=$(CDPATH='' cd -- "$(dirname "$socket")" 2>/dev/null && printf '%s/%s' "$(pwd -P)" "$(basename "$socket")") || real=$socket + while IFS= read -r line; do + case "$line" in + p*) pid=${line#p} ;; + n*) + [ -n "$pid" ] || continue + case "${line#n}" in + "$socket"|"$real") candidates="${candidates}${pid}"$'\n' ;; + esac + ;; + esac + done < <(lsof -U -a -c herdr -F pn 2>/dev/null) + [ -n "$candidates" ] || return 0 + while IFS= read -r candidate; do + [ -n "$candidate" ] || continue + cmd=$(ps -o command= -p "$candidate" 2>/dev/null || true) + case " $cmd " in *' server '*) printf '%s\n' "$candidate"; return 0 ;; esac + done <<EOF2 +$candidates +EOF2 + printf '%s\n' "${candidates%%$'\n'*}" +} + +fm_remote_herdr_process_env() { # <pid> + local pid=$1 + case "$pid" in ''|*[!0-9]*) return 1 ;; esac + if [ -r "/proc/$pid/environ" ]; then + tr '\0' '\n' < "/proc/$pid/environ" + return 0 + fi + ps -Eww -o command= -p "$pid" 2>/dev/null | tr ' ' '\n' | grep -E '^[A-Za-z_][A-Za-z0-9_]*=' || true +} + +fm_remote_herdr_process_ancestry() { # <pid> + local pid=$1 depth=0 line ppid + while [ "$depth" -lt 64 ]; do + case "$pid" in ''|*[!0-9]*) return 0 ;; esac + [ "$pid" -gt 0 ] || return 0 + line=$(ps -o ppid=,command= -p "$pid" 2>/dev/null) || return 0 + [ -n "$line" ] || return 0 + ppid=$(printf '%s' "$line" | awk '{print $1}') + printf '%s %s\n' "$pid" "$(printf '%s' "$line" | sed 's/^[[:space:]]*[0-9]*[[:space:]]*//')" + [ "$pid" -ne 1 ] || return 0 + pid=$ppid + depth=$((depth + 1)) + done +} + +fm_remote_herdr_gui_job_proves_owner() { # <uid> <label> <pid> + local uid=$1 label=$2 pid=$3 job + [ -n "$label" ] && [ "$label" != 0 ] || return 1 + job=$(launchctl print "gui/$uid/$label" 2>/dev/null) || return 1 + if printf '%s\n' "$job" | awk -v expected="$pid" ' + $1 == "pid" && $2 == "=" && $3 == expected { found = 1 } + END { exit found ? 0 : 1 } + '; then + return 0 + fi + ! launchctl print "user/$uid/$label" >/dev/null 2>&1 +} + +fm_remote_herdr_gui_job_is_exclusive() { # <uid> <label> + local uid=$1 label=$2 + launchctl print "gui/$uid/$label" >/dev/null 2>&1 \ + && ! launchctl print "user/$uid/$label" >/dev/null 2>&1 +} + +fm_remote_herdr_owner_birth() { # <pid> + local pid=$1 env uid xpc_line label + env=$(fm_remote_herdr_process_env "$pid") || { printf 'unknown\n'; return 0; } + if printf '%s\n' "$env" | grep -q -E '^SSH_(CONNECTION|CLIENT|TTY)='; then + printf 'ssh\n' + return 0 + fi + uid=$(id -u 2>/dev/null) || uid= + xpc_line=$(printf '%s\n' "$env" | grep -E '^XPC_SERVICE_NAME=' | head -1 || true) + label=${xpc_line#XPC_SERVICE_NAME=} + if [ -n "$uid" ] && [ -n "$xpc_line" ] \ + && fm_remote_herdr_gui_job_proves_owner "$uid" "$label" "$pid"; then + printf 'launchd\n' + return 0 + fi + if printf '%s\n' "$env" | grep -q -E '^FM_REMOTE_JOB_ACTIVE=1$' \ + && [ -n "$uid" ] \ + && fm_remote_herdr_gui_job_is_exclusive "$uid" dev.firstmate.remote-job; then + printf 'worker\n' + return 0 + fi + if fm_remote_herdr_process_ancestry "$pid" | fm_remote_herdr_ancestry_has_ssh_origin; then + printf 'ssh\n' + return 0 + fi + printf 'unknown\n' +} + +# Reads "<pid> <command>" ancestry lines on stdin and succeeds when one of +# them IS sshd (argv[0] sshd or sshd-session, including the "sshd-session: +# user@notty" process title) or IS herdr's SSH remote attach (argv[0] herdr +# with the whole-word argument remote-client-bridge). Only argv[0] and whole +# arguments are matched: an ancestor whose free-text arguments merely mention +# those words, such as an agent carrying a brief, must not count. +fm_remote_herdr_ancestry_has_ssh_origin() { + awk ' + { + argv0 = $2 + sub(/.*\//, "", argv0) + sub(/:$/, "", argv0) + if (argv0 == "sshd" || argv0 == "sshd-session") { found = 1; exit } + if (argv0 == "herdr") { + for (i = 3; i <= NF; i++) if ($i == "remote-client-bridge") { found = 1; exit } + } + } + END { exit found ? 0 : 1 } + ' +} + +fm_remote_herdr_birth_is_aqua() { # <birth> + case "$1" in launchd|worker) return 0 ;; esac + return 1 +} diff --git a/bin/fm-remote-secondmate-control.sh b/bin/fm-remote-secondmate-control.sh index d6788fa3ca9..16e44a6d4ba 100755 --- a/bin/fm-remote-secondmate-control.sh +++ b/bin/fm-remote-secondmate-control.sh @@ -162,7 +162,10 @@ cmd_launch() { claude|codex|opencode|pi|pi-signed|grok|kimi|cursor) ;; *) die "unverified remote secondmate harness: $harness" ;; esac - case "$effort" in -|low|medium|high|xhigh|max) ;; *) die "invalid remote secondmate effort: $effort" ;; esac + case "$effort" in -|low|medium|high|xhigh|max|ultra) ;; *) die "invalid remote secondmate effort: $effort" ;; esac + if [ "$effort" = ultra ]; then + "$SCRIPT_DIR/fm-harness.sh" validate-native-effort "$harness" "$model" "$effort" || return 1 + fi # Herdr is required on this host, not merely preferred: its server belongs to # the GUI login session, so the endpoint survives every SSH disconnection that # a remote route depends on. bin/fm-remote-doctor.sh is the readiness owner. @@ -231,8 +234,11 @@ cmd_relaunch() { claude|codex|opencode|pi|pi-signed|grok|kimi|cursor) ;; *) die "unverified remote secondmate harness: $harness" ;; esac - case "$effort" in -|default|low|medium|high|xhigh|max) ;; *) die "invalid remote secondmate effort: $effort" ;; esac + case "$effort" in -|default|low|medium|high|xhigh|max|ultra) ;; *) die "invalid remote secondmate effort: $effort" ;; esac case "$model" in *[[:space:]]*) die "invalid remote secondmate model: $model" ;; esac + if [ "$effort" = ultra ]; then + "$SCRIPT_DIR/fm-harness.sh" validate-native-effort "$harness" "$model" "$effort" || return 1 + fi remote_endpoint_require "$id" [ "$model" != - ] || model=default [ "$effort" != - ] || effort=default diff --git a/bin/fm-secondmate-restart.sh b/bin/fm-secondmate-restart.sh index 9f3b90ceffa..be720ea45fc 100755 --- a/bin/fm-secondmate-restart.sh +++ b/bin/fm-secondmate-restart.sh @@ -276,9 +276,14 @@ while [ "$i" -lt "${#IDS[@]}" ]; do MODEL[i]=$("$SCRIPT_DIR/fm-harness.sh" secondmate-model 2>/dev/null || true) EFFORT[i]=$("$SCRIPT_DIR/fm-harness.sh" secondmate-effort 2>/dev/null || true) case "${EFFORT[i]}" in - ''|low|medium|high|xhigh|max) ;; + ''|low|medium|high|xhigh|max|ultra) ;; *) EFFORT[i]="" ;; esac + if [ "${EFFORT[i]}" = ultra ] && ! "$SCRIPT_DIR/fm-harness.sh" validate-native-effort "${HARNESS[i]}" "${MODEL[i]}" "${EFFORT[i]}"; then + REASON[i]="the configured Ultra profile does not select native Codex through Pi" + i=$((i + 1)) + continue + fi fi if ! corr=$(fm_pending_reply_create "$FM_HOME" "$STATE" "$id" \ diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index 79f8cd115f3..996fb34563f 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -47,7 +47,9 @@ # represented by the two digests below. # 6. fleet digest - a compact data/backlog.md identity/metadata listing, # every state/*.meta, a bounded state/*.status tail, -# state/.afk, and a cheap per-task endpoint-liveness read: +# the away posture (state/.afk-contract and the legacy +# state/.afk daemon flag), and a cheap per-task +# endpoint-liveness read: # read-only, always runs. # 7. network checks - the result of the deferred network stage started back at # step 1, harvested WITHOUT waiting for it. @@ -106,10 +108,10 @@ # # Why lock first: the old documented order (bootstrap, THEN lock) let a # SECOND concurrent session run bootstrap's mutating sweeps - converging -# secondmate homes, retrying pending handoff outboxes, writing X-mode artifacts, -# and fetching or fast-forwarding every project clone - before ever discovering -# another session already holds the lock. Two sessions racing those sweeps is -# exactly the hazard the lock exists to prevent, so locking first closes the +# secondmate homes, retrying pending handoff outboxes and receiver wakes, writing +# X-mode artifacts, and fetching or fast-forwarding every project clone - before +# ever discovering another session already holds the lock. Two sessions racing +# those sweeps is exactly the hazard the lock exists to prevent, so locking first closes the # hole outright: only the session that actually wins the lock ever touches # shared mutable state. # @@ -873,8 +875,18 @@ done [ "$ORPHAN_STATUS_FOUND" -eq 1 ] || printf '(none)\n' subsection "AFK" -if [ -e "$STATE/.afk" ]; then - printf 'present - away-mode supervision is active; the daemon owns the watcher.\n' +# The away posture is the record (bin/fm-afk-contract.sh); the legacy flag +# still marks a running daemon on the harnesses that launch one. +if [ -f "$STATE/.afk-contract" ]; then + printf 'present - away posture recorded at %s (hold-for-return only; bin/fm-afk-contract.sh readback for the mandate)' \ + "$("$SCRIPT_DIR/fm-afk-contract.sh" field entered 2>/dev/null || printf unknown)" + if [ -e "$STATE/.afk" ]; then + printf '; the away daemon owns the watcher.\n' + else + printf '; no daemon runs, the ordinary supervision session continues.\n' + fi +elif [ -e "$STATE/.afk" ]; then + printf 'present - away-mode supervision is active; the daemon owns the watcher (legacy flag with no posture record).\n' else printf 'absent\n' fi diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 6978cc4da0d..312fc394d43 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -48,15 +48,18 @@ # model, and effort may change, which is what makes a harness switch one # ordinary relaunch. It refuses unless the recorded endpoint is positively # agent-free on a backend with a recovery-grade agent-state classifier (tmux -# or herdr), refuses unless the endpoint's shell is sitting in the recorded -# worktree, and clears the previous harness's per-task wiring before arming -# the new incarnation. +# or herdr), and clears the previous harness's per-task wiring before arming +# the new incarnation. The replacement still never starts outside the copy +# holding the work: a Herdr shell that has drifted out of the recorded +# worktree is told once to return, and only a shell that will not go refuses. # --harness <name> is the explicit per-spawn harness/profile adapter. The old # positional harness arg still works for back-compat. -# --model <name> and --effort <low|medium|high|xhigh|max> are concrete profile +# --model <name> and --effort <low|medium|high|xhigh|max|ultra> are concrete profile # axes chosen by firstmate at intake. They are only threaded into harnesses whose # installed CLIs were verified to support that axis; unsupported axes are omitted -# from that harness's launch rather than guessed. +# from that harness's launch rather than guessed. Ultra is the explicit +# exception: bin/fm-harness.sh validate-native-effort owns its model scope; +# supported Pi launches receive --codex-effort ultra, never --thinking ultra. # --backend <name> is the explicit runtime session-provider backend for this # exact task only (docs/configuration.md "Runtime backend" owns when that flag # is authorized). Without it, the script resolves FM_BACKEND, then @@ -342,9 +345,10 @@ # re-running the transition, so an eligible In-flight item is left untouched. # The transition is # skipped entirely for --secondmate spawns (persistent agents are not work -# items), on a config/backlog-backend=manual home, and in a home that keeps no -# data/backlog.md. An automatic-backend home with a backlog but no compatible -# tasks-axi refuses before creating any lifecycle state. +# items), on a config/backlog-backend=manual home, and in a markdown home that +# keeps no data/backlog.md. A configured non-markdown adapter remains +# active without a markdown file; any active automatic backend without +# compatible tasks-axi refuses before creating lifecycle state. # On success prints: spawned <id> harness=<name> kind=<ship|design|scout|secondmate> [mode=<mode> yolo=<on|off>] window=<backend-target> worktree=<path> # A ship or design task records the explicit mode/yolo it was passed; a secondmate spawn records # mode=secondmate, yolo=off, home=, and projects=; a scout records neither, and both the @@ -567,8 +571,8 @@ if [ "$TRACEPARENT_SET" -eq 1 ]; then } fi case "$EFFORT" in - ''|low|medium|high|xhigh|max) ;; - *) echo "error: --effort must be one of low, medium, high, xhigh, max" >&2; exit 1 ;; + ''|low|medium|high|xhigh|max|ultra) ;; + *) echo "error: --effort must be one of low, medium, high, xhigh, max, ultra" >&2; exit 1 ;; esac # --relaunch reuses an existing task's endpoint, worktree, project, and kind, @@ -694,7 +698,7 @@ spawn_remote_secondmate() { ;; esac case "$effort" in - -|low|medium|high|xhigh|max) ;; + -|low|medium|high|xhigh|max|ultra) ;; *) fm_lock_release "$registry_lock" || true fm_lock_release "$SPAWN_TASK_LOCK" || true @@ -702,6 +706,11 @@ spawn_remote_secondmate() { return 1 ;; esac + if [ "$effort" = ultra ] && ! "$SCRIPT_DIR/fm-harness.sh" validate-native-effort "$harness" "$model" "$effort"; then + fm_lock_release "$registry_lock" || true + fm_lock_release "$SPAWN_TASK_LOCK" || true + return 1 + fi meta="$STATE/$id.meta" if [ -e "$meta" ] || [ -L "$meta" ]; then if ! fm_backlog_record_present "$meta" "task record" "$STATE" \ @@ -1629,12 +1638,21 @@ if [ "$KIND" = secondmate ] && [ -z "$ARG3" ]; then SM_EFFORT=$("$SCRIPT_DIR/fm-harness.sh" secondmate-effort) if [ -n "$SM_EFFORT" ]; then case "$SM_EFFORT" in - low|medium|high|xhigh|max) EFFORT=$SM_EFFORT ;; - *) echo "warning: config/secondmate-harness effort token '$SM_EFFORT' is not one of low, medium, high, xhigh, max; ignoring" >&2 ;; + low|medium|high|xhigh|max|ultra) EFFORT=$SM_EFFORT ;; + *) echo "warning: config/secondmate-harness effort token '$SM_EFFORT' is not one of low, medium, high, xhigh, max, ultra; ignoring" >&2 ;; esac fi fi fi +# Ultra is an explicit native capability, never a Pi thinking-level alias. +# Validate the fully resolved profile before worktree or endpoint provisioning. +if [ "$EFFORT" = ultra ]; then + "$SCRIPT_DIR/fm-harness.sh" validate-native-effort "$HARNESS" "$MODEL" "$EFFORT" || exit 1 + [ "$RAW_LAUNCH" = 0 ] || { + echo "error: --effort ultra requires the canonical --harness pi or pi-signed launch so its native flag cannot be omitted" >&2 + exit 1 + } +fi if [ "$HARNESS" = omp ]; then omp_model_validate "$OMP_BIN" "$MODEL" || exit 1 fi @@ -3034,8 +3052,24 @@ if [ "$RELAUNCH" -eq 1 ]; then sleep 0.5 done if [ -z "$relaunch_seen" ] || [ "$(real_path_or_raw "$relaunch_seen")" != "$relaunch_wt_real" ]; then - echo "error: task $ID's endpoint is in '${relaunch_seen:-unknown}', not its recorded worktree '$WT'; refusing to relaunch an agent outside the copy holding its work" >&2 - exit 1 + if [ "$BACKEND" != herdr ]; then + echo "error: task $ID's endpoint is in '${relaunch_seen:-unknown}', not its recorded worktree '$WT'; refusing to relaunch an agent outside the copy holding its work" >&2 + exit 1 + fi + relaunch_cd_path=${WT//\'/\'\\\'\'} + spawn_send_text_line "$WT_TARGET" "cd -- '$relaunch_cd_path'" || { + echo "error: task $ID's endpoint is in '${relaunch_seen:-unknown}' and could not be told to return to its recorded worktree '$WT'; refusing to relaunch an agent outside the copy holding its work" >&2 + exit 1 + } + for _ in $(seq 1 10); do + relaunch_seen=$(spawn_current_path "$WT_TARGET" || true) + [ -z "$relaunch_seen" ] || [ "$(real_path_or_raw "$relaunch_seen")" != "$relaunch_wt_real" ] || break + sleep 0.5 + done + if [ -z "$relaunch_seen" ] || [ "$(real_path_or_raw "$relaunch_seen")" != "$relaunch_wt_real" ]; then + echo "error: task $ID's endpoint is in '${relaunch_seen:-unknown}' and did not return to its recorded worktree '$WT' when told to; refusing to relaunch an agent outside the copy holding its work" >&2 + exit 1 + fi fi [ "$KIND" = secondmate ] || validate_spawn_worktree "relaunch" "$T" elif [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then @@ -3478,6 +3512,17 @@ export default function (pi: any) { return busyEvent("idle", "agent-settled"); }); pi.on("turn_end", () => execFile("touch", ["$TURNEND"])); + // A native harness can make progress inside one Pi turn. This separate + // marker prevents false wedge alarms without fabricating a completed turn. + let lastProgress = 0; + pi.events?.on?.("codex-native:progress", () => { + const now = Date.now(); + if (now - lastProgress < 1000) return; + lastProgress = now; + execFile("$FM_ROOT/bin/fm-busy-event.sh", [ + "progress", "$STATE_REAL", "$ID", "--gen", "$BUSY_GEN", + ]); + }); } EOF ;; @@ -3936,7 +3981,7 @@ sq_piturnend=$(fm_launch_shell_quote "$PROJ_ABS/.pi/extensions/fm-primary-turnen sq_piwatch=$(fm_launch_shell_quote "$PROJ_ABS/.pi/extensions/fm-primary-pi-watch.ts") sq_opinput=$(fm_launch_shell_quote "$FM_ROOT/bin/fm-operational-input.sh") MODELFLAG=$(fm_launch_model_flag "$HARNESS" "$MODEL") -EFFORTFLAG=$(fm_launch_effort_flag "$HARNESS" "$EFFORT") +EFFORTFLAG=$(fm_launch_effort_flag "$HARNESS" "$EFFORT" "$MODEL") || exit 1 ROVOCONFIGOVERRIDE= if [ "$HARNESS" = rovo ]; then ROVOCONFIGOVERRIDE=$(fm_launch_rovo_config_override_flag "$EFFORT" "$DATA" "$STATE" "$ID") || { diff --git a/bin/fm-supervise-daemon.sh b/bin/fm-supervise-daemon.sh index 5f924228171..52b9a5cc1b8 100755 --- a/bin/fm-supervise-daemon.sh +++ b/bin/fm-supervise-daemon.sh @@ -187,6 +187,10 @@ FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" # classification predicates have exactly one definition. # shellcheck source=bin/fm-classify-lib.sh . "$FM_DAEMON_DIR/fm-classify-lib.sh" +# The away-posture record owner; declared waits retain the shared bounded +# recheck cadence while this daemon owns away-mode supervision. +# shellcheck source=bin/fm-afk-contract.sh +. "$FM_DAEMON_DIR/fm-afk-contract.sh" # Supervisor-pane discovery (FM_SUPERVISOR_TARGET_DEFAULT, # FM_SUPERVISOR_BACKEND_DEFAULT, discover_supervisor_target, @@ -294,7 +298,8 @@ afk_exit() { # <state> } # should_exit_afk: encodes firstmate's afk-exit contract as a testable function. -# afk inactive -> 1 (nothing to exit) +# away posture inactive -> 1 (nothing to exit; the posture is the record +# bin/fm-afk-contract.sh owns, or the legacy flag) # message has marker -> 1 (internal escalation; stay afk) # message is /afk command -> 1 (re-entering/extending afk; stay afk) # anything else -> 0 (captain is back; exit afk) @@ -302,7 +307,7 @@ afk_exit() { # <state> # alive. A false exit is self-correcting (the captain re-runs /afk). should_exit_afk() { # <state> <message-text> local state=$1 msg=$2 - afk_active "$state" || return 1 + afk_active "$state" || fm_afk_contract_present "$state" || return 1 message_is_injection "$msg" && return 1 case "$msg" in /afk*) return 1 ;; @@ -543,7 +548,9 @@ pause_marker_record() { # <window> <state> - create if absent task=$(window_to_task "$win" "$state") key=$(_stale_key "$task") marker="$state/.subsuper-paused-$key" - pause_streak_sync "$(pause_streak_path "$key" "$state")" "$(last_status_line "$state/$task.status")" || true + if pause_streak_sync "$(pause_streak_path "$key" "$state")" "$(last_status_line "$state/$task.status")"; then + rm -f "$state/.subsuper-pause-until-due-$key" + fi [ -e "$marker" ] || _now > "$marker" } @@ -553,7 +560,7 @@ pause_marker_record() { # <window> <state> - create if absent pause_marker_remove() { # <window> <state> local win=$1 state=$2 key key=$(_stale_key "$(window_to_task "$win" "$state")") - rm -f "$state/.subsuper-paused-$key" "$(pause_streak_path "$key" "$state")" + rm -f "$state/.subsuper-paused-$key" "$state/.subsuper-pause-until-due-$key" "$(pause_streak_path "$key" "$state")" } clear_pause_tracking() { # <window> <state> @@ -561,10 +568,10 @@ clear_pause_tracking() { # <window> <state> task=$(window_to_task "$win" "$state") key=$(_stale_key "$task") watcher_key=$(_stale_key "$win") - rm -f "$state/.subsuper-paused-$key" "$(pause_streak_path "$key" "$state")" "$state/.subsuper-stale-$key" \ + rm -f "$state/.subsuper-paused-$key" "$state/.subsuper-pause-until-due-$key" "$(pause_streak_path "$key" "$state")" "$state/.subsuper-stale-$key" \ "$(wedge_holds_path "$key" "$state")" \ "$state/.paused-$watcher_key" "$state/.paused-rechecked-$watcher_key" "$state/.paused-resurfaced-$watcher_key" \ - "$state/.paused-streak-$watcher_key" \ + "$state/.paused-streak-$watcher_key" "$state/.paused-until-due-$watcher_key" \ "$state/.stale-$watcher_key" "$state/.stale-since-$watcher_key" "$state/.wedge-escalations-$watcher_key" \ "$state/.wedge-holds-$watcher_key" \ "$state/.writing-since-$watcher_key" "$state/.writing-resurfaced-$watcher_key" @@ -1083,7 +1090,7 @@ _oldest_line_age() { # <buf> -> seconds since the oldest buffered item first ar # captain-relevant line the per-wake classifier missed and escalate it. housekeeping() { # <state> local state=$1 now due f key task win marker age last max_defer oldest pause_secs streak_file progress \ - holds_file holds hold_max escalated + holds_file holds hold_max escalated marker_epoch until bounded_until pause_reason now=$(_now) migrate_watcher_pause_markers "$state" @@ -1220,11 +1227,28 @@ housekeeping() { # <state> reconcile_pause_tracking "$win" "$state" "$last" continue fi - age=$(( now - $(cat "$marker" 2>/dev/null || echo "$now") )) streak_file=$(pause_streak_path "$key" "$state") - pause_streak_sync "$streak_file" "$last" || true + if pause_streak_sync "$streak_file" "$last"; then + rm -f "$state/.subsuper-pause-until-due-$key" + fi pause_secs=$(pause_resurface_window "$(pause_streak_count "$streak_file")") - [ "$age" -ge "$pause_secs" ] || continue + marker_epoch=$(cat "$marker" 2>/dev/null || echo "$now") + case "$marker_epoch" in ''|*[!0-9]*) marker_epoch=$now ;; esac + age=$(( now - marker_epoch )) + due="$state/.subsuper-pause-until-due-$key" + until= + bounded_until=0 + if until=$(status_paused_until "$last"); then + if [ "$now" -lt "$until" ] && [ "$age" -lt "$pause_secs" ]; then + continue + elif [ "$now" -lt "$until" ]; then + bounded_until=1 + elif [ "$(cat "$due" 2>/dev/null || true)" = "$until" ]; then + [ "$age" -ge "$pause_secs" ] || continue + fi + else + [ "$age" -ge "$pause_secs" ] || continue + fi # Endpoint-readability probe only: exit code 2 means the capture failed, so the # endpoint is gone and there is nothing left to re-surface. The busy/idle verdict # is deliberately discarded here. Do NOT reinstate a `0)` arm dropping the marker @@ -1233,17 +1257,26 @@ housekeeping() { # <state> # restart forever and the wait would never mature into its one recheck. stale_window_is_busy "$win" "$state" case "$?" in - 2) rm -f "$marker" "$streak_file" ;; + 2) rm -f "$marker" "$streak_file" "$due" ;; *) last=$(last_status_line "$state/$task.status") if [ -n "$last" ] && status_is_captain_held "$last"; then if escalate_add "$state" "captain-held ${age}s (awaiting the captain, answer the held decision or release the hold): $win"; then _now > "$marker" + pause_streak_bump "$streak_file" "$last" fi elif [ -n "$last" ] && status_is_paused "$last"; then - if escalate_add "$state" "paused ${age}s (awaiting external, recheck whether the wait still holds): $win"; then + if [ "$bounded_until" -eq 1 ]; then + pause_reason="paused ${age}s (awaiting external, the declared time is beyond the recheck cadence; confirm the wait still holds): $win" + else + pause_reason="paused ${age}s (awaiting external, recheck whether the wait still holds): $win" + fi + if escalate_add "$state" "$pause_reason"; then _now > "$marker" pause_streak_bump "$streak_file" "$last" + if [ -n "$until" ] && [ "$now" -ge "$until" ]; then + printf '%s\n' "$until" > "$due" + fi fi else rm -f "$marker" "$streak_file" @@ -1416,6 +1449,10 @@ is_wake_reason() { # <reason> # --- dispatch one wake reason to self-handle or escalate -------------------- # Side effects: logging, marker records, escalation buffer appends. +# A decision-owned queued row arrives as needs-decision:<files> rather than +# signal:<files> (bin/fm-watch.sh). Classify it as a signal so the capture file +# is populated, suppression markers commit, and the digest names the decision +# instead of "unknown wake:". handle_wake() { # <reason> <state> local reason=$1 state=$2 decision action distilled task last stale_detail local capture="$state/.subsuper-classified-end.$$" span_record='' span_rc='' endpoint ident rest sig marker @@ -1427,7 +1464,12 @@ handle_wake() { # <reason> <state> return fi case "$reason" in - signal:*) kind=signal; arg="${reason#signal: }" + signal:*|needs-decision:*) + kind=signal + case "$reason" in + needs-decision:*) arg="${reason#needs-decision: }" ;; + *) arg="${reason#signal: }" ;; + esac decision=$(FM_STATUS_SPAN_ENDPOINT_FILE="$capture" classify_signal "$arg" "$state") ;; stale:*) kind=stale; arg="${reason#stale: }"; stale_detail="${arg#"$arg"}" case "$arg" in *" ("*) stale_detail="${arg#*" ("}"; arg="${arg%% \(*}" ;; esac diff --git a/bin/fm-tasks-axi-lib.sh b/bin/fm-tasks-axi-lib.sh index b3cfb78878f..fe6f53641f8 100644 --- a/bin/fm-tasks-axi-lib.sh +++ b/bin/fm-tasks-axi-lib.sh @@ -15,8 +15,14 @@ # backlog mutations, but validated secondmate handoffs always use `tasks-axi mv`. # Absent or any other value keeps the default tasks-axi backend path, falling # back to manual mutation when the tool is not compatible. -# fm_tasks_axi_backend mirrors tasks-axi's environment, project, home-config, -# and default backend precedence for callers that need backend-specific flags. +# fm_tasks_axi_backend_resolve owns backend precedence: TASKS_AXI_BACKEND when +# set, then a backend in the working root's .tasks.toml, then one in +# $HOME/.tasks-axi/config.toml, then markdown. Lower-priority sources are read +# only when no earlier source supplies a backend; absent files keep that fallback. +# A detected unreadable or nonregular configuration file, including a dangling +# symlink, returns 2 with a path diagnostic on stderr and no backend on stdout. +# fm_tasks_axi_backend delegates to that resolver and preserves its status; +# callers must check it before selecting backend-specific flags or exemptions. # # This file is the single owner of FM_TASKS_AXI_MIN. bin/fm-bootstrap.sh turns a # failing check into the operator-facing MISSING diagnostic. @@ -135,24 +141,41 @@ fm_tasks_axi_backend_from_toml() { # <toml-path> } # Resolve the active tasks-axi backend with the same precedence as tasks-axi. -fm_tasks_axi_backend() { # <tasks-axi-working-directory> +fm_tasks_axi_backend_resolve() { # <tasks-axi-working-directory> local root=$1 backend if [ "${TASKS_AXI_BACKEND+x}" = x ]; then printf '%s\n' "$TASKS_AXI_BACKEND" return 0 fi - if backend=$(fm_tasks_axi_backend_from_toml "$root/.tasks.toml"); then - printf '%s\n' "$backend" - return 0 + local config="$root/.tasks.toml" + if { [ -d "${config%/*}" ] && [ ! -x "${config%/*}" ]; } || + { { [ -e "$config" ] || [ -L "$config" ]; } && { [ ! -f "$config" ] || [ ! -r "$config" ]; }; }; then + printf 'tasks-axi backend configuration cannot be read at %s\n' "$config" >&2 + return 2 fi - if [ -n "${HOME:-}" ] \ - && backend=$(fm_tasks_axi_backend_from_toml "$HOME/.tasks-axi/config.toml"); then + if backend=$(fm_tasks_axi_backend_from_toml "$config"); then printf '%s\n' "$backend" return 0 fi + if [ -n "${HOME:-}" ]; then + config="$HOME/.tasks-axi/config.toml" + if { [ -d "${config%/*}" ] && [ ! -x "${config%/*}" ]; } || + { { [ -e "$config" ] || [ -L "$config" ]; } && { [ ! -f "$config" ] || [ ! -r "$config" ]; }; }; then + printf 'tasks-axi backend configuration cannot be read at %s\n' "$config" >&2 + return 2 + fi + if backend=$(fm_tasks_axi_backend_from_toml "$config"); then + printf '%s\n' "$backend" + return 0 + fi + fi printf '%s\n' markdown } +fm_tasks_axi_backend() { # <tasks-axi-working-directory> + fm_tasks_axi_backend_resolve "$1" +} + fm_backlog_backend_value() { local config_dir=$1 backend_file value backend_file="$config_dir/backlog-backend" diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index e0cfa2d618c..cd24de3612a 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -21,11 +21,12 @@ # that withdrawal the next session start would replay the close and remove the # very record those refusals just preserved, which is the opposite of what each # of them is for. -# A transition that fails is fatal and loud, preserves its pending-close record, and +# A close that fails is fatal and loud, preserves its pending-close record, and # is retried by the next session start. The transition is skipped on a -# config/backlog-backend=manual home and in a home that keeps no -# data/backlog.md; those cases print the manual follow-up. An automatic-backend -# home with a backlog but no compatible tasks-axi refuses before cleanup. +# config/backlog-backend=manual home and in a markdown home that keeps no +# data/backlog.md; those cases print the manual follow-up. A configured +# non-markdown adapter remains active without a markdown file; any active +# automatic backend without compatible tasks-axi refuses before cleanup. # None of this loosens the landed-work gates below: the transition runs only on # the paths that already proceed to remove the record. # Ordinary tracked-output cleanup also reaps exactly refs/heads/fm/<task-id>, never the @@ -55,11 +56,12 @@ # The close - and only the close - is replaced by `tasks-axi reopen` with the # deliverable recorded while the backlog item is still an open captain call # (bin/fm-captain-hold.sh `open` owns that predicate), because the policy holds +# the very work item a question gates and cleanup must never retire the +# captain's own question. # NOTE: this uses `open`'s silent default and depends only on its unchanged # 0/1/2 exit-code contract. The optional `--identity` output that bin/fm-watch.sh # asks for prints only on an exit 0 and changes nothing read here. -# the very work item a question gates and cleanup must never retire the -# captain's own question. The same pending-close record carries that intent as +# The same pending-close record carries that intent as # `mode=retain`, so an interrupted cleanup replays the retention rather than a # close. "Cannot tell" refuses before any destructive step, --force does not # lift the deferral (it authorizes discarding unlanded WORK, never the @@ -1010,7 +1012,7 @@ remote_secondmate_teardown() { mv -f -- "$tmp" "$SECONDMATE_REG" status_retire_presentation_task "$STATE" "$ID" || return 1 fm_backlog_atomic_transition remove "$STATE/$ID.meta" "task record" "$STATE" || return 1 - rm -f -- "$STATE/$ID.status" "$STATE/$ID.turn-ended" \ + rm -f -- "$STATE/$ID.status" "$STATE/$ID.turn-ended" "$STATE/$ID.progress" \ "$STATE/$ID.pr-status" "$STATE/$ID.gbrain" "$STATE/$ID.usage-sessions" \ "$STATE/.$ID.open-decisions-cursor" printf 'teardown %s complete (remote %s:%s)\n' "$ID" "$remote_host" "$remote_home" @@ -1959,10 +1961,13 @@ withdraw_pending_backlog_close() { # invariant). This prints what already happened, so the follow-up wording stays # only where a human still owes the edit. backlog_refresh_reminder() { - local backlog_display root + local backlog_display root backend=markdown [ "$KIND" = secondmate ] && return 0 [ "$CLEANUP_RECOVERY" = orca ] && return 0 - if root=$(fm_backlog_root "$DATA") && [ "$(fm_tasks_axi_backend "$root")" != markdown ]; then + if root=$(fm_backlog_root "$DATA"); then + backend=$(fm_tasks_axi_backend "$root") || return 2 + fi + if [ "$backend" != markdown ]; then backlog_display="this home's configured tasks-axi backend (data directory $DATA)" elif backlog_display=$(fm_backlog_file "$DATA"); then : @@ -3499,7 +3504,7 @@ cleanup_firstmate_home_children() { retire_busy_state "$sub_state" "$child_id" "$child_busy_gen" || return 1 status_retire_presentation_task "$sub_state" "$child_id" || return 1 fm_backlog_atomic_transition remove "$sub_state/$child_id.meta" "task record" "$sub_state" || return 1 - rm -f "$sub_state/$child_id.turn-ended" \ + rm -f "$sub_state/$child_id.turn-ended" "$sub_state/$child_id.progress" \ "$sub_state/$child_id.pi-ext.ts" "$sub_state/$child_id.omp-ext.ts" \ "$sub_state/$child_id.run-step" \ "$sub_state/$child_id.grok-turnend-token" "$sub_state/$child_id.kimi-turnend-token" \ @@ -3972,7 +3977,7 @@ fi remove_pr_poll_artifacts "$STATE" "$ID" || exit 1 retire_busy_state "$STATE" "$ID" "$BUSY_GEN" || exit 1 status_retire_presentation_task "$STATE" "$ID" || exit 1 -rm -f "$STATE/$ID.status" "$STATE/$ID.turn-ended" \ +rm -f "$STATE/$ID.status" "$STATE/$ID.turn-ended" "$STATE/$ID.progress" \ "$STATE/$ID.pi-ext.ts" "$STATE/$ID.omp-ext.ts" "$STATE/$ID.run-step" \ "$STATE/$ID.grok-turnend-token" "$STATE/$ID.kimi-turnend-token" \ "$STATE/$ID.pr-status" "$STATE/$ID.gbrain" "$STATE/$ID.usage-sessions" \ diff --git a/bin/fm-test-isolation-proof.sh b/bin/fm-test-isolation-proof.sh index 3100e746f74..7c685c20af2 100755 --- a/bin/fm-test-isolation-proof.sh +++ b/bin/fm-test-isolation-proof.sh @@ -139,6 +139,7 @@ exclusion_reason() { fm-backend-herdr-presentation-e2e.test.sh|fm-backend-herdr-recovery-e2e.test.sh|\ fm-backend-herdr-prune-safety-e2e.test.sh|\ fm-backend-herdr-respawn-idem-e2e.test.sh|fm-backend-herdr-smoke.test.sh|\ + fm-backend-herdr-agent-exit-shell-e2e.test.sh|\ fm-backend-herdr-workspace-per-home-e2e.test.sh|fm-herdr-session-cleanup-e2e.test.sh) printf '%s\n' 'real Herdr-gated; Herdr lane is a later phase' ;; diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index b037e7c315f..fc0134f7098 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -184,15 +184,14 @@ PER_SCRIPT_TIMEOUT_SET= # is accepted rather than widened (HelloWorldSungin/firstmate#256). # # The arithmetic is replacement, not addition: a hung script spends the bound -# INSTEAD of its own healthy slot. The current portable-serial hint table totals -# about 6188s over eight shards, with the slowest near 774s. Replacing its ~31s -# average script with 480s puts script time near 1223s, beyond the 1200s CI cap -# before checkout and bootstrap. A hung script can therefore lose per-script -# attribution to the enclosing job cancellation; healthy estimates retain about -# seven minutes for setup and runner variability. Neither deadline is widened. +# INSTEAD of its own healthy slot. The merged portable-serial hint table totals +# about 6572s over eight shards, with the slowest near 822s. Replacing its ~32s +# average script with 480s puts script time near 1270s, inside the 1800s CI cap +# with about nine minutes left for setup and runner variability. The script +# bound stays unchanged; the serial job adopts upstream's 30-minute cap. # real-Herdr is 420s less its ~35s mean slot plus 480s, inside its 1200s step cap. -# portable-parallel is tighter: CI runs that lane serially, so ~422s less its -# ~38s mean slot plus 480s is ~864s, already past its 600s job cap before setup. +# portable-parallel is tighter: CI runs that lane serially, so ~545s less its +# ~45s mean slot plus 480s is ~980s, already past its 600s job cap before setup. # That lane can therefore lose per-script attribution to a job cancellation. # Raising the script bound would not remedy that enclosing-job limit. # @@ -325,6 +324,7 @@ family_for_basename() { fm-supervision-events.test.sh|fm-turnend-guard.test.sh|fm-wake-daemon-lifecycle-e2e.test.sh|\ fm-wake-drain-unread-status.test.sh|\ fm-tool-update-check.test.sh|\ + fm-mail.test.sh|fm-mail-check.test.sh|\ fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-checkpoint.test.sh|fm-watch-recovery-loop.test.sh|\ fm-watch-triage.test.sh|fm-watch-triage-waits.test.sh|fm-task-inbox.test.sh|\ fm-watcher-lock.test.sh|fm-inactive-reconcile.test.sh) @@ -336,13 +336,15 @@ family_for_basename() { fm-backend-herdr-launcher-workspace-e2e.test.sh|\ fm-backend-herdr-prune-safety-e2e.test.sh|fm-backend-herdr-respawn-idem-e2e.test.sh|\ fm-backend-herdr-focus-flash-e2e.test.sh|\ + fm-backend-herdr-stale-active-tab-e2e.test.sh|\ + fm-backend-herdr-agent-exit-shell-e2e.test.sh|\ fm-herdr-session-cleanup-e2e.test.sh|\ fm-backend-herdr-smoke.test.sh|fm-backend-herdr-workspace-per-home-e2e.test.sh|\ fm-control-herdr-smoke.test.sh) printf '%s\n' real-herdr-gated ;; fm-backlog-handoff.test.sh|fm-on.test.sh|fm-remote-backlog-handoff.test.sh|\ - fm-remote-doctor.test.sh|fm-remote-job.test.sh|fm-remote-job-orphan-reap.test.sh|\ + fm-remote-doctor.test.sh|fm-remote-herdr-guard.test.sh|fm-remote-job.test.sh|fm-remote-job-orphan-reap.test.sh|\ fm-remote-job-worker-leak.test.sh|fm-remote-transport-lanes.test.sh|\ fm-remote-reply.test.sh|fm-remote-secondmate-lifecycle-e2e.test.sh|\ fm-remote-secondmate-trace-context.test.sh|\ @@ -366,6 +368,7 @@ family_for_basename() { printf '%s\n' session-bootstrap ;; fm-afk-pi-dual-supervision-e2e.test.sh|fm-afk-pi-herdr-return-e2e.test.sh|\ + fm-pi-codex-native.test.sh|\ fm-bearings-board-lavish-live-e2e.test.sh|\ fm-claude-stop-autoarm-live-e2e.test.sh|\ fm-composer-matrix-live-e2e.test.sh|\ @@ -404,7 +407,7 @@ family_for_basename() { fm-teardown.test.sh|fm-x-mode.test.sh) printf '%s\n' pr-forge ;; - fm-afk-inject-e2e.test.sh|fm-afk-return.test.sh) + fm-afk-contract.test.sh|fm-afk-inject-e2e.test.sh|fm-afk-return.test.sh) printf '%s\n' afk ;; fm-bearings-board-render.test.sh|fm-bearings-snapshot.test.sh|\ @@ -525,16 +528,17 @@ EOF list_portable_parallel_1() { cat <<'EOF' tests/fm-captain-hold-lifecycle.test.sh -tests/fm-pr-merge.test.sh -tests/fm-crew-state.test.sh -tests/fm-backend-herdr.test.sh -tests/fm-cd-pretool-check.test.sh +tests/fm-test-run.test.sh +tests/fm-x-mode.test.sh +tests/fm-brief.test.sh +tests/fm-send-strict.test.sh tests/fm-grok-harness.test.sh -tests/fm-herdr-lab.test.sh +tests/fm-send-popup-settle.test.sh tests/fm-pi-primary-types.test.sh -tests/fm-review-diff.test.sh +tests/fm-spawn-batch.test.sh tests/fm-composer-ghost.test.sh -tests/fm-send-settle.test.sh +tests/fm-ensure-agents-md.test.sh +tests/fm-transition-lib.test.sh EOF } @@ -542,18 +546,17 @@ EOF list_portable_parallel_2() { cat <<'EOF' tests/fm-lint.test.sh -tests/fm-test-run.test.sh -tests/fm-x-mode.test.sh +tests/fm-pr-merge.test.sh +tests/fm-crew-state.test.sh tests/fm-arm-pretool-check.test.sh -tests/fm-brief.test.sh -tests/fm-send-strict.test.sh -tests/fm-send-popup-settle.test.sh +tests/fm-backend-herdr.test.sh +tests/fm-cd-pretool-check.test.sh +tests/fm-herdr-lab.test.sh tests/fm-composer-lib.test.sh +tests/fm-review-diff.test.sh tests/fm-tmux-submit-busy.test.sh -tests/fm-spawn-batch.test.sh -tests/fm-ensure-agents-md.test.sh +tests/fm-send-settle.test.sh tests/fm-supervision-instructions.test.sh -tests/fm-transition-lib.test.sh EOF } @@ -645,6 +648,7 @@ list_portable_serial() { # balance rather than coverage. That doc owns the refresh procedure. portable_serial_weight_hints() { cat <<'EOF' +tests/fm-afk-contract.test.sh 3000 tests/fm-afk-inject-e2e.test.sh 35792 tests/fm-afk-pi-dual-supervision-e2e.test.sh 109 tests/fm-afk-pi-herdr-return-e2e.test.sh 100 @@ -661,7 +665,7 @@ tests/fm-backend-tmux-smoke.test.sh 393 tests/fm-backend-zellij-smoke.test.sh 55 tests/fm-backend-zellij.test.sh 10313 tests/fm-backend.test.sh 22748 -tests/fm-backlog-atomicity.test.sh 154917 +tests/fm-backlog-atomicity.test.sh 161989 tests/fm-backlog-handoff.test.sh 54825 tests/fm-bearings-board-lavish-live-e2e.test.sh 117 tests/fm-bearings-board-render.test.sh 17024 @@ -671,7 +675,7 @@ tests/fm-bootstrap-network-parallel.test.sh 20354 tests/fm-bootstrap.test.sh 75613 tests/fm-branch-supervision.test.sh 9222 tests/fm-brief-repo-lib.test.sh 227 -tests/fm-busy-adapter-wiring.test.sh 29970 +tests/fm-busy-adapter-wiring.test.sh 49731 tests/fm-busy-state.test.sh 3171 tests/fm-calm-pi-extension.test.sh 52972 tests/fm-check-unregister.test.sh 565 @@ -684,7 +688,7 @@ tests/fm-cmux-claude-composer-live-e2e.test.sh 92 tests/fm-codex-continuity-live-e2e.test.sh 108 tests/fm-composer-matrix-live-e2e.test.sh 114 tests/fm-control-relaunch.test.sh 72543 -tests/fm-control.test.sh 38952 +tests/fm-control.test.sh 54301 tests/fm-cursor-harness.test.sh 30129 tests/fm-cursor-primary-live-e2e.test.sh 131 tests/fm-cursor-primary.test.sh 55508 @@ -720,7 +724,7 @@ tests/fm-gitignore-config.test.sh 112 tests/fm-gotmp.test.sh 2157 tests/fm-grok-continuity-live-e2e.test.sh 159 tests/fm-grok-stop-live-e2e.test.sh 92 -tests/fm-guard-stale-banner.test.sh 13176 +tests/fm-guard-stale-banner.test.sh 32981 tests/fm-harness-adapter-instructions-live-e2e.test.sh 107 tests/fm-harness-adapter-references.test.sh 118 tests/fm-harness-liveness-drift-live-e2e.test.sh 856 @@ -728,7 +732,7 @@ tests/fm-herdr-session-cleanup.test.sh 6858 tests/fm-herdr-submit-confirm-live-e2e.test.sh 118 tests/fm-herdr-version-floor-live-e2e.test.sh 93 tests/fm-home-summary-refresh.test.sh 39231 -tests/fm-inactive-reconcile.test.sh 48508 +tests/fm-inactive-reconcile.test.sh 74399 tests/fm-issue-linkage.test.sh 17659 tests/fm-issue-writeback.test.sh 29552 tests/fm-kimi-harness.test.sh 20521 @@ -742,14 +746,14 @@ tests/fm-muse-signals-live-e2e.test.sh 90 tests/fm-nm-test-contract.test.sh 812 tests/fm-no-mistakes-required-gate.test.sh 4578 tests/fm-no-mistakes-required.test.sh 370 -tests/fm-omp-harness.test.sh 20362 +tests/fm-omp-harness.test.sh 59969 tests/fm-omp-primary-live-e2e.test.sh 187 -tests/fm-on.test.sh 12020 +tests/fm-on.test.sh 34087 tests/fm-opencode-primary-live-e2e.test.sh 112 tests/fm-operational-input.test.sh 308 tests/fm-outcome-manifest.test.sh 8021 tests/fm-peek-remote.test.sh 1018 -tests/fm-pending-reply.test.sh 30068 +tests/fm-pending-reply.test.sh 86711 tests/fm-pi-branch-extension.test.sh 176128 tests/fm-pi-branch-live-e2e.test.sh 163 tests/fm-pi-branch-responsiveness-live-e2e.test.sh 13245 @@ -758,7 +762,7 @@ tests/fm-pi-primary-live-e2e.test.sh 132 tests/fm-pi-watch-extension.test.sh 62408 tests/fm-pi-windows-shell-invocation.test.sh 5121 tests/fm-pointer-check.test.sh 1314 -tests/fm-pr-check-security.test.sh 171211 +tests/fm-pr-check-security.test.sh 172215 tests/fm-pr-status.test.sh 420 tests/fm-procevent-quota.test.sh 1949 tests/fm-procevent-when.test.sh 17550 @@ -772,6 +776,7 @@ tests/fm-recall.test.sh 60802 tests/fm-remote-backlog-handoff.test.sh 109295 tests/fm-remote-doctor.test.sh 5335 tests/fm-remote-entrypoint.test.sh 167 +tests/fm-remote-herdr-guard.test.sh 1500 tests/fm-remote-job-orphan-reap.test.sh 2972 tests/fm-remote-job-worker-leak.test.sh 3854 tests/fm-remote-job.test.sh 97985 @@ -779,7 +784,7 @@ tests/fm-remote-reply.test.sh 101690 tests/fm-remote-secondmate-lifecycle-e2e.test.sh 292054 tests/fm-remote-secondmate-parent-binding.test.sh 103889 tests/fm-remote-secondmate-trace-context.test.sh 69915 -tests/fm-remote-transport-lanes.test.sh 63140 +tests/fm-remote-transport-lanes.test.sh 63976 tests/fm-rovo-harness.test.sh 14983 tests/fm-rovo-signals-live-e2e.test.sh 113 tests/fm-run-attribution-legacy-transition.test.sh 3737 @@ -788,7 +793,7 @@ tests/fm-secondmate-harness.test.sh 157114 tests/fm-secondmate-lifecycle-e2e.test.sh 8793 tests/fm-secondmate-liveness.test.sh 18146 tests/fm-secondmate-reconcile.test.sh 98643 -tests/fm-secondmate-restart.test.sh 108981 +tests/fm-secondmate-restart.test.sh 119085 tests/fm-secondmate-safety.test.sh 65616 tests/fm-secondmate-sync.test.sh 56818 tests/fm-send-inbox-doorbell-live-e2e.test.sh 113 @@ -1467,7 +1472,7 @@ families_for_changed_path() { printf '%s\n' "__script__:fm-quota-choose.test.sh" ;; .pi/extensions/fm-branch-supervision.ts|.pi/extensions/lib/fm-async-exec.ts|\ - .pi/extensions/lib/fm-branch-dispatch.ts) + .pi/extensions/lib/fm-branch-dispatch.ts|.pi/extensions/lib/fm-native-contract.ts) # The portable suites that actually load these files, named one by one. # Left unmapped, a Pi extension library resolves through the reference # scan, which widens to each referencing suite's WHOLE family - and diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 8963d464011..1ee40021360 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -188,7 +188,9 @@ fm_watcher_healthy() { # autoarm Claude's Stop-hook auto-arm and Cursor's stop-hook park: the # watcher is armed at each turn end and exits on its wake, so it # runs only BETWEEN turns. Mid-turn a fresh beacon with no live -# watcher process is the healthy state. +# watcher process is healthy, and a stale beacon is still healthy +# while a Claude auto-arm generation explains the gap +# (fm_autoarm_midturn_healthy). # extension Pi (and pi-signed): .pi/extensions/fm-primary-pi-watch.ts owns # continuity. It tears the watcher down on every actionable wake and # spawns the replacement itself, so a genuinely unheld singleton lock @@ -336,7 +338,11 @@ fm_afk_daemon_owns_supervision() { # stale-beacon - the beacon is stale beyond grace or # absent (a genuine supervision lapse) # autoarm: a fresh beacon within grace is healthy even with no live watcher, -# because the watcher only runs between turns; only a stale beacon is a lapse. +# because the watcher only runs between turns. A stale beacon is still healthy +# while fm_autoarm_midturn_healthy proves a Claude auto-arm generation +# explains the gap (a rewake bound to the current recovery generation and +# live session lock), because turn-end re-arms. +# Without that proof a stale or absent beacon is a genuine lapse. # extension: a live identity-matched watcher is the ordinary healthy state, but a # genuinely unheld lock is also healthy while the beacon is fresh AND a live Pi # session provably owns continuity (fm_extension_owns_supervision: the Pi or the @@ -366,7 +372,9 @@ fm_watcher_supervision_verdict() { esac model=$(fm_supervision_model) if [ "$model" = autoarm ]; then - [ "$fresh" = true ] && FM_WATCHER_VERDICT_OK=true + if [ "$fresh" = true ] || fm_autoarm_midturn_healthy "$state" "$grace"; then + FM_WATCHER_VERDICT_OK=true + fi return 0 fi if fm_watcher_healthy "$state" "$watch" "$grace" "$home"; then @@ -1245,9 +1253,11 @@ fm_failure_episode_reset() { # state/.claude-autoarm-epoch, whose monotonic epoch sequence IS the claim # generation. This is an optimistic, generation-based single-flight design: # -# - The CURRENT claim is the ledger's latest entry: line 1 is the classic -# "epoch=N owner_pid=P outcome=O updated_at=T" record, and line 2 is the -# claiming process's pid-identity, the same identity every other +# - The CURRENT claim is the ledger's latest entry: line 1 begins with the +# "epoch=N owner_pid=P outcome=O updated_at=T" record. A "rewake" outcome +# also records "session_pid=S recovery_generation=G", binding that +# handling turn to its live session-lock owner and watcher recovery episode. +# Line 2 is the claiming process's pid-identity, the same identity every other # supervision lock in this repo records (fm_pid_identity above). The # identity is MANDATORY: a claimant that cannot record it does not claim # (continuity falls to the synchronous guard), and the identity is read @@ -1328,7 +1338,8 @@ _fm_autoarm_epoch_field() { # <epoch-file> <field> } # Parse the current ledger claim. Sets FM_AUTOARM_GEN, FM_AUTOARM_OWNER, -# FM_AUTOARM_OUTCOME, and FM_AUTOARM_IDENTITY (line 2 of the entry, and ONLY +# FM_AUTOARM_OUTCOME, FM_AUTOARM_SESSION, FM_AUTOARM_RECOVERY, and +# FM_AUTOARM_IDENTITY (line 2 of the entry, and ONLY # line 2 - identity is never substituted from a lock, so a transient # micro-mutex hold or a reused pid can never authenticate a stale entry). fm_autoarm_ledger_read() { # <state-dir> @@ -1337,10 +1348,14 @@ fm_autoarm_ledger_read() { # <state-dir> FM_AUTOARM_GEN= FM_AUTOARM_OWNER= FM_AUTOARM_OUTCOME= + FM_AUTOARM_SESSION= + FM_AUTOARM_RECOVERY= FM_AUTOARM_IDENTITY= FM_AUTOARM_GEN=$(_fm_autoarm_epoch_field "$epoch" epoch) || return 1 FM_AUTOARM_OWNER=$(_fm_autoarm_epoch_field "$epoch" owner_pid) || return 1 FM_AUTOARM_OUTCOME=$(_fm_autoarm_epoch_field "$epoch" outcome) || return 1 + FM_AUTOARM_SESSION=$(_fm_autoarm_epoch_field "$epoch" session_pid 2>/dev/null || true) + FM_AUTOARM_RECOVERY=$(_fm_autoarm_epoch_field "$epoch" recovery_generation 2>/dev/null || true) case "$FM_AUTOARM_GEN" in ''|*[!0-9]*) return 1 ;; esac @@ -1377,6 +1392,40 @@ fm_autoarm_claim_open() { # <state-dir> [grace] return 0 } +# True when a stale mid-turn beacon is explained by a healthy Claude Stop +# auto-arm generation, so the pull guard must not cry supervision-off. +# The watcher runs only between turns; turn-end re-arms. +# +# Healthy means outcome=rewake with no exhausted-failure marker, bound to the +# current session-lock pid and current watcher recovery generation. The rewake +# ledger must also be at least as new as the last watcher beacon: a later beacon +# proves another between-turns watcher cycle has begun, so the rewake belongs to +# an earlier handling turn. +# +# A missing generation, a failed or exhausted episode, an open arming claim, a +# changed or dead session lock, a moved recovery generation, or an absent/later +# beacon all fail it, so a genuine lapse stays loud. Cursor autoarm homes have no +# Claude epoch ledger and fail this, keeping their existing fresh-beacon-only +# pull-guard contract. The rewake and beacon may both be older than grace: a +# legitimate handling turn can outrun grace, which is the false alarm this +# exists to stop. +fm_autoarm_midturn_healthy() { # <state-dir> [grace] + local state=$1 lock_pid recovery epoch_mtime beacon_mtime + [ -e "$state/.claude-autoarm-failure-notified" ] && return 1 + [ -e "$state/.claude-autoarm-failure-alarmed" ] && return 1 + fm_autoarm_ledger_read "$state" || return 1 + [ "$FM_AUTOARM_OUTCOME" = rewake ] || return 1 + lock_pid=$(sed -n '1p' "$state/.lock" 2>/dev/null || true) + [ -n "$FM_AUTOARM_SESSION" ] && [ "$FM_AUTOARM_SESSION" = "$lock_pid" ] || return 1 + fm_pid_alive "$lock_pid" || return 1 + fm_recovery_marker_read "$state/.watcher-down" || return 1 + recovery=${FM_RECOVERY_MARKER_TOKEN##*:} + [ -n "$FM_AUTOARM_RECOVERY" ] && [ "$FM_AUTOARM_RECOVERY" = "$recovery" ] || return 1 + epoch_mtime=$(fm_path_mtime "$state/.claude-autoarm-epoch") || return 1 + beacon_mtime=$(fm_path_mtime "$state/.last-watcher-beat") || return 1 + [ "$epoch_mtime" -ge "$beacon_mtime" ] +} + # Atomically publish this process as the owner of generation N+1, under one # short micro-mutex hold. Returns 0 with FM_AUTOARM_MY_GEN set on success, 2 # when a competing claimant won the race (the ledger holds an open claim), and @@ -1425,8 +1474,8 @@ fm_autoarm_claim_next() { # <state-dir> [grace] # ordering could permanently suppress a notice whose ledger write never won. # Returns 0 committed, 2 refused (superseded or required-marker failure), and 1 # unable (bounded contention or ledger-write failure). -fm_autoarm_write_owned() { # <state-dir> <gen> <outcome> [marker-file] - local state=$1 gen=$2 outcome=$3 marker=${4:-} lock epoch pid identity tmp i +fm_autoarm_write_owned() { # <state-dir> <gen> <outcome> [marker-file] [session-pid] [recovery-generation] + local state=$1 gen=$2 outcome=$3 marker=${4:-} session=${5:-} recovery=${6:-} lock epoch pid identity tmp i lock="$state/.claude-autoarm.lock" epoch="$state/.claude-autoarm-epoch" pid=${BASHPID:-$$} @@ -1444,8 +1493,11 @@ fm_autoarm_write_owned() { # <state-dir> <gen> <outcome> [marker-file] identity=$FM_AUTOARM_IDENTITY tmp="$epoch.tmp.$pid" if ! { - printf 'epoch=%s owner_pid=%s outcome=%s updated_at=%s\n' \ + printf 'epoch=%s owner_pid=%s outcome=%s updated_at=%s' \ "$gen" "$pid" "$outcome" "$(date +%s)" + [ -z "$session" ] || printf ' session_pid=%s' "$session" + [ -z "$recovery" ] || printf ' recovery_generation=%s' "$recovery" + printf '\n' [ -z "$identity" ] || printf '%s\n' "$identity" } > "$tmp" 2>/dev/null || ! mv -f "$tmp" "$epoch" 2>/dev/null; then rm -f "$tmp" 2>/dev/null || true @@ -1614,6 +1666,20 @@ fm_wake_clean_field() { } fm_wake_append() { + local status=0 + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + fm_wake_append_locked "$@" || status=$? + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return "$status" +} + +# fm_wake_append_locked <kind> <key> <payload> +# Locked core of fm_wake_append: appends the wake row under an already-held +# FM_WAKE_QUEUE_LOCK. Callers that must commit another durable record atomically +# with the append (holding this lock excludes the drain's acknowledgement, which +# deletes consumed rows under the same lock) acquire the lock once, run this and +# their own write, then release. +fm_wake_append_locked() { local kind=$1 key=$2 payload=$3 clean_key clean_payload epoch seq seq_file status local recovery_marker case "$kind" in @@ -1628,7 +1694,6 @@ fm_wake_append() { recovery_marker="$STATE/.watcher-down" status=0 - fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" _fm_recovery_marker_publish "$recovery_marker" downtime || status=$? if [ "$status" -eq 0 ]; then seq=$(cat "$seq_file" 2>/dev/null || echo 0) @@ -1641,7 +1706,6 @@ fm_wake_append() { if [ "$status" -eq 0 ]; then printf '%s\t%s\t%s\t%s\t%s\n' "$epoch" "$seq" "$kind" "$clean_key" "$clean_payload" >> "$FM_WAKE_QUEUE" || status=$? fi - fm_lock_release "$FM_WAKE_QUEUE_LOCK" return "$status" } diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 5e267492732..246a137474e 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -1135,17 +1135,17 @@ wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count- esac } -# busy_turn_over_age: 0 iff <task>'s latest turn-boundary wake marker is at least -# BUSY_TURN_MAX_SECS old. Ages the per-task turn-ended marker, the harness-neutral -# wake written by a verified turn-end producer or the cursor/agy native-idle -# detector; before any such wake arrives, ages the task's spawn record so a -# fresh task gets a bound. The caller checks that the pane is busy and routes a -# crossed bound through busy_turn_bound_check, never anything that touches the -# worker itself. +# busy_turn_over_age: 0 iff the last completed turn or explicit native-harness +# progress is at least BUSY_TURN_MAX_SECS old. Progress is actual observed model +# or tool activity, never a timer or a busy footer. It does not emit a wake or +# change semantic busy state. Before either marker exists, age the spawn record. +# The caller checks busy state and routes a crossed bound through inspection. busy_turn_over_age() { # <task> - local task=$1 f + local task=$1 f progress f="$STATE/$task.turn-ended" [ -e "$f" ] || f="$STATE/$task.meta" + progress="$STATE/$task.progress" + if [ -f "$progress" ] && [ "$progress" -nt "$f" ]; then f="$progress"; fi [ "$(age_of "$f")" -ge "$BUSY_TURN_MAX_SECS" ] } @@ -1176,7 +1176,7 @@ busy_turn_over_age() { # <task> # wording; a caller that reached the bounded cadence off pause tracking alone, with # no declaring verb left on the log, keeps the external-wait wording it always had. handle_paused_stale() { # <window> <task> <hash> - local win=$1 task=$2 h=$3 key statusf mtime age detail reason rf rf_age wait_line streak_file resurface_window + local win=$1 task=$2 h=$3 key statusf mtime age detail reason rf rf_age wait_line streak_file resurface_window until now due due_now=0 key=$(window_key "$win") printf '%s' "$h" > "$STATE/.stale-$key" : > "$STATE/.paused-$key" @@ -1185,12 +1185,14 @@ handle_paused_stale() { # <window> <task> <hash> statusf="$STATE/$task.status" mtime=$(stat_mtime "$statusf") case "$mtime" in ''|*[!0-9]*) mtime=$(date +%s) ;; esac - age=$(( $(date +%s) - mtime )) + now=$(date +%s) + age=$(( now - mtime )) rf="$STATE/.paused-resurfaced-$key" wait_line=$(last_status_line "$statusf") streak_file="$STATE/.paused-streak-$key" + due="$STATE/.paused-until-due-$key" if pause_streak_sync "$streak_file" "$wait_line"; then - rm -f "$rf" + rm -f "$rf" "$due" fi rf_age=$(age_of "$rf") # 999999 when no prior re-surface resurface_window=$(pause_resurface_window "$(pause_streak_count "$streak_file")") @@ -1201,8 +1203,23 @@ handle_paused_stale() { # <window> <task> <hash> detail="paused, awaiting external" reason="stale: $win (paused ${age}s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds)" fi - if [ "$age" -ge "$resurface_window" ] && [ "$rf_age" -ge "$resurface_window" ]; then + until= + if status_is_paused "$wait_line" && until=$(status_paused_until "$wait_line"); then + if [ "$now" -lt "$until" ] && [ "$age" -lt "$resurface_window" ]; then + triage_log "absorbed stale (paused, declared time not reached): $win" + return 0 + elif [ "$now" -lt "$until" ]; then + reason="stale: $win (paused ${age}s, awaiting external - the declared time is beyond the recheck cadence; confirm the wait still holds)" + elif [ "$(cat "$due" 2>/dev/null || true)" != "$until" ]; then + due_now=1 + reason="stale: $win (paused ${age}s, awaiting external - the declared clearing time has passed; confirm the wait cleared)" + fi + fi + if [ "$due_now" = 1 ] || { [ "$age" -ge "$resurface_window" ] && [ "$rf_age" -ge "$resurface_window" ]; }; then fm_wake_append stale "$win" "$reason" || exit 1 + if [ -n "$until" ] && [ "$now" -ge "$until" ]; then + printf '%s\n' "$until" > "$due" + fi date +%s > "$rf" pause_streak_bump "$streak_file" "$wait_line" wake "$reason" @@ -1270,7 +1287,7 @@ busy_turn_bound_check() { # <window> <task> <hash> <since-file> <escalation-fil clear_pause_state() { # <window-key> local key=$1 rm -f "$STATE/.paused-$key" "$STATE/.paused-rechecked-$key" "$STATE/.paused-resurfaced-$key" \ - "$STATE/.paused-streak-$key" + "$STATE/.paused-streak-$key" "$STATE/.paused-until-due-$key" } # The hash-scoped half of clear_pause_tracking: the stale suppressor, its wedge @@ -2393,11 +2410,9 @@ EOF # instead of the ordinary "signal:" below (other files in the same batch # keep the ordinary payload). The wake reason line itself, and every # harness-arm consumer that pattern-matches it, stays byte-identical - - # only the per-row payload changes, which is what - # docs/pi-supervision-branch.md's Pi-only branch dispatcher reads to keep a - # decision-owned row off the supervision branch (fm-branch-dispatch.ts, - # fm-primary-pi-watch.ts). Every other harness and script keeps seeing the - # exact same "signal:$files" wake it always has. + # only the per-row payload changes. The supervision branch dispatcher uses + # it to exclude decision-owned rows, and the away daemon passes it through + # handle_durable_wakes to handle_wake for once-per-declaration escalation. # shellcheck disable=SC2086 # same space-separated status-path list if afk_present || [ "$signal_actionable" -eq 0 ] \ || { ! signal_crew_provably_working $files && ! signal_turnend_panes_churned $files; }; then diff --git a/docs/agent-control.md b/docs/agent-control.md index 532767e6aa6..21aa5f23cf9 100644 --- a/docs/agent-control.md +++ b/docs/agent-control.md @@ -102,7 +102,8 @@ Switching harness is therefore one ordinary relaunch rather than a separate mech zellij, orca, and cmux are refused rather than reported as successful blind. - An ambiguous or unreadable endpoint state refuses. Only a positively classified state acts. -- `fm-spawn --relaunch` independently refuses unless the recorded endpoint is positively agent-free and its shell is sitting in the recorded worktree, so a replacement can never join a live agent or start outside the copy holding the work. +- `fm-spawn --relaunch` independently refuses unless the recorded endpoint is positively agent-free, so a replacement can never join a live agent. + It also requires the shell to be in the recorded worktree: tmux refuses immediately when it is not, while Herdr sends one `cd` to the recorded path and refuses unless a subsequent path read confirms the move. ## Capability matrix diff --git a/docs/architecture.md b/docs/architecture.md index 3a37c8da36a..7dd66a53364 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -88,7 +88,7 @@ A secondmate home's terminal child ledger lines, PR registrations, captain holds Absorbed wakes advance their suppression markers, log to `state/.watch-triage.log`, and keep the watcher blocking without a queue record or LLM turn. Each `fm-wake-drain.sh` presentation runs the same liveness guard as the supervision scripts, so a lapsed watcher chain surfaces even on a turn that only handles queued wakes. Routine watcher polling, supervision no-ops, elapsed waiting time, and absorbed benign wakes stay silent. -A declared external wait or verified captain-held transfer trades that silence for one bounded recheck per pause window, naming which human the wait is on, so neither a forgotten pause nor a forgotten hold can remain invisible indefinitely. +A declared external wait or an attended verified captain-held transfer trades that silence for one bounded recheck per pause window, naming which human the wait is on; while the away-posture record exists, captain-held work waits without rechecks and remains visible in the return brief. Crew status files are append-only wake-event logs, not current-state fields. Because of that, a per-wake read of only the latest line can bury an earlier still-open `needs-decision`/`blocked` under later unrelated appends; `fm-wake-drain.sh` prints a separate, fleet-wide OPEN DECISIONS section on every presentation (including the empty-queue path session-start relies on), built through `fm-classify-lib.sh`'s cursor-backed incremental scan using the authoritative `status_open_decisions` fold semantics so the buried decision keeps surfacing until it is explicitly resolved while each presentation folds only new status-log appends. The drain coordinates that fold and its annotations through a locked fleet-wide snapshot whose `.status-presentation-cursor` manifest records each status file's identity plus independent annotation and outcome-backstop byte offsets. @@ -113,6 +113,7 @@ Pre-transition briefs have no branch marker, work items do not bind branches, an [`bin/fm-nm-run-lib.sh`](../bin/fm-nm-run-lib.sh)'s header owns the exact branch, head, and pipeline-custody attribution rules, and the script header owns the current-versus-historical source, lookup-conclusiveness, cache, and source-precedence rules. Architecturally, an in-progress pipeline may advance the crew tip without losing attribution, and while the pipeline holds custody of the branch an active run keeps attribution on any head at all, including one this worktree cannot resolve or one that diverged; outside that custody the library also owns the narrowly approved ledger-anchored current-state path, while teardown remains strict. An inconclusive run lookup remains distinct from a confirmed absence and may replay that crew's recent observed step under source `run-step-degraded` only within a finite window after endpoint-liveness and exact-busy checks, preserving wedge detection when the crew stops, the lookup stays unavailable, or the reported run head remains unresolved in the worktree. +It also owns which binding run wins when more than one recorded run binds to the same worktree: a live run outranks a terminal one, so a crashed run sitting at the worktree's own commit never reports a healthy task as failed while its live successor is still validating. During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`. The most recent recognized ci log marker wins, so checks-green monitoring reports done while a later re-arm, failed-check, or issue marker returns the crew to working. A terminal failed run whose only failure is the ci monitor step, after every substantive step completed and the same marker reads checks green, also reports done with the run's PR URL, because a monitor whose only remaining job is to observe a human merge decision must not convert the absence of that decision into a failure verdict. @@ -170,7 +171,15 @@ It leads with a prominent bordered tangle banner, while `bin/fm-guard.sh` owns t On every verified primary harness, tracked hook integration gives the primary session a push-based backstop: when work, a process-event source, a registered custom check, Relay polling, or a pending wake queue needs supervision and no supervision owner provably holds this home with a fresh beacon, blocking-capable Stop hooks block and nonblocking turn-end integrations force one bounded follow-up. The guard covers the main primary and genuinely marked secondmate homes, exempts child crewmate/scout worktrees, is loop-safe per harness, and is documented in [turnend-guard.md](turnend-guard.md). -A presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) extends this for walk-away supervision: the `/afk` skill starts it through the tracked foreground helper `bin/fm-afk-start.sh`, after which the watcher reverts to daemon-managed one-shot mode and the daemon self-handles routine wakes in bash. +Away mode is a posture of the one supervision session, recorded in `state/.afk-contract` by `bin/fm-afk-contract.sh` after the captain confirms a read-back of their away words and mandate clauses, and announced at entry as hold-for-return only because no phone channel exists. +The record owner's header is the single owner of the record schema and clause fields, and by the captain's mandate no static parser reads the clause text: the object and precondition are recorded verbatim, structural presence and the verb list are checked, and the coarse best-effort never-set flag can miss spellings including joined compounds such as `oneTimeCode`. +That scan flags a clause without refusing it and is not authoritative; never-set, forbidden-action, and precondition judgment belongs to the supervision session at execution time in phase 4. +Forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, and no recorded clause is authority by itself. +The record's presence is the posture on every harness, `bin/fm-afk-launch.sh` owns entry and exit, and `bin/fm-afk-return.sh` archives the record and renders the return brief (supervisor health first, then the recorded clauses, what waits on the captain, what could not be fixed, what was handled, and cost) from the outcome store, the held set, and the status logs. +Declared external waits may name their clearing time with `until`; the shared declared-wait cadence above bounds those rechecks and continues to cover verified held transfers. +This release records clauses and does not execute them. +Pi and OMP extensions yield their cycle to the away daemon while `state/.afk` exists, as owned by [`watcher-continuity.md`](watcher-continuity.md). +A presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) extends this for walk-away supervision on every supported primary harness: the `/afk` skill starts it through the tracked foreground helper `bin/fm-afk-start.sh` once the record exists, after which the watcher reverts to daemon-managed one-shot mode and the daemon self-handles routine wakes in bash. The watcher and daemon share `bin/fm-classify-lib.sh` for captain-relevant status verbs, declared-wait vocabulary (a `paused:` external wait and a verified `captain-held` transfer alike, through one combined predicate), and status-scan primitives. Terminal verbs remain captain-relevant, while a nonterminal progress verb cannot become terminal merely because its prose contains a legacy free-text token such as `merged`; bare legacy free-text lines remain compatible. Both supervisors classify the status bytes appended since they last classified that log, never its last line alone, and report every actionable event through the captured endpoint before committing that position. @@ -182,7 +191,7 @@ The daemon's declared-wait window ages against the crew's own latest status line A wake already decorated as a possible wedge does not override the daemon's own declared-wait verdict either, so a declaration keeps its pane on the recheck cadence instead of the wedge cadence. In away mode, seen-status dedupe does not clear possible-wedge aging for nonterminal progress, so housekeeping still re-escalates an unchanged idle pane at the configured bound - subject to the same run-progress hold the always-on watcher applies, through the same `crew_wedge_progress` owner, so a walk-away digest does not report a possible wedge for a worker parked on a run that is demonstrably moving. Away-mode housekeeping has no worktree-write deferral of its own, so while `state/.afk` exists a quiet crew that is writing its own worktree still escalates as a possible wedge at that bound. -The daemon escalates captain-relevant events, plus a bounded recheck for a declared pause or a verified captain-held transfer that is still declared, naming which human that wait is on, as one batched, single-line digest using the canonical `away-supervisor` kind from `bin/fm-operational-input.sh` so firstmate can distinguish it structurally from real messages. +The daemon escalates captain-relevant events, plus a bounded recheck for a declared external wait that is still declared, as one batched, single-line digest using the canonical `away-supervisor` kind from `bin/fm-operational-input.sh` so firstmate can distinguish it structurally from real messages; captain-held transfers remain silent until return while the posture record exists. Its supervisor injection path supports tmux and herdr panes, with `FM_SUPERVISOR_BACKEND` and `FM_SUPERVISOR_TARGET` resolved independently from the task-spawn backend. Pane existence, busy checks, composer checks, capture, and verified submit route through `bin/fm-backend.sh`: tmux keeps the same submit core used by the tmux send backend, while herdr uses native agent-state submit confirmation on idle baselines, a composer empty fallback when native stays idle, and a pre-Enter rendered-footer transition when that baseline is unavailable. The retries-exhausted queued-Enter decision is owned by `fm_composer_queued_enter_verdict` in `bin/fm-composer-lib.sh`; tmux and herdr provide only their backend-specific busy signals. @@ -192,7 +201,7 @@ The daemon injects only into an affirmatively `empty` composer, so every other o The current operator boundary is in [Composer and injection safety](herdr-backend.md#composer-and-injection-safety). Unsupported supervisor backends refuse at daemon startup. Stalled escalation delivery writes `state/.subsuper-inject-wedged` and attempts a configured backend-independent active alert after `FM_MAX_DEFER_SECS` instead of silently deferring forever. -On an unmarked return, `bin/fm-afk-return.sh` owns ordered shutdown, durable catch-up evidence, and the fail-closed gate that keeps ordinary work behind every live firstmate-actionable blocker. +On an unmarked return, `bin/fm-afk-return.sh` owns ordered shutdown, the record archive, durable catch-up evidence, the return brief, and the fail-closed gate that keeps ordinary work behind every live firstmate-actionable blocker the away session could not fix. `fm-send.sh` delivers every remote text steer and ordinary local text steer as a durable steering-inbox record plus a best-effort constant doorbell line (`bin/fm-task-inbox-lib.sh`). The doorbell line is a shell no-op and is never typed into an endpoint classified as dead or missing; that record surfaces once for recovery instead of walking the re-ring ladder (`bin/fm-task-inbox-lib.sh` header). Its local-only typed plane - harness-native invocations and explicit backend targets - selects a pre-Enter popup-settle for slash commands and for codex `$...` skill invocations using metadata-routed target `harness=` values, then adds its own `FM_SEND_SETTLE` pause after successful typed sends so immediate peeks catch the receiving turn starting; the sub-supervisor uses only the shared submit core and does not pay that post-submit pause. @@ -307,10 +316,9 @@ Secondmates are idle by default: after startup recovery reconciles only work alr When called with `FM_HOME=<this-firstmate-home>` or when `FM_HOME` is already set to the active firstmate home, metadata-routed `fm-send.sh` requests to a live `kind=secondmate` use the live-charter-compatible `from-firstmate` carrier owned by `bin/fm-operational-input.sh`, so the secondmate returns terse answers through status lines and detailed answers through docs plus status pointers instead of replying only in its own chat. The parent guards every reply-bearing marked request against a missing correlated report without reading the secondmate conversation; `bin/fm-pending-reply-lib.sh` owns the correlation, recovery, escalation, and retention contract, while `bin/fm-send.sh` owns the explicit fire-and-forget exception. Explicit backend-target sends and direct human typing stay unmarked, so captain intervention in a secondmate pane remains conversational. -After seeding a secondmate, `fm-backlog-handoff.sh` validates the fleet-specific handoff, atomically delegates already-judged in-scope queued item moves to `tasks-axi mv`, and then sends a marked routed-work wake through the receiver's recorded endpoint. -A durable move with a missing, failed, or unresolved wake is reported as failure rather than success; rerunning the same handoff recovers known-undelivered wake intent without moving the item again, while an unresolved delivery is never blindly resent. -Remote routes move that dependency-closed set into a non-dispatchable backlog-format outbox before transfer, then use an idempotent remote receive under the destination backlog's own lock and retain the outbox until the receiver wake is confirmed. -The script header owns the wake correlation and recovery mechanics; `tests/fm-backlog-handoff.test.sh` and `tests/fm-remote-backlog-handoff.test.sh` pin the local and remote delivery boundaries. +After seeding a secondmate, `fm-backlog-handoff.sh` validates the fleet-specific handoff, atomically delegates already-judged in-scope queued item moves to `tasks-axi mv`, and then attempts a marked routed-work wake through the receiver's recorded endpoint. +The [`fm-backlog-handoff.sh`](../bin/fm-backlog-handoff.sh) header owns route-specific wake outcomes, remote outbox release after receipt, and stable wake-correlation retry behavior. +`tests/fm-backlog-handoff.test.sh` and `tests/fm-remote-backlog-handoff.test.sh` pin the local and remote delivery boundaries. An unreachable remote host is unknown rather than dead, preserves its route and durable work, and is never failed over or relaunched locally. Idle secondmate panes are healthy; teardown is explicit and refuses while the secondmate home has in-flight work unless the captain has approved discard with `--force`. @@ -469,4 +477,4 @@ Use `/stow` before an intentional reset when the conversation may hold durable k ## Development notes The current watcher reliability work combines always-on bash triage with a durable queue for actionable wakes, generation-bound post-handling acknowledgement, deterministic re-arm recovery after watcher downtime, a race-proof singleton lock, duplicate self-eviction, drain-time liveness assertion, and a self-verifying tracked-child arm wrapper. -The presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) provides walk-away supervision via the `/afk` skill while reusing the same shared wake classifier as the always-on watcher. +The away posture is the record `bin/fm-afk-contract.sh` owns; on the harnesses other than Pi the presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) still provides walk-away delivery via the `/afk` skill while reusing the same shared wake classifier as the always-on watcher. diff --git a/docs/captain-hold-lifecycle.md b/docs/captain-hold-lifecycle.md index dd5bdb5ae65..21a11805706 100644 --- a/docs/captain-hold-lifecycle.md +++ b/docs/captain-hold-lifecycle.md @@ -7,15 +7,15 @@ This document records the deterministic mechanism, structured surfaces, compatib A decision is not a separate thing in this system: it is an ordinary backlog task held for the captain, and the task id is the identity every surface and channel uses. `bin/fm-captain-hold.sh` is the only lifecycle command layered on that primitive. -The command addresses the active home's configured data directory the same way `bin/fm-backlog-transition-lib.sh` addresses every backlog transition, so the existing backlog remains the only durable work database and a secondmate-owned captain call stays in the secondmate home. +The command addresses the active home's configured data directory, so the existing backlog remains the only durable work database and a secondmate-owned captain call stays in the secondmate home. It never reads report bodies, review artifacts, terminal output, or chat. The `hold` subcommand is the mandatory captain-hold creation path: it uses an existing task or creates one when nothing exists to hold, records its UTC hold-set timestamp as the leading line of the task body, then invokes the underlying tasks-axi hold operation and verifies both records. Publishing the stamp first ensures a snapshot cannot observe a newly captain-held task without the timestamp that defines its age. Retries of an active hold preserve its hold-set timestamp, while re-holding released work starts a new timestamped lifecycle; a closed task is refused rather than reopened, and `--until` stores the captain's own deferral date through tasks-axi's date gate. -The `answer` subcommand records the captain's exact words and closes the call in the same act. -It requires a non-empty captain decision file of at most 8192 bytes, durably writes a resolution block carrying the decision digest and a `Resolution mode:` while retaining the leading hold-set stamp until `tasks-axi done` or, under `answer --release`, `tasks-axi unhold` succeeds, then restores the successful record's resolution-first body ordering (the previous body remains preserved below the block and archived through tasks-axi `--archive-body`). +The `answer` subcommand records the captain's exact words and resolves the call in the same act: it closes a question-shaped call, while `answer --release` frees a captain-gated work item to proceed without completing it. +It requires a non-empty captain decision file of at most 8192 bytes, durably writes a resolution block carrying the decision digest and a `Resolution mode:` while retaining the leading hold-set stamp until the selected `tasks-axi done` or `tasks-axi unhold` transition succeeds, then restores the successful record's resolution-first body ordering (the previous body remains preserved below the block and archived through tasks-axi `--archive-body`). If the close is interrupted, the still-held task therefore keeps its original age basis. A matching retry also completes any resolution-first normalization left unfinished after the close itself succeeded. An exact retry is idempotent only when the requested close mode matches the newest record; a drifted answer or mode mismatch is rejected, while a re-held task accepts a new answer as a new record on top. @@ -36,15 +36,20 @@ The `--force` path remains the explicit captain-approved discard escape hatch. The policy prefers holding the very work item a question gates, so the backlog row a finished task's cleanup is about to close is routinely the captain's own call. `bin/fm-teardown.sh` therefore asks the read-only `open` subcommand before its automatic close: exit 0 means the row is still an open captain call (not Done, `hold_kind: captain`), 1 means it is not, and 2 means the answer could not be established, which teardown treats as a refusal before any destructive step rather than as permission to close. -On 0 only the close changes: after cleanup and still under the task's own lock, teardown records one `Deliverable of the finished work: ...` line at the end of the task body and runs `tasks-axi reopen`, so the row returns to Queued with its hold intact and remains on the appropriate Captain's Call or Charted Next decision surface instead of reading as work still under way. -The pending-close record teardown already stages before destructive cleanup carries that intent as a `mode=retain` line, so an interrupted cleanup replays the retention at the next session start through the same record, validator, and lock as an ordinary close and never closes the row; an answer that closed the row first simply retires the record. -`--force` does not lift the deferral, because it authorizes discarding unlanded work, never the captain's question; only `answer` with the captain's words or evidence-backed `reconcile close` closes the call. +On 0 only the close changes: after cleanup and still under the task's own lock, teardown records one `Deliverable of the finished work: ...` line at the end of the task body, copies a supported pull request or canonical `data/<id>/report.md` into the row's structured artifact fields, and runs `tasks-axi reopen`, so the row returns to Queued with its hold intact and remains on the appropriate Captain's Call or Charted Next decision surface instead of reading as work still under way. +The pending-close record teardown already stages before destructive cleanup carries that intent as a `mode=retain` line, so an interrupted cleanup replays the retention at the next session start through the same record, validator, and lock as an ordinary close and never closes the row; if the captain answers before replay, `answer` validates that record and copies any supported retained pull request or report into the row before closing it, after which replay retires the record. +Two retained-delivery gaps remain bounded by tasks-axi 0.2.5 and are recorded for separate upstream work rather than representing defects introduced by this branch. +A retained local-only delivery cannot reach the row because `--note` exists on `tasks-axi done` but not on `tasks-axi update`, while the durable pending-close record carrying that note is retired when retention completes. +A relocated retained report cannot reach the row because tasks-axi accepts only `data/<id>/report.md`: `done` reports `Task report link must be a data/<id>/report.md path`, and `update` reports `--report must be a data/<id>/report.md path`. +When an interrupted retention leaves such a relocated report in the validated pending-close record, `answer` skips only that known-unsupported row artifact and closes normally, so the delivery remains absent from Recently Landed instead of wedging the captain's answer. +A pending-close record that fails validation outright is a different case and still refuses the answer, but the refusal names the record and the validation reason so the captain can repair it rather than facing a bare failure. +`--force` does not lift the deferral, because it authorizes discarding unlanded work, never the captain's question; only `answer` with the captain's words or evidence-backed `reconcile close` resolves the call, by either closing the question or releasing the gated work. `bin/fm-backlog-transition-lib.sh` owns the transition and its record, and `bin/fm-captain-hold.sh --help` owns the predicate's contract. -## Answer-time closure +## Answer-time resolution -"A keyed answer closes its matching captain-held task" is one capability with one owner. -`answers` is its channel-agnostic entry point: it reads `<task-id>\t<answer>\t<label>[\t<mode>]` lines and closes each named task through the same `answer` path, so every guard applies identically no matter which channel the answer arrived on. +"A keyed answer resolves its matching captain-held task" is one capability with one owner. +`answers` is its channel-agnostic entry point: it reads `<task-id>\t<answer>\t<label>[\t<mode>]` lines and resolves each named task through the same `answer` path, so every guard applies identically no matter which channel the answer arrived on. The optional mode column carries a card-declared close: `done` (default) completes the task and `release` lifts the hold so held work resumes; any other value is skipped. A key that names no task, names a task that is not captain-held, or names a task already closed is reported as `skipped:` and feeds nothing; a replay whose answer and requested close mode match the newest record is an idempotent `closed:`, while a mode mismatch is skipped; and the command exits nonzero when any key was skipped. `--source` is provenance text recorded in the durable decision, never a behavior switch, and the command carries no per-channel branch. @@ -100,7 +105,7 @@ Three checks run, all on exact identity and none on prose: - The card's key is the captain-held task id, so `bin/fm-captain-hold.sh open --distinguish-absent` is asked whether that task is still an open captain call. Exit 1 - present but closed, or no longer held for the captain - drops the card. - Exit 2 means the answer could not be established and exit 3 means the task is absent from the main backlog; both keep the card, because a card wrongly shown is recoverable and a call wrongly hidden is not. + Exit 2 means the answer could not be established and exit 3 means the task is absent from the main backlog, which includes a home carrying no backlog file at all; both keep the card, because a card wrongly shown is recoverable and a call wrongly hidden is not. - The payload's own `landed` rows are the recently-landed artifacts. A decision card whose task id or `pr_url` appears among them has already shipped its subject, so it drops. - A version decision can carry a structured `subject` with an artifact and numeric three-part version. @@ -141,9 +146,16 @@ Three accepted limits remain deliberate: - Cross-home summaries remain bounded by `FM_SNAPSHOT_SECONDMATE_DECISIONS` and `FM_SNAPSHOT_SECONDMATE_QUEUED`; a remote deferred hold beyond those bounds is not exported, so it can be neither gated nor revealed. Re-holding through the wrapper with `--until` remains the durable fix rather than relying on the projection safety net. -Recently Landed excludes a record that closed while still held for the captain (surviving `hold-kind: captain` on a Done row), so answered questions do not masquerade as shipped work; a work item released before completion keeps no hold annotations and lands normally. +[`bin/fm-landed-lib.sh`](../bin/fm-landed-lib.sh) owns Recently Landed's shared selection and artifact-display compatibility rules. +A local-only landing's note is written by `tasks-axi done --note` as the last of the row's indented body lines rather than into the row title, so the snapshot reads that final line as the note as well as parsing the title, and the landing is published carrying its recorded note. +A body that carries a captain resolution record is the captain's own prose and is never mined for that note, so a decision worded `local main` does not become a delivery artifact. The projection remains read-only and uses the canonical snapshot's structured fields, including the machine-written hold-set timestamp. +The window between a merge landing and cleanup is an accepted structural residual rather than an oversight. +That local window is normally only seconds wide and requires re-holding a task whose merge has just landed. +A re-hold inside the window makes cleanup retain the row rather than publish it, so the delivery is omitted until the stale hold is cleared from that row. +Queued forge merges cannot be covered locally because the forge performs the merge asynchronously after the local command has returned, when no lock this code could hold would still be held. + ## Record divergence A captain call can have two records, and closing one does not close the other. @@ -178,7 +190,8 @@ The shim recognizes an exact replay of a pre-collapse routed resolution by its h ## Verification record The focused end-to-end regression suite is `tests/fm-captain-hold-lifecycle.test.sh`, using only synthetic `sample` identities and decision text. -It proves: cleanup of a finished task whose own row is the captain call leaves that call open, queued, held, carrying its deliverable, and visible in Bearings' Captain's Call, leaves no pending record behind, survives a `--force` cleanup, and closes only when `answer` records the captain's words, while an ordinary finished task in the same home still closes with its report link; an interrupted cleanup leaves the row In flight and untouched with its pending record, and the next session start retains it as queued and held with the deliverable recorded; a relocated data directory keeps the retention in its one configured backlog; a ship row whose captain hold cannot be read refuses cleanup before any destructive step and surfaces the read failure; the reconstructed silent-divergence case is signalled - a status resolution over a still-open captain-held task reaches both `diverged` and the drain's `RECORD DIVERGENCE` section, under the collapsed and the legacy identity alike, while the backlog task, its hold, and the status log all survive the report unchanged and the printed hint names both reconciliation directions; the false-signal boundary holds - a captain call with no routed work item, a verified `captain-held` transfer, a still-open status decision, an already answered call, and an ordinary task whose keyed question was answered all stay silent; a report-only unresolved captain call refuses `--none` completion before teardown can erase the source; non-forced scout and design teardown always requires the durable inventory verification; the recorded-answer guard (a bare `tasks-axi done` close fails `verify` until `answer` records the captain's word, and an ordinary finished task cannot be dressed up as an answered call); answer-time closure through a bound channel with task-id keys, including the `release` close mode, mode-matched replay idempotence, and the refusal of drifted, mode-mismatched, absent, unheld, and already-closed keys; the chat channel reaching the same intake; hold-set stamping that precedes visible hold state, preserves an active lifecycle's timestamp, and resets after release; interrupted answer closure retaining the stamp until close and restoring resolution-first ordering on retry; deferral through `--until` leaving `captain_actionable` false until due; and every legacy path (composed identities through the shim, pre-collapse `decision_keys=` metadata, routed-resolution replay, and a concrete-origin binding). +It proves: cleanup of a finished task whose own row is the captain call leaves that call open, queued, held, carrying its deliverable, and visible in Bearings' Captain's Call, leaves no pending record behind, survives a `--force` cleanup, and closes only when `answer` records the captain's words, while an ordinary finished task in the same home still closes with its report link; an interrupted cleanup leaves the row In flight and untouched with its pending record, the next session start retains it as queued and held with the deliverable recorded when it remains unanswered, and an answer before replay preserves that record's completed report while closing the call so the next session start retires the satisfied record without losing the delivery from Recently Landed; a pending-close record that cannot be validated refuses the answer while naming the record and the reason; a relocated data directory keeps the retention in its one configured backlog; direct PR and local-only merge entrypoint calls refuse a still-held task before reaching the forge or moving local main, while a released pull request passes the guarded PR entrypoint, cleanup records its artifact, and Recently Landed publishes it; an ordinary release still survives zero-retention cleanup and archives when configured; a ship row whose captain hold cannot be read refuses cleanup before any destructive step and surfaces the read failure; the reconstructed silent-divergence case is signalled - a status resolution over a still-open captain-held task reaches both `diverged` and the drain's `RECORD DIVERGENCE` section, under the collapsed and the legacy identity alike, while the backlog task, its hold, and the status log all survive the report unchanged and the printed hint names both reconciliation directions; the false-signal boundary holds - a captain call with no routed work item, a verified `captain-held` transfer, a still-open status decision, an already answered call, and an ordinary task whose keyed question was answered all stay silent; a released call whose decision text is `local main`, closed with no artifact, is not published as a local-only landing; a report-only unresolved captain call refuses `--none` completion before teardown can erase the source; non-forced scout and design teardown always requires the durable inventory verification; the recorded-answer guard (a bare `tasks-axi done` close fails `verify` until `answer` records the captain's word, and an ordinary finished task cannot be dressed up as an answered call); answer-time resolution through a bound channel with task-id keys, including the `release` mode, mode-matched replay idempotence, and the refusal of drifted, mode-mismatched, absent, unheld, and already-closed keys; the chat channel reaching the same intake; hold-set stamping that precedes visible hold state, preserves an active lifecycle's timestamp, and resets after release; interrupted answer closure retaining the stamp until close and restoring resolution-first ordering on retry; deferral through `--until` leaving `captain_actionable` false until due; and every legacy path (composed identities through the shim, pre-collapse `decision_keys=` metadata, routed-resolution replay, and a concrete-origin binding). +The suite does not test the accepted merge-to-cleanup re-hold window or asynchronous queued-forge landing because those events occur after the locally serialized merge command has returned. The markdown-to-beads migration family runs the same suite's beads fixture (bd-driven scratch graph, self-skipping on markdown-only tasks-axi installs) and proves: `verify` and `complete` resolve an attested legacy id through a migrated row's marker note, through the configured prefix when no row carries a note - naming the resolved row in the completion line - and through the marker note of a pre-collapse derived identity; a marker-noted row wins over an unrelated captain-held row occupying the bare prefix namesake; an unresolvable id is refused once naming the id (never an empty name); and the attested id stays in `decision_keys=` for idempotent re-verification. One case in that family needs no beads install and always runs: a stubbed tasks-axi that fails any markdown file override proves the captain-hold hold, answer, and close mutations reach a beads-configured home without one. @@ -191,5 +204,5 @@ That suite drives its Lavish session through a protocol-shaped stub, and `tests/ `tests/fm-classify-decision-key.test.sh` pins `status_key_closing_verb` itself: it separates a resolution from the durable-transfer close and from a still-open key, reports the last real transition across re-openings and both key positions, and treats a prose mention as no transition. -Projection regressions live in `tests/fm-fleet-snapshot-view.test.sh` (the total structured-only bucket classifier, hold-until parsing, kind-independent captain actionability, undated-hold aging, and title stripping) and `tests/fm-bearings-snapshot.test.sh` (default and expanded decision-bucket membership, deferral explanations, blocker-overflow disclosure, working-hold dual surfaces, remote-summary schema invalidation, and the landed exclusion by surviving captain-hold annotations). +Projection regressions live in `tests/fm-fleet-snapshot-view.test.sh` (the total structured-only bucket classifier, hold-until parsing, kind-independent captain actionability, undated-hold aging, and title stripping) and `tests/fm-bearings-snapshot.test.sh` (default and expanded decision-bucket membership, deferral explanations, blocker-overflow disclosure, working-hold dual surfaces, remote-summary schema invalidation, exact leading-kind inference, artifact-kind mismatch and answered-question exclusion, kind-bearing and kindless local-only landings publishing their recorded note, and scout-report precedence over competing pull-request links). The exact commands and their summarized outputs are recorded in the shipping PR's evidence; run the four suites above plus `tests/fm-send-resolve-key.test.sh`, `tests/fm-bearings-board.test.sh`, `tests/fm-procevent.test.sh`, and `bin/fm-lint.sh` to refresh this record, and `FM_BEARINGS_LAVISH_LIVE=1 tests/fm-bearings-board-lavish-live-e2e.test.sh` after a lavish-axi upgrade. diff --git a/docs/configuration.md b/docs/configuration.md index a2818cf2ddb..f0c0c571803 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -189,7 +189,7 @@ This preference is local to each Firstmate home and is not part of secondmate in On a Pi primary, an in-process supervision branch handles eligible task-local wake rows and selected heartbeat reviews while keeping main-only rows on the captain-facing path; [docs/pi-supervision-branch.md](pi-supervision-branch.md) owns its conversation lifecycle, row eligibility, mixed-queue dispatch, heartbeat routing, and pre-drain recheck. Supervision is default-on: once a Pi primary session owns this home's fleet lock, the branch is eligible for every task with no captain grant file required. A genuinely no-op heartbeat is absorbed in bash and never reaches Pi, and every watcher-failure alarm stays on the captain-facing main path. -Away mode still declines every wake offer, and a broken branch still falls back to today's wake-to-main path. +A legacy `state/.afk` daemon flag still declines every wake offer, the away-posture record alone does not, and a broken branch still falls back to today's wake-to-main path. The branch's role stays bounded exactly as the captain-approved architecture set it: it cannot merge a PR, land local work, or freshly spawn, and every existing captain gate remains unchanged. Homes on any other primary harness never load this feature and are entirely unaffected. `AGENTS.md`'s `state/` inventory routes the branch's runtime files to their format and lifecycle owners. @@ -213,11 +213,15 @@ A provider that exists only because an extension registered it inside the captai Stored OAuth and API-key credentials retain their native credential type because Firstmate never copies, converts, installs, or overwrites credentials for the branch runtime. The file holds one `<provider>/<model-id>` line followed by one newline, split at the first `/` so a provider-qualified model id such as `openrouter/anthropic/claude-sonnet-4-5` survives intact. An absent, unreadable, or unparseable file means no pin, and the branch then follows main's own current model, applied explicitly and live whenever main changes models mid-session. +When main uses `codex-native`, following main explicitly selects the same model through ordinary Pi's `openai-codex` provider, so the background branch owns an independent Pi conversation. +If that ordinary Pi model is unavailable, the branch refuses to build and returns the notification to main; it never inherits the main native thread or silently selects a different model. +Picking "Follow main" under a `codex-native` main reports that same `openai-codex` model, or that same refusal, because the command and the branch build share one follow rule. +A `codex-native` branch pin is refused and excluded from the picker. A valid pin wins over main and remains unaffected by main's model changes. Picking "Follow main" removes the file, and the command writes a pin at mode `0600` and replaces it atomically so a failed write leaves the current choice unchanged rather than claiming persistence. The file's current state decides the branch model on every branch build - the new conversation each main session start opens and the reopen after a model or effort change inside one session - and it overrides Pi's restore of whatever model a reopened branch session recorded, so the choice survives all of them. That override is what keeps "Follow main" honest: a branch conversation that ran under an earlier pin still records that model, so clearing the file explicitly applies main's model rather than letting the reopened session restore the old one. -Only when main's own model is unknown, or this home's stored credentials cannot run it in the isolated branch runtime, does an unpinned build fall back to passing no override at all, which is the behavior from before this file existed; the wake is never lost over model choice, and the command says plainly when main's model could not be applied instead of reporting a change that did not take effect. +For ordinary Pi providers, only when main's own model is unknown, or this home's stored credentials cannot run it in the isolated branch runtime, does an unpinned build fall back to passing no override at all, which is the behavior from before this file existed; the wake is never lost over model choice, and the command says plainly when main's model could not be applied instead of reporting a change that did not take effect. A pin naming a model Pi cannot hand back, because the model is unknown or has no configured credentials, is never silently downgraded onto main's model: the branch refuses to build and rejects the accepted wake to the watcher's captain-facing main path, exactly as any other unreachable branch does. Picking also releases the live branch so the next wake reopens this session's own branch conversation under the new model without waiting for a session replacement. @@ -240,17 +244,19 @@ Both choices are local to each Firstmate home and are not part of secondmate inh ## Backlog backend (.tasks.toml / config/backlog-backend) The tracked `.tasks.toml` pins the default `tasks-axi` markdown backend to `data/backlog.md`, with `done_keep = 10` and an archive at `data/done-archive.md`. -When the default backend is selected and compatible `tasks-axi` is on `PATH`, firstmate uses its verbs for routine backlog mutations. +A home may instead select another tasks-axi adapter such as Beads through its own `.tasks.toml` or `TASKS_AXI_BACKEND`; firstmate still uses only tasks-axi verbs for routine backlog reads and mutations, and the adapter maps `start` and evidence-bearing `done` transitions to its native statuses and evidence fields. When the automatic transition gate applies, dispatch and completion are not separate operator actions: each moves its work item inside the same run that creates or removes the task's record, so the ordinary successful path cannot leave the backlog and live task set out of sync ([`bin/fm-backlog-transition-lib.sh`](../bin/fm-backlog-transition-lib.sh)). Under that gate, dispatch accepts only an unheld, unblocked Queued or In flight item in this home; a missing, Done, held, or dependency-blocked item is refused before any endpoint or local copy is created. Completion refuses to report success until the item is closed, and session start reconciles this home's own books after an interrupted run. When a spawn is interrupted after launch delivery began, its exit path re-reads the paired task record and the backlog row under the same per-task lock as the commit, repairs a row the commit believed it had moved, and reports only what was verified or honestly attempted, never intent phrased as outcome ([`bin/fm-spawn.sh`](../bin/fm-spawn.sh); [`tests/fm-backlog-atomicity.test.sh`](../tests/fm-backlog-atomicity.test.sh)). -Automatic transition mutations address the configured `<data>/backlog.md` explicitly from the data directory's parent, keeping relocated backlog configuration, archives, and relative scout-report links together. -That explicit markdown file belongs to the markdown backend only: a home whose resolved tasks-axi backend is non-markdown never receives a markdown file override and requires no markdown backlog file, so its reads, probes, and mutations address the backend its own configuration selects. +Automatic transitions run from the configured data directory's parent, letting that home's effective tasks-axi configuration address its selected adapter while keeping relative scout-report links rooted there. +A markdown backlog is additionally addressed by an explicit `--file` at `<data>/backlog.md`, so the change lands in the home that owns the task regardless of the caller's working directory. +Any other configured adapter is addressed by that root alone, because `--file` would override the adapter's own workspace path. +The gate does not apply to persistent secondmates, manual-backend homes, or markdown homes without a backlog file, preserving their existing persistent-agent, manual, or ad-hoc lifecycle behavior while configured non-markdown adapters remain active without that file. Migrated-hold resolution on a beads home reads its graph path, binary, and prefix from the root `.tasks.toml` `[beads]` section only, and refuses (rc=2) when the beads backend is selected elsewhere (a `TASKS_AXI_BACKEND` override or user-level config) with no root-level `[beads]` section. -The gate does not apply to persistent secondmates, manual-backend homes, or markdown homes without a backlog file, preserving their existing persistent-agent, manual, or ad-hoc lifecycle behavior. -On an automatic-backend home with a backlog, missing or incompatible `tasks-axi`, an unresolvable configured data directory, or one containing a control byte fails lifecycle work before mutation. -Secondmate handoffs bypass that routine-backend choice: `fm-backlog-handoff.sh` keeps only its own fleet-level validation, delegates the item move to `tasks-axi mv`, and requires a verified receiver wake after a new move becomes durable. +On an automatic-backend home, missing or incompatible `tasks-axi`, an unresolvable configured data directory, or one containing a control byte fails lifecycle work before mutation. +An unreadable backend configuration can refuse lifecycle work before the no-backlog exemption applies; repair the configuration named in the diagnostic ([backend resolution contract](../bin/fm-tasks-axi-lib.sh)). +Secondmate handoffs bypass that routine-backend choice: `fm-backlog-handoff.sh` keeps only its own fleet-level validation and delegates the item move to `tasks-axi mv`; its [script header](../bin/fm-backlog-handoff.sh) owns route-specific wake outcomes and remote outbox release. It moves in-scope `## Queued` items only and refuses `## In flight` and historical `## Done` records, which stay with their home for pruning or archiving. Handoff item bodies must use at least two leading spaces, and the helper refuses a selected item with a single-space or tab-indented continuation rather than risk orphaning it. Because bootstrap requires `tasks-axi` on `PATH` on every profile, that delegation works fleet-wide, and the `config/backlog-backend=manual` knob governs firstmate's own hand-editing of its backlog, not this validated helper. @@ -258,8 +264,8 @@ Compatible means the installed build passes the shared version and feature probe Bootstrap requires compatible `tasks-axi` on every profile; see "Toolchain" below for missing-tool reporting and silent default-backend behavior. Set the local, gitignored `config/backlog-backend` file to `manual` to force manual backlog editing and suppress the verbose `BOOTSTRAP_INFO: tasks-axi available` fact, not missing-tool reporting. A `manual` home owns its backlog file outright: the lifecycle transitions above are skipped there, dispatch and completion never fail over the file's contents, and a completed teardown prints the hand edit that is owed instead. -Absent or `tasks-axi` selects the default tasks-axi backend. -The file format is unchanged in both modes; tasks-axi and manual edits produce the same `## In flight`, `## Queued`, and `## Done` sections. +Absent or `tasks-axi` selects the tasks-axi path. +On the default markdown adapter, tasks-axi and manual edits produce the same `## In flight`, `## Queued`, and `## Done` sections. ## Runtime backend (config/backend / FM_BACKEND) @@ -440,8 +446,7 @@ The lease is held under the secondmate id until explicit retirement or seed roll Teardown of a leased home fails closed if `treehouse return` cannot release the lease; plain-clone homes with no treehouse pool slot are removed directly. Secondmate routes cover `no-mistakes` and `direct-PR` projects; `local-only` projects remain main-firstmate work. For `no-mistakes` projects, seeding initializes only projects newly cloned into a secondmate home and refuses to mutate a preexisting clone that is not already initialized. -After creating a secondmate, move existing main-backlog queued items that you have judged in-scope with `fm-backlog-handoff.sh <secondmate-id> <item-key>...`; it refuses In flight, Done, or non-secondmate homes, and a new move succeeds only after waking the recorded receiver. -If the wake is known to have failed, the moved item remains durable and rerunning the same handoff retries it idempotently; an unresolved delivery is reported and never blindly resent. +After creating a secondmate, move existing main-backlog queued items that you have judged in-scope with `fm-backlog-handoff.sh <secondmate-id> <item-key>...`; it refuses In flight, Done, or non-secondmate homes, and its [script header](../bin/fm-backlog-handoff.sh) owns route-specific wake outcomes and retries. Set `FM_SECONDMATE_CHARTER` to seed from inline charter text when no filled charter brief exists; set `FM_SECONDMATE_SCOPE` when the routing scope should differ from the charter text. The seeded home's `data/charter.md` owns the standard secondmate lifecycle and escalation contract; the route file points to it through the existing `home:` field instead of adding another pointer. Each seed writes an `.fm-secondmate-home` identity marker at the home root, alongside a durable `.fm-secondmate-parent` record of the home's route to its parent (see "Provision a route" in [`docs/remote-secondmates.md`](remote-secondmates.md)). @@ -580,7 +585,7 @@ This section is the single owner of the canonical schema and its per-field seman { "when": "<natural-language condition describing a kind of task>", "use": [ - { "harness": "<adapter>", "model": "<optional model>", "effort": "<low|medium|high|xhigh|max, optional>" } + { "harness": "<adapter>", "model": "<optional model>", "effort": "<low|medium|high|xhigh|max|ultra, optional>" } ], "why": "<optional rationale that helps firstmate choose>" } @@ -595,10 +600,12 @@ Per rule, `when` and `use` are required. Both `use` and the optional top-level `default` accept either one profile object or a non-empty array of profile objects. The single-object form stays fully backward-compatible, and every profile needs `harness`. Profile `model` and `effort` fields and rule `why` are optional. +`ultra` is native-only: the model-aware validation contract and launch mapping are owned by `bin/fm-harness.sh validate-native-effort` and `bin/fm-spawn.sh` respectively. An omitted model or effort means the selected harness uses its own default for that axis. Every profile array is an implicit quota-aware choice resolved through `quota-array-dispatch`. If no dispatch rule fits, firstmate resolves `default` through the same object-or-array path before falling back to `config/crew-harness`. -If a selected profile carries an effort value the chosen harness does not accept, `fm-spawn.sh` records the requested `effort=` in task meta for traceability but omits the launch flag, and bootstrap reports the invalid harness/effort pair as a `CREW_DISPATCH` diagnostic when it is visible in the file. +Except for `ultra`, which refuses unsupported profiles under the native-effort contract above, an effort value the chosen harness does not accept is recorded as `effort=` in task meta for traceability but omitted from the launch flags. +Bootstrap reports unsupported harness/model/effort combinations as a `CREW_DISPATCH` diagnostic when they are visible in the file. See [`docs/examples/crew-dispatch.json`](examples/crew-dispatch.json) for a starting point to copy into local `config/crew-dispatch.json`. When the file exists, bootstrap validates it with `jq`. Valid files stay silent by default; with `FM_BOOTSTRAP_VERBOSE_FACTS=1`, bootstrap emits `BOOTSTRAP_INFO: crew dispatch active config/crew-dispatch.json`, one `BOOTSTRAP_INFO:` fact per rule, and one fact for the optional default profile set. @@ -673,7 +680,7 @@ A herdr, zellij, or cmux home is therefore never told `tmux` is missing, and the When `config/crew-dispatch.json` exists, bootstrap also requires `jq` for dispatch profile validation. When Relay is opted in, bootstrap also requires `curl` and `jq` before arming the relay poll shim. `tasks-axi` and `quota-axi` are required bootstrap tools in every profile, the same class as `lavish-axi`. -An absent or incompatible `tasks-axi` reports `MISSING: tasks-axi (install: npm install -g tasks-axi)`; when `config/backlog-backend` is not `manual`, a home with a backlog refuses lifecycle mutation until compatible `tasks-axi` is on `PATH`, while a manual-backend home keeps its backlog hand-edited. +An absent or incompatible `tasks-axi` reports `MISSING: tasks-axi (install: npm install -g tasks-axi)`; when `config/backlog-backend` is not `manual`, a home with a configured non-markdown adapter or a markdown backlog refuses lifecycle mutation until compatible `tasks-axi` is on `PATH`, while a manual-backend home keeps its backlog hand-edited. An absent or incompatible `gh-axi` reports `MISSING: gh-axi (install: npm install -g gh-axi && gh-axi setup hooks)`. An absent or incompatible `lavish-axi` reports `MISSING: lavish-axi (install: npm install -g lavish-axi && lavish-axi setup hooks)`. An absent or too-old `quota-axi` reports `MISSING: quota-axi (install: npm install -g quota-axi)`; firstmate cannot resolve a profile array without a compatible binary. @@ -776,6 +783,38 @@ The sweep must finish inside `FM_CHECK_TIMEOUT` (default 30), because a run the So a budget larger than that timeout allows is cut down to what fits instead of being refused, and the cut is reported in the report line. A budget that is not a whole number from 1 to 120 is still refused outright. +## Mail plane (.env) + +The mail plane (bin/fm-mail.sh) reads unseen IMAP messages and sends one SMTP message. +Its `poll` command surfaces each new message as a durable `check: mail <uid>` wake, which is also what the standing received-mail check runs each watcher cycle. +Poll emission is exactly-once-recovering: a published wake always carries a durable journal record, and a poll interrupted before recording its uid is healed from that journal, so inbound mail is never silently missed. +A duplicate wake is possible if the process is killed between the queue append and the journal write and the drain acknowledges that row before the next poll heals it, or under a triple write fault that leaves a queued row with no durable record; neither case drops mail. +IMAP and SMTP use implicit TLS on the default ports 993 and 465 (`IMAP4_SSL` / `SMTP_SSL`). +STARTTLS and port 587 are not supported. +It is off unless the home's gitignored `.env` provides the connection values. +This section is the single owner of the mail-plane configuration schema; for direct invocations, environment values override `.env`, matching the Relay contract. + +Required, in the home's gitignored `.env`: + +```sh +FM_MAIL_USER= # IMAP/SMTP login +FM_MAIL_PASS= # IMAP/SMTP password +FM_IMAP_HOST= # IMAP server hostname +FM_SMTP_HOST= # SMTP server hostname +``` + +`FM_IMAP_PORT` (default 993), `FM_SMTP_PORT` (default 465), `FM_MAIL_TIMEOUT` (default 20 seconds), and `FM_MAIL_POLL_MAX_WAKES` (default 20, valid 1..200) are optional. +The per-poll wake cap bounds the wakes of one `poll` run; header fetches scan a larger bounded window of new unseen uids plus already-surfaced retry-set uids, so a flood or large backlog still makes bounded progress every poll, keeping the durable wake queue bounded without ever dropping mail. +A message whose header cannot be fetched is surfaced with a degraded summary instead of being skipped, so it is never missed and cannot block later mail. +A later poll retries that fetch and, on success, surfaces the real sender and subject; a persistently unfetchable message stays degraded without repeating that wake. + +A home that wants mail polled unattended arms the standing check in the live home: `bin/fm-mail-check.sh arm`. +Arming writes `state/mail.check.sh` and registers it with the watcher's slow-check cadence (`FM_CHECK_INTERVAL`), so the plane's `poll` runs on its own: new mail still surfaces as `check: mail <uid>` wakes from the poll, and the standing check itself also prints a line (and the watcher turns that line into a wake) unless the poll is a proven no-op. +Same-line silence is only for a proven no-op: a successful poll with no new mail, or a repeated identical pre-wake failure that cannot have queued mail. +A fail-closed poll that already queued a wake, and a timeout, always print so the watcher wakes to drain it. +`FM_MAIL_CHECK_BUDGET` (default 15, valid 5..25) bounds one standing poll and is cut down to fit `FM_CHECK_TIMEOUT`. +`bin/fm-mail-check.sh disarm` removes the standing check. + ## Relay (.env) Relay lets a firstmate instance answer public mentions and act on normal reversible mention requests through firstmate's normal lifecycle. @@ -804,7 +843,7 @@ The watcher accepts the shim only when its bytes match the expected generated co This section is the single owner of the Relay cadence contract: a Relay instance polls every 30 seconds instead of the default 300, only a Relay instance speeds up because a non-Relay home has no `config/x-mode.env`, and the session-start supervision operating block includes the cadence instruction when that file exists. The active primary-harness supervision protocol owns how that sourced cadence reaches the watcher process. Because `bin/fm-watch.sh` reads `FM_CHECK_INTERVAL` only at process start, a cadence transition - opt-in while a watcher is already running, or opt-out - is applied by restarting the home-scoped watcher through the emitted harness protocol; bootstrap deliberately never restarts the watcher itself. -While away mode is active the daemon owns the watcher and its default cadence applies; away-mode Relay cadence is a deferred follow-up. +While a legacy daemon flag is active the daemon owns the watcher and its default cadence applies; on Pi the away-posture record alone leaves the ordinary Relay watcher cadence active, and daemon-backed Relay cadence remains a deferred follow-up. When the token is removed or empty, the next locked session-start bootstrap step removes those artifacts. Steady-state off is silent and writes nothing. Relay remains additive to non-Relay lifecycle behavior: homes without the generated artifacts keep the default watcher cadence and do not run the Relay poll. @@ -1034,14 +1073,18 @@ Claims live under `$XDG_STATE_HOME/firstmate/procevent-claims` (override with `F Each claim binds its caller-reported home and runner PID to a process identity, unique claim generation, exact registration-file generation, and resolved state-root identity. Registration, acquisition, replacement, retirement, and generation-bound release are serialized at one machine-wide boundary per source. A live identity-matched owner is never displaced, and release removes only the exact generation the caller acquired. -Retirement and orphan reconciliation select a runner process group for signalling only while its recorded process identity still matches and the live runner still leads that group. +Every stop proves ownership before its first signal: the live runner's recorded process identity must match and it must still lead its process group. +Once that stop has proved ownership and sent TERM, its own escalation to KILL checks only whether the proved group still has members; it does not re-read the leader's identity or group membership, which can change or become unreadable as TERM ends the leader. +This proof belongs only to that stop's own escalation and cannot authorize another caller that encounters an unproved group. A claim counts as reclaimable only when its owner is stale and an independent process-group check finds no members; a crashed leader or reused pid whose process group still has members cannot relax ownership cleanup, so reconcile preserves the claim without signalling the ambiguous group or starting a replacement. +If the leader dies to anything other than the stop's own signal, `retire`, `reconcile`, `sweep-home`, and the guard all refuse its surviving group permanently, and the source silently stops listening. +Whether that group may ever be signalled remains an open decision; the repaired guard does not close this gap. Reclaiming a generation that IS gone is not gated on tidying its capture-reservation records. Those records are keyed by claim token and every replacement claims a fresh one, so a leftover that can no longer be located - a state-root identity a claim recorded before its home was re-created, for example - is stale bytes rather than an ownership hazard. Ordinary release and reclamation still attempt reservation cleanup and require it unless both owner staleness and whole-group absence prove the generation gone. The narrow live-owner terminal-self-retirement path also attempts cleanup but tolerates its own still-in-flight reservation, which the runner removes on the normal end-of-capture path; exact home, PID, and claim-token ownership remains mandatory before the claim is released. -If identity cannot be established for a live PID, or a surviving owned group cannot be proved stopped, the operation preserves the registration and claim for safe retry rather than adding a second owner. -A live PID whose identity no longer matches is a reused PID, so cleanup refuses it before signalling. +If identity cannot be established before the first signal, or a surviving owned group cannot be proved stopped, the operation preserves the registration and claim for safe retry rather than adding a second owner. +A live PID whose identity no longer matches is refused before the first signal. Identity and process-group verification cannot be made atomic with signalling in portable shell: the reaper signals only a target it has verified as the recorded generation, but PID and group reuse remain possible in the narrow interval between verification and the signal. Launch pacing is the primary host-wedge protection; watchdog cleanup is a backstop. @@ -1063,13 +1106,16 @@ Detaching a runner into its own process group is what lets a persistent source o So a home's process-event state carries a lease that registration, attached start, reconciliation, acknowledgement, and listing refresh, and the watcher's reconcile cycle is what keeps it fresh in a live home. An attached public `start` continues refreshing the lease while its caller remains attached. Each runner fails closed unless a small guard starts successfully beside it in a separate process group. -That guard accepts the lease only while the state root retains the device/inode identity recorded by the runner's claim, and stops the runner's whole process group after two consecutive checks cannot prove that identity and lease freshness. +That guard accepts the lease only while the state root retains the device/inode identity recorded by the runner's claim, and initiates the verified stop after two consecutive reads cannot prove that identity and lease freshness, so one unreadable read cannot kill a live runner. +Those two reads are spaced half a check interval apart, so the pair the debounce requires completes inside one check interval instead of costing two of them. +For a runner whose ownership can still be proved, the nominal detection bound is therefore the lease plus one check interval, after which the verified stop runs within its own grace period; the lease age is compared in whole seconds, so a configured lease is honoured until that age reads one second past it, and scheduling delays or failed inspection and signalling can extend the whole bound. +That grace is a ceiling rather than a delay every stop pays: two seconds for the ordinary signal and two more for the forced one, spent only by a group that outlives the signal it was sent, which is why a healthy runner's stop completes in a fraction of a second. The group signal reaches the blocking child and everything under it exactly as retirement does. A runner exports the inherited `FM_PROCEVENT_IN_RUNNER` marker and every lease refresh is skipped under it, so a runner and its ordinary children do not certify their own owner, and the next reconcile in a live home simply starts a replacement runner. That no-self-refresh rule is CONFUSED-AGENT-GRADE, the same deliberate captain-decided grade `bin/fm-lease-lib.sh` documents: it stops the accidental case this boundary exists for, an orphaned or test-scaffolding source tree that would otherwise keep its own owner alive. A source that DELIBERATELY strips the marker from its environment can still refresh the lease, so adversarial-grade unforgeability is explicitly out of scope here and tracked as separate follow-up design work. Scope is the owning state root and one runner generation, never a script or process name, so a live source in another home is untouched: that home refreshes its own lease. -`FM_PROCEVENT_OWNER_LEASE_SECONDS` (default 600, range 1..86400) is how long a runner keeps going with no sign of activity in its owning home, and `FM_PROCEVENT_OWNER_CHECK_SECONDS` (default 15, range 1..3600) is how often its guard re-reads the lease. +`FM_PROCEVENT_OWNER_LEASE_SECONDS` (default 600, range 1..86400) is how long a runner keeps going with no sign of activity in its owning home, and `FM_PROCEVENT_OWNER_CHECK_SECONDS` (default 15, range 1..3600) is the guard's detection interval: it re-reads the lease and the recorded state-root identity twice within each interval, half an interval apart, so the two reads its debounce needs fit inside one interval rather than costing two. `FM_PROCEVENT_LAUNCH_FLOOR_SECONDS` (default 1, range 1..3600) is the minimum time between consecutive launches of one registration generation's stored command, bounding the launch rate of an immediately returning source during that lease window. The generation's first launch is immediate, later launches share its monotonic pacing timestamp, a timestamp from before a reboot is treated as expired, and replacing the registration starts a fresh pacing generation. @@ -1170,6 +1216,9 @@ FM_GBRAIN_MAINTENANCE_STATE= # optional operator announcement fm-gbrain-health FM_GBRAIN_MAINTENANCE_DETAIL= # optional free text shown with that announcement, e.g. the release being installed FM_RECALL_TIMEOUT= # optional seconds per fm-recall.sh retrieval call, overriding its per-command defaults (search 60, think 300); search sizes its result-provenance pass from the same value, once per corpus it reads, and takes its per-corpus autocut-floor read out of whatever is left of it, disclosing the pinned default on the answer when that remainder is too small to hold one FM_RECALL_JSONL_MAX_BYTES=262144 # size cap for the home-wide fm-recall.sh search-read log at state/recall.jsonl; past the cap the oldest lines are dropped and the newest tail is kept +FM_MAIL_CHECK_BUDGET=15 # seconds allowed for one standing mail poll; valid 5..25, cut to fit FM_CHECK_TIMEOUT +FM_MAIL_POLL_MAX_WAKES=20 # per-poll wake cap for a mail poll; valid 1..200, keeps a flood from flooding firstmate +FM_MAIL_TIMEOUT=20 # mail-plane IMAP/SMTP socket timeout in seconds; invalid or non-positive values become 20 FM_TOOL_UPDATE_INTERVAL=900 # seconds between watched-tool probe sweeps; 0 probes on every run, other values must be 60..86400 FM_TOOL_UPDATE_PROBE_SECS=5 # 1..30 seconds allowed for one version or git probe FM_TOOL_UPDATE_BUDGET_SECS=20 # 1..120 seconds allowed for a whole watched-tool sweep; cut to fit FM_CHECK_TIMEOUT, and the cut is reported @@ -1177,13 +1226,13 @@ FM_TOOL_UPDATE_NOW= # test override for the watched-tool sweep clock; the sw FM_PROCEVENT_MAX_OUTPUT_BYTES=1048576 # bound on one captured process-to-event result FM_PROCEVENT_CLAIM_ROOT= # machine-wide source claim root; default $XDG_STATE_HOME/firstmate/procevent-claims FM_PROCEVENT_OWNER_LEASE_SECONDS=600 # how long a source runner keeps going with no activity in its owning home; 1..86400 -FM_PROCEVENT_OWNER_CHECK_SECONDS=15 # how often a runner's guard re-reads that lease; 1..3600 +FM_PROCEVENT_OWNER_CHECK_SECONDS=15 # a runner guard's detection interval, read twice per interval; 1..3600 FM_PROCEVENT_LAUNCH_FLOOR_SECONDS=1 # minimum interval between launches of one registration generation's source command; 1..3600 FM_WHEN_OUTPUT_TAIL_BYTES=8192 # bound on the command-output tail inside one condition->action outcome document FM_CODEX_WATCH_CHECKPOINT=180 # seconds per foreground watcher checkpoint in Codex primary supervision FM_CREW_STATE_NM_TIMEOUT=10 # seconds allowed per no-mistakes query inside fm-crew-state.sh; bin/fm-fleet-snapshot.sh derives its own value from its per-task bound for the reads it makes, so this override does not reach those, and that script's header owns the derivation FM_TEARDOWN_NM_TIMEOUT=10 # seconds allowed per no-mistakes query or abort inside fm-teardown.sh -FM_CREW_STATE_RUNS_LIMIT=200 # recent no-mistakes run rows scanned when axi status cannot be attributed to the worktree +FM_CREW_STATE_RUNS_LIMIT=200 # recent no-mistakes run rows scanned when the runs ledger is consulted: axi status cannot be attributed directly, or its answer is terminal and may have a live sibling run FM_CREW_STATE_DEGRADED_MAX_AGE=900 # seconds a recorded run-step may stand in as the degraded answer while the no-mistakes lookup cannot complete; 0 disables the degrade FM_CREW_STATE_BIN=bin/fm-crew-state.sh # test override for the current-state reader used by working/paused watcher triage FM_RUN_PROGRESS_BIN=bin/fm-run-progress.sh # test override for the validation-run progress reader consulted at the wedge-escalation point @@ -1198,7 +1247,7 @@ FMX_FOLLOWUP_MAX_AGE_SECS=604800 # local window for posting Relay completion f FMX_FOLLOWUP_MAX_COUNT=3 # local cap on Relay completion follow-ups per linked mention FM_PF_RETRY_BACKOFF_SECS=900 # seconds before the next attempt after a retryable promised-public-reply delivery error FM_LOCK_STALE_AFTER=2 # grace seconds for missing or nonnumeric lock-owner PIDs (minimum 2s); dead numeric PIDs have no age grace -FM_GUARD_GRACE=300 # seconds before guard warnings, arm health checks, and the primary turn-end guard treat a watcher beacon as stale +FM_GUARD_GRACE=300 # beacon freshness threshold for guard verdicts, arm health checks, and the primary turn-end guard; see docs/turnend-guard.md for model-aware exceptions FM_CLAUDE_AUTOARM_ATTEMPTS=2 # bounded Stop-owned arm attempts per Claude auto-arm cycle; accepted values are 1, 2, or 3 FM_CLAUDE_AUTOARM_SYNC_WAIT_MS=800 # milliseconds the --claude turn-end guard waits for watcher health, an open Stop auto-arm generation claim, or a fresh epoch before deciding recovery ownership or failure progression FM_CLAUDE_AUTOARM_EPOCH_FRESH=15 # seconds a recorded auto-arm outcome remains eligible for the current event epoch's recovery or failure decision @@ -1220,7 +1269,7 @@ FM_TURNEND_CHURN_ABSORB_SECS=900 # longest one endpoint's bare turn-ends may b FM_CAPTAIN_RE='done:|needs-decision:|blocked:|failed:|PR ready|checks green|ready in branch|merged' # captain-relevant status regex; nonterminal progress verbs remain excluded even when their prose matches FM_CLASSIFY_PAUSED_VERB=paused # leading status verb for a declared external wait; excluded from FM_CAPTAIN_RE and distinct from blocked FM_STALE_ESCALATE_SECS=240 # idle seconds before a provably-working stale pane or a confidently dead parked-decision repeat escalates; other first-sighting stale states surface immediately unless they declare the pause verb -FM_BUSY_TURN_MAX_SECS=3600 # maximum age of a busy pane's latest state/<id>.turn-ended marker, or its state/<id>.meta spawn record before any turn-boundary wake arrives, before the same wedge escalation used for a provably-working non-busy stale takes over; inspection-only, never an automatic interrupt or restart; a declared external wait or verified captain-held transfer takes the FM_PAUSE_RESURFACE_SECS recheck below instead. bin/fm-supervision-lib.sh owns the window, and bin/fm-fleet-snapshot.sh publishes it as supervision.watcher.quiet_allowance_seconds so the dashboard's Task activity signal judges quiet against this same tolerance instead of a constant of its own; docs/dashboard-inbox-policy.md owns that signal and what renders it today +FM_BUSY_TURN_MAX_SECS=3600 # maximum age of a busy pane's latest state/<id>.turn-ended or observed native-harness state/<id>.progress marker, or its state/<id>.meta spawn record before any turn-boundary wake arrives, before the same wedge escalation used for a provably-working non-busy stale takes over; inspection-only, never an automatic interrupt or restart; a declared external wait or verified captain-held transfer takes the FM_PAUSE_RESURFACE_SECS recheck below instead. bin/fm-supervision-lib.sh owns the window, and bin/fm-fleet-snapshot.sh publishes it as supervision.watcher.quiet_allowance_seconds so the dashboard's Task activity signal judges quiet against this same tolerance instead of a constant of its own; docs/dashboard-inbox-policy.md owns that signal and what renders it today FM_PAUSE_RESURFACE_SECS=3600 # seconds before a declared external wait or verified captain-held transfer re-surfaces for a recheck in the watcher, and before repeated new-hash alarms for an ordinary crew task with an open backlog call, including a live busy pane past FM_BUSY_TURN_MAX_SECS, and before a declared external wait re-surfaces in the away-mode daemon, which ages its window against the crew's own latest status line rather than pane busy state so only a status append that stops declaring the wait ends that routing; the away-mode clock runs from when the hold began and is never restarted by the crew rewriting its paused reason FM_PAUSE_RESURFACE_MAX_STREAK=3 # how many times an unchanged declared wait may double the WIDTH of its recheck window before the cadence stops widening; 0 restores a fixed FM_PAUSE_RESURFACE_SECS cadence, and the streak restarts whenever the wait itself changes, which only ever narrows the window back toward the base. Clamped internally to 12 doublings and a one-day window, so a misconfigured value cannot overflow into a permanent re-surface FM_SECONDMATE_WAKE_STALL_SECS=1800 # no-progress interval for an oldest actionable foreign queue identity when the semantic busy verdict is unknown, dead, or unreadable; also used after proven busy exceeds FM_BUSY_TURN_MAX_SECS. Progress resets the interval, declared external-wait rows are excluded, and one notification covers each episode; zero or invalid values use 1800 diff --git a/docs/fm-test-portable-shards.md b/docs/fm-test-portable-shards.md index 07ab62d89ab..ba8207a9678 100644 --- a/docs/fm-test-portable-shards.md +++ b/docs/fm-test-portable-shards.md @@ -6,38 +6,38 @@ ## Verification inputs The proven-isolated candidate set remains the 24-script concurrent proof recorded in [fm-test-isolation-proof.md](fm-test-isolation-proof.md). -Placement timings are refreshed from completed script measurements in [CI run 34800278334](https://github.com/HelloWorldSungin/firstmate/actions/runs/34800278334). -Its first parallel lane reached the unchanged ten-minute job limit after passing nine scripts; the old two-minute estimate no longer represented the enlarged suites. +The 2026-09-14 placement refresh uses completed script measurements in [CI run 34837015817](https://github.com/HelloWorldSungin/firstmate/actions/runs/34837015817). +Its first parallel lane reached the unchanged ten-minute job limit after five completed scripts, with the enlarged hold and merge suites accounting for 504 seconds together. The second parallel lane completed successfully. -The two scripts not completed before the first lane was cancelled retain the passing local full-sweep measurements shown explicitly below. +The six scripts not completed before the first lane was cancelled retain the earlier passing measurements from [CI run 34800278334](https://github.com/HelloWorldSungin/firstmate/actions/runs/34800278334), shown explicitly below. These measurements change placement, not isolation eligibility or execution deadlines. | duration_ms | script | Measurement | |---:|---|---| -| 203500 | `tests/fm-captain-hold-lifecycle.test.sh` | CI completed script | -| 179803 | `tests/fm-lint.test.sh` | CI completed script | -| 150192 | `tests/fm-test-run.test.sh` | CI completed script | -| 122702 | `tests/fm-pr-merge.test.sh` | CI completed script | -| 32818 | `tests/fm-crew-state.test.sh` | CI completed script | -| 28065 | `tests/fm-x-mode.test.sh` | CI completed script | -| 26899 | `tests/fm-arm-pretool-check.test.sh` | CI completed script | -| 21249 | `tests/fm-backend-herdr.test.sh` | CI completed script | -| 15995 | `tests/fm-cd-pretool-check.test.sh` | CI completed script | -| 14889 | `tests/fm-brief.test.sh` | Local full sweep, 2026-09-12 | -| 6983 | `tests/fm-grok-harness.test.sh` | CI completed script | -| 6788 | `tests/fm-send-strict.test.sh` | CI completed script | -| 6351 | `tests/fm-herdr-lab.test.sh` | CI completed script | -| 4464 | `tests/fm-send-popup-settle.test.sh` | CI completed script | -| 4220 | `tests/fm-pi-primary-types.test.sh` | CI completed script | -| 4219 | `tests/fm-composer-lib.test.sh` | CI completed script | -| 3832 | `tests/fm-review-diff.test.sh` | CI completed script | -| 2271 | `tests/fm-tmux-submit-busy.test.sh` | CI completed script | -| 2241 | `tests/fm-spawn-batch.test.sh` | CI completed script | -| 2049 | `tests/fm-composer-ghost.test.sh` | CI completed script | -| 1834 | `tests/fm-send-settle.test.sh` | CI completed script | -| 810 | `tests/fm-ensure-agents-md.test.sh` | CI completed script | -| 307 | `tests/fm-supervision-instructions.test.sh` | CI completed script | -| 93 | `tests/fm-transition-lib.test.sh` | Local full sweep, 2026-09-12 | +| 305369 | `tests/fm-captain-hold-lifecycle.test.sh` | CI completed script | +| 209643 | `tests/fm-lint.test.sh` | CI completed script | +| 198766 | `tests/fm-pr-merge.test.sh` | CI completed script | +| 157536 | `tests/fm-test-run.test.sh` | CI completed script | +| 41660 | `tests/fm-crew-state.test.sh` | CI completed script | +| 30805 | `tests/fm-arm-pretool-check.test.sh` | CI completed script | +| 29237 | `tests/fm-x-mode.test.sh` | CI completed script | +| 27380 | `tests/fm-backend-herdr.test.sh` | CI completed script | +| 23099 | `tests/fm-brief.test.sh` | CI completed script | +| 16799 | `tests/fm-cd-pretool-check.test.sh` | CI completed script | +| 7584 | `tests/fm-send-strict.test.sh` | CI completed script | +| 6983 | `tests/fm-grok-harness.test.sh` | Previous recorded measurement | +| 6351 | `tests/fm-herdr-lab.test.sh` | Previous recorded measurement | +| 4883 | `tests/fm-send-popup-settle.test.sh` | CI completed script | +| 4644 | `tests/fm-composer-lib.test.sh` | CI completed script | +| 4220 | `tests/fm-pi-primary-types.test.sh` | Previous recorded measurement | +| 3832 | `tests/fm-review-diff.test.sh` | Previous recorded measurement | +| 2505 | `tests/fm-spawn-batch.test.sh` | CI completed script | +| 2453 | `tests/fm-tmux-submit-busy.test.sh` | CI completed script | +| 2049 | `tests/fm-composer-ghost.test.sh` | Previous recorded measurement | +| 1834 | `tests/fm-send-settle.test.sh` | Previous recorded measurement | +| 906 | `tests/fm-ensure-agents-md.test.sh` | CI completed script | +| 342 | `tests/fm-supervision-instructions.test.sh` | CI completed script | +| 171 | `tests/fm-transition-lib.test.sh` | CI completed script | ## Parallel lanes @@ -45,9 +45,9 @@ The two parallel lanes use longest-processing-time assignment from those measure | Lane | Script count | Estimated duration | |---|---:|---:| -| `portable-parallel-1` | 11 | 421533 ms (~7.03 min) | -| `portable-parallel-2` | 13 | 421041 ms (~7.02 min) | -| imbalance | | 492 ms | +| `portable-parallel-1` | 12 | 544542 ms (~9.08 min) | +| `portable-parallel-2` | 12 | 544509 ms (~9.08 min) | +| imbalance | | 33 ms | `bin/fm-test-run.sh` contains the exact ordered memberships in `list_portable_parallel_1` and `list_portable_parallel_2`. @@ -73,7 +73,8 @@ Shared scripts use those upstream per-script maxima; fork-only scripts retain th The round also includes upstream's retained native-Windows measurement for `tests/fm-pi-windows-shell-invocation.test.sh` and its new live-guard weights. The 2026-09-14 refresh takes the maximum of those retained hints and completed passing-script measurements from fork CI runs [34802687283](https://github.com/HelloWorldSungin/firstmate/actions/runs/34802687283) and [34804089007](https://github.com/HelloWorldSungin/firstmate/actions/runs/34804089007). The latter's serial lane 5 reached its unchanged 20-minute cap after 16 passing scripts; its completed `FM_TEST_END` records are included explicitly because cancellation prevented a timing artifact. -These runs provide passing measurements for 200 of the 201 serial scripts; `tests/fm-test-isolation-proof.test.sh` retains its earlier 2567 ms hint, with its corrected assertion passing locally. +Before the next upstream prefix was included, these runs provided passing measurements for 200 of the 201 serial scripts; `tests/fm-test-isolation-proof.test.sh` retains its earlier 2567 ms hint, with its corrected assertion passing locally. +The next prefix takes the maximum of each retained fork hint and the upstream endpoint's existing hint. Taking maxima preserves native-Windows measurements and earlier slow-run evidence rather than replacing them with portable gate-skip durations. A script with no hint receives `PORTABLE_SERIAL_DEFAULT_WEIGHT_MS`; the runner's coverage output reports that unmeasured share. Hints only affect balance: the coverage guard keeps the partition complete and disjoint whatever they say, so a stale hint costs a slower shard rather than lost coverage. @@ -84,22 +85,24 @@ Refresh the hints whenever the serial lane gains scripts, rather than waiting fo Shard count is sized from that total rather than left where an earlier, smaller remainder put it. The lane grew from about 19 minutes across 69 scripts to about 58 minutes across 154, which four shards could no longer carry inside the job timeout: on the run above, `portable-serial-2of4` was cancelled at 15 minutes having finished 24 of its 32 scripts, and the hints then put a perfectly balanced quarter at 14.5 minutes, still on the tripwire rather than inside it. -The refreshed shared hints and retained fork-only hints now put the slowest of eight shards at about 12.89 minutes. +The merged shared maxima and retained fork-only hints put the slowest of eight shards at about 13.69 minutes. +The current 209-script serial lane has six unhinted scripts, using the runner's conservative default, and totals 6572397 ms of assignment weight. +The 30-minute serial job cap adopted from upstream leaves setup and runner-speed margin; the fork's 480-second per-script bound remains unchanged. | Lane | Script count | Estimated duration | |---|---:|---:| -| `portable-serial-1of8` | 24 | 773453 ms (~12.89 min) | -| `portable-serial-2of8` | 25 | 773490 ms (~12.89 min) | -| `portable-serial-3of8` | 26 | 773503 ms (~12.89 min) | -| `portable-serial-4of8` | 25 | 773453 ms (~12.89 min) | -| `portable-serial-5of8` | 25 | 773452 ms (~12.89 min) | -| `portable-serial-6of8` | 25 | 773452 ms (~12.89 min) | -| `portable-serial-7of8` | 25 | 773453 ms (~12.89 min) | -| `portable-serial-8of8` | 26 | 773502 ms (~12.89 min) | -| imbalance | | 51 ms | +| `portable-serial-1of8` | 26 | 821551 ms (~13.69 min) | +| `portable-serial-2of8` | 26 | 821546 ms (~13.69 min) | +| `portable-serial-3of8` | 26 | 821547 ms (~13.69 min) | +| `portable-serial-4of8` | 26 | 821552 ms (~13.69 min) | +| `portable-serial-5of8` | 27 | 821587 ms (~13.69 min) | +| `portable-serial-6of8` | 26 | 821546 ms (~13.69 min) | +| `portable-serial-7of8` | 26 | 821534 ms (~13.69 min) | +| `portable-serial-8of8` | 26 | 821534 ms (~13.69 min) | +| imbalance | | 53 ms | The watcher triage cases are split into core and wait/decision scripts with one shared fixture owner in `tests/watch-triage-helpers.sh`. -All 125 original cases remain in exactly one script. +All 125 original cases remain in exactly one script, with compatible new upstream progress and declared-deadline cases added beside them. Their initial passing local measurements were 160205 ms and 191775 ms after CI reached the unchanged 480-second combined-script limit while still passing cases. The refreshed CI hints are 236467 ms for core and 292716 ms for waits. @@ -140,8 +143,8 @@ Portable shards, each portable serial shard, and the Herdr lane upload runner-ge | Lane | Bound | Rationale | |---|---|---| -| portable parallel 1/2 | job `timeout-minutes: 10` | The measured shard sums are about 7 minutes and the timeout is a hang tripwire. | -| portable serial 1-8 | job `timeout-minutes: 20` | The slowest estimated shard is about 12.89 minutes, leaving roughly seven minutes for setup and runner-speed spread. | +| portable parallel 1/2 | job `timeout-minutes: 10` | The measured shard sums are about 9.08 minutes, leaving about 55 seconds for setup and runner variation. | +| portable serial 1-8 | job `timeout-minutes: 30` | The slowest estimated shard is about 13.69 minutes, leaving setup and runner-speed margin. | | Herdr | family-run step `timeout-minutes: 20`; job `timeout-minutes: 75` backstop | The required lane is bounded independently of the per-script deadline; refresh timings from its uploaded artifacts. Previous healthy runs finished around 7 minutes, so the step bound is the hang tripwire (cleanup and timing artifacts still upload) while the job cap stays a last-resort backstop. | Timeouts are hang tripwires rather than expected healthy durations. @@ -149,7 +152,7 @@ Timeouts are hang tripwires rather than expected healthy durations. Inside each lane, `bin/fm-test-run.sh` applies its own default per-script bound, so a hung script usually turns red with per-script attribution before the job cap cancels the lane; its `--help` owns that bound's value and opt-out, and the rationale beside `DEFAULT_PER_SCRIPT_TIMEOUT_SECS` owns the per-lane margin arithmetic. Neither portable lane has room to spare, because a hung script spends the bound instead of its own healthy slot. -On the slowest estimated serial shard, replacing its average script with the 480-second bound puts script time around 20.4 minutes, past the 20-minute cap before checkout and bootstrap overhead. -A hung script may therefore reach the job limit before the per-script limit can report it; the healthy estimate retains roughly seven minutes for setup and runner-speed variation. +On the slowest estimated serial shard, replacing its average script with the 480-second bound puts script time around 21 minutes, within the 30-minute cap before checkout and bootstrap overhead. +The healthy estimate retains roughly sixteen minutes for setup and runner-speed variation. The portable parallel cap is tighter still: the same arithmetic already lands past its 10-minute cap before setup, so expect the job timeout rather than per-script attribution when a script hangs there. On the required Herdr lane the bound has the thinnest margin over its slowest measured script, so a healthy but unusually slow Herdr end-to-end script can turn red as `exit=124`; that margin is accepted rather than widened, tracked in `HelloWorldSungin/firstmate#256`. diff --git a/docs/fork-divergence.md b/docs/fork-divergence.md index 82a1814d624..f61f03b4205 100644 --- a/docs/fork-divergence.md +++ b/docs/fork-divergence.md @@ -129,6 +129,11 @@ Upstream `kunchenguid/firstmate#3842` adds backlog-backed call identity to the s That identity and its release/re-hold reset are adopted while a worker's own paused declaration still routes directly to the fork's widening-cadence owner. The backlog read remains outside the secondmate ordinary-poll path. +Upstream `kunchenguid/firstmate#4048` also silences held-task rechecks under its away-posture record and raises the default cadence to four hours. +Firstmate retained the fork's one-hour base and widening cadence for held tasks and external waits, including during a confirmed away posture. +The compatible explicit `until` deadline wakes a cleared wait early while a distant deadline remains bounded by the shared cadence. +`tests/fm-daemon.test.sh` verifies a confirmed posture still resurfaces a held task, and the watcher wait tests exercise early, future, and distant deadlines. + ### Herdr pre-Enter footer read on a native working baseline The fork skips the pre-Enter rendered-footer read entirely when herdr's native agent-state baseline is already `working`, because the rendered-footer conversion refuses a `working` baseline outright and the read can therefore produce no verdict. @@ -170,7 +175,8 @@ The rationale beside the constant owns why 480s and what the bound costs each CI [`tests/fm-test-run.test.sh`](../tests/fm-test-run.test.sh) pins the default arming, the opt-out, the exit 124 versus exit 125 distinction, and the signal relay, so an upstream round that rewrites the timeout wiring cannot retire the default silently. The upstream `kunchenguid/firstmate#3489` rebalance refreshes shared duration hints and raises the portable serial job cap to 20 minutes. -The fork retains eight shards and its fork-only timing hints, while adopting that cap; the runner still owns the 480-second per-script bound and the updated margin arithmetic. +The fork retains eight shards and its fork-only timing hints. +The fork adopts the upstream 30-minute serial job cap and takes per-script maxima across both parents; the runner still owns the 480-second per-script bound and the updated margin arithmetic. Watcher triage cases are partitioned between `tests/fm-watch-triage.test.sh` and `tests/fm-watch-triage-waits.test.sh`, with shared case definitions in `tests/watch-triage-helpers.sh`, so suite growth does not weaken the per-script bound. @@ -213,6 +219,10 @@ Compatible row-format validation lives in the existing relation-table reader; ma `tests/fm-teardown.test.sh` requires an unfetched parked run to remain untouched even with a valid terminal exact-HEAD anchor, while fetched equal-or-descendant heads retain their existing abort behavior. `tests/fm-crew-state.test.sh` pins valid and invalid calendar dates, malformed rows and anchors, and degraded evidence versus confirmed absence. +Upstream `kunchenguid/firstmate#2881` makes an attributable live run outrank an earlier terminal result. +That precedence is adopted in the relation-table reader and full-detail reconciliation without widening teardown attribution or replacing the fork's degraded and abandoned verdicts. +`tests/fm-crew-state.test.sh` covers live-versus-terminal selection alongside the retained attribution cases. + ### Definition-of-done owner carries this fork's ready-to-validate handoff Upstream's `kunchenguid/firstmate#3269` made [`bin/fm-dod-lib.sh`](../bin/fm-dod-lib.sh) the one owner of a ship task's mode-specific definition of done so a promoted scout receives the same delivery contract a briefed worker does. @@ -285,13 +295,17 @@ The streaming-follow-up regression also runs a replacement while away, proving t Upstream's OMP watcher port in `kunchenguid/firstmate#3867` receives the same standby protection in [`.omp/extensions/fm-primary-omp-watch.ts`](../.omp/extensions/fm-primary-omp-watch.ts), preserving the shared daemon's ownership instead of starting a competing cycle. `tests/fm-omp-harness.test.sh` exercises flag-present launch, child retirement, pending-wake preservation, one-cycle return, and replacement through the extension interface. -This is the existing `.afk` ownership contract, not the later upstream AFK-posture design, and portable extension tests do not establish live OMP vendor compatibility. +Upstream `kunchenguid/firstmate#4048` adds durable away-posture records and removes the Pi daemon entry path. +Firstmate retained the Pi/OMP daemon and extension standby contract while adopting the compatible proposal, confirmation, archive, and return flow. +`tests/fm-afk-launch.test.sh` covers confirmed posture plus Pi/OMP lifecycle preparation, and the existing extension suites retain delivery and one-cycle resumption coverage. +Portable extension tests do not establish live OMP vendor compatibility. ### Herdr presentation fixture ownership and cleanup The fork separates presentation, focus, and negative-path coverage in [`tests/fm-backend-herdr-presentation-e2e.test.sh`](../tests/fm-backend-herdr-presentation-e2e.test.sh) from multi-home ownership and restart coverage in [`tests/fm-backend-herdr-recovery-e2e.test.sh`](../tests/fm-backend-herdr-recovery-e2e.test.sh), keeping both within ordinary runner bounds. They share common setup and instrumentation functions in [`tests/herdr-presentation-fixture.sh`](../tests/herdr-presentation-fixture.sh), while each entrypoint owns independent source, home, lab, evidence, and task identities. +The shared fixture supports a task-specific lab label and derives its version probe from the guarded lab status response, keeping Herdr command execution inside the lab helper. The recovery fixture retires completed multi-home task records before whole-session restart scenarios. Its secondmate homes carry local parent bindings and registry entries, so concurrent positive cases exercise the shared project lock and ownership checks, with a bounded retry only for the expected lock-contention refusals. Both entrypoints belong to the existing real-Herdr family and use the ordinary default timeout in local and CI runs; no per-script timeout allowance is required. @@ -299,6 +313,7 @@ Treehouse leases belong to processes, so leaving those completed records after t Both fixtures keep their worktree journals across command-substitution subshells and delegate cleanup to [`tests/herdr-presentation-cleanup.sh`](../tests/herdr-presentation-cleanup.sh). That owner waits for outstanding fixture operations, shuts down the named lab before ordinary slot returns, verifies each copy's source repository, and preserves source Git metadata and separate evidence after any failure. Repeated cleanup retains the first verdict without repeating partial mutations. +The presentation fixture restores its move audit from the same separate evidence directory that owns the saved snapshot, with copy failures stopping the fixture before later ordering assertions. [`tests/fm-test-fixture-cleanup.test.sh`](../tests/fm-test-fixture-cleanup.test.sh) covers return failure, lifecycle failure, foreign ownership, outstanding operations, and repeated cleanup without using real Herdr. Production allocation and slot-exclusivity policy remain unchanged. @@ -384,6 +399,13 @@ A full upstream merge must not restore that instruction, because pointing a fork No test guards `CONTRIBUTING.md`'s content, so this ledger entry is the only standing record of intent for that half and must be consulted when a sync round touches the contributor workflow section. [`README.md`](../README.md)'s install instruction still clones upstream rather than this fork, a known and deliberately unresolved divergence tracked in `HelloWorldSungin/firstmate#172`, and a sync round must not read it as evidence that this fork intends upstream as its PR base. +### Supervisor-only chat address + +Upstream `kunchenguid/firstmate#4075` binds its chat address requirement to every agent reading `AGENTS.md`. +Firstmate retained the fork's supervisor-role boundary while adopting the upstream chat-only and non-artifact scope. +`AGENTS.md` owns the address scope, and `bin/fm-dod-lib.sh` owns the worker role delivered at launch. +`tests/fm-spawn-dispatch-profile.test.sh` consumes actual generated launch commands and verifies that the worker reporting and address exception reaches the harness without rewriting the authored brief. + ### GBrain per-home knowledge memory The fork carries GBrain as per-home task-knowledge memory - search through [`bin/fm-recall.sh`](../bin/fm-recall.sh), capture through [`bin/fm-gbrain-capture.sh`](../bin/fm-gbrain-capture.sh), and the serving, pin, scoping, evaluation, and deployment tooling around [`bin/fm-gbrain.sh`](../bin/fm-gbrain.sh) - and upstream has no equivalent of any of it. diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index f8c6bb5ac0b..c3eb6e87010 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -35,6 +35,16 @@ The required CI lane uses the pinned installers in `bin/fm-install-herdr.sh` and Those script headers own release assets, checksums, download bounds, and post-install gates. Real harness credential tests remain opt-in rather than part of default CI. +## Client selection + +Each operation routed through the adapter's session-scoped CLI helper starts with the first `herdr` on `PATH` unless that session has already selected another client. +A host can carry more than one client, such as a self-updated copy in `~/.local/bin` beside a package-managed one, and a client older than the running server can receive error code `protocol_mismatch` on operational commands. +On that refusal the adapter reads `status --json --session <name>` from each distinct `herdr` on `PATH` in order, adopts the first one the running server reports compatible, and retries the command on it once. +The choice is reused only for later calls to the same session in that process; another session starts with the `PATH` default, and a later mismatch forces selection again so a changed server can return to that default. +Ordinary adapter operations make no selection read on the happy path, status that supplies neither `.server.compatible` nor both client and server protocols leaves compatibility unknown, and no other failure triggers a reselection. +`fm-remote-doctor.sh` reports the client selected for the remote session. +Removing or upgrading the shadowing client is the durable fix; `bin/backends/herdr.sh` "client selection" owns the mechanics. + ## Watching and task containers The ordinary topology puts one task tab per endpoint in the exact workspace of the Firstmate or secondmate that launches it. @@ -118,11 +128,14 @@ The worker remains on the ordinary flat or Herdr-current-order path. Normal task metadata remains the sole endpoint authority after creation. Cleanup closes only the exact recorded task pane and never calls `workspace close`. Herdr 0.7.5's explicit close moves focus to a neighbor whenever it empties a non-focused workspace, while its pane-death removal preserves the focused workspace whenever the dying workspace sits behind it or the focused workspace is last; both behaviors are fixed in Herdr 0.8.0, and the exact rules live in the adapter header of `bin/backends/herdr.sh`. -Projected cleanup therefore runs under the same session lock, captures the exact active tab, refuses to delete the active tab, and treats a workspace-emptying close as a focus-safe removal: it verifies the close would empty the workspace, repositions the doomed workspace behind the focused one through the verified `workspace.move` transport when needed, proves the pane holds one lone idle shell, and ends that shell so Herdr removes the emptied workspace through its focus-preserving pane-death path. +Projected cleanup therefore runs under the same session lock, refuses to delete the tab a live foreground client is viewing, and treats a workspace-emptying close as a focus-safe removal: it verifies the close would empty the workspace, repositions the doomed workspace behind the focused one through the verified `workspace.move` transport when needed, proves the pane holds one lone idle shell, and ends that shell so Herdr removes the emptied workspace through its focus-preserving pane-death path. +The persisted `.focused` pointer is not a live viewer: when `herdr terminal title clear` reports `no_foreground_client`, cleanup proceeds on that tab because no human is attached and skips restoration of the tab it destroys. +Herdr currently has no atomic client-aware mutation, so a fresh target-focus and foreground-client checkpoint runs immediately before each move, signal, or explicit close; when a live viewer has switched to another tab, that fresh tab becomes the restore target. +A client can still attach or switch focus in the residual checkpoint-to-mutation window, and a durable atomic close is deferred until Herdr exposes that primitive. The repositioning move-to-last preserves every surviving workspace's relative order, and removal is confirmed against the exact moved workspace rather than inferred from pane disappearance before an unconfirmed removal makes one verified attempt under the same session lock to roll the doomed workspace back to its exact original position. If that rollback cannot restore the verified original order, cleanup warns loudly and leaves the retained records for inspection rather than retrying the shared-layout mutation. The pane-death signals are pid-exact: the escalation re-reads the pane's process information and refuses unless the same shell pid still passes the strict bare-idle ownership proof, so an exited and reused pid is never signaled. -Any ambiguity, unsupported or failed move, or unproved shell falls back to the plain explicit close, and the exact prior-tab restore remains the backstop behind every close, so degraded behavior is never worse than the pre-mitigation sub-second restore. +A move-plan ambiguity, unsupported or failed move, or unproved shell falls back to the plain explicit close, and exact tab restoration remains the backstop whenever a surviving tab must be preserved, so degraded behavior is never worse than the pre-mitigation sub-second restore. Ordinary non-projected task removal serializes through the same session lock, applies the same focus-safe plan when its close would empty a non-focused workspace, keeps the legitimate plain close when the target is the active tab, and refuses an unlocked close if the lock cannot be acquired. Task cleanup acquires that session lock before the task's isolated copy is returned, so a contended lock refuses up front while the copy, every durable record, and the endpoint are all intact for a plain rerun. Forced secondmate cleanup recursively preflights every Herdr child endpoint and acquires every affected named-session lock before mutating any child, then retains each child's durable identity unless that exact pane returns structured not-found after its close. @@ -175,6 +188,7 @@ Each entrypoint creates its own source, homes, lab, and evidence; both are selec `tests/fm-herdr-session-cleanup.test.sh` covers every discovery, ownership, topology, process, locking, revalidation, focus, retirement, and continue-on-error boundary. `tests/fm-herdr-session-cleanup-e2e.test.sh` covers the restored-shell cleanup in a guarded non-default named lab. `tests/fm-backend-herdr-focus-flash-e2e.test.sh` reproduces the raw explicit-close focus steal on the installed release and proves the focus-safe emptying-close plan removes a doomed workspace with no wrong-focus interval; [`verification/runtime-backends.md`](verification/runtime-backends.md#workspace-removal-focus-safety) owns the active versioned evidence. +`tests/fm-backend-herdr-stale-active-tab-e2e.test.sh` proves a persisted-focused tab still closes when no foreground client is attached. ## Default-tab prune safety @@ -283,9 +297,11 @@ A restored same-labeled tab with a missing pane or no registered agent is a husk Create replaces only a confidently dead or no-agent husk, creates the replacement before closing the old tab, and refuses live or unknown states. This prevents closing the workspace's last tab before a replacement exists. -The generic Herdr agent-liveness probe reuses the same classifier. -A structurally gone pane becomes `missing`, a restored agent-less shell becomes `dead`, a registered agent becomes `alive` unless its registration is proven stale, and an unexpected read becomes `unreadable`. +The generic Herdr agent-liveness probe reuses the same pane classifier, then applies one recovery-only exception. +A structurally gone pane or a pane read from a session positively reported as having no running server becomes `missing`, a restored agent-less shell becomes `dead`, a registered agent becomes `alive` unless its registration is proven stale, and every other unexpected read becomes `unreadable`. +The stopped-server exception does not widen husk detection or any close authority; those paths still refuse an unreadable pane. Unlike tmux process-name inspection, native registration can classify Pi without guessing from a generic interpreter name. +`tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh` pins the live-Pi versus leftover-shell distinction; [`verification/runtime-backends.md`](verification/runtime-backends.md#agent-lifecycle-control) owns the versioned evidence. A registration is not trusted on its own, because Herdr can hold one that outlives its agent. An agent whose lifecycle reporting is hook-authoritative (Pi's `herdr:pi` extension) never deregisters on exit, Herdr skips screen detection for it, and Herdr exposes no agent-deregister verb, so `agent get` keeps reporting the last agent state forever after any exit, clean or killed. @@ -304,7 +320,7 @@ Mid-session secondmate agent-process liveness is not implemented because idle se Protocol 16 can subscribe to `pane.agent_status_changed` over one bounded Unix-socket reader. `bin/fm-transition-lib.sh` owns the backend-neutral transition vocabulary and policy. The Herdr adapter subscribes before reconciling current levels, buffers edges during reconciliation, and returns fresh blocked transitions for this home's panes. -The watcher maps the pane back to the task and skips secondmate endpoints, declared `paused:` waits, and verified `captain-held` transfers, because a declared wait already names the human the fast escalation would report and is left to the watcher's own bounded pause cadence. +The watcher maps the pane back to the task and skips secondmate endpoints, declared `paused:` waits, and verified `captain-held` transfers, because a declared wait already names the human the fast escalation would report and is left to the watcher's own bounded pause cadence; a captain-held transfer remains silent without rechecks while the away-posture record exists. The push path only shortens latency. Polling runs every cycle and remains the permanent fallback when protocol 16, the event schema, Python, connection, subscription, or repeated reader execution is unavailable. @@ -320,8 +336,8 @@ For Herdr, target existence, native state, capture, composer state, and verified The pane-independent max-defer alert is configured in [`wedge-alarm.md`](wedge-alarm.md). Harnesses with native tracked background execution can run the daemon in their terminal. -Pi has no such mechanism. -`bin/fm-afk-launch.sh` therefore creates a dedicated unfocused Herdr workspace, runs the daemon there with an explicit supervisor target and backend, records the exact daemon pane, and closes only that pane on stop. +Pi and pi-signed launch the same away daemon, with extension standby owned by [`watcher-continuity.md`](watcher-continuity.md). +For another harness without native tracked background execution, `bin/fm-afk-launch.sh` creates a dedicated unfocused Herdr workspace, runs the daemon there with an explicit supervisor target and backend, records the exact daemon pane, and closes only that pane on stop. It never splits the captain's active tab and never uses shell `&`. Recovery reconciles only the recorded exact id. @@ -367,7 +383,9 @@ tests/fm-backend-herdr-workspace-per-home-e2e.test.sh tests/fm-backend-herdr-launcher-workspace-e2e.test.sh tests/fm-backend-herdr-presentation-e2e.test.sh tests/fm-backend-herdr-recovery-e2e.test.sh +tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh tests/fm-backend-herdr-eventwait-smoke.test.sh +tests/fm-control-herdr-smoke.test.sh tests/fm-herdr-session-cleanup.test.sh tests/fm-herdr-session-cleanup-e2e.test.sh tests/fm-afk-inject-herdr-e2e.test.sh diff --git a/docs/pi-supervision-branch.md b/docs/pi-supervision-branch.md index 76cb84a8b85..12f62616b34 100644 --- a/docs/pi-supervision-branch.md +++ b/docs/pi-supervision-branch.md @@ -22,7 +22,7 @@ The supervision branch itself is Pi-only by construction: ## Components and their owners - Wake dispatch: `.pi/extensions/fm-primary-pi-watch.ts` stays the dispatcher; `.pi/extensions/lib/fm-branch-dispatch.ts` owns the offer handshake and row eligibility, while [`watcher-continuity.md`](watcher-continuity.md#per-actor-acknowledgement) owns the per-actor consume contract. - A successful row grant transfers ownership of exactly the currently branch-eligible rows to the branch; a check-kind triggering close (merge-confirmation polls, Relay mentions, credential/auth failures, and every other legitimately main-only class) is never offered even when other rows are eligible, no acceptor (extension absent, away mode, branch broken) keeps today's wake-to-main path for that close, and watcher-failure alarms always go to main because only main can repair the watcher cycle. + A successful row grant transfers ownership of exactly the currently branch-eligible rows to the branch; a check-kind triggering close (merge-confirmation polls, Relay mentions, credential/auth failures, and every other legitimately main-only class) is never offered even when other rows are eligible, no acceptor (extension absent, legacy away daemon flag, branch broken) keeps today's wake-to-main path for that close, and watcher-failure alarms always go to main because only main can repair the watcher cycle. A decision-owned event surfaced by `bin/fm-watch.sh`'s signal path gets the identical treatment even though it keeps the ordinary `signal` kind. `signal_files_actionable` marks the queued payload `needs-decision:` for a newly surfaced `needs-decision`, a `captain-held` declaration surfaced through the no-verb fallback, or a pending-reply second-mate escalation; `scopeForUnreadWake` excludes every marked row from what the branch may claim. For a stale row, `scopeForUnreadWake` folds the mapped task's status log and excludes the row when any `needs-decision` remains open or the current meaningful declaration is `captain-held`; an unreadable or symlinked status log fails the scope closed rather than influencing routing. @@ -64,7 +64,7 @@ The supervision branch itself is Pi-only by construction: [`watcher-continuity.md`](watcher-continuity.md#per-actor-acknowledgement) owns the consume-side guarantee that neither actor can present or acknowledge the other's claim. Heartbeat keeps its own all-or-nothing recheck over the rows it can claim: it takes every branch-ownable unread row or none of them, and an unresolvable task-local row still defers the whole review to main. A producer can still append a row in the instant between that final check and drain startup; this accepted residual follows the confused-agent-grade boundary above rather than claiming adversarial queue isolation. - Away mode and a broken branch between its bounded recovery probes keep today's wake-to-main behavior. + A legacy away daemon flag and a broken branch between its bounded recovery probes keep today's wake-to-main behavior; the away-posture record alone leaves the branch active. ## Off-thread delivery @@ -150,8 +150,9 @@ No caching machinery beyond this exists, deliberately: any later dynamic content ## Away mode -Away mode carries over unchanged: while `state/.afk` exists the away daemon owns supervision, and the branch declines every wake offer for the duration. -What is new is only the attended path: outside away mode, the branch absorbs the routine majority that previously interrupted the captain's conversation, applying the same escalation etiquette the daemon applies while away. +On Pi, `/afk` confirms the away-posture record (`state/.afk-contract`, owned by `bin/fm-afk-contract.sh`) before launching the away daemon. +While `state/.afk` exists, the branch and primary watcher extension stand by under [`watcher-continuity.md`](watcher-continuity.md), so only the daemon owns the supervision cycle. +After away mode clears, the extension resumes one cycle and ordinary branch routing returns. ## Verification diff --git a/docs/remote-secondmates.md b/docs/remote-secondmates.md index 90664116731..878d5334d7d 100644 --- a/docs/remote-secondmates.md +++ b/docs/remote-secondmates.md @@ -45,7 +45,7 @@ The origin URL named for each project must be reachable from the remote account ## Non-interactive tool contract -No login or interactive shell ever runs on the remote host, so `~/.profile`, `~/.bashrc`, and `~/.zshrc` never contribute to the runtime `PATH`. +Remote job execution never runs a login or interactive shell, so `~/.profile`, `~/.bashrc`, and `~/.zshrc` never contribute to the job worker's runtime `PATH`. `bin/fm-remote-job-lib.sh` is the single owner of the worker `PATH` and builds it by filesystem discovery rather than by evaluating shell startup files. The authorized child sees `<remote-root>/bin` first, then a genuine account `~/.local/bin`, the nvm default version bin, asdf shims and install bins, mise shims and install bins, Nix directories, Homebrew directories, and the system tail `/usr/bin:/bin:/usr/sbin:/sbin`. Nvm selection follows the filesystem `alias/default` chain and chooses the highest matching installed semantic version, falling back to the highest installed semantic version when the alias is absent or has no installed match. @@ -54,6 +54,7 @@ The Nix and package-manager order after version-manager discovery is `~/.nix-pro Exact repeated entries are omitted. For the three Nix locations, a final `bin` symlink is resolved to its physical directory, while a path reached through symlinked ancestors remains in its documented position. Other final-component symlink directories, including `~/.local/bin`, are excluded. +Because `~/.local/bin` precedes the package-manager directories, a stale self-updated `herdr` there shadows the one the account's login shell may resolve; the Herdr adapter steps around a client the running server refuses and `fm-remote-doctor.sh` names which client it selected ([`herdr-backend.md`](herdr-backend.md#client-selection)). The entrypoint resolves `git` only from the operator portion before prepending `<remote-root>/bin` for the authorized child. A checkout-local `bin/git` therefore cannot authorize an untracked command, and a host with no operator `git` receives an install-or-wrapper diagnostic before command execution. @@ -99,6 +100,10 @@ bin/fm-on.sh <secondmate-id|ssh-alias> fm-remote-doctor.sh --fix ``` Over the plain SSH doctor bootstrap, it writes and reloads the Firstmate-owned `dev.firstmate.remote-job` and `dev.firstmate.herdr.fm-remote` launch agents on macOS, both scoped with `LimitLoadToSessionType=Aqua` and bootstrapped in `gui/<uid>`. +The Herdr agent runs [`bin/fm-remote-herdr-guard.sh`](../bin/fm-remote-herdr-guard.sh) through a shell in login mode with separate `-l` and `-c` arguments, resolving the remote account's executable labeled Directory Services `UserShell`, then an executable `$SHELL`, and finally `/bin/sh`, so the server inherits the account's own environment. +The `gui/<uid>` domain, not the login shell, is what gives that server and every pane it spawns the Aqua audit session and login-keychain access; a server born in any other session cannot read the login keychain, and every claude pane under it falls back to a stale plaintext credentials file and reports "Login expired". +Herdr's own SSH remote attach starts such a server when it finds none, and at boot it wins the `fm-remote` socket because sshd accepts connections before the login session exists, so the guard is what makes the launch agent converge: it execs the server in the foreground under launchd when nothing owns the socket, exits 0 when an Aqua-born server already does, and otherwise stops the foreign server and takes the session over, closing its panes so the parent firstmate relaunches its mates into the Aqua-born server. +`KeepAlive={SuccessfulExit=false}` lets that exit 0 rest instead of respawning against a held socket; the guard's header owns the decision table and [`bin/fm-remote-herdr-owner-lib.sh`](../bin/fm-remote-herdr-owner-lib.sh) owns the birth markers it reads. It starts the same workers directly on Linux, recreates the `~/.local/bin/fm-remote-entrypoint.sh` symlink when it is absent, and creates only Firstmate-owned required-tool wrappers that it can prove resolve to a version-manager target, stopping after one harness satisfies the at-least-one requirement. It never installs packages or overwrites a non-Firstmate file at a reserved wrapper path. The dedicated Herdr launch agent owns only the remote-secondmate `fm-remote` server and does not inspect, rewrite, start, stop, or require the user's interactive `default` session or its `dev.firstmate.herdr` launch agent. @@ -109,7 +114,7 @@ These steps are never automated and are always reported rather than silently att - The first console login on that Mac, and automatic login in System Settings > Users & Groups when the machine runs headless and must come back on its own after a reboot. - FileVault, which holds a reboot at pre-boot authentication before any login session exists. - Installing any missing required tool that no safe wrapper can resolve. -- The required remote tool set is `git`, `jq`, `herdr`, compatible `tasks-axi`, `treehouse`, and at least one of `claude`, `codex`, `opencode`, `pi`, `pi-signed`, `grok`, or `kimi`. +- The required remote tool set is `git`, `jq`, `herdr`, compatible `tasks-axi`, `treehouse`, and at least one of `claude`, `codex`, `opencode`, `pi`, `pi-signed`, `grok`, or `kimi`; macOS additionally requires `lsof` so the doctor and guard can prove which process owns the session socket. - Each worker runtime's own `/login`, and any keychain password prompt that login needs. Firstmate never writes an auto-login password, never changes FileVault, and never stores an account password. @@ -218,9 +223,8 @@ bin/fm-backlog-handoff.sh <id> <item-key>... For a remote route, `tasks-axi mv` first moves the dependency-closed set atomically from the primary backlog into `data/handoff/<id>.outbox.md`. The outbox is then copied to the remote handoff scratch directory and `fm-backlog-receive.sh` atomically ingests every destination-absent key under the remote backlog's own lock. -After receipt, the helper sends a marked routed-work instruction through the recorded remote endpoint and removes the outbox only after that wake is confirmed. -A failed wake leaves the remote backlog intact and the outbox available for `--resume-pending`; an unresolved send is reported without a blind resend. -Bootstrap retries pending outboxes and emits `SECONDMATE_HANDOFF:` only when one remains. +The [`bin/fm-backlog-handoff.sh`](../bin/fm-backlog-handoff.sh) header owns remote outbox release after receipt and stable wake-correlation retry behavior. +Bootstrap retries pending outboxes and wakes, and emits `SECONDMATE_HANDOFF:` only when an outbox remains. There is no two-phase journal and no additional tasks-axi release requirement. ## Sync, update, and retirement @@ -265,6 +269,7 @@ bin/fm-test-run.sh tests/fm-crew-state.test.sh bin/fm-test-run.sh tests/fm-remote-job.test.sh bin/fm-test-run.sh tests/fm-remote-transport-lanes.test.sh bin/fm-test-run.sh tests/fm-remote-doctor.test.sh +bin/fm-test-run.sh tests/fm-remote-herdr-guard.test.sh bin/fm-test-run.sh tests/fm-project-origin.test.sh bin/fm-test-run.sh tests/fm-secondmate-sync.test.sh bin/fm-test-run.sh tests/fm-remote-reply.test.sh @@ -274,6 +279,7 @@ bin/fm-test-run.sh tests/fm-remote-secondmate-trace-context.test.sh ``` The account-level checks the doctor performs - a real Aqua login session, a real `launchctl` domain, and a real herdr server - are only ever exercised against fixtures here, so the readiness gate's behavior on a genuine Mac remains an operator-run smoke test. +The audit-session facts the guard relies on are recorded with their commands in [runtime backend verification](verification/runtime-backends.md#fm-remote-server-birth-and-login-keychain-access). For a real-host smoke test, provision a disposable remote account and project, run the doctor and its repair against that account, launch the second mate, send one marked request, verify its correlated reply and structured fleet projection, simulate an unreachable host to confirm unknown-without-failover behavior, then retire only after the remote queue is empty. The deterministic suite is automated; real-host validation is still an operator-run smoke test and is not claimed by the repository tests. diff --git a/docs/scripts.md b/docs/scripts.md index 24084d1d894..f26513354fc 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -38,7 +38,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-remote-job-worker.sh` | Long-lived remote queue worker for tracked `fm-*.sh` commands in the account runtime | | `fm-remote-job-reap-orphans.sh` | Stop remote job workers left running by a pruned code root, never one whose checkout still exists | | `fm-remote-doctor.sh` | Check, and with `--fix` repair, one remote account's second-mate readiness (remote job worker, Herdr, Aqua launch agents, PATH, and required tools) | -| `fm-backlog-handoff.sh` | Move queued backlog items into a secondmate home and durably wake its recorded receiver | +| [`fm-backlog-handoff.sh`](../bin/fm-backlog-handoff.sh) | Move queued backlog items into a secondmate home; its header owns route-specific wake outcomes and retries | | `fm-backlog-receive.sh` | Idempotently ingest one confined remote handoff outbox through tasks-axi | | `fm-captain-hold.sh` | Hold tasks for the captain, record answers, gate completion, and report status/backlog divergence | | `fm-decision-hold.sh` | One-release compatibility shim mapping retired decision commands onto `fm-captain-hold.sh` | @@ -105,9 +105,10 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-watch-checkpoint.sh` | Run one bounded foreground watcher checkpoint for Codex-style supervision | | `fm-watch.sh` | Singleton-safe watcher: absorb benign wakes, detect stalled local-secondmate wake queues, and exit on actionable ones | | `fm-inactive-reconcile.sh` | Reconcile long-inactive direct crewmate terminal outcomes without forge access | +| `fm-afk-contract.sh` | Own the away-posture record: schema, mandate-clause fields and never-set scan, refusal naming the missing part, read-back, entry announcement, archive | | `fm-afk-start.sh` | Run the common sourceable away-mode daemon entry in the foreground | -| `fm-afk-launch.sh` | Own away-mode entry, exit, rollback, and any backend terminal lifecycle | -| `fm-afk-return.sh` | Own deterministic return shutdown, catch-up evidence, and the firstmate-actionable blocker gate | +| `fm-afk-launch.sh` | Own away-mode entry (read-back, confirm, record), exit, rollback, and any backend terminal lifecycle | +| `fm-afk-return.sh` | Own deterministic return shutdown, the return brief, catch-up evidence, and the firstmate-actionable blocker gate | | `fm-supervisor-target-lib.sh` | Resolve the shared supervisor target and backend for the daemon and launcher | | `fm-supervise-daemon.sh` | Presence-gated away-mode sub-supervisor: self-handle routine wakes, guard injection by the detected primary harness, escalate batched digests, alert on failed delivery | | `fm-crew-state.sh` | Print one deterministic current-state line for a crew | @@ -141,7 +142,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-control.sh` | Agent lifecycle control plane: allowlisted `interrupt`, `exit`, and transactional `relaunch` verbs for an exact task id ([agent-control.md](agent-control.md)) | | `fm-control-lib.sh` | One executable owner of the control-plane verb allowlist, per-harness interrupt/exit mechanics, and per-backend capability | | `fm-busy-lib.sh` | Single owner of the semantic busy-state contract: verdicts, source attribution, and per-harness sources | -| `fm-busy-event.sh` | The only writer of a task's semantic busy-state record; arms an incarnation and applies lifecycle events | +| `fm-busy-event.sh` | The only writer of a task's semantic busy-state record and native-harness progress marker; arms an incarnation and applies lifecycle events | | `fm-tmux-lib.sh` | Shared tmux pane primitives for composer capture, verified submit, and the submit-time busy check | | `fm-peek.sh` | Print a bounded tail of a crewmate endpoint | | `fm-check-register.sh` | Bind an intentional custom watcher check to its current bytes | @@ -171,7 +172,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-parent-channel-lib.sh` | Resolve a secondmate home's parent channel and append a captain-facing outcome line to it at most once | | `fm-promote.sh` | Promote a scout task in place to a protected ship task with an explicit delivery mode, and write the ship instructions carrying that mode's definition of done | | `fm-teardown.sh` | Fail-closed teardown: return landed ship or design worktrees, reap that one task's branch once its merge is proven, close this home's backlog item, require design decision inventory or completed scout deliverables, retire secondmate homes | -| `fm-harness.sh` | Detect the running harness and resolve crew or secondmate harness, model, and effort | +| `fm-harness.sh` | Detect the running harness, resolve crew or secondmate harness, model, and effort, and validate the native-only `ultra` effort | | `fm-lock.sh` | Per-home firstmate session lock | | `fm-x-lib.sh` | Shared Relay config, relay, and reply-threading helpers | | `fm-x-poll.sh` | One bounded Relay poll: stash newly offered mentions and emit their once-only wake | @@ -184,6 +185,9 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-public-followup-emit.sh` | Report one typed terminal work result into the home that owes the public reply, or stage it when that home is on another machine | | `fm-public-followup-collect.sh` | Read and retire the typed terminal results a remote work home staged for the home that owes the public reply | | `fm-inbox.sh` | The captain's out-of-band capture surface: queue a note, dictate one, read status, ask a side question | +| `fm-mail.sh` | General-purpose mail plane: read unseen IMAP mail, send one SMTP message, or surface new mail as a `check` wake via `poll` (configuration in the home's gitignored `.env`) | +| `fm-mail.py` | The IMAP/SMTP engine behind `fm-mail.sh` | +| `fm-mail-check.sh` | Standing received-mail poll: `arm` registers a watcher check that runs `fm-mail.sh poll` on the watcher cadence (new mail still wakes via the poll; the check's own line also wakes unless the poll is a proven no-op), `disarm` removes it | | `fm-voice-relay.py` | Hold the spoken conversation on this host, answer from the records, and hand real work to `fm-inbox.sh` ([voice-relay.md](voice-relay.md)) | | `fm-voice-client.py` | The laptop end of the spoken interface: capture, playback, and turn timing over SSH; audio devices unverified | | `fm_voice_frame.py` | The wire format both machines share, copied to the laptop beside the client | diff --git a/docs/secondmate-parent-channel.md b/docs/secondmate-parent-channel.md index 3e68e812ae4..a9e682c9945 100644 --- a/docs/secondmate-parent-channel.md +++ b/docs/secondmate-parent-channel.md @@ -9,7 +9,7 @@ This note records why a secondmate home's captain-facing outcomes are delivered A secondmate is a firstmate in its own home, and nobody reads its chat: the captain and the main firstmate see only what is appended to the parent channel. On 2026-09-02 four outcomes across two mate homes never reached the captain. The watcher had delivered the parent's request within a minute each time, the mate did the work, and then the mate addressed "captain" in its own chat instead of appending to the channel. -The cause is structural rather than a one-off lapse: `AGENTS.md` tells every firstmate to reach the captain and to address the captain in every response, while the charter's return-channel rule is a smaller, later instruction. +The cause is structural rather than a one-off lapse: the mate can satisfy the [address rule in `AGENTS.md`](../AGENTS.md#firstmate) in local chat while missing the charter's later return-channel instruction. The captain's framing of the requirement was: "the root problem is not specific to PRs, right? it looks like any message or outcomes from second mates can miss. we need to make sure our fixes are addressing this in a principled, fundamental way, not surgically treating the symptoms of just this PR update miss." A PR-ready report was the observed symptom, but a finding, a decision, a blocker, and a failure all fail the same way, because every one of them depended on the mate model remembering to write one line. @@ -42,7 +42,7 @@ A missed-reply escalation includes the complete first sighting path and line num ## What is deliberately not built -- No mirror of the mate's chat: every firstmate turn contains captain-facing text by mandate, so choosing which sentence is an outcome would itself be model behavior, and every harness exposes turn text differently. +- No mirror of the mate's chat: chat can mix outcomes with other conversation, so choosing which sentence is an outcome would itself be model behavior, and every harness exposes turn text differently. - No threshold escalation of a child's open decision or blocker: a decision the mate escalates is a captain hold, which is published; a decision the mate neither answers nor escalates is a supervision-quality question, separable from channel delivery. - No second watcher or standalone scanner: a lightweight ledger pass runs inside the existing inactive-outcome command on every watcher poll and reuses its receipts and upstream append. - No orphan lifecycle: teardown refuses instead of removing an undelivered outcome, the same way it refuses on other unlanded conditions. diff --git a/docs/supervision-protocols/claude.md b/docs/supervision-protocols/claude.md index f0d631f6493..477476da54d 100644 --- a/docs/supervision-protocols/claude.md +++ b/docs/supervision-protocols/claude.md @@ -18,7 +18,7 @@ When this session owns supervision and away mode is not active: No PreToolUse hook denies fleet commands based on watcher status. [`watcher-continuity.md`](../watcher-continuity.md) owns the exact session-lock recovery boundary. 8. The turn-end guard (`bin/fm-turnend-guard.sh --claude`) remains the final backstop. - It requires the PID-strict live-watcher and fresh-beacon predicate at the Stop boundary, while the mid-turn pull guard accepts a fresh beacon without a live process under Claude's between-turns auto-arm model. + It requires the PID-strict live-watcher and fresh-beacon predicate at the Stop boundary; [`turnend-guard.md`](../turnend-guard.md#guard-predicates) owns the distinct model-aware mid-turn pull-guard rules. It allows the stop when a watcher is healthy or an open auto-arm generation claim owns recovery, while fresh failure epochs advance the bounded one-time attended fail-open progression described in [`turnend-guard.md`](../turnend-guard.md). 9. Waiting on the hook-owned cycle is silent: do not send idle progress while the watcher is parked. diff --git a/docs/supervision-protocols/pi.md b/docs/supervision-protocols/pi.md index 898288eb35f..79694982bd6 100644 --- a/docs/supervision-protocols/pi.md +++ b/docs/supervision-protocols/pi.md @@ -1,6 +1,6 @@ Mode: Pi extension background wake. -When this session owns supervision and away mode is not active: +When this session owns supervision and no legacy away daemon flag is active: 1. Drain first with `bin/fm-wake-drain.sh`. After handling all emitted wakes and reconciling open decisions and unread status lines, run the exact `--ack-through` command printed as `WAKE_ACK_REQUIRED`; until then the work remains durable for idempotent re-handling after interruption. 2. Confirm the Pi primary auto-loaded both project extensions (plain `pi` or `pi-signed`, after approving project trust once per clone); if not, restart the selected executable with `-e __FM_PI_TURNEND_EXT__ -e __FM_PI_EXT__` as a trust-free fallback. @@ -29,7 +29,7 @@ That request is the one turn in which MAIN processes the outcome: give the capta Only that call closes the outcome; an unrelated, empty, or paraphrased answer leaves it open, and the current unprocessed sequence set is presented again at the next run boundary and at session start until it is acknowledged. The persisted entry is already the captain-visible record, so MAIN must not re-emit it verbatim merely because it appeared. Before MAIN steers, controls lifecycle, or cleans up a task, claim its lease with `bin/fm-lease.sh claim <task>` and release it afterwards; a refused claim means the branch is acting on that task right now. -This conversation still receives every other fleet-wide or unresolvable wake, the branch's wakes when it is unavailable or away mode is active, and every watcher-failure alarm regardless, so the arm and repair contract above is unchanged. +This conversation still receives every other fleet-wide or unresolvable wake, the branch's wakes when it is unavailable or a legacy away daemon flag is active, and every watcher-failure alarm regardless, so the arm and repair contract above is unchanged. Treat the merged fleet event as already handled for fleet operations: MAIN must not re-drain, re-run, or acknowledge it. Separately, MAIN applies judgment about whether and how to surface, summarize, reference, or incorporate a merged sailboat outcome in the captain conversation; event ownership does not decide the conversational treatment. Read the durable outcome store with the fm_branch_outcomes tool when the captain asks what happened. diff --git a/docs/turnend-guard.md b/docs/turnend-guard.md index abf95bc0425..4a50b2a4c20 100644 --- a/docs/turnend-guard.md +++ b/docs/turnend-guard.md @@ -36,7 +36,9 @@ A custom check registered with `bin/fm-check-register.sh` counts the same way, s Otherwise it calls `fm_watcher_healthy <state-dir> <watch-path> [grace-seconds] [home]` from `bin/fm-wake-lib.sh`, the same PID-strict identity-matched lock and fresh-beacon check used by `bin/fm-watch-arm.sh`: a stale beacon blocks even when a watcher pid is live, and a fresh leftover beacon blocks when the lock is missing, dead, or identity-mismatched. The turn-end guard needs that strict check because it fires at the turn boundary, where the auto-arm is bringing a fresh watcher up for the upcoming idle period, and it cooperates with that arm rather than trusting a beacon left by the cycle that just ended. `bin/fm-guard.sh`, the pull warning, instead uses the model-aware `fm_watcher_supervision_verdict` from the same library, because it fires mid-turn when the auto-arm model runs no watcher at all. -Under the Claude Stop auto-arm model a beacon fresh within grace is healthy even with no live watcher process, and only a beacon stale beyond grace (or absent) alarms. +Under the Claude Stop auto-arm model a beacon fresh within grace is healthy even with no live watcher process. +A stale beacon is still healthy while `fm_autoarm_midturn_healthy` in `bin/fm-wake-lib.sh` proves a Claude rewake explains the mid-turn gap: the rewake is bound to the current recovery generation and live session-lock owner, and no later watcher beacon or exhausted-failure marker supersedes it, because that session's turn-end will re-arm. +Without that proof a stale or absent beacon is a genuine lapse and alarms. Under the extension model (Pi, pi-signed, and omp) a live identity-matched watcher is the ordinary healthy state, but a genuinely unheld lock with a beacon fresh within grace is also healthy while a live Pi or omp session provably owns continuity, because `.pi/extensions/fm-primary-pi-watch.ts` and `.omp/extensions/fm-primary-omp-watch.ts` tear the watcher down on every actionable wake and spawn the replacement themselves. A lock is genuinely unheld only when the lock directory or its symlinked owner directory is absent, or when the existing lock records no pid at all. Any lock with a recorded pid remains down when its pid, home, watcher path, or process identity fails the strict watcher health check. @@ -95,7 +97,7 @@ In the default Codex mode, a true value lets the second stop finish after one fo Claude runs the guard with `--claude`, which ignores `stop_hook_active` and cooperates with the Stop-owned auto-arm. Claude Code sets `stop_hook_active=true` on every stop after any stop-hook continuation, including `asyncRewake` rewakes, which re-opened the 2026-07-21 blind window under the default one-shot behavior. The Claude mode waits up to `FM_CLAUDE_AUTOARM_SYNC_WAIT_MS` (default 800 milliseconds) and allows the stop when the watcher is healthy, the auto-arm's generation claim is open, or `state/.claude-autoarm-epoch` contains a fresh actionable rewake owned by this event epoch. -The claim is the ledger entry itself: the epoch sequence in `state/.claude-autoarm-epoch` is a monotonic claim generation, line 1 is the classic epoch record, and line 2 records the claiming process's mandatory pid-identity (`fm_autoarm_claim_open` and `fm_autoarm_claim_next` in `bin/fm-wake-lib.sh` own the contract). +The claim is the ledger entry itself: the epoch sequence in `state/.claude-autoarm-epoch` is a monotonic claim generation, line 1 records the claim and terminal outcome, and line 2 records the claiming process's mandatory pid-identity; `fm_autoarm_claim_open` and `fm_autoarm_claim_next` in `bin/fm-wake-lib.sh` own the format contract. A claim is open while its outcome is `arming`, its owner pid is alive, its recorded identity successfully recomputes and matches that pid, and it is not stuck - stuck meaning the entry and the watcher beacon are both older than the guard grace, which proves the owner hung mid-arm (a healthy hours-long foregrounded cycle keeps the beacon beating, and every arming phase with no watcher is bounded in seconds). Anything else - a finished outcome, a dead or identity-mismatched owner, a stuck owner, an identityless entry, or no entry - lets the next Stop-owned firing take the next generation and arm; taking a newer generation is the reclaim, and a steady-state predecessor is never signalled or revoked. No mutex is held across arming or output: `state/.claude-autoarm.lock` survives only as a micro-mutex serializing individual ledger writes, and a superseded owner goes completely silent - ownership is re-verified before every arm invocation, episode-state mutation, ledger write, and continuation. @@ -183,7 +185,7 @@ That warning uses `bin/fm-supervision-instructions.sh --repair-line`, so it alwa ## Regression coverage `tests/fm-turnend-guard.test.sh` covers the predicate, main and secondmate primary scope, child-worktree exclusion, `FM_HOME` and `FM_STATE_OVERRIDE` precedence, the live-lock and fresh-beacon guard predicate, the cooperative `--claude` open-generation claim wait, monotonic failed-epoch progression, bounded attended fail-open, post-alarm continuation suppression, positive recovery reset, generation and legacy claim cases that must block or clear instead of allowing a blind stop, away-mode daemon ownership between watcher cycles and over a watcher lock left behind by an exited watcher, plus its dead, pid-reused, absent, stale-beacon, and away-mode-off negatives, the away-mode beacon's poll-derived grace widening for a live daemon still mid-cycle and its bound against a dead daemon, a beacon older than that wider grace, and FM_POLL's inapplicability with away mode off, Pi logical-run latching, missing-`jq` behavior, all five primary registrations, Grok native and legacy selection, typed field precedence, malformed input, and exactly-one-path safety. -`tests/fm-guard-stale-banner.test.sh` covers the pull-guard predicate, including the persistent-model fresh-leftover-beacon negative control, the auto-arm model's healthy fresh-beacon-without-a-watcher case and stale-beacon alarm, and the extension model's live-watcher path, ownership-qualified fresh hand-off, held-lock failures, independently broken ownership signals, stale-beacon alarm, queued-wake warning, and Pi and pi-signed harness routing. +`tests/fm-guard-stale-banner.test.sh` covers the pull-guard predicate, including the persistent-model fresh-leftover-beacon negative control; the auto-arm model's healthy fresh-beacon-without-a-watcher case, session-and-recovery-bound long-turn rewake tolerance, independently broken tolerance signals, open-claim negative control, stale-beacon alarm, and isolation from other models; and the extension model's live-watcher path, ownership-qualified fresh hand-off, held-lock failures, independently broken ownership signals, stale-beacon alarm, queued-wake warning, and Pi and pi-signed harness routing. It also covers true-reason banner wording and reason-keyed episode dedup surviving a beacon mtime change. `tests/fm-cursor-primary.test.sh` covers the Cursor park end to end over real processes with no harness installed: each tracked Claude-shaped entrypoint standing down on a Cursor payload, both follow-up sources, the bounded repair nag and its reset, the nested loop bounds, supersession, away-mode and lock-ownership inertness, Pi-host stand-down without Cursor identity and continued parking when `PI_CODING_AGENT` leaks alongside `CURSOR_AGENT` or `CURSOR_INVOKED_AS`, child-worktree exclusion, and that the adapter never exits 2. `FM_CURSOR_PRIMARY_LIVE_E2E=1 tests/fm-cursor-primary-live-e2e.test.sh` is the opt-in guard that proves the same behavior against the installed cursor-agent and fails naming the harness and version. diff --git a/docs/verification/process-event-sources.md b/docs/verification/process-event-sources.md index 669a5b2c247..a2e52f9b5fe 100644 --- a/docs/verification/process-event-sources.md +++ b/docs/verification/process-event-sources.md @@ -124,7 +124,7 @@ Exercised by `tests/fm-procevent.test.sh` against a fake blocking source whose c | PID-reuse safety | retirement refuses a live PID whose identity differs from the claim before signalling, and a surviving process group prevents stale-generation cleanup on both ordinary and failed reservation-removal paths | | coherent ownership reads | a claim replacement held inside the source boundary blocks `list` until one complete generation is visible | | retire-start exclusion | a queued start revalidates registration after the serialized retirement boundary and executes no child | -| uncertain identity | a live owner whose identity probe transiently fails is not signaled or released, and its registration remains for retry | +| uncertain identity before the first signal | a live owner whose identity probe transiently fails is not signaled or released, and its registration remains for retry | | bounded home sweep | a non-mutating full-tree preflight precedes teardown, then registrations and claim-only owned sources retire through the ordinary safe path at each home-removal boundary | | sweep refusal | uncertain identity preserves the runner, claim, registration, home, lease, and parent retirement evidence for retry | | foreign ownership | sweeping one home removes its registration without signaling or releasing another home's live claim | @@ -178,30 +178,29 @@ The 2026-08-27 review inspected `bin/fm-harness.sh`, `bin/fm-supervision-instruc ## Runner lifetime and cleanup -A runner started by `reconcile` is its own process group leader and is reparented to init, so it outlives the shell that started it by design. -Removing a home's state directory does not stop an already-running child, and signalling only the runner leaves the blocking child alive. - -Three paths stop a runner generation through its verified process group: - -- The runner starts only after its separate owner guard confirms initialization; the guard stops the runner group after two consecutive checks cannot prove the owning home's recorded physical identity and lease freshness. -- `retire` resolves the runner PID and identity from this home's machine-wide claim, so retirement still works when the home's state is already gone. -- `reconcile` stops a runner this home owns whose source registration has been removed, and reports it as `stopped=N`. - -The owner guard and explicit cleanup paths reach the blocking source and its descendants through the runner's group. -The registration launch floor independently bounds repeated runner launches while an owner-loss lease is still valid. -The Lavish adapter's start-to-start poll governor separately bounds its internal retry loop under shipped defaults without delaying a normally blocking poll. -An attached public `start` maintains the lease for its caller's lifetime. -At the accepted confused-agent/accidental grade, the inherited `FM_PROCEVENT_IN_RUNNER` marker prevents detached runners and their ordinary children from refreshing it; adversarial unforgeability against a source that deliberately strips that marker is out of scope. - -The same group rule decides when a claim may be reclaimed, not only when a runner may be signalled. -A leader that died while its process group kept running is not a gone generation. -Because the leaderless group cannot be proved to belong to the recorded generation, `reconcile` preserves its claim without signalling it or starting a replacement. -Once a stale owner and an independent group check prove the whole generation gone, an unreachable token-keyed capture reservation cannot veto reclamation. -Known limit: when either a live reused PID or an absent leader makes group ownership ambiguous, the reaper does not act because it cannot prove the group is the orphan generation; storm-rate containment plus ordinary lease and reconcile cleanup are the confused-agent-grade backstop. -Known limit: identity and process-group verification cannot be made atomic with signalling in portable shell. -The reaper signals only a target it has verified as the orphan generation, but PID and group reuse remain possible in the narrow interval between verification and the signal; launch pacing is the primary host-wedge protection and watchdog cleanup is a backstop. - -`tests/fm-procevent.test.sh` covers owner-loss reaping, descendant churn cessation, cross-home scope, launch pacing, guard startup failure, attached-start continuity, explicit retirement, and stale-group reconciliation. +The [operating contract](../configuration.md#process-to-event-sources-stateprocevent) owns stop authority, the guard's lease and two-read debounce, claim reclamation, and the permanent leak and silent loss of listening after an unrelated leader death. + +Measured on 2026-09-08 on macOS (Darwin 25.5.0) against a stand-in poll child that traps TERM, INT, and HUP and keeps blocking: before the repair the guard signalled, lost the leader to that signal, and exited leaving the child running past 70 seconds. +The guard caused the permanent leak by destroying the leader needed to prove ownership; a guard that causes that leak is worse than no guard. +After the repair, the guard cleared that child in 7.7 seconds with a 5-second lease and 1-second check, and `retire` cleared the same shape in about 2.4 seconds. +The bound is the lease term plus ONE check interval plus the stop's grace period, roughly 620 seconds at the shipped 600-second lease and 15-second check - a 601-second lease term, one 15-second interval, and up to 4 seconds of stop, with two reads still required so a single unreadable read cannot kill a live runner. +What was tightened is the spacing of those two reads, not their number: half a check interval apart they both fit inside the single interval the bound budgets, where a full interval between them cost a second one. +The lease term is the configured lease plus one second because the age comparison is in whole seconds, and that rounding is part of the bound rather than slack. +The figure and that reason belong together: a number recorded without why it is that number is the one a later reader shortens. +On the same date and host, retiring a healthy runner fell from about 2.8 seconds with a forced group signal every time to about 0.6 seconds with the ordinary signal alone. +The circular lock wait described at `release_start_claim` in [`bin/fm-procevent.sh`](../../bin/fm-procevent.sh) explains why healthy runners required the forced signal; the measured delay was the stop waiting for exit cleanup that could not acquire its lock. + +Measured on 2026-09-09 on the same host, reaping an orphaned listener whose home stopped refreshing its lease and sampling the phase between the guard's check clock and the lease clock across eight runs per variant: 3.5 to 4.7 seconds with the two reads half an interval apart against 4.4 to 5.3 seconds with a full interval between them, at a 2-second lease and 1-second check, and 5.9 to 6.1 against 7.7 to 8.1 seconds at a 2-second lease and 4-second check. +The regression pins that phase rather than sampling it, because a sampled phase lets a guard spending two intervals pass on a lucky alignment; it prints its own figure, 13.2 seconds after the last owner activity against a documented 15-second bound at a 7-second lease and 6-second check, and a guard given a full interval between its two reads breached that deadline. +A guard that acted on a single failed read instead reached the same reaping in 9.9 seconds, so the UNSAFE variant is the faster one. +That is why the bound and the debounce are pinned by separate cases: a change trading one away for the other would otherwise register only as an improvement. + +[`tests/fm-procevent.test.sh`](../../tests/fm-procevent.test.sh) exercises these reproductions through the executable interface: a TERM-surviving child under both `retire` and the guard, escalation with an absent or zombie leader or probes configured to become unreadable after TERM, and refusal of mismatched live identities or nonleaders before the first signal. +The healthy-runner case requires the attached `start` to return status 143 (TERM); the printed retirement duration and sampled stop windows are supplementary evidence, not a timing-based pass condition. +Two cases pin the guard's own numbers rather than only its outcome: one reaps an orphaned listener within the lease term plus a single check interval, with the lease expiry deliberately placed late in that interval, and one fails exactly one lease read against a home that is still alive and requires the runner to survive it. +They fail for opposite reasons, which is the point of keeping them apart. +The crashed-leader cases separately pin refusal and claim preservation when a leader dies outside the stop's own signal, so successful escalation cannot be mistaken for closing that limit. +Refresh the regressions with `bash tests/fm-procevent.test.sh`; the dated measurements above are recorded observations, not fixed timing thresholds. ## Portability finding diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index eaffdc064f9..71b2af21e2f 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -734,6 +734,69 @@ The CLI matrix was checked directly: All destructive verification used `bin/fm-herdr-lab.sh` with a non-default `fm-lab-` name and a byte-identical default-session tripwire. No ambient `herdr server stop` command is a supported test operation. +### fm-remote server birth and login-keychain access + +Measured 2026-09-09 on macOS 26 (Darwin 25.6.0) aarch64 with Claude Code 2.1.266 and Herdr 0.9.0, the guarantee behind `bin/fm-remote-herdr-guard.sh` and the doctor's `herdr-server` check: login-keychain access follows the audit session a process was born into, never the launch shape or the shell. + +Same user, same `HOME`, same login keychain item, three births, probed with `launchctl managername`, `getaudit_addr` (a compiled probe), `security find-generic-password -a "$USER" -w -s "Claude Code-credentials"` (output withheld), and `claude auth status`: + +| Birth | `managername` | audit session | `security ... -w` | `claude auth status` | +| --- | --- | --- | --- | --- | +| `gui/501` LaunchAgent, bare `ProgramArguments`, `launchctl bootstrap` + `kickstart -k` mid-session | Aqua | asid 100038 (the `gui/501` asid), `HAS_GRAPHIC_ACCESS HAS_TTY HAS_CONSOLE_ACCESS HAS_AUTHENTICATED` | exit 0 | `loggedIn: true` | +| `gui/501` LaunchAgent, `zsh -l -c 'exec ...'`, same reload | Aqua | asid 100038, same flags | exit 0 | `loggedIn: true` | +| `user/501` LaunchAgent (`LimitLoadToSessionType=Background`), same reload | Background | asid 100056, flags `0x0` | exit 36 `User interaction is not allowed.`, item metadata still readable | `loggedIn: false`, `authMethod: none` | + +Claude Code 2.1.266 maps that exit 36 (and 44) to "no keychain data" and reads `~/.claude/.credentials.json` instead; with a stale file it prints `Failed to authenticate: OAuth session expired and could not be refreshed` (interactive: `Login expired · Please run /login`). + +Candidate birth markers were read with `ps -Eww -o command= -p <pid>` for own-uid processes, noting that macOS hides the environment of Apple platform binaries such as `/bin/sleep` and that a herdr server is never one. + +```text +launchd-born herdr server (child of launchd, gui/501): XPC_SERVICE_NAME=org.nix-community.home.herdr-server no SSH_* +SSH-born herdr server (child of `herdr --session fm-remote remote-client-bridge` under `sshd-session: user@notty`): SSH_CLIENT=... SSH_CONNECTION=... no XPC_SERVICE_NAME +``` + +`XPC_SERVICE_NAME` identifies a launchd label but does not identify its domain, because the Background `user/501` job also carried that variable while lacking keychain access. +The owner classifier therefore accepts that label only when `launchctl print gui/<uid>/<label>` identifies the owner pid or the label is loaded in `gui/<uid>` but not `user/<uid>`. +`XPC_SERVICE_NAME=0`, including a value inherited by a herdr live-handoff child, remains unknown. +`FM_REMOTE_JOB_ACTIVE=1` proves the Aqua worker only when `dev.firstmate.remote-job` is loaded in `gui/<uid>` but not `user/<uid>`. + +The SSH-born row was read on the remote host whose `dev.firstmate.herdr.fm-remote` job showed `state = spawn scheduled`, `runs = 239`, `last exit code = 1` and a log repeating `error: herdr server is already running`: herdr's remote attach had started the session's server as its own child before the login session existed, and launchd's copy lost the socket on every retry. +`pgrep -f` did not list the herdr server's argv on macOS; `lsof -U -a -c herdr -F pn` named the socket owner. + +A separate foreground-supervision check ran on 2026-09-09 on macOS 26 (Darwin 25.6.0) with Herdr 0.9.0 using the throwaway Aqua launch agent `dev.fm-rca.herdr-fg`. +Its `ProgramArguments` ran `/run/current-system/sw/bin/zsh -l -c "exec /etc/profiles/per-user/kunchen/bin/herdr server --session fm-lab-fg-90381-18985"`, with `KeepAlive={SuccessfulExit=false}` and `ThrottleInterval=10`, after `launchctl bootstrap gui/501 <plist>` and `launchctl kickstart -k gui/501/dev.fm-rca.herdr-fg`. +`launchctl print gui/501/dev.fm-rca.herdr-fg` reported `state = running` and `pid = 4806`. +`lsof -U -a -c herdr -F pn` named pid 4806 as the owner of `~/.config/herdr/sessions/fm-lab-fg-90381-18985/herdr.sock`. +`ps -o pid,ppid,command -p 4806` reported `4806 1 /etc/profiles/per-user/kunchen/bin/herdr server --session fm-lab-fg-90381-18985`, and its environment carried `XPC_SERVICE_NAME=dev.fm-rca.herdr-fg`. +No other herdr process existed for that session, and after 15 seconds the job remained running with pid 4806. +After a guarded `herdr session stop`, the job reported `state = not running` and `last exit code = 0`, and it stayed at rest through the throttle interval. +A second `launchctl kickstart -k gui/501/dev.fm-rca.herdr-fg` started pid 45574, which was also the new socket owner. +This proves that `herdr server` remains in the foreground as the launchd job, so the guard's final `exec` supplies the intended supervision and the earlier server that survived `launchctl bootout` was the unrelated SSH-bridge-born process. + +`bin/fm-test-run.sh tests/fm-remote-herdr-guard.test.sh` pins the resulting decision table against real marker-carrying processes, and `tests/fm-remote-doctor.test.sh` pins the doctor's verdicts on the same markers. + +### Client selection + +Measured 2026-09-08 on a macOS aarch64 host running a Herdr 0.9.0 server (protocol 22) for the `fm-remote` session while `~/.local/bin/herdr` still held the self-updated 0.8.2 client (protocol 20) ahead of the Nix-managed 0.9.0 client on the remote-job `PATH`. + +```sh +~/.local/bin/herdr --version +~/.local/bin/herdr pane get wCY:p2 --session fm-remote; echo "rc=$?" +~/.local/bin/herdr status --json --session fm-remote | jq -c '{c:.client.protocol,s:{running:.server.running,protocol:.server.protocol,compatible:.server.compatible}}' +herdr status --json --session fm-remote | jq -c '{c:.client.protocol,s:{running:.server.running,protocol:.server.protocol,compatible:.server.compatible}}' +``` + +```text +herdr 0.8.2 +{"id":"cli:pane:get","error":{"code":"protocol_mismatch","message":"client protocol 20 is older than server protocol 22; upgrade the Herdr client before using this command"}} +rc=1 +{"c":20,"s":{"running":true,"protocol":22,"compatible":false}} +{"c":22,"s":{"running":true,"protocol":22,"compatible":true}} +``` + +The refusal is a JSON error on stderr with exit 1 and empty stdout, and both client generations report `.server.compatible` and `.server.protocol` per named session, which is what the selection in `bin/backends/herdr.sh` reads. +`tests/fm-backend-herdr.test.sh` pins the bypass, same-process same-session caching, cross-session isolation, forced reselection, and both status shapes against fakes; `tests/fm-backend-herdr-smoke.test.sh` refreshes the real status normalization against the installed binary's running lab server. + ### Submit confirmation Measured 2026-08-19 against Herdr 0.8.0 and Claude Code 2.1.236 in an isolated `fm-lab-` session. @@ -872,7 +935,7 @@ ok - real Herdr lab: multi-home exact-pane teardowns restore captain focus witho ok - real Herdr lab validation completed on Herdr 0.7.4 with the default-session tripwire intact ``` -The suite also covers lost or failed move responses, active-tab refusal, restart husks, missing and duplicate tokens, manual renames, concurrent cleanup, and exact focus restoration. +The suite also covers lost or failed move responses, restart husks, missing and duplicate tokens, manual renames, concurrent cleanup, and exact focus restoration. The mandatory projection suite ran again on 2026-07-24 against Herdr 0.7.5 protocol 16: @@ -1111,6 +1174,70 @@ A separate live reading on 2026-08-15 found the other dead-pane shape the classi The classifier read that endpoint `dead` while four concurrently live Pi and Claude panes on the same server all read `alive`; the portable `test_agent_state_*` cases in `tests/fm-backend-herdr.test.sh` pin the chain and negative shapes. That command is the guard that refreshes this record; run it after every Herdr upgrade rather than trusting the version above. +For Pi on Herdr 0.9.0, `herdr agent get` reflects whether the agent process remains live; its registration does not persist merely because the pane and parent shell do. +A Pi launched as a child of the pane shell (not via `exec`) that then `/quit`s or is SIGKILL'd leaves the pane and shell in place, and `agent get` returns `agent_not_found`. +A sibling live idle Pi stays `agent=pi` with `agent_status=idle`. +`fm_backend_herdr_pane_agent_state` maps that `agent_not_found` leftover shell to `no-agent` and `fm_backend_herdr_agent_state` maps it to `dead` (relaunch-allowed), while the live idle pane stays `alive`. +`herdr pane get` `.agent_status` can still read `idle` after the occupant is gone; liveness is `agent get`, never that pane field. + +```sh +tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh +``` + +Refresh that live pair after every Herdr upgrade. Observed 2026-09-10 on Herdr 0.9.0 / protocol 22 with Pi 0.82.0 in an isolated `fm-lab-` session: + +```text +ok - agent get distinguishes leftover-shell (dead/no-agent) from live idle Pi +ok - pane get agent_status lag cannot keep an exited occupant classified alive +``` + +### Endpoint recovery classification + +Measured 2026-09-10 on macOS aarch64 against Herdr 0.9.0 (protocol 22) in an isolated `fm-lab-` session. + +An endpoint recorded in a session whose server is not running cannot be read by any operational call, and `status` is the one command that answers with a body instead of refusing: + +```sh +herdr pane get w1:p2 --session fm-lab-never-started +herdr status --json --session fm-lab-never-started | jq -c "{running: .server.running, status: .server.status}" +``` + +```text +{"id":"cli:pane:get","error":{"code":"server_not_running","message":"no herdr server is running at /Users/kunchen/.config/herdr/sessions/fm-lab-never-started/herdr.sock; run `herdr session attach fm-lab-never-started` to start or attach it"}} +{"running":false,"status":"not_running"} +``` + +`fm_backend_herdr_agent_state` therefore settles an uninterpretable pane read with `.server.running` rather than with the `server_not_running` error code, which keeps the verdict working across the supported range: the field is present on 0.8.2 protocol 20 and 0.9.0 protocol 22 alike (measured in "Client selection" above), while the code is not. +Only that recovery-grade read is widened; the husk classifier under it stays strict, because it licenses closing panes. +Observed in the lab, in one run: + +```text +live agent-free pane dead +endpoint in a session with no server missing +malformed target unreadable +``` + +The same run drove `bin/fm-spawn.sh --relaunch` against a real Herdr pane whose shell had been moved outside its recorded worktree: the shell was told once to return, ended in the recorded worktree, and the replacement was launched into the SAME pane, leaving one task tab. + +Herdr 0.8.x is not installed on this host, so protocol-20 coverage is structural plus the adapter fixture exercising both response shapes; it is not a live result. +Refresh the live half, which fails naming the installed version, with: + +```sh +tests/fm-control-herdr-smoke.test.sh +``` + +Observed 2026-09-10: + +```text +ok - real herdr 0.9.0: a gone session reads recoverable while a live pane and a malformed target do not +ok - real herdr: a drifted agent-free shell returns to its worktree and reuses the same endpoint +``` + +`tests/fm-backend-herdr.test.sh` pins the logic portably by driving the two signals apart - the same failed pane read yields `missing` under a stopped server and `unreadable` under a running one - and asserts that the husk classifier still refuses on that identical read. +`tests/fm-control-herdr-smoke.test.sh` proves the Herdr-only drift recovery against a real binary in an isolated lab session. +`tests/fm-control-relaunch.test.sh` drives a tmux stub and proves that tmux retains its prior refusal without sending `cd` or any other input to the pane. +The Herdr refusal when a shell accepts the command but does not move is not exercised in this change. + ### Away-mode transport The Pi/Herdr return and injection path was reverified on Herdr 0.7.3 and Pi 0.80.7: @@ -1695,6 +1822,20 @@ ok - tracked Pi extensions pass strict no-emit typecheck against Pi 0.85.1 These are samples from this host; the guard compares each run with its own unloaded floor and does not assert those exact durations on another run. No provider request or credentialed live-model behavior is covered by these probes. +## Native Codex through Pi + +Verified on 2026-09-08 with Pi 0.85.1 and the installed `pi-codex-native` 0.2.1 adapter. +Run this token-free guard after updating Pi, Codex, or the adapter: + +```sh +FM_PI_CODEX_NATIVE_LIVE=1 bash tests/fm-pi-codex-native.test.sh +``` + +Observed result: `"result": "PASS"`. +The guard runs the real Pi runtime, native adapter, three FirstMate primary extensions, native MCP transport, and FirstMate's durable outcome scripts in an isolated home. +It verifies native `ultra` on initial and operational turns and after restart, startup operational input, watcher arming, a notification while main is idle, outcome read and one acknowledgement, refusal of a duplicate acknowledgement, and no reprocessing after restart. +Its native App Server peer and watcher-close process are deterministic fixtures; it does not claim a real backend or a live model was tested by that command. +`tests/fm-busy-state.test.sh`, `tests/fm-busy-adapter-wiring.test.sh`, and `tests/fm-watch-triage.test.sh` cover separate progress notification, unchanged semantic busy state, rejection of a superseded worker's events, and progress refreshing the busy-age bound without fabricating a completed turn. ### 2026-09-14 overlapping primary prompts and loaded-marker writers diff --git a/docs/verification/supervision.md b/docs/verification/supervision.md index 683924616be..0abc7f88581 100644 --- a/docs/verification/supervision.md +++ b/docs/verification/supervision.md @@ -369,7 +369,7 @@ fm-doc-audience-check: ok surfaces=61 local_links=174 FM_TEST_SUMMARY total=4 failed=0 skipped_gate=0 duration_ms=102585 ``` -The model-aware pull-guard predicate correction (`bin/fm-guard.sh` no longer reports a false watcher-down mid-turn under the Claude Stop auto-arm model, where the watcher runs only between turns) was verified on 2026-08-04 with the installed ShellCheck 0.11.0 and the same isolated behavior suites. +The fresh-beacon portion of the model-aware pull-guard predicate (`bin/fm-guard.sh` accepts a beacon within grace without a live watcher under the Claude Stop auto-arm model) was verified on 2026-08-04 with the installed ShellCheck 0.11.0 and the same isolated behavior suites. ```sh bin/fm-lint.sh diff --git a/tests/fixtures/fm-brief-no-issue.sha256 b/tests/fixtures/fm-brief-no-issue.sha256 index c5b929b62e0..c49aedbd5d5 100644 --- a/tests/fixtures/fm-brief-no-issue.sha256 +++ b/tests/fixtures/fm-brief-no-issue.sha256 @@ -2,7 +2,7 @@ 6e21288c5c74cbce14f1727661ac3c2f1e4681592f576479bcff11db06e9648e 8118 golden-direct 86ea14d4c05cae9af4b72e993aa78f144d5b15d9fc0d7037e2eac3aabec23bd7 8335 golden-local f4cf36f17260bbac40ed7176d22e051061cd816adf1915b9334f522359bdab56 14432 golden-ship-herdr -f3f5919dd5d778c8fda98693ebc1a3a61e759c98d5479169ba1ec323fe759b04 5978 golden-scout -0c86493c6d0a27658c5646b938cc55b164dc81bd2223aeaa3c1ffdb60f58604e 7651 golden-scout-herdr -dcdf12271a3ac2f675608112acbbc53e05de8451017433d34f1fac0da8cc7593 7809 golden-secondmate -501e98877e2814986279f14ce6ebf8dac2e2d406cbf0914aa8f989b701e30185 8072 golden-secondmate-empty +cdd95282b547074c5e6a5e3387d0633268767f9230f79df0fbee4048b0a4f189 6123 golden-scout +4acd66839504297ff310e55868912c5e11837ad71abd3b26a2bbd01928f2fc0b 7796 golden-scout-herdr +9b490e1e5657cf429609c87ed39befc5ad84ee1577567583c389efed7faca723 7885 golden-secondmate +a1205e99bae20d234ef4e4d9e7f14acd6d0528f87efee6381a07e7fe8eebcafc 8148 golden-secondmate-empty diff --git a/tests/fm-afk-contract.test.sh b/tests/fm-afk-contract.test.sh new file mode 100755 index 00000000000..1ad4a6cd05b --- /dev/null +++ b/tests/fm-afk-contract.test.sh @@ -0,0 +1,546 @@ +#!/usr/bin/env bash +# tests/fm-afk-contract.test.sh - the away-posture record owner +# (bin/fm-afk-contract.sh): the mandate-clause fields, the structural refusal +# naming the missing part, the never-set scan, the read-back rendering, the entry announcement (hold-for- +# return only), the propose/confirm lifecycle with verbatim words, the refresh +# and replace rules, the archive at return, and the read subcommands every +# consumer uses instead of parsing the file. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +CONTRACT="$ROOT/bin/fm-afk-contract.sh" +TMP_ROOT=$(fm_test_tmproot fm-afk-contract-tests) + +make_home() { # <name> -> prints the home dir + local dir="$TMP_ROOT/$1" + mkdir -p "$dir/state" + printf '%s\n' "$dir" +} + +contract() { # <home> <args...> + local home=$1 + shift + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" "$CONTRACT" "$@" +} + +# compile_refusal <expected-missing-fragment> <label> <field flags...> +compile_refusal() { + local expected=$1 label=$2 home out rc + shift 2 + home=$(make_home "refuse-$RANDOM-$$") + set +e + out=$(contract "$home" propose "$@" 2>&1) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "$label: expected exit 3 for a refused clause, got $rc: $out" + assert_contains "$out" "refused: missing $expected" "$label: the refusal did not name the missing part" + assert_contains "$out" ' (none)' "$label: a refused-only proposal should list no accepted clause" +} + +# compile_accept <expected-readback-line> <label> <field flags...> +compile_accept() { + local expected=$1 label=$2 home out rc + shift 2 + home=$(make_home "accept-$RANDOM-$$") + set +e + out=$(contract "$home" propose "$@" 2>&1) + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "$label: expected exit 0 for an accepted clause, got $rc: $out" + assert_contains "$out" "$expected" "$label: the accepted clause was not read back as given" +} + +# compile_flagged <concept> <label> <field flags...>: the clause is recorded +# (exit 0, listed as accepted) and carries the best-effort never-set flag. +compile_flagged() { + local concept=$1 label=$2 home out rc + shift 2 + home=$(make_home "flag-$RANDOM-$$") + set +e + out=$(contract "$home" propose "$@" 2>&1) + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "$label: a flagged clause must still be recorded (exit 0), got $rc: $out" + assert_contains "$out" "flagged: names '$concept', a never-set concept that is never pre-authorizable; recorded, judged at execution" "$label: the read-back did not show the flag" + assert_not_contains "$out" 'refused: missing object' "$label: a never-set match must flag, never refuse" + [ "$(contract "$home" flags --proposal | cut -f2)" = "$concept" ] || fail "$label: flags did not name the concept: $(contract "$home" flags --proposal)" +} + +# compile_unflagged <label> <field flags...>: an ordinary name is neither +# refused nor flagged. +compile_unflagged() { + local label=$1 home out rc + shift + home=$(make_home "plain-$RANDOM-$$") + set +e + out=$(contract "$home" propose "$@" 2>&1) + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "$label: an ordinary clause was refused: $out" + assert_not_contains "$out" 'flagged:' "$label: an ordinary name was flagged" + [ -z "$(contract "$home" flags --proposal)" ] || fail "$label: flags listed an ordinary clause" +} + +# The structural check refuses only a missing field or an unlisted verb, and +# names the missing part every time. +test_fields_refuse_each_missing_part_by_name() { + compile_refusal "action - 'fix' is not a mandate verb" 'unknown verb' --action fix --object 'whatever breaks' --when 'it breaks' + compile_refusal 'action - the clause names no action' 'empty verb' --action '' --object 'task x PR' --when 'checks green' + compile_refusal 'object - the clause names no thing to act on' 'no object' --action merge --when 'checks green' + compile_refusal 'object - the clause names no thing to act on' 'blank object' --action merge --object ' ' --when 'checks green' + compile_refusal 'when - the clause states no precondition' 'no when' --action merge --object 'task x PR' + compile_refusal 'when - the clause states no precondition' 'blank when' --action merge --object 'task x PR' --when ' ' + compile_refusal 'stop - --stop was given with no text' 'blank explicit stop' --action merge --object 'task x PR' --when 'checks green' --stop ' ' + pass "structural refusals name their missing part" +} + +test_omitted_stop_confirms_as_no_stop() { + local home row out + home=$(make_home omitted-stop) + contract "$home" propose --action merge --object 'task x PR' --when 'checks green' >/dev/null || fail "proposal without stop failed" + row=$(contract "$home" clauses --proposal) + [ "$(printf '%s' "$row" | cut -f5)" = - ] || fail "an omitted stop was not serialized as the no-stop marker" + out=$(contract "$home" confirm 2>&1) || fail "confirmation without stop failed: $out" + assert_contains "$out" '1 mandate clause(s) recorded, 0 refused' "confirmation did not accept the omitted stop" + pass "an omitted stop uses the no-stop marker and confirms" +} + +# The never-set is a coarse best-effort flag: a listed concept, exact or plainly +# inflected, across punctuation boundaries, flags the clause without refusing it; +# an unrelated name never matches; joined compounds are a documented miss. +test_never_set_flags_without_refusing_and_never_over_matches() { + compile_flagged credential 'credentials' --action answer --object 'the credential prompt on task q' --when asked + compile_flagged legal 'legal' --action answer --object 'the legal acceptance on task q' --when asked + compile_flagged 'attended prompt' 'attended prompt' --action answer --object 'the attended prompt on task q' --when asked + compile_flagged credential 'credential compound' --action answer --object 'task q credential-prompt' --when 'prompt starts' + compile_flagged credential 'credential plural with punctuation' --action answer --object 'task q credentials/keys' --when 'prompt starts' + compile_flagged 'attended prompt' 'attended plural compound' --action answer --object 'task q attended-prompts' --when 'it appears' + compile_flagged payment 'payment plural' --action answer --object 'task q payments' --when 'prompt starts' + compile_flagged 'one time code' 'one-time code' --action answer --object 'task q one-time-code prompt' --when 'it appears' + compile_flagged 'one time code' 'one-time codes plural' --action answer --object 'task q one-time-codes prompt' --when 'it appears' + compile_flagged 'api key' 'api keys plural' --action answer --object 'task q api-keys prompt' --when 'it appears' + compile_flagged login 'in the precondition' --action merge --object 'task x PR' --when 'after the Login/2FA prompt clears' + compile_flagged password 'in the stop' --action merge --object 'task x PR' --when 'checks green' --stop 'if a PASSWORD is asked' + compile_unflagged 'ping-service is not pin' --action merge --object 'task ping-service PR' --when 'checks green' + compile_unflagged 'tokenize-worker is not token' --action rerun --object 'task tokenize-worker' --when 'after clause 1' + compile_unflagged 'pinned is not pin' --action merge --object 'task pinned-deps PR' --when 'checks green' + compile_unflagged 'legally is not legal' --action rerun --object 'task legally-named' --when 'after clause 1' + compile_unflagged 'joined compound is a documented miss' --action answer --object 'task q oneTimeCode prompt' --when 'it appears' + pass "the never-set flags listed concepts and their inflections without refusing, and never fires on unrelated names" +} + +# No parser reads the object or precondition: any text the captain gives is +# recorded verbatim, including wording a grammar would have judged. +test_fields_record_the_captain_wording_verbatim() { + compile_accept '1. merge task nm-windows-fix-r1 PR when checks green' 'green merge' \ + --action merge --object 'task nm-windows-fix-r1 PR' --when 'checks green' + compile_accept "1. merge task x's PR when checks green" 'possessive PR role' \ + --action merge --object "task x's PR" --when 'checks green' + compile_accept '1. merge task y PR when red on nm-ci-windows' 'red merge with the failing check named' \ + --action Merge --object 'task y PR' --when 'red on nm-ci-windows' + compile_accept '1. merge task y PR when even if nm-ci-windows is red stop the captain returns' 'stop field' \ + --action merge --object 'task y PR' --when 'even if nm-ci-windows is red' --stop 'the captain returns' + compile_accept '1. abort-run no-mistakes run for task nm-ci-windows-git-shard-split-r1 when install deadlocks' 'named event' \ + --action abort-run --object 'no-mistakes run for task nm-ci-windows-git-shard-split-r1' --when 'install deadlocks' + compile_accept '1. wake-me task fix-windows when at 2026-09-08T08:00Z' 'time precondition' \ + --action wake-me --object 'task fix-windows' --when 'at 2026-09-08T08:00Z' + compile_accept '1. discard the worktree of task w when its rerun fails twice' 'named discard' \ + --action discard --object 'the worktree of task w' --when 'its rerun fails twice' + compile_accept '1. merge task x PR when looks red enough, honestly' 'wording is recorded, never judged' \ + --action merge --object 'task x PR' --when 'looks red enough, honestly' + compile_accept '1. dispatch these queued items when the windows lane is green' 'dispatch' \ + --action dispatch --object 'these queued items' --when 'the windows lane is green' + pass "clause fields are recorded verbatim, and no static parser judges the wording" +} + +test_clause_fields_round_trip_reversible_whitespace() { + local home rows out object when stop expected + home=$(make_home clause-whitespace) + object='task x PR' + when=$'checks\tgreen\nthen done' + stop=$'stop\\literal\n' + contract "$home" propose --action merge --object "$object" --when "$when" --stop "$stop" >/dev/null \ + || fail "proposal with whitespace-bearing clause fields failed" + rows=$(contract "$home" clauses --proposal) + expected=$(printf '1\tmerge\ttask x PR\tchecks\\tgreen\\nthen done\tstop\\\\literal\\n') + [ "$rows" = "$expected" ] || fail "clause TSV did not reversibly preserve whitespace: $rows" + out=$(contract "$home" readback --proposal; printf x) + out=${out%x} + assert_contains "$out" $'1. merge task x PR when checks\tgreen\nthen done stop stop\\literal' \ + "read-back did not render clause fields verbatim" + pass "clause fields preserve repeated spaces, tabs, newlines, and backslashes" +} + +test_clause_ids_are_input_ordinals_across_accepted_and_refused() { + local home out rc + home=$(make_home ordinals) + set +e + out=$(contract "$home" propose \ + --action merge --object 'task a PR' --when 'checks green' \ + --action merge --object regardless \ + --action prerelease --object 'repo r' --when 'after clause 1' \ + --action install --object 'the prerelease on mini' --when 'after clause 3' \ + --action rerun --object 'task t' --when 'after clause 4' 2>&1) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "a mixed proposal should exit 3 (rc=$rc): $out" + assert_contains "$out" '1. merge task a PR when checks green' 'clause 1 accepted' + assert_contains "$out" '2. "action=merge object=regardless when=(none)" - refused: missing when - the clause states no precondition' 'clause 2 refused for its missing precondition' + assert_contains "$out" '3. prerelease repo r when after clause 1' 'clause 3 keeps its input ordinal' + assert_contains "$out" '4. install the prerelease on mini when after clause 3' 'clause 4 keeps its input ordinal' + assert_contains "$out" '5. rerun task t when after clause 4' 'clause 5 keeps its input ordinal' + [ "$(contract "$home" propose --action merge --object 'task a PR' --when 'checks green' --action merge --object regardless --action rerun --object 'task t' --when 'after clause 1' 2>/dev/null | grep -c '^ [0-9]')" -eq 3 ] \ + || fail "the read-back did not list every clause once" + pass "clause ids are input ordinals across accepted and refused clauses" +} + +test_readback_renders_words_verbatim_and_both_lists() { + local home out words + home=$(make_home readback) + words="$home/words.txt" + printf 'drive the windows fix to green and merge it,\n cut a prerelease; then re-run "nm-ci-windows"\n\tif the install deadlocks abort the competing pipeline\n' > "$words" + out=$(contract "$home" propose --words-file "$words" --expected-return 2026-09-08T08:00Z --spend 3 \ + --action merge --object 'task nm-windows-fix-r1 PR' --when 'checks green' \ + --action merge --object 'regardless of checks' 2>&1) || true + assert_contains "$out" 'Away posture read-back (proposed, not yet confirmed):' 'read-back title' + assert_contains "$out" 'expected return: 2026-09-08T08:00Z' 'expected return rendered' + assert_contains "$out" 'spend cap: 3 concurrent workers' 'spend cap rendered' + assert_contains "$out" 'reach: hold-for-return only. No phone channel is configured; anything that needs you waits for your return.' 'reach rendered' + assert_contains "$out" ' drive the windows fix to green and merge it,' 'words line 1' + assert_contains "$out" ' cut a prerelease; then re-run "nm-ci-windows"' 'words line 2 keeps its own indentation and quotes' + assert_contains "$out" "$(printf ' \tif the install deadlocks')" 'words line 3 keeps its tab' + assert_contains "$out" ' accepted clauses:' 'accepted list header' + assert_contains "$out" ' 1. merge task nm-windows-fix-r1 PR when checks green' 'accepted clause' + assert_contains "$out" ' refused clauses:' 'refused list header' + assert_contains "$out" ' 2. "action=merge object=regardless of checks when=(none)" - refused: missing when' 'refused clause' + assert_contains "$out" 'every clause expires at return' 'the never-set reminder' + assert_contains "$out" 'recorded clauses are held for the return brief and are not executed by this release' 'the not-executed notice' + assert_contains "$out" 'forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text; no recorded clause is authority by itself' 'the hard authority invariant' + assert_contains "$out" 'Say go to confirm' 'confirmation prompt' + # The verbatim words survive the record byte for byte, trailing newline included. + [ "$(contract "$home" words --proposal; printf x)" = "$(cat "$words"; printf x)" ] || fail "the proposal did not keep the words verbatim" + pass "the read-back renders the words verbatim beside the accepted and refused lists" +} + +test_words_preserve_final_newline_shape() { + local home without with trailing out + home=$(make_home words-newline-shape) + without="$home/without.txt" + with="$home/with.txt" + trailing="$home/trailing.txt" + printf 'merge when green' > "$without" + printf 'merge when green\n' > "$with" + printf 'first line\n\n' > "$trailing" + contract "$home" propose --words-file "$without" >/dev/null || fail "proposal without a final newline failed" + [ "$(contract "$home" words --proposal; printf x)" = "$(cat "$without"; printf x)" ] \ + || fail "words without a final newline did not round-trip byte-exact" + contract "$home" propose --words-file "$with" >/dev/null || fail "proposal with a final newline failed" + [ "$(contract "$home" words --proposal; printf x)" = "$(cat "$with"; printf x)" ] \ + || fail "words with a final newline did not round-trip byte-exact" + out=$(contract "$home" propose --words-file "$trailing"; printf x) || fail "proposal with trailing blank lines failed" + out=${out%x} + assert_contains "$out" $' first line\n \n accepted clauses:' \ + "read-back dropped a trailing blank line from the captain's words" + pass "words preserve their final newline shape in storage and read-back" +} + +test_propose_confirm_writes_the_record_and_announces_hold_for_return() { + local home out record proposed_epoch + home=$(make_home lifecycle) + contract "$home" propose --words 'merge it when green' --action merge --object 'task a PR' --when 'checks green' \ + --action merge --object everything >/dev/null 2>&1 || true + [ -f "$home/state/.afk-contract.proposed" ] || fail "propose did not write the proposal" + proposed_epoch=$(contract "$home" field entered_epoch --proposal) + [ ! -f "$home/state/.afk-contract" ] || fail "a proposal alone must not count as the posture" + sleep 1 + out=$(contract "$home" confirm 2>&1) || fail "confirm failed: $out" + record="$home/state/.afk-contract" + [ -f "$record" ] || fail "confirm did not write the record" + [ ! -f "$home/state/.afk-contract.proposed" ] || fail "confirm left the proposal behind" + [ -f "$record" ] || fail "the confirmed posture record is absent" + assert_contains "$out" 'Away posture confirmed at ' 'announcement opens with the confirmation time' + assert_contains "$out" 'hold-for-return only. No phone channel is configured; anything that needs you waits for your return.' 'announcement says hold-for-return only, aloud' + assert_contains "$out" '1 mandate clause(s) recorded, 1 refused, and 0 flagged as naming a never-set concept; recorded clauses are held for the return brief and are not executed by this release; forbidden, destructive, irreversible, and security-sensitive actions are never pre-authorizable regardless of clause text, and no recorded clause is authority by itself.' 'announcement counts clauses and states the hard authority invariant' + assert_contains "$out" 'Expected return: not given. Spend cap: 4 concurrent workers.' 'announcement carries the defaults' + [ "$(contract "$home" field version)" = 1 ] || fail "record version is not 1" + [ "$(contract "$home" field reach_channels)" = none ] || fail "reach channels are not none" + case "$(contract "$home" field confirmed_epoch)" in ''|*[!0-9]*) fail "confirmed_epoch is not numeric" ;; esac + case "$(contract "$home" field entered_epoch)" in ''|*[!0-9]*) fail "entered_epoch is not numeric" ;; esac + [ "$(contract "$home" field entered_epoch)" -gt "$proposed_epoch" ] || fail "entry time was not stamped at confirmation" + [ "$(contract "$home" words)" = 'merge it when green' ] || fail "words did not round-trip" + [ "$(contract "$home" clauses)" = "$(printf '1\tmerge\ttask a PR\tchecks green\t-')" ] || fail "clauses TSV is wrong: $(contract "$home" clauses)" + [ "$(contract "$home" refused | cut -f1,2)" = "$(printf '2\taction=merge object=everything when=(none)')" ] || fail "refused TSV is wrong: $(contract "$home" refused)" + pass "propose then confirm writes the record, announces hold-for-return only, and every read subcommand reflects it" +} + +test_confirm_requires_readback_and_refresh_is_a_no_op() { + local home out first rc + home=$(make_home defaults) + set +e + out=$(contract "$home" confirm 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "confirm without a proposal wrote a record" + assert_contains "$out" 'run propose before confirm' 'confirm refusal names the required read-back step' + [ ! -e "$home/state/.afk-contract" ] || fail "confirm without a proposal created posture state" + contract "$home" propose >/dev/null || fail "plain proposal failed" + out=$(contract "$home" confirm 2>&1) || fail "plain confirmation failed: $out" + assert_contains "$out" 'No mandate clauses recorded.' 'plain announcement' + assert_contains "$out" 'hold-for-return only.' 'plain announcement says hold-for-return' + first=$(cat "$home/state/.afk-contract") + sleep 1 + out=$(contract "$home" confirm 2>&1) || fail "refresh confirm failed: $out" + assert_contains "$out" 'already recorded at' 'refresh names the standing record' + [ "$(cat "$home/state/.afk-contract")" = "$first" ] || fail "a refresh rewrote the standing record" + pass "confirmation requires a read-back, and refresh leaves the standing record untouched" +} + +test_confirming_a_new_proposal_archives_the_standing_record() { + local home first_epoch archived + home=$(make_home replace) + contract "$home" propose >/dev/null 2>&1 || fail "first propose failed" + contract "$home" confirm >/dev/null 2>&1 || fail "first confirm failed" + first_epoch=$(contract "$home" field entered_epoch) + sleep 1 + contract "$home" propose --action merge --object 'task a PR' --when 'checks green' >/dev/null 2>&1 || fail "second propose failed" + contract "$home" confirm >/dev/null 2>&1 || fail "second confirm failed" + archived=$(find "$home/state/afk-contracts" -name "$first_epoch-superseded-*.afk-contract" -print -quit) + [ -f "$archived" ] || fail "the superseded record was not archived" + [ "$(contract "$home" field entered_epoch)" = "$first_epoch" ] || fail "replacement changed the away session start" + [ "$(contract "$home" clauses | cut -f2)" = merge ] || fail "the new record does not carry the new clause" + pass "a replacement archives the old mandate and keeps the session start" +} + +test_failed_replacement_keeps_the_standing_record() { + local home before out rc + home=$(make_home replace-failure) + contract "$home" propose --words 'original posture' >/dev/null || fail "first propose failed" + contract "$home" confirm >/dev/null || fail "first confirm failed" + before=$(cat "$home/state/.afk-contract") + contract "$home" propose --words 'replacement posture' >/dev/null || fail "replacement propose failed" + printf 'not a directory\n' > "$home/state/afk-contracts" + set +e + out=$(contract "$home" confirm 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "replacement succeeded without an archive destination" + [ "$(cat "$home/state/.afk-contract")" = "$before" ] || fail "failed replacement removed or changed the standing posture" + [ -f "$home/state/.afk-contract.proposed" ] || fail "failed replacement discarded the pending proposal" + pass "a failed replacement keeps the standing posture live" +} + +test_failed_final_replacement_rolls_back_the_superseded_archive() { + local home before out rc + home=$(make_home replace-final-move-failure) + contract "$home" propose --words 'original posture' >/dev/null || fail "first propose failed" + contract "$home" confirm >/dev/null || fail "first confirm failed" + before=$(cat "$home/state/.afk-contract") + contract "$home" propose --words 'replacement posture' >/dev/null || fail "replacement propose failed" + mkdir -p "$home/fakebin" + cat > "$home/fakebin/mv" <<'SH' +#!/usr/bin/env bash +case "${1:-}:${2:-}" in + *.afk-contract.confirming.*:*/.afk-contract) exit 1 ;; +esac +exec /bin/mv "$@" +SH + chmod +x "$home/fakebin/mv" + set +e + out=$(PATH="$home/fakebin:$PATH" contract "$home" confirm 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "replacement succeeded after its final publication failed" + [ "$(cat "$home/state/.afk-contract")" = "$before" ] || fail "failed final publication changed the standing posture" + [ -f "$home/state/.afk-contract.proposed" ] || fail "failed final publication discarded the pending proposal" + [ -z "$(find "$home/state/afk-contracts" -name '*-superseded-*.afk-contract' -print -quit)" ] \ + || fail "failed final publication left a duplicate superseded mandate" + pass "a failed final replacement publication rolls back its superseded archive" +} + +test_validation_rejects_incomplete_clause_rows() { + local home record out rc + home=$(make_home malformed-clause-row) + contract "$home" propose --action merge --object 'task a PR' --when 'checks green' >/dev/null || fail "proposal failed" + contract "$home" confirm >/dev/null || fail "confirmation failed" + record="$home/state/.afk-contract" + grep -v '^ object: ' "$record" > "$home/truncated" + mv "$home/truncated" "$record" + set +e + out=$(contract "$home" validate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "validation accepted a clause row without its object field" + assert_contains "$out" 'malformed clauses row 1: missing or invalid object' "validation did not name the malformed clause row" + set +e + out=$(contract "$home" archive 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "archive accepted a clause row without its object field" + [ -f "$record" ] || fail "archive moved the malformed clause record" + pass "validation and archive refuse incomplete clause rows by name" +} + +test_validation_rejects_blank_decoded_clause_fields() { + local field home record out rc + for field in object when; do + home=$(make_home "blank-$field-row") + contract "$home" propose --action merge --object 'task a PR' --when 'checks green' >/dev/null || fail "$field proposal failed" + contract "$home" confirm >/dev/null || fail "$field confirmation failed" + record="$home/state/.afk-contract" + sed "s/^ $field: e:.*/ $field: e:/" "$record" > "$home/damaged" + mv "$home/damaged" "$record" + set +e + out=$(contract "$home" validate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "validation accepted a blank decoded $field" + assert_contains "$out" "malformed clauses row 1: missing or invalid $field" "validation did not name the blank $field" + set +e + contract "$home" archive >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "archive accepted a blank decoded $field" + [ -f "$record" ] || fail "archive moved the record with a blank $field" + done + pass "validation and archive refuse blank decoded clause fields" +} + +test_validation_rejects_blank_stop_and_refused_text() { + local kind home record out rc + for kind in stop refused-text; do + home=$(make_home "blank-$kind") + if [ "$kind" = stop ]; then + contract "$home" propose --action merge --object 'task a PR' --when 'checks green' --stop 'captain returns' >/dev/null || fail "stop proposal failed" + else + contract "$home" propose --action merge --object 'task a PR' >/dev/null 2>&1 || true + fi + contract "$home" confirm >/dev/null || fail "$kind confirmation failed" + record="$home/state/.afk-contract" + if [ "$kind" = stop ]; then + sed 's/^ stop: e:.*/ stop: e:/' "$record" > "$home/damaged" + else + sed 's/^ text: e:.*/ text: e:/' "$record" > "$home/damaged" + fi + mv "$home/damaged" "$record" + set +e + out=$(contract "$home" validate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "validation accepted blank $kind data" + if [ "$kind" = stop ]; then + assert_contains "$out" 'malformed clauses row 1: missing or invalid stop' "validation did not name the blank stop" + else + assert_contains "$out" 'malformed refused row 1: missing or invalid text' "validation did not name the blank refused text" + fi + set +e + contract "$home" archive >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "archive accepted blank $kind data" + [ -f "$record" ] || fail "archive moved the record with blank $kind data" + done + pass "validation and archive refuse blank stop and refused text" +} + +test_validation_rejects_damaged_words_blocks() { + local mode home record out rc + for mode in unindented empty; do + home=$(make_home "damaged-words-$mode") + contract "$home" propose --words 'captain words' >/dev/null || fail "$mode words proposal failed" + contract "$home" confirm >/dev/null || fail "$mode words confirmation failed" + record="$home/state/.afk-contract" + if [ "$mode" = unindented ]; then + sed 's/^ captain words$/captain words/' "$record" > "$home/damaged" + else + grep -v '^ captain words$' "$record" > "$home/damaged" + fi + mv "$home/damaged" "$record" + set +e + out=$(contract "$home" validate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "validation accepted the $mode words block" + assert_contains "$out" 'invalid words block:' "validation did not name the damaged words block" + set +e + contract "$home" archive >/dev/null 2>&1 + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "archive accepted the $mode words block" + [ -f "$record" ] || fail "archive moved the record with $mode words" + done + pass "validation and archive refuse damaged words blocks" +} + +test_archive_moves_the_record_aside_and_is_idempotent() { + local home epoch path + home=$(make_home archive) + contract "$home" propose >/dev/null 2>&1 || fail "propose failed" + contract "$home" confirm >/dev/null 2>&1 || fail "confirm failed" + epoch=$(contract "$home" field entered_epoch) + path=$(contract "$home" archive) || fail "archive failed" + [ "$path" = "$home/state/afk-contracts/$epoch.afk-contract" ] || fail "archive path is not keyed by entered_epoch: $path" + [ -f "$path" ] || fail "archived record missing" + [ ! -f "$home/state/.afk-contract" ] || fail "the record still stands after archive" + contract "$home" archive || fail "a second archive with no record must succeed as a no-op" + [ "$(contract "$home" archived "$epoch")" = "$path" ] || fail "archived lookup did not find the record" + [ "$(contract "$home" words --path "$path")" = '' ] || fail "reading an archived record by path failed" + pass "archive keys the record by its entry time, empties the posture, and is idempotent" +} + +test_inputs_are_validated() { + local home out rc + home=$(make_home inputs) + set +e + out=$(contract "$home" propose --expected-return 'tomorrow morning' 2>&1) + rc=$? + set -e + [ "$rc" -eq 2 ] || fail "a non-ISO expected return should be a usage error (rc=$rc): $out" + assert_contains "$out" '--expected-return must be UTC ISO 8601' 'expected-return refusal wording' + set +e + out=$(contract "$home" propose --spend 0 2>&1) + rc=$? + set -e + [ "$rc" -eq 2 ] || fail "a zero spend cap should be a usage error (rc=$rc): $out" + set +e + out=$(contract "$home" propose --object 'task x PR' 2>&1) + rc=$? + set -e + [ "$rc" -eq 2 ] || fail "an empty clause should be a usage error, not a silent skip (rc=$rc): $out" + assert_contains "$out" '--object must follow the --action that opens its clause' 'a field with no open clause is a usage error' + [ ! -f "$home/state/.afk-contract.proposed" ] || fail "an invalid proposal was written" + set +e + out=$(contract "$home" validate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "validate with no record should fail" + printf 'version: 9\nentered_epoch: 1\nclauses:\nrefused:\n' > "$home/state/.afk-contract" + set +e + out=$(contract "$home" validate 2>&1) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "a foreign record version must be refused" + assert_contains "$out" "carries version '9', expected 1" 'version refusal wording' + pass "malformed inputs and foreign record versions are refused rather than guessed" +} + +test_fields_refuse_each_missing_part_by_name +test_omitted_stop_confirms_as_no_stop +test_never_set_flags_without_refusing_and_never_over_matches +test_fields_record_the_captain_wording_verbatim +test_clause_fields_round_trip_reversible_whitespace +test_clause_ids_are_input_ordinals_across_accepted_and_refused +test_readback_renders_words_verbatim_and_both_lists +test_words_preserve_final_newline_shape +test_propose_confirm_writes_the_record_and_announces_hold_for_return +test_confirm_requires_readback_and_refresh_is_a_no_op +test_confirming_a_new_proposal_archives_the_standing_record +test_failed_replacement_keeps_the_standing_record +test_failed_final_replacement_rolls_back_the_superseded_archive +test_validation_rejects_incomplete_clause_rows +test_validation_rejects_blank_decoded_clause_fields +test_validation_rejects_blank_stop_and_refused_text +test_validation_rejects_damaged_words_blocks +test_archive_moves_the_record_aside_and_is_idempotent +test_inputs_are_validated diff --git a/tests/fm-afk-launch.test.sh b/tests/fm-afk-launch.test.sh index 93f2fc4c45c..1a70f571f47 100755 --- a/tests/fm-afk-launch.test.sh +++ b/tests/fm-afk-launch.test.sh @@ -22,6 +22,11 @@ set -u ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" LAUNCH="$ROOT/bin/fm-afk-launch.sh" START="$ROOT/bin/fm-afk-start.sh" +CONTRACT="$ROOT/bin/fm-afk-contract.sh" +# Pin the default unit harness; Pi and OMP lifecycle ownership also receive +# explicit coverage in unit_pi_preserves_the_daemon_lifecycle. +unset PI_CODING_AGENT FM_PI_HARNESS CURSOR_AGENT CURSOR_INVOKED_AS GEMINI_CLI ATLASSIAN_AGENT_TYPE ROVODEV_CLI +export CLAUDECODE=1 FAILED=0 fail() { printf 'not ok - %s\n' "$1" >&2; FAILED=1; } @@ -40,6 +45,127 @@ GLOBAL_CLEANUP() { } trap GLOBAL_CLEANUP EXIT +confirm_posture() { # <home> + FM_HOME="$1" FM_STATE_OVERRIDE="$1/state" "$CONTRACT" propose >/dev/null 2>&1 \ + && FM_HOME="$1" FM_STATE_OVERRIDE="$1/state" "$CONTRACT" confirm >/dev/null 2>&1 +} + +# --------------------------------------------------------------------------- +# UNIT 0: the away-posture record is the entry. `propose` reads the mandate +# back, `confirm` records it and announces hold-for-return, and every daemon +# path requires that confirmed record. +# --------------------------------------------------------------------------- +unit_propose_confirm_records_the_posture_without_a_daemon() { + local st out rc + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-propose.XXXXXX") + mkdir -p "$st/state" + out=$(FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" propose \ + --words 'merge the windows fix when green' --action merge --object 'task fix-windows PR' --when 'checks green' \ + --action merge --object regardless 2>&1) + rc=$? + if [ "$rc" -eq 3 ] && [ -f "$st/state/.afk-contract.proposed" ] \ + && printf '%s' "$out" | grep -F '1. merge task fix-windows PR when checks green' >/dev/null \ + && printf '%s' "$out" | grep -F '2. "action=merge object=regardless when=(none)" - refused: missing when' >/dev/null \ + && [ ! -e "$st/state/.afk-contract" ]; then + pass "propose: the read-back lists accepted and refused clauses and writes only a proposal" + else + fail "propose: read-back or proposal wrong (rc=$rc): $out" + fi + out=$(FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" confirm 2>&1) + rc=$? + if [ "$rc" -eq 0 ] && [ -f "$st/state/.afk-contract" ] && [ ! -e "$st/state/.afk-contract.proposed" ] \ + && [ ! -e "$st/state/.afk" ] && [ ! -e "$st/state/.afk-daemon-terminal" ] \ + && printf '%s' "$out" | grep -F 'hold-for-return only. No phone channel is configured; anything that needs you waits for your return.' >/dev/null; then + pass "confirm: records the posture, announces hold-for-return only, and launches no daemon" + else + fail "confirm: record, announcement, or daemon state wrong (rc=$rc): $out" + fi + printf 'schema\tfm-afk-return.v1\nphase\tblocked\n' > "$st/state/.afk-return-catchup" + if FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" propose --action merge --object 'task a PR' --when 'checks green' >/dev/null 2>&1; then + fail "propose: accepted a new mandate while the prior return catch-up was pending" + else + pass "propose: refuses while the prior return catch-up is pending" + fi + rm -rf "$st" +} + +unit_pi_preserves_the_daemon_lifecycle() { + local st harness out rc + for harness in pi pi-signed omp; do + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-pi.XXXXXX") + mkdir -p "$st/state" + confirm_posture "$st" || fail "$harness: could not confirm fixture posture" + out=$(FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" FM_TEST_HARNESS="$harness" \ + bash -c '. "$1"; fm_afk_launch_primary_harness() { printf "%s" "$FM_TEST_HARNESS"; }; fm_afk_launch_main start-native' _ "$LAUNCH" 2>&1) + rc=$? + if [ "$rc" -eq 0 ] && [ -f "$st/state/.afk-contract" ] && [ -e "$st/state/.afk" ]; then + pass "$harness: confirmed posture prepares the daemon lifecycle and extension standby flag" + else + fail "$harness: daemon lifecycle was refused or lost ownership state (rc=$rc): $out" + fi + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" stop >/dev/null 2>&1 \ + || fail "$harness: daemon lifecycle cleanup failed" + [ ! -e "$st/state/.afk" ] || fail "$harness: stop left the standby flag" + rm -rf "$st" + done +} + +unit_daemon_entry_requires_confirmation() { + local st out rc + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-entry-record.XXXXXX") + mkdir -p "$st/state" + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$CONTRACT" propose --action merge --object 'task a PR' --when 'checks green' >/dev/null 2>&1 + out=$(FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" start-native 2>&1) + rc=$? + if [ "$rc" -ne 0 ] && [ -f "$st/state/.afk-contract.proposed" ] && [ ! -e "$st/state/.afk-contract" ] \ + && [ ! -e "$st/state/.afk" ] && printf '%s' "$out" | grep -F 'a confirmed away-posture record is required' >/dev/null; then + pass "daemon entry: a pending proposal cannot bypass captain confirmation" + else + fail "daemon entry: pending proposal was promoted or refusal was unclear (rc=$rc): $out" + fi + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$CONTRACT" confirm >/dev/null 2>&1 + if FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" start-native >/dev/null 2>&1 \ + && [ -e "$st/state/.afk" ]; then + pass "daemon entry: an explicitly confirmed record permits lifecycle preparation" + else + fail "daemon entry: rejected an explicitly confirmed record" + fi + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" stop >/dev/null 2>&1 + rm -rf "$st" +} + +unit_failed_daemon_launch_preserves_confirmed_record() { + local st + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-failed-record.XXXXXX") + mkdir -p "$st/state" + confirm_posture "$st" || fail "failed start: could not confirm fixture posture" + if ! FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" FM_SUPERVISOR_TARGET=unused \ + FM_SUPERVISOR_BACKEND=unsupported "$LAUNCH" start >/dev/null 2>&1 \ + && [ -f "$st/state/.afk-contract" ] && [ ! -e "$st/state/afk-contracts" ]; then + pass "failed start: preserves the pre-confirmed posture record" + else + fail "failed start: changed the pre-confirmed posture record" + fi + rm -rf "$st" +} + +unit_stop_archives_the_record_last() { + local st epoch + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-stop-archive.XXXXXX") + mkdir -p "$st/state" + confirm_posture "$st" || fail "stop archive: could not confirm fixture posture" + FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" start-native >/dev/null 2>&1 || fail "stop archive: native entry failed" + epoch=$(FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$CONTRACT" field entered_epoch) + if FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" stop >/dev/null 2>&1 \ + && [ ! -e "$st/state/.afk" ] && [ ! -e "$st/state/.afk-contract" ] \ + && [ -f "$st/state/afk-contracts/$epoch.afk-contract" ]; then + pass "stop: clears the away flag and archives the posture record under its entry time" + else + fail "stop: the posture record was not archived (state: $(ls -a "$st/state"))" + fi + rm -rf "$st" +} + # --------------------------------------------------------------------------- # UNIT 1: fm_afk_clear_stale_artifacts removes exactly the three stale artifacts. # --------------------------------------------------------------------------- @@ -222,6 +348,7 @@ unit_failed_start_rolls_back_state() { mkdir -p "$st/state" printf 'pending\n' > "$st/state/.subsuper-escalations" printf 'wedged\n' > "$st/state/.subsuper-inject-wedged" + confirm_posture "$st" || fail "failed start: could not confirm fixture posture" if FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" FM_SUPERVISOR_TARGET=unused \ FM_SUPERVISOR_BACKEND=unsupported "$LAUNCH" start >/dev/null 2>&1; then fail "failed start: unsupported backend unexpectedly succeeded" @@ -243,6 +370,7 @@ unit_concurrent_start_serialized() { tmux new-session -d -s "$cap_session" 2>/dev/null || { fail "concurrent start: captain session creation failed"; rm -rf "$st"; return 0; } TRACK_TMUX_SESSIONS="$TRACK_TMUX_SESSIONS $cap_session" cap_pane=$(tmux display-message -p -t "$cap_session" '#{pane_id}') + confirm_posture "$st" || fail "concurrent start: could not confirm fixture posture" FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" FM_SUPERVISOR_TARGET="$cap_pane" \ FM_SUPERVISOR_BACKEND=tmux FM_AFK_LAUNCH_ENTRY="$SLEEPER" "$LAUNCH" start >/dev/null 2>&1 & # shellcheck disable=SC2031 # The background PID is captured immediately in this shell. @@ -499,6 +627,7 @@ unit_native_lifecycle() { st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-native.XXXXXX") mkdir -p "$st/state" : > "$st/state/.subsuper-escalations" + confirm_posture "$st" || fail "native lifecycle: could not confirm fixture posture" if FM_HOME="$st" FM_STATE_OVERRIDE="$st/state" "$LAUNCH" start-native >/dev/null 2>&1 \ && [ "$(cut -f1 "$st/state/.afk-daemon-terminal")" = none ] \ && [ -e "$st/state/.afk" ] \ @@ -877,6 +1006,7 @@ e2e_herdr() { cap_pane=$(printf '%s' "$out" | jq -r '.result.root_pane.pane_id // empty') if [ -z "$cap_ws" ] || [ -z "$cap_pane" ]; then E2E_HERDR_CLEANUP; fail "herdr e2e: could not create captain workspace"; return 0; fi target="$SESSION:$cap_pane" + confirm_posture "$home_tmp" || fail "herdr e2e: could not confirm fixture posture" before=$(fm_backend_herdr_cli "$SESSION" pane list --workspace "$cap_ws" 2>/dev/null | jq --arg t "$cap_tab" '[.result.panes[]?|select(.tab_id==$t)]|length') ws_before=$(fm_backend_herdr_cli "$SESSION" workspace list 2>/dev/null | jq '[.result.workspaces[]?]|length') @@ -918,6 +1048,7 @@ e2e_tmux() { tmux new-session -d -s "$cap_session" 2>/dev/null || { fail "tmux e2e: could not create captain session"; rm -rf "$home_tmp"; return 0; } TRACK_TMUX_SESSIONS="$TRACK_TMUX_SESSIONS $cap_session" cap_pane=$(tmux display-message -p -t "$cap_session" '#{pane_id}') + confirm_posture "$home_tmp" || fail "tmux e2e: could not confirm fixture posture" before=$(tmux list-panes -t "$cap_session" | wc -l | tr -d ' ') FM_HOME="$home_tmp" FM_STATE_OVERRIDE="$home_tmp/state" \ @@ -943,6 +1074,11 @@ e2e_tmux() { } unit_clear_stale +unit_propose_confirm_records_the_posture_without_a_daemon +unit_pi_preserves_the_daemon_lifecycle +unit_daemon_entry_requires_confirmation +unit_failed_daemon_launch_preserves_confirmed_record +unit_stop_archives_the_record_last unit_relative_paths_are_absolute_before_daemon_launch unit_fresh_vs_refresh unit_stop_ordering diff --git a/tests/fm-afk-pi-herdr-return-e2e.test.sh b/tests/fm-afk-pi-herdr-return-e2e.test.sh index 1976ae87b79..ccbe496860a 100755 --- a/tests/fm-afk-pi-herdr-return-e2e.test.sh +++ b/tests/fm-afk-pi-herdr-return-e2e.test.sh @@ -187,6 +187,12 @@ cat > "$HOME_DIR/data/backlog.md" <<'EOF' ## Done EOF +PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" \ + FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="$PRIMARY_TARGET" FM_AFK_LAUNCH_ENTRY="$TMP_ROOT/daemon-entry" \ + "$ROOT/bin/fm-afk-launch.sh" propose --words "" >/dev/null +PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" \ + FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="$PRIMARY_TARGET" FM_AFK_LAUNCH_ENTRY="$TMP_ROOT/daemon-entry" \ + "$ROOT/bin/fm-afk-launch.sh" confirm >/dev/null PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" \ FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="$PRIMARY_TARGET" FM_AFK_LAUNCH_ENTRY="$TMP_ROOT/daemon-entry" \ "$ROOT/bin/fm-afk-launch.sh" start >/dev/null @@ -279,6 +285,12 @@ PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_ROOT_OVERRIDE="$PROJE # A clean re-entry creates no stale delivery or alert, and an immediate return is # idempotently clear because the keyed blocker is resolved. +PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" \ + FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="$PRIMARY_TARGET" FM_AFK_LAUNCH_ENTRY="$TMP_ROOT/daemon-entry" \ + "$ROOT/bin/fm-afk-launch.sh" propose --words "" >/dev/null +PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" \ + FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="$PRIMARY_TARGET" FM_AFK_LAUNCH_ENTRY="$TMP_ROOT/daemon-entry" \ + "$ROOT/bin/fm-afk-launch.sh" confirm >/dev/null PATH="$FAKEBIN:$ORIGINAL_PATH" HERDR_SESSION="$SESSION" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$STATE" \ FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="$PRIMARY_TARGET" FM_AFK_LAUNCH_ENTRY="$TMP_ROOT/daemon-entry" \ "$ROOT/bin/fm-afk-launch.sh" start >/dev/null diff --git a/tests/fm-afk-return.test.sh b/tests/fm-afk-return.test.sh index 9c5629563e9..a510e70d9bf 100755 --- a/tests/fm-afk-return.test.sh +++ b/tests/fm-afk-return.test.sh @@ -1,11 +1,15 @@ #!/usr/bin/env bash -# Deterministic return-catch-up gate regression. +# Deterministic return-catch-up gate and return-brief regression. # # Covers the second half of the 2026-07-14 incident: an away-mode blocked event # survived in durable state, but the ordinary return request could proceed to # Bearings before Firstmate owned remediation. The shared script now stops, # drains, preserves evidence, and refuses ordinary work until every live open # `blocked:` event is resolved or durably reclassified. +# The brief cases pin the away-posture redesign's return: the brief is composed +# from the archived posture record, the outcome store, the held set, and the +# status logs, health first, and the gate shrinks to what the away session could +# not fix. set -u # shellcheck source=tests/lib.sh @@ -22,6 +26,16 @@ install_runner() { # <case-dir> # fm-timeout-lib.sh: the shared hard bound fm-classify-lib.sh sources for the # wedge detector's bounded worktree write probe. cp "$ROOT/bin/fm-timeout-lib.sh" "$dir/bin/" + # The return brief's durable sources: the posture-record owner, the outcome + # store owner, and the backlog reader with its tasks-axi probe. + cp "$ROOT/bin/fm-afk-contract.sh" "$dir/bin/" + cp "$ROOT/bin/fm-branch-outcome.sh" "$dir/bin/" + cp "$ROOT/bin/fm-tasks-axi-lib.sh" "$dir/bin/" + cp "$ROOT/bin/fm-backlog-transition-lib.sh" "$dir/bin/" + cp "$ROOT/.tasks.toml" "$dir/home/.tasks.toml" + printf '## In flight\n\n## Queued\n\n## Done\n' > "$dir/home/data/backlog.md" + # The fake stop mirrors the real one's ordering: the away flag goes, then the + # posture record is archived through its owner. cat > "$dir/bin/fm-afk-launch.sh" <<'SH' #!/usr/bin/env bash [ "${1:-}" = stop ] || exit 2 @@ -32,6 +46,7 @@ if [ -e "$FM_HOME/state/.fail-terminal-stop-once" ]; then exit 1 fi rm -f "$FM_HOME/state/.afk-daemon-terminal" +"$(dirname "$0")/fm-afk-contract.sh" archive >/dev/null SH cat > "$dir/bin/fm-wake-drain.sh" <<'SH' #!/usr/bin/env bash @@ -275,10 +290,426 @@ test_check_retries_recorded_terminal_teardown() { pass "check retries recorded terminal teardown and keeps catch-up gated until success" } +# --- the return brief ------------------------------------------------------- +# Rendered from durable records only: the archived away-posture record, the +# outcome store, the held set, and the status logs. Health comes first, then +# the mandate, then what waits on the captain, then what could not be fixed; +# the blocker gate shrinks to what the away session could not fix. + +contract_in() { # <case-dir> <args...> + local dir=$1 + shift + FM_HOME="$dir/home" FM_STATE_OVERRIDE="$dir/home/state" "$dir/bin/fm-afk-contract.sh" "$@" +} + +outcome_in() { # <case-dir> <args...> + local dir=$1 + shift + FM_HOME="$dir/home" FM_STATE_OVERRIDE="$dir/home/state" "$dir/bin/fm-branch-outcome.sh" "$@" +} + +line_of() { # <haystack> <needle> -> 1-based line number of the first match, or empty + printf '%s\n' "$1" | grep -n -F -- "$2" | head -1 | cut -d: -f1 +} + +test_return_brief_composes_from_record_store_and_held_set() { + local dir out rc gate health_line clauses_line waiting_line failed_line second + dir="$TMP_ROOT/brief" + install_runner "$dir" + (cd "$dir/home" && tasks-axi add fix-windows 'Fix the windows lane' --file data/backlog.md >/dev/null \ + && tasks-axi hold fix-windows --reason 'awaiting the captain on the merge' --kind captain --file data/backlog.md >/dev/null) \ + || fail "could not seed the held backlog" + contract_in "$dir" propose --words 'merge the windows fix when green, then cut a prerelease' \ + --action merge --object 'task fix-windows PR' --when 'checks green' \ + --action prerelease --object 'repo no-mistakes' --when 'after clause 1' \ + --action merge --object everything >/dev/null 2>&1 || true + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not confirm the away-posture record" + # Two live blockers, one on a task with a captain-verdict outcome and one on a + # task with a routine outcome. A third task failed outright. + printf 'window=synthetic:fm-fix-windows\nbackend=tmux\nkind=ship\n' > "$dir/home/state/fix-windows.meta" + printf 'blocked [key=token]: firstmate can refresh the token\n' > "$dir/home/state/fix-windows.status" + printf 'window=synthetic:fm-other\nbackend=tmux\nkind=ship\n' > "$dir/home/state/other.meta" + printf 'blocked [key=dep]: needs the upstream dependency\nneeds-decision [key=pick]: choose the target\n' > "$dir/home/state/other.status" + printf 'window=synthetic:fm-dead\nbackend=tmux\nkind=scout\n' > "$dir/home/state/dead.meta" + printf 'failed: the reproduction never compiled\n' > "$dir/home/state/dead.status" + outcome_in "$dir" append --task fix-windows --verdict captain \ + --summary 'blocked on a token only the captain holds; held for return' --wake 'signal: fix-windows.status' >/dev/null \ + || fail "could not seed the captain outcome row" + outcome_in "$dir" append --task other --verdict routine \ + --summary 'resent the steer; worker resumed' --wake 'stale: synthetic:fm-other' >/dev/null \ + || fail "could not seed the routine outcome row" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "the unreached blocker should still gate the return (rc=$rc): $out" + gate="$dir/home/state/.afk-return-catchup" + [ -e "$dir/home/state/afk-contracts" ] || fail "the return did not archive the away-posture record" + [ ! -e "$dir/home/state/.afk-contract" ] || fail "the live away-posture record survived the return" + assert_contains "$out" '=== Return brief (away ' "the brief did not open with the away window" + assert_contains "$out" 'supervision ran through the away window with no detected gap' "health did not report the clean window" + health_line=$(line_of "$out" 'Supervisor health:') + clauses_line=$(line_of "$out" 'Mandate clauses:') + waiting_line=$(line_of "$out" 'Waiting on you:') + failed_line=$(line_of "$out" 'Tried and failed, or could not be fixed:') + [ -n "$health_line" ] && [ -n "$clauses_line" ] && [ -n "$waiting_line" ] && [ -n "$failed_line" ] \ + || fail "the brief is missing a section: $out" + [ "$health_line" -lt "$clauses_line" ] && [ "$clauses_line" -lt "$waiting_line" ] && [ "$waiting_line" -lt "$failed_line" ] \ + || fail "the brief sections are out of order (health $health_line, clauses $clauses_line, waiting $waiting_line, failed $failed_line)" + assert_contains "$out" '1. merge task fix-windows PR when checks green - recorded, not executed by this release' "the accepted clause was not listed as recorded-only" + assert_contains "$out" '2. prerelease repo no-mistakes when after clause 1 - recorded, not executed by this release' "the second clause was not listed" + assert_contains "$out" '3. "action=merge object=everything when=(none)" - refused at entry: missing when' "the refused clause was not listed with its missing part" + assert_contains "$out" 'merge the windows fix when green, then cut a prerelease' "the captain's verbatim words were not carried into the brief" + assert_contains "$out" 'fix-windows,queued,task' "the held backlog item was not listed under waiting on you" + assert_contains "$out" 'awaiting the captain on the merge' "the hold reason was not listed" + assert_contains "$out" 'other [key=pick] needs your decision: choose the target' "the open decision was not listed under waiting on you" + assert_contains "$out" 'fix-windows: blocked on a token only the captain holds; held for return' "the captain-verdict outcome was not listed" + assert_contains "$out" 'fix-windows [key=token] still blocked, firstmate remediates before ordinary work' "the blocker sharing a task with a captain outcome was exempted" + assert_contains "$out" 'other [key=dep] still blocked, firstmate remediates before ordinary work' "the unreached blocker was not listed as could-not-fix" + assert_contains "$out" 'dead: failed: the reproduction never compiled' "the failed task was not listed" + assert_contains "$out" '1 routine outcome(s) recorded' "the routine outcome count was not reported" + assert_contains "$out" 'other: resent the steer; worker resumed' "the routine outcome was not listed" + assert_contains "$out" 'Cost: 2 supervision outcome(s) recorded (1 routine, 1 captain); 3 task(s) live at return.' "the cost line is wrong" + assert_contains "$out" 'firstmate-actionable blocker: other [key=dep]' "the unreached blocker did not gate" + assert_contains "$out" 'firstmate-actionable blocker: fix-windows [key=token]' "a captain outcome incorrectly exempted an open blocker" + grep -F "$(printf 'contract\t')" "$gate" >/dev/null || fail "the gate did not retain the posture-record window" + grep -F "$(printf 'evidence\thealth\t')" "$gate" >/dev/null || fail "the gate did not retain the health snapshot" + + # Remediate both blockers; the check re-renders the same brief from the + # archived record and clears. + printf 'resolved [key=dep]: the upstream dependency landed\n' >> "$dir/home/state/other.status" + printf 'resolved [key=token]: the token was refreshed\n' >> "$dir/home/state/fix-windows.status" + second=$(run_return "$dir" check) || fail "the remediated return did not clear: $second" + assert_contains "$second" '1. merge task fix-windows PR when checks green - recorded, not executed by this release' "check did not re-render the mandate from the archived record" + assert_contains "$second" 'supervision ran through the away window with no detected gap' "check lost the health snapshot taken at begin" + assert_contains "$second" 'catch-up clear' "check did not clear the gate" + [ ! -e "$gate" ] || fail "the cleared check left the gate behind" + FM_HOME="$dir/home" FM_STATE_OVERRIDE="$dir/home/state" "$dir/bin/fm-afk-return.sh" guard \ + || fail "guard still refused after the record was archived and the gate cleared" + pass "the return brief renders health, mandate, waiting, could-not-fix, handled, and cost from durable records, and the gate shrinks to what the away session could not fix" +} + +test_return_brief_keeps_refresh_history() { + local dir out first_epoch + dir="$TMP_ROOT/brief-refresh" + install_runner "$dir" + contract_in "$dir" propose --words 'first mandate' \ + --action merge --object 'task first PR' --when 'checks green' >/dev/null 2>&1 || fail "could not propose the first mandate" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not confirm the first mandate" + first_epoch=$(contract_in "$dir" field entered_epoch) + outcome_in "$dir" append --task first --verdict routine \ + --summary 'completed before the mandate refresh' --wake 'signal: first.status' >/dev/null \ + || fail "could not seed the pre-refresh outcome" + contract_in "$dir" propose --words $'replacement mandate\n\n' \ + --action wake-me --object 'task second' --when 'at 2026-09-08T08:00Z' >/dev/null 2>&1 || fail "could not propose the replacement mandate" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not confirm the replacement mandate" + [ "$(contract_in "$dir" field entered_epoch)" = "$first_epoch" ] || fail "refresh changed the away-window boundary" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + out=$(run_return "$dir" begin) || fail "refreshed posture return did not clear: $out" + assert_contains "$out" 'merge task first PR when checks green - superseded at ' "the superseded mandate was omitted" + assert_contains "$out" 'wake-me task second when at 2026-09-08T08:00Z - recorded' "the final mandate was omitted" + assert_contains "$out" 'first: completed before the mandate refresh' "the pre-refresh outcome was omitted" + assert_contains "$out" $' replacement mandate\n \nWaiting on you:' "the return brief dropped a trailing blank line from the final words" + [ -f "$dir/home/state/afk-contracts/$first_epoch.afk-contract" ] || fail "return did not archive the final session record at the canonical path" + pass "a refreshed posture keeps its original window, superseded mandate, and earlier outcomes" +} + +test_malformed_posture_record_keeps_catchup_gated() { + local dir out rc gate record before + dir="$TMP_ROOT/malformed-posture-record" + install_runner "$dir" + gate="$dir/home/state/.afk-return-catchup" + record="$dir/home/state/.afk-contract" + printf 'version: 1\nentered: 2026-09-08T08:00:00Z\nentered_epoch: 123\nwords: |-\n captain words survive\n' > "$record" + before=$(cat "$record"; printf x) + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "a malformed posture record should keep catch-up gated (rc=$rc): $out" + [ -f "$gate" ] || fail "a malformed posture record did not retain the return gate" + [ -f "$record" ] || fail "a malformed posture record was archived or deleted" + [ "$(cat "$record"; printf x)" = "$before" ] || fail "a malformed posture record lost its captain words" + [ ! -e "$dir/home/state/afk-contracts/123.afk-contract" ] || fail "a malformed posture record was archived" + assert_contains "$out" "away-posture record unreadable: $record; catch-up stays gated" "return did not name the malformed posture record" + pass "a malformed posture record stays live and gates return catch-up" +} + +test_missing_epoch_record_stays_required_after_disappearing() { + local dir out rc gate record backup epoch entered + dir="$TMP_ROOT/missing-epoch-posture-record" + install_runner "$dir" + gate="$dir/home/state/.afk-return-catchup" + record="$dir/home/state/.afk-contract" + backup="$dir/valid-record.backup" + contract_in "$dir" propose --words 'captain words survive' \ + --action merge --object 'task restored PR' --when 'checks green' >/dev/null || fail "could not propose the posture record" + contract_in "$dir" confirm >/dev/null || fail "could not confirm the posture record" + epoch=$(contract_in "$dir" field entered_epoch) + entered=$(contract_in "$dir" field entered) + cp "$record" "$backup" + grep -v '^entered_epoch: ' "$record" > "$dir/damaged-record" + mv "$dir/damaged-record" "$record" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "a malformed record without an epoch should retain catch-up (rc=$rc): $out" + [ -f "$gate" ] || fail "the malformed record without an epoch did not retain the gate" + assert_contains "$out" "away-posture record unreadable: $record; catch-up stays gated" "begin did not retain the unreadable record path" + rm "$record" + set +e + out=$(run_return "$dir" check) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "deleting the malformed epochless record cleared catch-up (rc=$rc): $out" + assert_contains "$out" "away-posture record missing: $record; catch-up stays gated" "check did not name the missing retained record" + [ -f "$gate" ] || fail "the missing retained record did not preserve the gate" + cp "$backup" "$record" + out=$(run_return "$dir" check) || fail "check did not clear after the retained record was restored valid: $out" + assert_contains "$out" "=== Return brief (away $entered ->" "the restored record did not recover its away window" + assert_contains "$out" 'merge task restored PR when checks green - recorded' "the restored clause was omitted from the brief" + assert_contains "$out" 'captain words survive' "the restored captain words were omitted from the brief" + [ -f "$dir/home/state/afk-contracts/$epoch.afk-contract" ] || fail "the restored record was not archived under its recovered epoch" + assert_contains "$out" 'catch-up clear' "the restored valid record did not clear catch-up" + [ ! -e "$gate" ] || fail "the restored valid record left the gate behind" + pass "an epochless malformed record remains required until restored valid" +} + +test_unreadable_outcome_store_keeps_catchup_gated() { + local dir out rc gate + dir="$TMP_ROOT/unreadable-outcome-store" + install_runner "$dir" + gate="$dir/home/state/.afk-return-catchup" + printf '{malformed json\n' > "$dir/home/state/branch-outcomes.jsonl" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "an unreadable outcome store should keep catch-up gated (rc=$rc): $out" + [ -f "$gate" ] || fail "an unreadable outcome store did not retain the return gate" + assert_contains "$out" 'outcome store unreadable, catch-up stays gated' "the partial brief did not disclose its unreadable store" + : > "$dir/home/state/branch-outcomes.jsonl" + out=$(run_return "$dir" check) || fail "catch-up did not clear after the outcome store was repaired: $out" + assert_contains "$out" 'catch-up clear' "the repaired outcome store did not clear catch-up" + assert_not_contains "$out" 'outcome store unreadable' "the repaired store retained stale failure evidence" + [ ! -e "$gate" ] || fail "the repaired outcome store left the return gate behind" + pass "an unreadable outcome store gates catch-up until a successful reread" +} + +test_failed_held_listing_keeps_catchup_gated() { + local dir out waiting rc gate + dir="$TMP_ROOT/held-list-failure" + install_runner "$dir" + mkdir -p "$dir/fakebin" + cat > "$dir/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +printf 'synthetic held backlog failure\n' >&2 +exit 1 +SH + chmod +x "$dir/fakebin/tasks-axi" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + gate="$dir/home/state/.afk-return-catchup" + set +e + out=$(PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_STATE_OVERRIDE="$dir/home/state" \ + "$dir/bin/fm-afk-return.sh" begin 2>&1) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "a failed held-set read should keep catch-up gated (rc=$rc): $out" + [ -f "$gate" ] || fail "a failed held-set read did not retain the return gate" + waiting=$(printf '%s\n' "$out" | awk '/^Waiting on you:/{show=1} /^Tried and failed, or could not be fixed:/{show=0} show') + assert_contains "$waiting" "held listing unavailable: $dir/home/data/backlog.md: synthetic held backlog failure; catch-up stays gated" "the failed held listing was not disclosed" + assert_not_contains "$waiting" '(nothing)' "an unavailable held set was also reported as empty" + out=$(run_return "$dir" check) || fail "catch-up did not clear after the held-set reader recovered: $out" + assert_contains "$out" 'catch-up clear' "the recovered held-set reader did not clear catch-up" + [ ! -e "$gate" ] || fail "the recovered held-set reader left the return gate behind" + pass "an unavailable held listing gates catch-up until a successful reread" +} + +test_unreadable_status_file_keeps_catchup_gated() { + local dir out rc gate status + dir="$TMP_ROOT/unreadable-status" + install_runner "$dir" + gate="$dir/home/state/.afk-return-catchup" + status="$dir/home/state/unreadable.status" + printf 'window=synthetic:fm-unreadable\nbackend=tmux\nkind=ship\n' > "$dir/home/state/unreadable.meta" + printf 'needs-decision [key=hidden]: private captain decision\nfailed: private failure detail\n' > "$dir/status-source" + ln -s "$dir/status-source" "$status" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "an unreadable status should keep catch-up gated (rc=$rc): $out" + [ -f "$gate" ] || fail "an unreadable status did not retain the return gate" + assert_contains "$out" "status file unreadable: $status; catch-up stays gated" "the partial brief did not name the unreadable status" + assert_not_contains "$out" 'private captain decision' "the brief followed the refused status symlink for a decision" + assert_not_contains "$out" 'private failure detail' "the brief followed the refused status symlink for a failure" + rm "$status" + : > "$status" + out=$(run_return "$dir" check) || fail "catch-up did not clear after the status file was repaired: $out" + assert_contains "$out" 'catch-up clear' "the repaired status did not clear catch-up" + assert_not_contains "$out" 'status file unreadable:' "the repaired status retained stale failure evidence" + [ ! -e "$gate" ] || fail "the repaired status left the return gate behind" + pass "an unreadable status stays private and gates until a successful reread" +} + +test_return_guard_refuses_while_the_record_exists() { + local dir out rc + dir="$TMP_ROOT/guard-record" + install_runner "$dir" + contract_in "$dir" propose >/dev/null 2>&1 || fail "could not propose the away-posture record" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not write the away-posture record" + set +e + out=$(FM_HOME="$dir/home" FM_STATE_OVERRIDE="$dir/home/state" "$dir/bin/fm-afk-return.sh" guard 2>&1) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "guard should refuse while the away-posture record exists (rc=$rc): $out" + assert_contains "$out" 'away mode is still active' "guard did not name the away posture" + [ ! -e "$dir/home/state/.afk" ] || fail "fixture error: the legacy flag should be absent in this case" + pass "the read-only guard treats the away-posture record as active away mode without the legacy flag" +} + +test_return_brief_health_leads_with_a_gap() { + local dir out gap_line clean_line + dir="$TMP_ROOT/brief-gap" + install_runner "$dir" + contract_in "$dir" propose >/dev/null 2>&1 || fail "could not propose the away-posture record" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not write the away-posture record" + : > "$dir/home/state/.watcher-down" + # A beacon older than the grace, on either date flavor. + touch "$dir/home/state/.last-watcher-beat" + if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$(( $(date +%s) - 900 ))" '+%Y%m%d%H%M.%S')" "$dir/home/state/.last-watcher-beat" + else touch -m -d "@$(( $(date +%s) - 900 ))" "$dir/home/state/.last-watcher-beat"; fi + : > "$dir/home/state/.fake-drain" + out=$(run_return "$dir" begin) || fail "a clean fleet with a supervision gap should still clear the gate: $out" + assert_contains "$out" 'GAP: watcher downtime was detected during the away window' "the downtime marker was not reported as a gap" + assert_contains "$out" 'GAP: the watcher beat was ' "the stale beacon was not reported as a gap" + assert_not_contains "$out" 'no detected gap' "a gap window was reported as clean" + gap_line=$(line_of "$out" 'GAP: watcher downtime') + clean_line=$(line_of "$out" 'Mandate clauses:') + [ "$gap_line" -lt "$clean_line" ] || fail "the gap was not reported before the mandate" + pass "the return brief leads with supervisor health and names every detected gap" +} + +test_return_brief_without_a_record_reports_the_legacy_flag() { + local dir out + dir="$TMP_ROOT/brief-legacy" + install_runner "$dir" + printf '%s\n' "$(( $(date +%s) - 7200 ))" > "$dir/home/state/.afk" + : > "$dir/home/state/.fake-drain" + out=$(run_return "$dir" begin) || fail "a legacy-flag return with no blockers should clear: $out" + assert_contains "$out" '(no away-posture record for this window; legacy away flag only)' "the legacy window was not named" + assert_contains "$out" ', 2h00m) ===' "the away window was not measured from the legacy flag's own timestamp" + [ ! -e "$dir/home/state/.afk" ] || fail "the legacy flag survived the return" + pass "a return with only the legacy away flag still renders the brief and measures the window from the flag" +} + + + +test_unreadable_superseded_archive_keeps_return_gated() { + local dir out rc epoch archive backup + dir="$TMP_ROOT/superseded-unreadable" + install_runner "$dir" + contract_in "$dir" propose --words 'first mandate' \ + --action merge --object 'task first PR' --when 'checks green' >/dev/null 2>&1 || fail "could not propose the first mandate" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not confirm the first mandate" + epoch=$(contract_in "$dir" field entered_epoch) + contract_in "$dir" propose --words 'replacement mandate' \ + --action wake-me --object 'task second' --when 'at 2026-09-08T08:00Z' >/dev/null 2>&1 || fail "could not propose the replacement mandate" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not confirm the replacement mandate" + archive="" + for archive in "$dir/home/state/afk-contracts/$epoch-superseded-"*.afk-contract; do break; done + [ -f "$archive" ] || fail "no superseded archive was written" + backup="$dir/superseded.backup" + cp "$archive" "$backup" + printf 'version: 1\n' > "$archive" + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "an unreadable superseded archive must keep catch-up gated (rc=$rc): $out" + assert_contains "$out" 'superseded away-posture record unreadable' "the gate did not name the unreadable superseded archive" + [ -e "$dir/home/state/.afk-return-catchup" ] || fail "the gate was not retained" + rm -f "$archive" + set +e + out=$(run_return "$dir" check) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "deleting the named superseded archive cleared catch-up (rc=$rc): $out" + assert_contains "$out" "superseded away-posture record missing: $archive" "check did not retain the exact missing archive" + cp "$backup" "$archive" + out=$(run_return "$dir" check) || fail "check did not clear once the superseded archive was restored: $out" + assert_contains "$out" 'catch-up clear' "check did not clear the gate" + pass "an unreadable superseded mandate stays required until its record validates" +} + +test_missing_final_archive_keeps_retained_contract_gated() { + local dir out rc epoch archive backup + dir="$TMP_ROOT/final-archive-missing" + install_runner "$dir" + contract_in "$dir" propose --words 'durable mandate' \ + --action merge --object 'task final PR' --when 'checks green' >/dev/null 2>&1 || fail "could not propose the mandate" + contract_in "$dir" confirm >/dev/null 2>&1 || fail "could not confirm the mandate" + epoch=$(contract_in "$dir" field entered_epoch) + seed_live_blocker "$dir" tmux repair-final + touch "$dir/home/state/.last-watcher-beat" + : > "$dir/home/state/.fake-drain" + set +e + out=$(run_return "$dir" begin) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "the live blocker did not retain catch-up (rc=$rc): $out" + archive="$dir/home/state/afk-contracts/$epoch.afk-contract" + [ -f "$archive" ] || fail "return did not archive the final posture record" + backup="$dir/final.backup" + cp "$archive" "$backup" + rm "$archive" + printf 'resolved [key=repair-final]: repaired the synthetic blocker\n' >> "$dir/home/state/repair-task.status" + set +e + out=$(run_return "$dir" check) + rc=$? + set -e + [ "$rc" -eq 3 ] || fail "check cleared after the retained final archive disappeared (rc=$rc): $out" + assert_contains "$out" "archived away-posture record missing for entered_epoch $epoch; catch-up stays gated" "check did not name the missing final archive" + [ -f "$dir/home/state/.afk-return-catchup" ] || fail "the missing final archive did not retain the gate" + cp "$backup" "$archive" + out=$(run_return "$dir" check) || fail "check did not clear after the final archive was restored: $out" + assert_contains "$out" 'catch-up clear' "a valid restored final archive did not clear catch-up" + pass "the retained contract epoch requires its final archive on every check" +} + test_return_gate_orders_catchup_before_bearings test_explicit_reclassification_requires_durable_reason test_captain_decision_does_not_masquerade_as_firstmate_blocker test_evidence_publication_failure_preserves_wake_for_redrain test_away_reentry_refuses_pending_return_gate test_check_retries_recorded_terminal_teardown +test_unreadable_superseded_archive_keeps_return_gated +test_missing_final_archive_keeps_retained_contract_gated +test_return_brief_composes_from_record_store_and_held_set +test_return_brief_keeps_refresh_history +test_malformed_posture_record_keeps_catchup_gated +test_missing_epoch_record_stays_required_after_disappearing +test_unreadable_outcome_store_keeps_catchup_gated +test_failed_held_listing_keeps_catchup_gated +test_unreadable_status_file_keeps_catchup_gated +test_return_guard_refuses_while_the_record_exists +test_return_brief_health_leads_with_a_gap +test_return_brief_without_a_record_reports_the_legacy_flag + printf '\nall fm-afk-return tests passed\n' diff --git a/tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh b/tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh new file mode 100755 index 00000000000..e3451e24755 --- /dev/null +++ b/tests/fm-backend-herdr-agent-exit-shell-e2e.test.sh @@ -0,0 +1,213 @@ +#!/usr/bin/env bash +# Real-Herdr regression: `herdr agent get` (not `pane get` `.agent_status`) +# distinguishes a Pi that exited to a leftover shell from a live idle Pi, so +# the recovery classifier maps the leftover shell to no-agent/dead +# (relaunch-allowed) and keeps the live idle pane alive. +# +# Do not launch Pi with `exec`: the pane shell must survive when the agent +# exits. That is the #4115 shape. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=tests/herdr-test-safety.sh +. "$ROOT/tests/herdr-test-safety.sh" + +herdr_forget_inherited_pane +fm_live_gate default-on FM_HERDR_AGENT_EXIT_SHELL_E2E herdr jq pi + +HERDR_LAB_HELPER=${HERDR_LAB_HELPER:-$ROOT/bin/fm-herdr-lab.sh} +[ -x "$HERDR_LAB_HELPER" ] || { echo "skip: live: Herdr lab helper not executable at $HERDR_LAB_HELPER"; exit 0; } + +HERDR_ORIGINAL_PATH=$PATH +TMP_ROOT=$(fm_test_tmproot fm-herdr-agent-exit-shell-e2e) +FAKEBIN="$TMP_ROOT/fakebin" +PROJECT="$TMP_ROOT/project" +PI_DIR="$TMP_ROOT/pi-agent" +mkdir -p "$FAKEBIN" "$PROJECT" "$PI_DIR" +printf '# Isolated herdr agent-exit lab\n' > "$PROJECT/AGENTS.md" + +HERDR_LAB_SESSION=$("$HERDR_LAB_HELPER" name fm-herdr-agent-exit-shell) +export HERDR_LAB_HELPER HERDR_LAB_SESSION HERDR_ORIGINAL_PATH + +cleanup() { + local status=$? + env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" teardown "$HERDR_LAB_SESSION" || status=1 + fm_test_cleanup + exit "$status" +} +trap cleanup EXIT +"$HERDR_LAB_HELPER" provision "$HERDR_LAB_SESSION" + +cat > "$FAKEBIN/herdr" <<'SH' +#!/usr/bin/env bash +set -u +args=("$@") +last=$((${#args[@]} - 1)) +flag=$((last - 1)) +if [ "${#args[@]}" -ge 2 ] \ + && [ "${args[$flag]}" = --session ] \ + && [ "${args[$last]}" = "$HERDR_LAB_SESSION" ]; then + unset "args[$last]" "args[$flag]" +fi +set -- "${args[@]}" +for arg in "$@"; do + case "$arg" in --session|--session=*) exit 9 ;; esac +done +exec env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" "$@" +SH +chmod +x "$FAKEBIN/herdr" + +lab() { env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" "$@"; } + +classify_pane() { # <pane_id> + PATH="$FAKEBIN:$HERDR_ORIGINAL_PATH" bash -c ' + set -u + . "$1/bin/backends/herdr.sh" + fm_backend_herdr_pane_agent_state "$2" "$3" + ' _ "$ROOT" "$HERDR_LAB_SESSION" "$1" +} + +classify_recovery() { # <pane_id> + PATH="$FAKEBIN:$HERDR_ORIGINAL_PATH" bash -c ' + set -u + . "$1/bin/backends/herdr.sh" + fm_backend_herdr_agent_state "$2:$3" + ' _ "$ROOT" "$HERDR_LAB_SESSION" "$1" +} + +agent_get_code() { # <pane_id> + local out + out=$(lab agent get "$1" 2>&1) || true + printf '%s' "$out" | jq -r '.error.code // empty' +} + +agent_get_status() { # <pane_id> + local out + out=$(lab agent get "$1" 2>&1) || true + printf '%s' "$out" | jq -r '.result.agent.agent_status // empty' +} + +agent_get_kind() { # <pane_id> + local out + out=$(lab agent get "$1" 2>&1) || true + printf '%s' "$out" | jq -r '.result.agent.agent // empty' +} + +pane_present() { # <pane_id> + lab pane get "$1" >/dev/null 2>&1 +} + +pane_agent_status_field() { # <pane_id> + lab pane get "$1" 2>/dev/null | jq -r '.result.pane.agent_status // empty' +} + +wait_until() { # <pane_id> <idle|gone> [attempts] + local pane=$1 want=$2 attempts=${3:-90} code status + for _ in $(seq 1 "$attempts"); do + code=$(agent_get_code "$pane") + status=$(agent_get_status "$pane") + case "$want" in + idle) + case "$status" in idle|done|blocked) return 0 ;; esac + ;; + gone) + [ "$code" = agent_not_found ] && return 0 + ;; + esac + sleep 0.5 + done + return 1 +} + +assert_live_idle() { # <pane_id> <label> + local pane=$1 label=$2 kind status pane_state recov pane_field + pane_present "$pane" || fail "$label: pane disappeared while the agent should still be live" + kind=$(agent_get_kind "$pane") + status=$(agent_get_status "$pane") + [ "$kind" = pi ] || fail "$label: agent get kind was '$kind', want pi" + [ "$status" = idle ] || fail "$label: agent get status was '$status', want idle (authority is agent get, not pane get)" + pane_state=$(classify_pane "$pane") + recov=$(classify_recovery "$pane") + [ "$pane_state" = live ] || fail "$label: pane classifier was '$pane_state', want live" + [ "$recov" = alive ] || fail "$label: recovery classifier was '$recov', want alive (must not reclaim a live idle agent)" + pane_field=$(pane_agent_status_field "$pane") + : "$pane_field" +} + +assert_exited_to_shell() { # <pane_id> <label> + local pane=$1 label=$2 code pane_state recov pane_field + pane_present "$pane" || fail "$label: pane was reaped; launch used exec or the shell did not survive" + code=$(agent_get_code "$pane") + [ "$code" = agent_not_found ] || fail "$label: agent get code was '$code', want agent_not_found" + pane_state=$(classify_pane "$pane") + recov=$(classify_recovery "$pane") + [ "$pane_state" = no-agent ] || fail "$label: pane classifier was '$pane_state', want no-agent" + [ "$recov" = dead ] || fail "$label: recovery classifier was '$recov', want dead (relaunch-allowed), not alive" + # pane get .agent_status may still read idle after the occupant is gone. + # Liveness comes from agent get; a lagged idle field must not keep the pane alive. + pane_field=$(pane_agent_status_field "$pane") + case "$pane_field" in + idle|done|blocked|working) + [ "$recov" = dead ] || fail "$label: pane get agent_status=$pane_field lagged but recovery was '$recov'" + ;; + esac +} + +TRUST="$TMP_ROOT/trust.ts" +cat > "$TRUST" <<'EOF' +import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; +export default function (pi: ExtensionAPI) { + pi.on("project_trust", () => ({ trusted: "yes", remember: false })); +} +EOF + +# Child of the pane shell, never exec, so /quit and SIGKILL return to zsh. +PI_CMD=$(printf 'env PI_CODING_AGENT_DIR=%q pi -e %q --no-context-files --no-session' "$PI_DIR" "$TRUST") + +CREATE=$(lab workspace create --cwd "$PROJECT" --label 'agent-exit-shell' --no-focus) \ + || fail 'could not create the lab workspace' +P_LIVE=$(printf '%s' "$CREATE" | jq -er '.result.root_pane.pane_id') \ + || fail 'could not read the live pane id' +WS=$(printf '%s' "$CREATE" | jq -er '.result.workspace.workspace_id') \ + || fail 'could not read the workspace id' + +QUIT_TAB=$(lab tab create --workspace "$WS" --cwd "$PROJECT" --label quit-to-shell --no-focus) \ + || fail 'could not create the /quit tab' +P_QUIT=$(printf '%s' "$QUIT_TAB" | jq -er '.result.root_pane.pane_id // .result.pane.pane_id') \ + || fail 'could not read the /quit pane id' +KILL_TAB=$(lab tab create --workspace "$WS" --cwd "$PROJECT" --label kill-to-shell --no-focus) \ + || fail 'could not create the SIGKILL tab' +P_KILL=$(printf '%s' "$KILL_TAB" | jq -er '.result.root_pane.pane_id // .result.pane.pane_id') \ + || fail 'could not read the SIGKILL pane id' + +lab pane run "$P_LIVE" "$PI_CMD" >/dev/null || fail 'could not launch live-idle Pi' +lab pane run "$P_QUIT" "$PI_CMD" >/dev/null || fail 'could not launch /quit Pi' +lab pane run "$P_KILL" "$PI_CMD" >/dev/null || fail 'could not launch SIGKILL Pi' + +wait_until "$P_LIVE" idle || fail 'live-idle Pi never became idle on agent get' +wait_until "$P_QUIT" idle || fail '/quit Pi never became idle on agent get' +wait_until "$P_KILL" idle || fail 'SIGKILL Pi never became idle on agent get' + +assert_live_idle "$P_LIVE" 'before-exit live-idle' + +lab pane run "$P_QUIT" '/quit' >/dev/null || fail 'could not send /quit' + +KILL_PID=$(lab pane process-info --pane "$P_KILL" | jq -r ' + .result.process_info.foreground_processes[]? + | select((.name // "") == "pi" or ((.argv0 // "") == "pi")) + | .pid +' | head -1) +[ -n "$KILL_PID" ] && [ "$KILL_PID" != null ] \ + || fail 'SIGKILL pane had no pi pid in process-info' +kill -KILL "$KILL_PID" || fail "kill -KILL $KILL_PID failed" + +wait_until "$P_QUIT" gone || fail '/quit pane never dropped off agent get' +wait_until "$P_KILL" gone || fail 'SIGKILL pane never dropped off agent get' + +assert_exited_to_shell "$P_QUIT" '/quit leftover shell' +assert_exited_to_shell "$P_KILL" 'SIGKILL leftover shell' +assert_live_idle "$P_LIVE" 'sibling live-idle after exits' + +pass 'agent get distinguishes leftover-shell (dead/no-agent) from live idle Pi' +pass 'pane get agent_status lag cannot keep an exited occupant classified alive' diff --git a/tests/fm-backend-herdr-presentation-e2e.test.sh b/tests/fm-backend-herdr-presentation-e2e.test.sh index 439ca95a5f5..3f27d544c5c 100755 --- a/tests/fm-backend-herdr-presentation-e2e.test.sh +++ b/tests/fm-backend-herdr-presentation-e2e.test.sh @@ -201,49 +201,33 @@ pass "real Herdr lab: every projected create, task-tab create, seeded prune, and mkdir -p "$ACTIVE_SEEDED_CONTROL" printf '%s\n' requested > "$ACTIVE_SEEDED_CONTROL/stage" ACTIVE_SEEDED_START=$(log_line_count) -ACTIVE_SEEDED_FOCUS_START=$(focus_audit_line_count) -if spawn_task active-seeded "$HOME_DIR" "$PROJECT_DIR" > "$EVIDENCE_ROOT/active-seeded.out" 2> "$EVIDENCE_ROOT/active-seeded.err"; then - fail "active seeded-tab projection should refuse the prune" +cp "$MOVE_CALL_LOG" "$EVIDENCE_ROOT/move-log-before-active-seeded" \ + || fail "could not save the move audit before the active seeded-tab fixture" +if ! spawn_task active-seeded "$HOME_DIR" "$PROJECT_DIR" > "$EVIDENCE_ROOT/active-seeded.out" 2> "$EVIDENCE_ROOT/active-seeded.err"; then + fail "detached persisted-focus seeded prune should succeed: $(cat "$EVIDENCE_ROOT/active-seeded.err")" +fi +if grep -F "target is the captain's active tab" "$EVIDENCE_ROOT/active-seeded.err" >/dev/null 2>&1; then + fail "detached persisted-focus seeded prune still used the live-viewer refusal" fi -grep -F "target is the captain's active tab" "$EVIDENCE_ROOT/active-seeded.err" >/dev/null 2>&1 \ - || fail "active seeded-tab projection did not report its exact refusal" -ACTIVE_SEEDED_WSID=$(cat "$ACTIVE_SEEDED_CONTROL/workspace") -ACTIVE_SEEDED_TAB=$(cat "$ACTIVE_SEEDED_CONTROL/seeded-tab") ACTIVE_SEEDED_PANE=$(cat "$ACTIVE_SEEDED_CONTROL/seeded-pane") ACTIVE_SEEDED_TASK_PANE=$(cat "$ACTIVE_SEEDED_CONTROL/task-pane") -ACTIVE_SEEDED_FOCUS="$ACTIVE_SEEDED_WSID/$ACTIVE_SEEDED_TAB" -assert_focus_is "$ACTIVE_SEEDED_FOCUS" "active seeded-tab prune refusal" -assert_raw_presentation_mutations_preserved_since "$ACTIVE_SEEDED_FOCUS_START" "active seeded-tab prune refusal" -lab pane get "$ACTIVE_SEEDED_PANE" >/dev/null 2>&1 \ - || fail "active seeded-tab refusal removed the exact seeded pane" -if lab pane get "$ACTIVE_SEEDED_TASK_PANE" >/dev/null 2>&1; then - fail "active seeded-tab failure did not abort-clean the non-active task pane" -fi -sed -n "$((ACTIVE_SEEDED_FOCUS_START + 1)),\$p" "$FOCUS_AUDIT_LOG" | awk -F '\t' -v focus="$ACTIVE_SEEDED_FOCUS" -v pane="$ACTIVE_SEEDED_PANE" ' - $1 == "seeded-prune-refusal" && $2 == focus && $3 == focus && $4 == pane { found = 1 } - END { exit(found ? 0 : 1) } -' || fail "guarded lab did not observe exact focus across the active seeded-tab refusal" -if sed -n "$((ACTIVE_SEEDED_START + 1)),\$p" "$HERDR_CALL_LOG" | grep -F $'pane\tclose\t'"$ACTIVE_SEEDED_PANE" >/dev/null 2>&1; then - fail "active seeded-tab refusal closed the exact active pane" +if lab pane get "$ACTIVE_SEEDED_PANE" >/dev/null 2>&1; then + fail "detached persisted-focus seeded prune left the seeded pane behind" fi +lab pane get "$ACTIVE_SEEDED_TASK_PANE" >/dev/null 2>&1 \ + || fail "detached persisted-focus seeded prune lost the task pane" +sed -n "$((ACTIVE_SEEDED_START + 1)),\$p" "$HERDR_CALL_LOG" | grep -F $'pane\tclose\t'"$ACTIVE_SEEDED_PANE" >/dev/null 2>&1 \ + || fail "detached persisted-focus seeded prune did not close the seeded pane" lab tab focus "$SECOND_TWO_TAB" >/dev/null || fail "could not restore the captured captain tab after the active seeded-tab fixture" assert_focus_is "$CAPTAIN_FOCUS" "active seeded-tab fixture restoration" rm -rf "$ACTIVE_SEEDED_CONTROL" -ACTIVE_SEEDED_CLEANUP_FOCUS_START=$(focus_audit_line_count) -ACTIVE_SEEDED_LOCK=$(session_presentation_lock_path) \ - || fail "could not resolve the session presentation lock for active-seeded cleanup" -PATH="$FAKEBIN:$PATH" FM_HOME="$HOME_DIR" bash -c ' - . "$0/bin/fm-wake-lib.sh" - . "$0/bin/backends/herdr.sh" - lock=$1 - fm_lock_acquire_wait "$lock" - fm_backend_herdr_projection_cleanup_exact "$2" "$3" "$4" - fm_lock_release "$lock" -' "$ROOT" "$ACTIVE_SEEDED_LOCK" "$HERDR_LAB_SESSION" "$ACTIVE_SEEDED_TASK_PANE" "$ACTIVE_SEEDED_PANE" +remember_meta_worktree "$HOME_DIR/state/active-seeded.meta" >/dev/null +teardown_task active-seeded "$HOME_DIR" > "$TMP_ROOT/active-seeded-teardown.out" 2> "$TMP_ROOT/active-seeded-teardown.err" \ + || fail "detached persisted-focus seeded prune leftover teardown failed: $(cat "$TMP_ROOT/active-seeded-teardown.err")" +cp "$EVIDENCE_ROOT/move-log-before-active-seeded" "$MOVE_CALL_LOG" \ + || fail "could not restore the move audit after the active seeded-tab fixture" assert_focus_is "$CAPTAIN_FOCUS" "active seeded-tab fixture cleanup" -assert_cleanup_focus_preserved "$ACTIVE_SEEDED_CLEANUP_FOCUS_START" "$ACTIVE_SEEDED_PANE" "$CAPTAIN_FOCUS" -rm -f "$HOME_DIR/state/active-seeded.herdr-presentation" -pass "real Herdr lab: active seeded-tab pruning refuses the exact pane and preserves exact focus" +pass "real Herdr lab: persisted-focused seeded prune proceeds when no live client is attached" LOCK_CONTENTION_READY="$TMP_ROOT/lock-contention-ready" LOCK_CONTENTION_RELEASE="$TMP_ROOT/lock-contention-release" @@ -293,7 +277,7 @@ fi assert_focus_is "$CAPTAIN_FOCUS" "bounded presentation lock flat fallback" assert_raw_presentation_mutations_preserved_since "$LOCK_CONTENTION_FOCUS_START" "bounded presentation lock flat fallback" teardown_task lock-contended "$HOME_DIR" > "$EVIDENCE_ROOT/lock-contended-teardown.out" 2> "$EVIDENCE_ROOT/lock-contended-teardown.err" \ - || fail "flat lock-contention fixture teardown failed" + || fail "flat lock-contention fixture teardown failed: $(cat "$EVIDENCE_ROOT/lock-contended-teardown.err")" assert_focus_is "$CAPTAIN_FOCUS" "bounded presentation lock flat fallback teardown" pass "real Herdr lab: bounded lock contention warns and falls back flat without projection or focus drift" PROJECTION_ORDER_START=$(log_line_count) diff --git a/tests/fm-backend-herdr-smoke.test.sh b/tests/fm-backend-herdr-smoke.test.sh index 61d6c58e1ae..2cb34ae44a6 100755 --- a/tests/fm-backend-herdr-smoke.test.sh +++ b/tests/fm-backend-herdr-smoke.test.sh @@ -71,6 +71,18 @@ esac [ -n "$SEEDED_TAB_ID" ] || fail "the first container_ensure in a brand-new isolated session must CREATE the workspace and report its seeded default tab id" pass "real herdr: container_ensure starts the isolated session's server, creates the firstmate workspace ($CONTAINER), and reports its seeded default tab id ($SEEDED_TAB_ID)" +# --- client selection: the real status shape the selection reads ------------ +# bin/backends/herdr.sh "client selection" steps around a client the running +# server refuses by reading .server.running/.server.compatible per session; a +# fixture can only restate that shape, so prove the installed binary against +# its own running lab server normalizes to running and compatible with equal +# protocols. +CLIENT_STATUS=$(fm_backend_herdr_client_status "$(command -v herdr)" "$SESSION") +IFS='|' read -r CS_RUNNING CS_COMPATIBLE <<< "$CLIENT_STATUS" +[ "$CS_RUNNING" = true ] || fail "real herdr: status for the running lab server normalized running=$CS_RUNNING (raw: $CLIENT_STATUS)" +[ "$CS_COMPATIBLE" = true ] || fail "real herdr: the installed client normalized compatible=$CS_COMPATIBLE against its own server (raw: $CLIENT_STATUS)" +pass "real herdr: session status normalizes running and compatible" + # A second container_ensure must reuse (ADOPT) the same workspace (idempotent) # and report an EMPTY seeded tab id - the created-vs-adopted gate that fixes # the 2026-07-02 self-kill incident (docs/herdr-backend.md "Default-tab diff --git a/tests/fm-backend-herdr-stale-active-tab-e2e.test.sh b/tests/fm-backend-herdr-stale-active-tab-e2e.test.sh new file mode 100755 index 00000000000..2fe0bca7650 --- /dev/null +++ b/tests/fm-backend-herdr-stale-active-tab-e2e.test.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +# Real-Herdr regression for the teardown active-tab guard: persisted +# .focused is not a live viewer. A pane on that tab must close when no +# foreground client is attached. +set -u + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +HERDR_LAB_HELPER=${HERDR_LAB_HELPER:-$ROOT/bin/fm-herdr-lab.sh} + +fail() { printf 'not ok - %s\n' "$1" >&2; exit 1; } +pass() { printf 'ok - %s\n' "$1"; } + +command -v herdr >/dev/null 2>&1 || { echo 'skip: herdr not found'; exit 0; } +command -v jq >/dev/null 2>&1 || { echo 'skip: jq not found'; exit 0; } +[ -x "$HERDR_LAB_HELPER" ] || { echo "skip: Herdr lab helper not executable at $HERDR_LAB_HELPER"; exit 0; } + +HERDR_ORIGINAL_PATH=$PATH +TMP_ROOT=$(mktemp -d "$(cd "${TMPDIR:-/tmp}" && pwd -P)/fm-herdr-stale-active-tab-e2e.XXXXXX") +FAKEBIN="$TMP_ROOT/fakebin" +mkdir -p "$FAKEBIN" + +HERDR_LAB_SESSION=$("$HERDR_LAB_HELPER" name fm-herdr-teardown-stale-active-tab-guard-r1) +export HERDR_LAB_HELPER HERDR_LAB_SESSION HERDR_ORIGINAL_PATH +cleanup() { + local status=$? + env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" teardown "$HERDR_LAB_SESSION" || status=1 + rm -rf "$TMP_ROOT" + exit "$status" +} +trap cleanup EXIT +"$HERDR_LAB_HELPER" provision "$HERDR_LAB_SESSION" + +cat > "$FAKEBIN/herdr" <<'SH' +#!/usr/bin/env bash +set -u +args=("$@") +last=$((${#args[@]} - 1)) +flag=$((last - 1)) +if [ "${#args[@]}" -ge 2 ] \ + && [ "${args[$flag]}" = --session ] \ + && [ "${args[$last]}" = "$HERDR_LAB_SESSION" ]; then + unset "args[$last]" "args[$flag]" +fi +set -- "${args[@]}" +for arg in "$@"; do + case "$arg" in --session|--session=*) exit 9 ;; esac +done +exec env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" "$@" +SH +chmod +x "$FAKEBIN/herdr" + +lab() { env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" "$@"; } + +CREATE=$(lab workspace create --cwd "$ROOT" --label 'stale-active-tab' --no-focus) \ + || fail 'could not create the persisted-focus workspace' +PANE=$(printf '%s' "$CREATE" | jq -er '.result.root_pane.pane_id') \ + || fail 'could not read the created pane id' +TAB=$(printf '%s' "$CREATE" | jq -er '.result.tab.tab_id') \ + || fail 'could not read the created tab id' +lab tab focus "$TAB" >/dev/null || fail 'could not persist focus onto the target tab' + +TITLE=$(lab terminal title clear) \ + || fail 'could not probe foreground-client attachment' +REASON=$(printf '%s' "$TITLE" | jq -er '.result.reason') \ + || fail "could not parse the title-clear reason: $TITLE" +[ "$REASON" = no_foreground_client ] \ + || fail "lab unexpectedly had a foreground client (reason=$REASON)" + +FOCUSED=$(lab workspace list | jq -er '[.result.workspaces[] | select(.focused == true)] | select(length == 1) | .[0].active_tab_id') \ + || fail 'could not read the persisted focused tab' +[ "$FOCUSED" = "$TAB" ] || fail "persisted focus was $FOCUSED, not the target tab $TAB" + +OUT=$(PATH="$FAKEBIN:$HERDR_ORIGINAL_PATH" bash -c ' + . "$1/bin/backends/herdr.sh" + fm_backend_herdr_cli() { + local session=$1 + shift + HERDR_SESSION="$session" herdr "$@" --session "$session" + } + fm_backend_herdr_projection_close_pane_focus_preserving "$2" "$3" +' _ "$ROOT" "$HERDR_LAB_SESSION" "$PANE" 2>&1) +STATUS=$? +[ "$STATUS" -eq 0 ] || fail "detached persisted-focus close failed (status $STATUS): $OUT" +case "$OUT" in + *"target is the captain's active tab"*) + fail "detached persisted-focus close still used the live-viewer refusal: $OUT" + ;; +esac +if lab pane get "$PANE" >/dev/null 2>&1; then + fail 'detached persisted-focus close left the pane behind' +fi +pass 'detached client: persisted .focused on the target tab does not block pane close' diff --git a/tests/fm-backend-herdr.test.sh b/tests/fm-backend-herdr.test.sh index 125a050551a..8697a3f2f07 100755 --- a/tests/fm-backend-herdr.test.sh +++ b/tests/fm-backend-herdr.test.sh @@ -14,6 +14,8 @@ set -u . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" # shellcheck source=tests/herdr-test-safety.sh . "$(dirname "${BASH_SOURCE[0]}")/herdr-test-safety.sh" +# shellcheck source=tests/herdr-client-pair-fixture.sh +. "$(dirname "${BASH_SOURCE[0]}")/herdr-client-pair-fixture.sh" command -v jq >/dev/null 2>&1 || { echo "skip: jq not found (required by the herdr adapter)"; exit 0; } @@ -59,6 +61,11 @@ if [ "${1:-}" = status ] && [ "${2:-}" = --json ] && [ "${FM_HERDR_SCRIPT_STATUS printf '{"client":{"version":"0.7.1","protocol":14},"server":{"running":true}}\n' exit 0 fi +if [ "${1:-}" = terminal ] && [ "${2:-}" = title ] && [ "${3:-}" = clear ]; then + reason=${FM_FAKE_HERDR_FOREGROUND_REASON:-no_foreground_client} + printf '{"result":{"reason":"%s"}}\n' "$reason" + exit 0 +fi n=$next echo "$n" > "$COUNT_FILE" if [ -f "$RESP/$n.exit" ]; then @@ -154,6 +161,9 @@ case "$cmd $sub" in "status --json") printf '{"client":{"version":"0.7.1","protocol":14},"server":{"running":true}}\n' ;; + "terminal title") + printf '{"result":{"reason":"no_foreground_client"}}\n' + ;; "workspace list") jq_state '{result:{workspaces:.workspaces}}' ;; @@ -329,6 +339,235 @@ test_cli_helper_sets_env_and_appends_trailing_session_flag() { pass "fm_backend_herdr_cli: sets HERDR_SESSION AND appends a trailing --session flag on every call" } +# --- client selection: a stale client shadowing a compatible one ------------- +# +# Two herdr clients on PATH is a real host shape (a self-updated ~/.local/bin +# copy next to a package-managed one), and the fixed remote-job PATH resolves +# ~/.local/bin first. A client older than the running server answers every +# command with error code protocol_mismatch (verified: herdr 0.8.2, protocol +# 20, against a 0.9.0 server, protocol 22), and until the adapter learned to +# step around it, a live remote secondmate read `unreadable`, every doorbell +# into it failed, and the relaunch that would repair it was refused. + +# run_with_clients <dir> <path-dirs...> -- <bash -c body>: sources the adapter +# in a fresh shell whose PATH holds exactly the named client directories plus +# jq and the system tail, so no herdr from the runner's own PATH can leak in. +# Bodies are bash -c sources, so their single-quoted $ expansions are +# deliberate (SC2016). +# shellcheck disable=SC2016 +run_with_clients() { # <dir> <path> <body> + local dir=$1 path=$2 body=$3 + FM_HERDR_PAIR_DIR="$dir" PATH="$path:$dir/tools:/usr/bin:/bin" \ + bash -c ". \"\$0/bin/backends/herdr.sh\"; $body" "$ROOT" +} + +# The #4091 widening, and the boundary it is deliberately confined to. +# +# A recovery-grade read that cannot confirm the pane is `missing` only when the +# session's server is POSITIVELY stopped - absence for that whole session - +# and stays `unreadable` otherwise. The two signals are driven apart here on +# purpose: the SAME failed pane read is settled two ways by the server state +# alone, so the case cannot go quietly vacuous if one signal stops being read. +# +# The second half matters as much as the first: the widening must not reach the +# husk classifier under it, because that one licenses CLOSING panes. +test_recovery_grade_read_widens_only_at_its_own_boundary() { + local dir log resp fb gone running husk + + herdr_state_with_server() { # <dir-suffix> <server-running-json> + local dir="$TMP_ROOT/recovery-widen-$1" resp log fb + mkdir -p "$dir/responses"; resp="$dir/responses"; log="$dir/log"; : > "$log" + # 1: the pane read, failing in a way this parse cannot interpret. + printf 'Error: socket unavailable\n' > "$resp/1.out" + printf '1\n' > "$resp/1.exit" + # 2: the server-state read that settles it. + printf '{"client":{"protocol":22},"server":{"running":%s}}\n' "$2" > "$resp/2.out" + fb=$(make_herdr_fakebin "$dir") + PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" FM_HERDR_SCRIPT_STATUS=1 \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_agent_state fmtest:w1:p2' "$ROOT" + } + + gone=$(herdr_state_with_server gone false) + running=$(herdr_state_with_server running true) + [ "$gone" = missing ] \ + || fail "an uninterpretable pane read against a positively stopped server must read missing, got '$gone'" + [ "$running" = unreadable ] \ + || fail "an uninterpretable pane read against a RUNNING server must stay unreadable, got '$running'" + [ "$gone" != "$running" ] \ + || fail "the server-state signal is not being consulted: both verdicts are '$gone'" + + # An unreadable server state is not evidence of absence either. + dir="$TMP_ROOT/recovery-widen-unknown"; mkdir -p "$dir/responses"; resp="$dir/responses"; log="$dir/log"; : > "$log" + printf 'Error: socket unavailable\n' > "$resp/1.out"; printf '1\n' > "$resp/1.exit" + printf 'not json at all\n' > "$resp/2.out"; printf '1\n' > "$resp/2.exit" + fb=$(make_herdr_fakebin "$dir") + out=$(PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" FM_HERDR_SCRIPT_STATUS=1 \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_agent_state fmtest:w1:p2' "$ROOT") + [ "$out" = unreadable ] \ + || fail "a server state that cannot itself be read must keep the conservative verdict, got '$out'" + + # The confinement: the husk classifier sees the SAME stopped-server read and + # must still refuse, because it is what licenses closing a pane. + dir="$TMP_ROOT/recovery-widen-husk"; mkdir -p "$dir/responses"; resp="$dir/responses"; log="$dir/log"; : > "$log" + printf 'Error: socket unavailable\n' > "$resp/1.out"; printf '1\n' > "$resp/1.exit" + printf '{"client":{"protocol":22},"server":{"running":false}}\n' > "$resp/2.out" + fb=$(make_herdr_fakebin "$dir") + husk=$(PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" FM_HERDR_SCRIPT_STATUS=1 \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_pane_agent_state fmtest w1:p2; printf " "; fm_backend_herdr_tab_is_husk fmtest w1:p2 && printf husk || printf refused' "$ROOT") + [ "$husk" = "unknown refused" ] \ + || fail "the stopped-server rule leaked into the husk classifier, which licenses closing panes: got '$husk'" + pass "herdr recovery-grade read: a stopped server means missing there, and nowhere else" +} + +test_agent_state_bypasses_a_stale_client_shadowing_a_compatible_one() { + local dir out err + dir="$TMP_ROOT/client-pair-bypass"; make_herdr_client_pair "$dir" + out=$(run_with_clients "$dir" "$dir/stale:$dir/current" 'fm_backend_herdr_agent_state fm-remote:wCY:p2' 2>"$dir/stderr") \ + || fail "agent-state read with a shadowing stale client should not fail" + err=$(cat "$dir/stderr") + [ "$out" = alive ] || fail "a live remote pane behind a stale shadowing client should read alive, got: $out (stderr: $err)" + assert_contains "$(cat "$dir/current.log")" "pane get wCY:p2" "the compatible client should have served the pane read" + assert_contains "$(cat "$dir/current.log")" "agent get wCY:p2" "the compatible client should have served the agent read" + [ -z "$err" ] || fail "a successful bypass must print nothing on stderr (callers merge stderr into parsed JSON), got: $err" + pass "herdr client selection: a live pane behind a stale shadowing client reads alive" +} + +# shellcheck disable=SC2016 +test_cli_caches_the_selected_client_within_a_process() { + local dir out + dir="$TMP_ROOT/client-pair-cache"; make_herdr_client_pair "$dir" + out=$(run_with_clients "$dir" "$dir/stale:$dir/current" \ + 'fm_backend_herdr_cli fm-remote pane get wCY:p2 >/dev/null 2>&1 + fm_backend_herdr_cli fm-remote agent get wCY:p2 >/dev/null 2>&1 + printf "%s" "${FM_BACKEND_HERDR_BIN:-unset}"') + [ "$out" = "$dir/current/herdr" ] \ + || fail "the compatible client should be selected and exported, got: $out" + [ "$(grep -c 'pane get\|agent get' "$dir/stale.log")" -eq 1 ] \ + || fail "after selection the stale client must not be retried in the same process, got: $(cat "$dir/stale.log")" + assert_contains "$(cat "$dir/current.log")" "agent get wCY:p2" "the second call should go straight to the selected client" + pass "herdr client selection: one selected client is reused per process" +} + +# shellcheck disable=SC2016 +test_cli_scopes_the_selected_client_to_its_session() { + local dir out + dir="$TMP_ROOT/client-pair-cross-session" + mkdir -p "$dir/stale" "$dir/current" "$dir/tools" + ln -sf "$(command -v jq)" "$dir/tools/jq" + cat > "$dir/stale/herdr" <<'SH' +#!/usr/bin/env bash +session=${!#} +printf '%s\n' "$*" >> "${FM_HERDR_PAIR_DIR:?}/stale.log" +if [ "${1:-} ${2:-}" = "status --json" ]; then + if [ "$session" = fresh ]; then + printf '{"client":{"version":"0.8.2","protocol":20},"server":{"running":false}}\n' + elif [ -e "$FM_HERDR_PAIR_DIR/switched" ]; then + printf '{"client":{"version":"0.8.2","protocol":20},"server":{"running":true,"protocol":20,"compatible":true}}\n' + else + printf '{"client":{"version":"0.8.2","protocol":20},"server":{"running":true,"protocol":22,"compatible":false}}\n' + fi +elif [ "$session" = fresh ] && [ "${1:-}" = server ]; then + printf 'path-default-server\n' +elif [ "$session" = modern ] && [ -e "$FM_HERDR_PAIR_DIR/switched" ]; then + printf 'legacy\n' +else + printf '{"error":{"code":"protocol_mismatch"}}\n' >&2 + exit 1 +fi +SH + cat > "$dir/current/herdr" <<'SH' +#!/usr/bin/env bash +session=${!#} +printf '%s\n' "$*" >> "${FM_HERDR_PAIR_DIR:?}/current.log" +if [ "${1:-} ${2:-}" = "status --json" ]; then + if [ "$session" = fresh ]; then + printf '{"client":{"version":"0.9.0","protocol":22},"server":{"running":false}}\n' + elif [ -e "$FM_HERDR_PAIR_DIR/switched" ]; then + printf '{"client":{"version":"0.9.0","protocol":22},"server":{"running":true,"protocol":20,"compatible":false}}\n' + else + printf '{"client":{"version":"0.9.0","protocol":22},"server":{"running":true,"protocol":22,"compatible":true}}\n' + fi +elif [ "$session" = fresh ] && [ "${1:-}" = server ]; then + printf 'selected-server\n' +elif [ "$session" = modern ] && [ ! -e "$FM_HERDR_PAIR_DIR/switched" ]; then + printf 'modern\n' +else + printf '{"error":{"code":"protocol_mismatch"}}\n' >&2 + exit 1 +fi +SH + chmod +x "$dir/stale/herdr" "$dir/current/herdr" + out=$(run_with_clients "$dir" "$dir/stale:$dir/current" \ + 'fm_backend_herdr_cli modern pane get w1:p1 > "$FM_HERDR_PAIR_DIR/modern.out" || exit 1 + fm_backend_herdr_cli fresh status --json > "$FM_HERDR_PAIR_DIR/fresh-status.out" || exit 1 + fm_backend_herdr_cli fresh server > "$FM_HERDR_PAIR_DIR/server.out" || exit 1 + touch "$FM_HERDR_PAIR_DIR/switched" + fm_backend_herdr_cli modern pane get w1:p1 > "$FM_HERDR_PAIR_DIR/legacy.out" || exit 1 + printf "%s|%s|%s|%s|%s" "$(cat "$FM_HERDR_PAIR_DIR/modern.out")" "$(jq -r .server.running "$FM_HERDR_PAIR_DIR/fresh-status.out")" "$(cat "$FM_HERDR_PAIR_DIR/server.out")" "$(cat "$FM_HERDR_PAIR_DIR/legacy.out")" "${FM_BACKEND_HERDR_BIN:-PATH-default}"') + [ "$out" = 'modern|false|path-default-server|legacy|PATH-default' ] \ + || fail "a selected client should stay scoped to its session while forced reselection still returns to the PATH default, got: $out" + assert_contains "$(cat "$dir/stale.log")" 'server --session fresh' "a stopped second session should start with the PATH-default client" + assert_not_contains "$(cat "$dir/current.log")" 'server --session fresh' "another session's selected client must not start the stopped session" + [ "$(grep -c 'pane get w1:p1' "$dir/current.log")" -eq 2 ] \ + || fail "the selected client should be retried after its own server compatibility changes: $(cat "$dir/current.log")" + assert_contains "$(cat "$dir/stale.log")" 'pane get w1:p1' "the changed session call should retry on the newly compatible PATH-default client" + pass "herdr client selection: selected clients remain scoped to their session" +} + +# shellcheck disable=SC2016 +test_cli_unrelated_failure_never_triggers_reselection() { + local dir out rc + dir="$TMP_ROOT/client-pair-unrelated"; make_herdr_client_pair "$dir" + # current first: its pane_not_found refusal is an ordinary business result, + # so the stale client behind it must never be consulted or selected. + out=$(run_with_clients "$dir" "$dir/current:$dir/stale" \ + 'fm_backend_herdr_cli fm-remote pane get wZZ:p9 2>&1; rc=$?; printf "\nrc=%s bin=%s\n" "$rc" "${FM_BACKEND_HERDR_BIN:-unset}"'); rc=$? + assert_contains "$out" 'pane_not_found' "the ordinary refusal must be replayed to the caller verbatim" + assert_contains "$out" 'rc=1 bin=unset' "an unrelated failure must keep the exit status and select nothing" + [ ! -e "$dir/stale.log" ] || fail "the shadowed client must not be consulted on an unrelated failure: $(cat "$dir/stale.log")" + pass "herdr client selection: only protocol_mismatch triggers reselection; other failures pass through untouched" +} + +# shellcheck disable=SC2016 +test_cli_single_client_pays_no_selection_read() { + local dir out + dir="$TMP_ROOT/client-single"; make_herdr_client_pair "$dir" + out=$(run_with_clients "$dir" "$dir/current" 'fm_backend_herdr_cli fm-remote pane get wCY:p2 >/dev/null; printf "%s" "${FM_BACKEND_HERDR_BIN:-unset}"') + [ "$out" = unset ] || fail "a single healthy client must stay the PATH default, got: $out" + [ "$(grep -c status "$dir/current.log")" -eq 0 ] \ + || fail "a healthy call must make no status read: $(cat "$dir/current.log")" + pass "herdr client selection: the happy path makes no extra call" +} + +test_client_status_reads_both_status_shapes() { + local dir out + dir="$TMP_ROOT/client-status-shapes"; mkdir -p "$dir/bin" "$dir/tools" + ln -sf "$(command -v jq)" "$dir/tools/jq" + # An older client that reports protocols but no .server.compatible field + # (the pre-0.8 status shape) must be judged by protocol equality. + cat > "$dir/bin/herdr" <<'SH' +#!/usr/bin/env bash +case "${FM_HERDR_STATUS_SHAPE:?}" in + legacy-equal) printf '{"client":{"version":"0.7.5","protocol":16},"server":{"running":true,"protocol":16}}\n' ;; + legacy-older) printf '{"client":{"version":"0.7.5","protocol":16},"server":{"running":true,"protocol":22}}\n' ;; + no-protocol) printf '{"client":{"version":"0.7.1"},"server":{"running":true}}\n' ;; + stopped) printf '{"client":{"version":"0.9.0","protocol":22},"server":{"running":false}}\n' ;; +esac +SH + chmod +x "$dir/bin/herdr" + for shape in legacy-equal legacy-older no-protocol stopped; do + out=$(FM_HERDR_STATUS_SHAPE=$shape PATH="$dir/tools:/usr/bin:/bin" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_client_status "$1" fm-remote' "$ROOT" "$dir/bin/herdr") + case "$shape" in + legacy-equal) [ "$out" = 'true|true' ] || fail "legacy equal protocols should read compatible, got: $out" ;; + legacy-older) [ "$out" = 'true|false' ] || fail "legacy older client should read incompatible, got: $out" ;; + no-protocol) [ "$out" = 'true|' ] || fail "a client reporting no protocol must read unknown, never false, got: $out" ;; + stopped) [ "$out" = 'false|' ] || fail "a stopped server must read not running, got: $out" ;; + esac + done + pass "herdr client status: .server.compatible, legacy protocol equality, unknown, and stopped shapes all normalize" +} + # --- launcher_identity: the exact workspace a worker must be placed in ------- # # Herdr injects HERDR_ENV/HERDR_PANE_ID/HERDR_SESSION/HERDR_SOCKET_PATH into @@ -1430,16 +1669,73 @@ test_projection_close_refuses_active_tab() { printf '%s\n' '{"result":{"workspaces":[{"workspace_id":"w9","active_tab_id":"w9:t2","focused":true}]}}' > "$resp/1.out" printf '%s\n' '{"result":{"tabs":[{"tab_id":"w9:t2","focused":true}]}}' > "$resp/2.out" printf '%s\n' '{"result":{"pane":{"pane_id":"w9:p2","tab_id":"w9:t2","workspace_id":"w9"}}}' > "$resp/3.out" + printf '%s\n' '{"result":{"tabs":[{"tab_id":"w9:t1","workspace_id":"w9"},{"tab_id":"w9:t2","workspace_id":"w9"}]}}' > "$resp/4.out" + cp "$resp/1.out" "$resp/5.out" + cp "$resp/2.out" "$resp/6.out" fb=$(make_herdr_fakebin "$dir") out=$(PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + FM_FAKE_HERDR_FOREGROUND_REASON=cleared \ bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_projection_close_pane_focus_preserving fmtest w9:p2' "$ROOT" 2>&1) status=$? - [ "$status" -ne 0 ] || fail "cleanup must refuse when exact active-tab preservation is impossible" + [ "$status" -ne 0 ] || fail "cleanup must refuse when a live client is viewing the active tab" assert_contains "$out" "target is the captain's active tab" \ "active-tab cleanup refusal did not explain the focus-safety boundary" + assert_contains "$(cat "$log")" $'terminal\x1ftitle\x1fclear' \ + "live-client active-tab refusal did not probe foreground attachment" assert_not_contains "$(cat "$log")" $'pane\x1fclose' \ "active-tab cleanup refusal still closed the pane" - pass "herdr presentation focus: cleanup refuses rather than close the captain's active tab" + pass "herdr presentation focus: cleanup refuses rather than close the tab a live client is viewing" +} + +test_projection_close_refuses_unknown_foreground_reason() { + local dir events out status + dir="$TMP_ROOT/projection-focus-unknown-foreground"; mkdir -p "$dir" + events="$dir/events"; : > "$events" + out=$(ROOT="$ROOT" EVENTS="$events" bash -c ' + . "$ROOT/bin/backends/herdr.sh" + fm_backend_herdr_projection_focus_snapshot() { printf "w9\tw9:t2"; } + fm_backend_herdr_emptying_close_plan() { printf "plain\n"; } + fm_backend_herdr_cli() { + case "$2 $3" in + "pane get") printf "{\"result\":{\"pane\":{\"pane_id\":\"w9:p2\",\"tab_id\":\"w9:t2\",\"workspace_id\":\"w9\"}}}\n" ;; + "terminal title") printf "{\"result\":{\"reason\":\"set\"}}\n" ;; + "pane close") printf "close\n" >> "$EVENTS" ;; + esac + } + fm_backend_herdr_projection_close_pane_focus_preserving fmtest w9:p2 + ' 2>&1) + status=$? + [ "$status" -ne 0 ] || fail "an unexpected foreground response must not authorize an active-tab close" + [ ! -s "$events" ] || fail "unexpected foreground response still closed the pane" + assert_contains "$out" "could not verify whether a foreground client is viewing the target tab" \ + "unexpected foreground response did not take the fail-closed unknown path" + pass "herdr presentation focus: unexpected foreground-client reasons fail closed" +} + +test_projection_close_allows_stale_active_tab_without_foreground_client() { + local dir log resp fb out status + dir="$TMP_ROOT/projection-focus-stale-active-allow"; mkdir -p "$dir/responses" + log="$dir/log"; resp="$dir/responses"; : > "$log" + printf '%s\n' '{"result":{"workspaces":[{"workspace_id":"w9","active_tab_id":"w9:t2","focused":true}]}}' > "$resp/1.out" + printf '%s\n' '{"result":{"tabs":[{"tab_id":"w9:t2","focused":true}]}}' > "$resp/2.out" + printf '%s\n' '{"result":{"pane":{"pane_id":"w9:p2","tab_id":"w9:t2","workspace_id":"w9"}}}' > "$resp/3.out" + printf '%s\n' '{"result":{"tabs":[{"tab_id":"w9:t1","workspace_id":"w9"},{"tab_id":"w9:t2","workspace_id":"w9"}]}}' > "$resp/4.out" + : > "$resp/5.out" + printf '%s\n' '{"error":{"code":"pane_not_found"}}' > "$resp/6.out" + fb=$(make_herdr_fakebin "$dir") + out=$(PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_projection_close_pane_focus_preserving fmtest w9:p2' "$ROOT" 2>&1) + status=$? + [ "$status" -eq 0 ] || fail "cleanup must close a persisted-focused tab when no live client is attached: $out" + assert_not_contains "$out" "target is the captain's active tab" \ + "detached persisted-focus close still used the live-viewer refusal" + assert_contains "$(cat "$log")" $'terminal\x1ftitle\x1fclear' \ + "detached persisted-focus close did not probe foreground attachment" + assert_contains "$(cat "$log")" $'pane\x1fclose\x1fw9:p2' \ + "detached persisted-focus close did not close the exact pane" + assert_not_contains "$(cat "$log")" $'tab\x1ffocus' \ + "detached persisted-focus close restored a persisted pointer with no live viewer" + pass "herdr presentation focus: cleanup closes a persisted-focused tab when no live client is attached" } test_projection_close_reports_focus_restore_failure() { @@ -1498,6 +1794,106 @@ test_projection_close_rechecks_required_agent_state_at_boundary() { pass "herdr presentation reclaim: live agent state at the close boundary refuses mutation" } +test_projection_close_rechecks_foreground_client_after_agent_validation() { + local dir events attached out status + dir="$TMP_ROOT/projection-close-foreground-boundary"; mkdir -p "$dir" + events="$dir/events"; attached="$dir/attached"; : > "$events" + out=$(ROOT="$ROOT" EVENTS="$events" ATTACHED="$attached" bash -c ' + . "$ROOT/bin/backends/herdr.sh" + fm_backend_herdr_projection_focus_snapshot() { printf "w9\tw9:t2"; } + fm_backend_herdr_pane_agent_state() { + printf "agent\n" >> "$EVENTS" + : > "$ATTACHED" + printf no-agent + } + fm_backend_herdr_cli() { + case "$2 $3" in + "pane get") printf "{\"result\":{\"pane\":{\"pane_id\":\"w9:p2\",\"tab_id\":\"w9:t2\"}}}\n" ;; + "terminal title") + printf "foreground\n" >> "$EVENTS" + if [ -e "$ATTACHED" ]; then + printf "{\"result\":{\"reason\":\"cleared\"}}\n" + else + printf "{\"result\":{\"reason\":\"no_foreground_client\"}}\n" + fi + ;; + "pane close") printf "close\n" >> "$EVENTS" ;; + esac + } + fm_backend_herdr_projection_close_pane_focus_preserving fmtest w9:p2 no-agent + ' 2>&1) + status=$? + [ "$status" -ne 0 ] || fail "a client attaching during agent validation must defer the active-tab close" + [ "$(cat "$events")" = $'agent\nforeground' ] \ + || fail "foreground attachment was not checked immediately after agent validation: $(cat "$events")" + assert_contains "$out" "target is the captain's active tab" \ + "fresh foreground-client refusal did not explain the active-tab boundary" + pass "herdr presentation focus: active-tab attachment is rechecked after agent validation" +} + +test_projection_close_rechecks_target_focus_after_planning() { + local dir events focused out status + dir="$TMP_ROOT/projection-close-focus-switch"; mkdir -p "$dir" + events="$dir/events"; focused="$dir/focused"; : > "$events" + out=$(ROOT="$ROOT" EVENTS="$events" FOCUSED="$focused" bash -c ' + . "$ROOT/bin/backends/herdr.sh" + fm_backend_herdr_projection_focus_snapshot() { + if [ -e "$FOCUSED" ]; then + printf "w9\tw9:t2" + else + printf "w1\tw1:t1" + fi + } + fm_backend_herdr_emptying_close_plan() { + : > "$FOCUSED" + printf "plain\n" + } + fm_backend_herdr_cli() { + case "$2 $3" in + "pane get") printf "{\"result\":{\"pane\":{\"pane_id\":\"w9:p2\",\"tab_id\":\"w9:t2\",\"workspace_id\":\"w9\"}}}\n" ;; + "terminal title") printf "{\"result\":{\"reason\":\"cleared\"}}\n" ;; + "pane close") printf "close\n" >> "$EVENTS" ;; + esac + } + fm_backend_herdr_projection_close_pane_focus_preserving fmtest w9:p2 + ' 2>&1) + status=$? + [ "$status" -ne 0 ] || fail "a target focused during planning must defer the pane close" + [ ! -s "$events" ] || fail "the focus-switched target was still mutated: $(cat "$events")" + assert_contains "$out" "target is the captain's active tab" \ + "focus-switch refusal did not explain the active-tab boundary" + pass "herdr presentation focus: pre-close checkpoint catches a target focused during planning" +} + +test_projection_close_preserves_live_focus_that_switched_away_from_target() { + local dir events sample out status + dir="$TMP_ROOT/projection-close-focus-switch-away"; mkdir -p "$dir" + events="$dir/events"; sample="$dir/sample"; : > "$events"; printf '0\n' > "$sample" + out=$(ROOT="$ROOT" EVENTS="$events" SAMPLE="$sample" bash -c ' + . "$ROOT/bin/backends/herdr.sh" + fm_backend_herdr_projection_focus_snapshot() { + local n + n=$(cat "$SAMPLE"); n=$((n + 1)); printf "%s\n" "$n" > "$SAMPLE" + if [ "$n" -eq 1 ]; then printf "w9\tw9:t2"; else printf "w1\tw1:t1"; fi + } + fm_backend_herdr_emptying_close_plan() { printf "plain\n"; } + fm_backend_herdr_cli() { + case "$2 $3" in + "pane get") printf "{\"result\":{\"pane\":{\"pane_id\":\"w9:p2\",\"tab_id\":\"w9:t2\",\"workspace_id\":\"w9\"}}}\n" ;; + "terminal title") printf "{\"result\":{\"reason\":\"cleared\"}}\n" ;; + esac + } + fm_backend_herdr_explicit_close_pane_confirmed() { printf "close\n" >> "$EVENTS"; } + fm_backend_herdr_projection_focus_restore() { printf "restore:%s\n" "$2" >> "$EVENTS"; } + fm_backend_herdr_projection_close_pane_focus_preserving fmtest w9:p2 + ' 2>&1) + status=$? + [ "$status" -eq 0 ] || fail "a client switching from the target to another tab should allow the target close: $out" + [ "$(cat "$events")" = $'close\nrestore:w1\tw1:t1' ] \ + || fail "close did not preserve the live client's fresh non-target focus: $(cat "$events")" + pass "herdr presentation focus: close preserves a live client that switches away from the target during planning" +} + # --- emptying-close focus-safe removal (Herdr 0.7.5 #1621 mitigation) ------ # # The fixtures below model the verified 0.7.5 rules: an explicit close that @@ -2524,8 +2920,12 @@ test_projection_seeded_prune_refuses_active_tab() { printf '%s\n' '{"result":{"workspaces":[{"workspace_id":"w9","active_tab_id":"w9:t1","focused":true}]}}' > "$resp/4.out" printf '%s\n' '{"result":{"tabs":[{"tab_id":"w9:t1","focused":true},{"tab_id":"w9:t2","focused":false}]}}' > "$resp/5.out" printf '%s\n' '{"result":{"pane":{"pane_id":"w9:p1","tab_id":"w9:t1","workspace_id":"w9"}}}' > "$resp/6.out" + cp "$resp/1.out" "$resp/7.out" + cp "$resp/4.out" "$resp/8.out" + cp "$resp/5.out" "$resp/9.out" fb=$(make_herdr_fakebin "$dir") out=$(PATH="$fb:$PATH" FM_HERDR_LOG="$log" FM_HERDR_RESPONSES="$resp" \ + FM_FAKE_HERDR_FOREGROUND_REASON=cleared \ bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_workspace_prune_seeded_default_tab fmtest w9 w9:t1 focus-preserving' "$ROOT" 2>&1) status=$? [ "$status" -ne 0 ] || fail "projected seeded pruning must refuse the active tab" @@ -3073,7 +3473,7 @@ test_projection_reclaim_replaces_only_exact_husk_and_advances_binding() { [ -n "$agent_line" ] && [ "$agent_line" -lt "$close_line" ] \ || fail "reclaim did not recheck the old pane agent state before the close" boundary_mutations=$(sed -n "$((agent_line + 1)),$((close_line - 1))p" "$log" \ - | grep -Ev $'\x1f(tab\x1flist|pane\x1flist|workspace\x1flist)' || true) + | grep -Ev $'\x1f(tab\x1flist|pane\x1flist|workspace\x1flist|terminal\x1ftitle\x1fclear)' || true) [ -z "$boundary_mutations" ] \ || fail "reclaim mutated between the old pane agent recheck and the close: $boundary_mutations" assert_not_contains "$calls" $'workspace\x1fclose' "reclaim introduced workspace-close authority" @@ -4888,6 +5288,13 @@ test_workspace_label_secondmate_marker_trims_whitespace test_workspace_label_empty_marker_falls_back_to_primary test_workspace_label_different_secondmates_get_different_labels test_cli_helper_sets_env_and_appends_trailing_session_flag +test_agent_state_bypasses_a_stale_client_shadowing_a_compatible_one +test_recovery_grade_read_widens_only_at_its_own_boundary +test_cli_caches_the_selected_client_within_a_process +test_cli_scopes_the_selected_client_to_its_session +test_cli_unrelated_failure_never_triggers_reselection +test_cli_single_client_pays_no_selection_read +test_client_status_reads_both_status_shapes test_launcher_identity_absent_without_a_herdr_pane test_launcher_identity_absent_when_herdr_env_alone_is_set test_launcher_identity_resolves_the_exact_pane_tab_and_workspace @@ -4941,8 +5348,13 @@ test_projection_create_never_closes_a_concurrent_same_label_tab test_projection_focus_snapshot_requires_exact_workspace_and_tab test_projection_close_restores_exact_prior_focus test_projection_close_refuses_active_tab +test_projection_close_refuses_unknown_foreground_reason +test_projection_close_allows_stale_active_tab_without_foreground_client test_projection_close_reports_focus_restore_failure test_projection_close_rechecks_required_agent_state_at_boundary +test_projection_close_rechecks_foreground_client_after_agent_validation +test_projection_close_rechecks_target_focus_after_planning +test_projection_close_preserves_live_focus_that_switched_away_from_target test_projection_close_emptying_after_focus_uses_pane_death_without_move test_projection_close_emptying_before_focus_repositions_then_uses_pane_death test_projection_close_emptying_before_last_focus_needs_no_move diff --git a/tests/fm-backlog-atomicity.test.sh b/tests/fm-backlog-atomicity.test.sh index 2bbeda6923d..ec44444a613 100755 --- a/tests/fm-backlog-atomicity.test.sh +++ b/tests/fm-backlog-atomicity.test.sh @@ -42,6 +42,15 @@ command -v tasks-axi >/dev/null 2>&1 || { # --- fixture ---------------------------------------------------------------- +# fm_tasks_axi_backend reads <addressing-root>/.tasks.toml and otherwise falls +# through to the developer's ambient ~/.tasks-axi/config.toml. make_home pins +# the home itself; a case that relocates its data directory is addressed from +# that directory's own parent instead, so it pins that root too. Cases that +# prove a root OUTSIDE the home is refused deliberately leave it unpinned. +pin_markdown_backend() { # <addressing-root> + printf '%s\n' 'backend = "markdown"' > "$1/.tasks.toml" +} + # A home with a real backlog, a real project clone with an origin, a pooled # worktree, and stubs for every tool the spawn path shells out to. make_home() { # <name> [task-id...] @@ -55,6 +64,15 @@ make_home() { # <name> [task-id...] printf '%s\n' claude > "$home/config/crew-harness" printf '%s\n' '# Backlog' '' '## In flight' '' '## Queued' '' '## Done' \ > "$home/data/backlog.md" + # Pin the adapter per case: without it fm_tasks_axi_backend would fall through + # to the developer's ambient tasks-axi config and silently exercise a + # different transition path. + cat > "$home/.tasks.toml" <<'EOF' +backend = "markdown" + +[markdown] +path = "data/backlog.md" +EOF for id in "$@"; do mkdir -p "$home/data/$id" cat > "$home/data/$id/brief.md" <<EOF @@ -102,6 +120,27 @@ row_state() { # <case-dir> <id> sed -n 's/^ state: *//p' | head -1 } +configure_env_backend_tasks_axi() { # <case-dir> + local case_dir=$1 + rm -f "$(backlog_of "$case_dir")" + cat > "$case_dir/fakebin/tasks-axi" <<SH +#!/usr/bin/env bash +case "\${1:-}" in + --version) printf '0.2.5\n' ;; + update) printf '%s\n' '--archive-body' ;; + mv) printf '%s\n' '[<id>...]' ;; + show) + printf 'task:\n state: queued\n held: no\n blocked: no\n' + ;; + start) + printf '%s\n' "\$*" > "$case_dir/env-backend-start" + ;; + *) exit 2 ;; +esac +SH + chmod +x "$case_dir/fakebin/tasks-axi" +} + # Shadow tasks-axi with a wrapper that fails one verb and delegates every other # verb to the real binary, so a test can drive a genuine mid-transition failure # without faking the reads around it. @@ -591,6 +630,148 @@ run_bootstrap() { # <case-dir> # --- dispatch --------------------------------------------------------------- +test_backend_resolution_preserves_config_errors() { + local case_dir project_config user_config config resolver out rc probe + case_dir="$TMP_ROOT/backend-resolution-errors" + project_config="$case_dir/home/.tasks.toml" + user_config="$case_dir/user-home/.tasks-axi/config.toml" + mkdir -p "$case_dir/home" "$case_dir/user-home/.tasks-axi" + # shellcheck disable=SC2016 # $1..$3 must expand when bash -c evaluates the probe with its supplied arguments. + probe='. "$1/bin/fm-tasks-axi-lib.sh"; "$2" "$3"' + printf '%s\n' 'backend = "beads"' > "$user_config" + printf '%s\n' 'backend = "markdown"' > "$project_config" + for config in "$project_config" "$user_config"; do + chmod 000 "$config" + [ ! -r "$config" ] || fail "the backend configuration fixture is still readable" + for resolver in fm_tasks_axi_backend fm_tasks_axi_backend_resolve; do + rc=0 + out=$(env -u TASKS_AXI_BACKEND HOME="$case_dir/user-home" bash -c "$probe" _ \ + "$ROOT" "$resolver" "$case_dir/home" 2>"$case_dir/stderr") || rc=$? + [ "$rc" -eq 2 ] || fail "$resolver concealed an unreadable configuration: $config (exit $rc)" + [ -z "$out" ] || fail "$resolver returned a backend for an unreadable configuration: $out" + assert_grep "tasks-axi backend configuration cannot be read at $config" "$case_dir/stderr" \ + "$resolver did not identify the unreadable configuration" + done + chmod 600 "$config" + rm "$config" + done + pass "backend resolution preserves unreadable configuration errors for every caller" +} + +test_backend_resolution_preserves_precedence_and_defaults() { + local case_dir resolver out probe + case_dir="$TMP_ROOT/backend-resolution-precedence" + mkdir -p "$case_dir/home" "$case_dir/user-home/.tasks-axi" + # shellcheck disable=SC2016 # $1..$3 must expand when bash -c evaluates the probe with its supplied arguments. + probe='. "$1/bin/fm-tasks-axi-lib.sh"; "$2" "$3"' + for resolver in fm_tasks_axi_backend fm_tasks_axi_backend_resolve; do + out=$(env -u TASKS_AXI_BACKEND HOME="$case_dir/user-home" bash -c "$probe" _ \ + "$ROOT" "$resolver" "$case_dir/home") || fail "$resolver rejected absent configuration" + [ "$out" = markdown ] || fail "$resolver changed the unconfigured default" + printf '%s\n' 'backend = "beads"' > "$case_dir/user-home/.tasks-axi/config.toml" + out=$(env -u TASKS_AXI_BACKEND HOME="$case_dir/user-home" bash -c "$probe" _ \ + "$ROOT" "$resolver" "$case_dir/home") || fail "$resolver rejected readable user configuration" + [ "$out" = beads ] || fail "$resolver ignored the user backend" + chmod 000 "$case_dir/user-home/.tasks-axi/config.toml" + printf '%s\n' 'backend = "markdown"' > "$case_dir/home/.tasks.toml" + out=$(env -u TASKS_AXI_BACKEND HOME="$case_dir/user-home" bash -c "$probe" _ \ + "$ROOT" "$resolver" "$case_dir/home") || fail "$resolver read a lower-priority user configuration" + [ "$out" = markdown ] || fail "$resolver ignored the project backend" + chmod 000 "$case_dir/home/.tasks.toml" + out=$(env TASKS_AXI_BACKEND=beads HOME="$case_dir/user-home" bash -c "$probe" _ \ + "$ROOT" "$resolver" "$case_dir/home") || fail "$resolver read configuration despite an environment override" + [ "$out" = beads ] || fail "$resolver ignored the environment backend" + chmod 600 "$case_dir/home/.tasks.toml" "$case_dir/user-home/.tasks-axi/config.toml" + rm "$case_dir/home/.tasks.toml" "$case_dir/user-home/.tasks-axi/config.toml" + done + pass "backend resolution preserves environment, project, user, and default precedence" +} + +test_backlog_callers_refuse_unreadable_backend_config() ( + local case_dir data id operation rc diagnostic + local args=() + case_dir="$TMP_ROOT/backend-callers" + data="$case_dir/records" + id='backend-callers-row' + mkdir -p "$data" "$case_dir/data" "$case_dir/config" + printf '%s\n' '# Backlog' '' '## In flight' '' '## Queued' '' '## Done' > "$data/backlog.md" + cp "$data/backlog.md" "$case_dir/data/backlog.md" + TASKS_AXI_BACKEND=markdown tasks-axi add "$id" "Configured row" --file "$data/backlog.md" >/dev/null \ + || fail "could not create the configured backlog" + TASKS_AXI_BACKEND=markdown tasks-axi add "$id" "Default row" --file "$case_dir/data/backlog.md" >/dev/null \ + || fail "could not create the default backlog" + cp "$data/backlog.md" "$case_dir/configured-before" + cp "$case_dir/data/backlog.md" "$case_dir/default-before" + ln -s missing-config "$case_dir/.tasks.toml" + . "$ROOT/bin/fm-tasks-axi-lib.sh" + . "$ROOT/bin/fm-backlog-transition-lib.sh" + unset TASKS_AXI_BACKEND + for operation in fm_backlog_transition_applies fm_backlog_row_show fm_backlog_row_list fm_backlog_row_probe fm_backlog_mutate; do + case "$operation" in + fm_backlog_transition_applies) args=("$case_dir/config" "$data" ship) ;; + fm_backlog_row_list) args=("$data") ;; + fm_backlog_mutate) args=("$data" start "$id") ;; + *) args=("$data" "$id") ;; + esac + rc=0 + "$operation" "${args[@]}" > "$case_dir/stdout" 2> "$case_dir/stderr" || rc=$? + [ "$rc" -eq 2 ] || fail "$operation ignored the backend error (exit $rc)" + [ ! -s "$case_dir/stdout" ] || fail "$operation returned data despite the backend error" + case "$operation" in + fm_backlog_row_probe) diagnostic=$FM_BACKLOG_ROW_ERROR ;; + fm_backlog_transition_applies|fm_backlog_mutate) diagnostic=$FM_BACKLOG_TRANSITION_ERROR ;; + *) diagnostic=$(cat "$case_dir/stderr") ;; + esac + assert_contains "$diagnostic" "tasks-axi backend configuration cannot be read at $case_dir/.tasks.toml" \ + "$operation lost the configuration diagnostic" + cmp -s "$case_dir/configured-before" "$data/backlog.md" || fail "$operation changed the configured backlog" + cmp -s "$case_dir/default-before" "$case_dir/data/backlog.md" || fail "$operation changed the default backlog" + done + pass "backlog readers and mutations refuse unresolved backends without touching either backlog" +) + +test_captain_hold_preserves_relocated_backlog_on_backend_error() { + local case_dir home data config_state id out rc show + id='backend-hold-row' + for config_state in dangling absent readable; do + case_dir="$TMP_ROOT/backend-hold-$config_state" + home="$case_dir/home" + data="$home/records" + mkdir -p "$data" "$home/data" "$home/config" "$home/state" "$case_dir/user-home" + printf '%s\n' '# Backlog' '' '## In flight' '' '## Queued' '' '## Done' > "$data/backlog.md" + cp "$data/backlog.md" "$home/data/backlog.md" + TASKS_AXI_BACKEND=markdown tasks-axi add "$id" "Hold regression" --file "$data/backlog.md" >/dev/null \ + || fail "could not create the configured hold row" + TASKS_AXI_BACKEND=markdown tasks-axi add "$id" "Hold regression" --file "$home/data/backlog.md" >/dev/null \ + || fail "could not create the default hold row" + cp "$data/backlog.md" "$case_dir/configured-before" + cp "$home/data/backlog.md" "$case_dir/default-before" + case "$config_state" in + dangling) ln -s missing-config "$home/.tasks.toml" ;; + readable) printf '%s\n' 'backend = "markdown"' > "$home/.tasks.toml" ;; + esac + rc=0 + out=$(env -u TASKS_AXI_BACKEND HOME="$case_dir/user-home" FM_HOME="$home" \ + FM_DATA_OVERRIDE="$data" "$ROOT/bin/fm-captain-hold.sh" hold "$id" \ + --title "Hold regression" --reason "Captain must choose" 2>&1) || rc=$? + if [ "$config_state" = dangling ]; then + [ "$rc" -ne 0 ] || fail "captain hold accepted an unresolved backend and changed the wrong backlog" + assert_contains "$out" "tasks-axi backend configuration cannot be read at $home/.tasks.toml" \ + "captain hold did not report its configuration error" + cmp -s "$case_dir/configured-before" "$data/backlog.md" \ + || fail "refused captain hold changed the configured backlog" + else + [ "$rc" -eq 0 ] || fail "captain hold rejected $config_state configuration: $out" + show=$(TASKS_AXI_BACKEND=markdown tasks-axi show "$id" --file "$data/backlog.md") \ + || fail "the configured hold row disappeared" + assert_contains "$show" "held: yes" "captain hold did not update the configured backlog" + fi + cmp -s "$case_dir/default-before" "$home/data/backlog.md" \ + || fail "captain hold changed the default backlog with $config_state configuration" + done + pass "captain hold refuses backend errors and preserves relocated addressing for valid configuration" +} + test_dispatch_moves_the_item_in_flight_in_the_same_run() { local case_dir id out id=atomic-dispatch-b1 @@ -628,6 +809,27 @@ test_dispatch_omits_the_file_for_a_beads_show() { pass "dispatch omits the markdown file when probing a Beads backlog" } +test_a_leftover_markdown_symlink_does_not_brick_a_beads_home() { + local case_dir home id out + id=atomic-dispatch-beads-b2 + case_dir=$(make_home dispatch-beads-leftover "$id") + home=$(home_of "$case_dir") + printf '%s\n' 'backend = "beads"' '[beads]' 'path = ".beads"' \ + 'prefix = "atomic"' > "$home/.tasks.toml" + mkdir -p "$home/archive" + mv "$home/data/backlog.md" "$home/archive/backlog.md" + ln -s ../archive/backlog.md "$home/data/backlog.md" + make_beads_tasks_axi_stub "$case_dir" "$id" + + out=$(run_ship_spawn "$case_dir" "$id") \ + || fail "Beads spawn failed over a leftover markdown symlink: $out" + assert_contains "$out" "spawned $id" "Beads spawn did not report success" + assert_present "$home/state/$id.meta" "spawn published no record" + assert_grep "show $id" "$case_dir/tasks-axi-calls" \ + "Beads dispatch did not probe the backlog row" + pass "a leftover markdown symlink does not brick a Beads home's dispatch" +} + test_completion_omits_the_file_for_a_beads_done() { local case_dir home id out id=atomic-completion-beads-b1 @@ -857,6 +1059,7 @@ test_recovery_uses_the_parent_of_a_trailing_slash_data_record() { case_dir=$(make_home recovery-relocated-root) relocated="$case_dir/fm-records" mkdir -p "$relocated" + pin_markdown_backend "$case_dir" backlog="$relocated/backlog.md" printf '%s\n' '# Backlog' '' '## In flight' '' '## Queued' '' '## Done' > "$backlog" tasks-axi add "$id" "item for $id" --kind ship --file "$backlog" >/dev/null @@ -879,6 +1082,7 @@ test_completion_targets_a_nested_relative_data_directory() { relative_data=relocated/data data="$case_dir/$relative_data" mkdir -p "$case_dir/relocated" + pin_markdown_backend "$case_dir/relocated" mv "$(home_of "$case_dir")/data" "$data" data_resolved=$(cd "$data" && pwd -P) backlog="$data/backlog.md" @@ -906,6 +1110,7 @@ test_immediate_child_absolute_data_dispatches_and_completes() { local case_dir id data data_resolved backlog out id=atomic-immediate-child-data-b2 case_dir=$(make_home immediate-child-data "$id") + pin_markdown_backend "$case_dir" data="$case_dir/fm-records" mv "$(home_of "$case_dir")/data" "$data" data_resolved=$(cd "$data" && pwd -P) @@ -931,6 +1136,7 @@ test_bare_relative_data_dispatches_and_completes() { local case_dir id data backlog out id=atomic-bare-relative-data-b2 case_dir=$(make_home bare-relative-data "$id") + pin_markdown_backend "$case_dir" data="$case_dir/records" mv "$(home_of "$case_dir")/data" "$data" backlog="$data/backlog.md" @@ -1437,6 +1643,7 @@ test_completion_records_a_relative_report_for_relocated_data() { case_dir=$(make_home close-relocated-scout) relocated="$case_dir/relocated/data" mkdir -p "$case_dir/relocated" + pin_markdown_backend "$case_dir/relocated" mv "$(home_of "$case_dir")/data" "$relocated" backlog="$relocated/backlog.md" tasks-axi add "$id" "item for $id" --kind scout --file "$backlog" >/dev/null @@ -1464,6 +1671,7 @@ test_space_containing_scout_report_marker_replays() { case_dir=$(make_home space-report-replay) data="$case_dir/crew space/data" mkdir -p "$case_dir/crew space" + pin_markdown_backend "$case_dir/crew space" mv "$(home_of "$case_dir")/data" "$data" backlog="$data/backlog.md" tasks-axi add "$id" "item for $id" --kind scout --file "$backlog" >/dev/null @@ -2574,6 +2782,12 @@ test_home_without_a_backlog_dispatches_and_completes() { id=atomic-no-backlog-b12 case_dir=$(make_home no-backlog "$id") rm -f "$(backlog_of "$case_dir")" + cat > "$(home_of "$case_dir")/.tasks.toml" <<'EOF' +backend = "markdown" + +[markdown] +path = "data/backlog.md" +EOF make_tasks_axi_incompatible "$case_dir" out=$(run_ship_spawn "$case_dir" "$id") || fail "no-backlog spawn failed: $out" @@ -2587,11 +2801,173 @@ test_home_without_a_backlog_dispatches_and_completes() { pass "a home with no backlog remains exempt from lifecycle transitions" } +test_spawn_refuses_a_special_file_tasks_config() { + local case_dir home id out rc=0 + id=atomic-special-config-b15 + case_dir=$(make_home special-config "$id") + home=$(home_of "$case_dir") + add_item "$case_dir" "$id" + rm -f "$home/.tasks.toml" + mkfifo "$home/.tasks.toml" + + out=$(FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_SPAWN_NO_GUARD=1 FM_FAKE_PANE_PATH="$case_dir/wt" TMUX="fake,1,0" \ + CLAUDE_CONFIG_DIR='' \ + PATH="$case_dir/fakebin:$PATH" \ + timeout 60 "$SPAWN" "$id" "$case_dir/project" --mode no-mistakes --yolo off 2>&1) || rc=$? + [ "$rc" -ne 124 ] || fail "spawn hung reading a special-file tasks-axi config" + [ "$rc" -ne 0 ] || fail "spawn accepted a special-file tasks-axi config" + assert_contains "$out" "tasks-axi config is not a regular file" \ + "spawn did not identify the unsafe tasks-axi config" + assert_absent "$home/state/$id.meta" \ + "spawn published a task record through an unsafe tasks-axi config" + pass "spawn refuses a special-file tasks-axi config instead of blocking on it" +} + +test_spawn_refuses_an_unsafe_tasks_config_before_exempting_a_missing_backlog() { + local case_dir home id out rc=0 + id=atomic-unsafe-config-b15 + case_dir=$(make_home unsafe-config "$id") + home=$(home_of "$case_dir") + rm -f "$(backlog_of "$case_dir")" + mkdir -p "$case_dir/outside" + cat > "$case_dir/outside/tasks.toml" <<'EOF' +backend = "markdown" + +[markdown] +path = "records/tasks.md" +EOF + rm -f "$home/.tasks.toml" + ln -s "$case_dir/outside/tasks.toml" "$home/.tasks.toml" + + out=$(run_ship_spawn "$case_dir" "$id") || rc=$? + [ "$rc" -ne 0 ] || fail "spawn accepted a tasks-axi config resolving outside the home" + assert_contains "$out" "tasks-axi config resolves outside its authorized directory" \ + "spawn silently exempted the home instead of reporting the unsafe config" + assert_absent "$home/state/$id.meta" \ + "spawn published a task record through an unsafe tasks-axi config" + pass "spawn refuses an unsafe tasks-axi config before any exemption is derived from it" +} + + +test_spawn_refuses_a_data_directory_symlinked_outside_the_home() { + local case_dir home id out rc=0 + id=atomic-external-data-b15 + case_dir=$(make_home external-data "$id") + home=$(home_of "$case_dir") + add_item "$case_dir" "$id" + mkdir -p "$case_dir/outside" + mv "$home/data/backlog.md" "$home/data/$id" "$case_dir/outside/" + rmdir "$home/data" + ln -s "$case_dir/outside" "$home/data" + + out=$(TASKS_AXI_BACKEND=markdown run_ship_spawn "$case_dir" "$id") || rc=$? + [ "$rc" -ne 0 ] || fail "spawn accepted a data directory resolving outside the home" + assert_contains "$out" "backlog file authorized directory resolves outside this home" \ + "spawn did not identify the data directory escaping the home" + assert_absent "$home/state/$id.meta" \ + "spawn published a task record through a data directory outside the home" + pass "spawn refuses a data directory symlinked outside the home" +} + + +test_configured_adapter_refuses_a_data_directory_outside_the_home() { + local case_dir home id out rc=0 + id=atomic-external-data-beads-b15 + case_dir=$(make_home external-data-beads "$id") + home=$(home_of "$case_dir") + add_item "$case_dir" "$id" + mkdir -p "$case_dir/outside" + mv "$home/data/backlog.md" "$home/data/$id" "$case_dir/outside/" + rmdir "$home/data" + ln -s "$case_dir/outside" "$home/data" + + out=$(TASKS_AXI_BACKEND=beads run_ship_spawn "$case_dir" "$id") || rc=$? + [ "$rc" -ne 0 ] \ + || fail "a configured adapter accepted a data directory resolving outside the home" + assert_contains "$out" "backlog data directory authorized directory resolves outside this home" \ + "a configured adapter did not identify the data directory escaping the home" + assert_absent "$home/state/$id.meta" \ + "a configured adapter published a task record outside the home" + pass "a configured adapter refuses a data directory outside the home" +} + + +test_dispatch_and_completion_are_structural() { + local case_dir home id meta out pr + id=fm-structural-b15 + pr=https://github.com/example/firstmate/pull/15 + case_dir=$(make_home structural "$id") + home=$(home_of "$case_dir") + add_item "$case_dir" "$id" + + out=$(run_ship_spawn "$case_dir" "$id") \ + || fail "structural spawn failed: $out" + [ "$(row_state "$case_dir" "$id")" = in_flight ] \ + || fail "spawn left the backlog item outside In flight" + + # Recovery may claim an already-live row repeatedly; the transition remains + # idempotent and does not reopen or duplicate the item. + run_bootstrap "$case_dir" >/dev/null \ + || fail "first idempotent reconciliation failed" + run_bootstrap "$case_dir" >/dev/null \ + || fail "second idempotent reconciliation failed" + [ "$(row_state "$case_dir" "$id")" = in_flight ] \ + || fail "repeated claims changed the live backlog state" + + meta="$home/state/$id.meta" + printf 'pr=%s\n' "$pr" >> "$meta" + out=$(run_teardown "$case_dir" "$id") \ + || fail "structural teardown failed: $out" + [ "$(row_state "$case_dir" "$id")" = "done" ] \ + || fail "teardown left the backlog item outside Done" + assert_grep "$pr" "$(backlog_of "$case_dir")" \ + "teardown closed the item without its recorded PR evidence" + pass "dispatch and completion transition structurally with evidence" +} + +test_refused_teardown_leaves_the_item_live() { + local case_dir home id out rc=0 + id=fm-structural-refusal-b15 + case_dir=$(make_home structural-refusal "$id") + home=$(home_of "$case_dir") + add_item "$case_dir" "$id" + out=$(run_ship_spawn "$case_dir" "$id") \ + || fail "refusal setup spawn failed: $out" + + printf '%s\n' unlanded > "$case_dir/wt/unlanded.txt" + git -C "$case_dir/wt" add unlanded.txt + git -C "$case_dir/wt" -c user.name=fmtest -c user.email=fmtest@example.invalid \ + commit -q -m "unlanded fixture work" + out=$(run_teardown "$case_dir" "$id") || rc=$? + + [ "$rc" -ne 0 ] || fail "teardown accepted unlanded work" + [ "$(row_state "$case_dir" "$id")" = in_flight ] \ + || fail "refused teardown changed the live backlog state" + assert_present "$home/state/$id.meta" \ + "refused teardown removed the live task record" + pass "refused teardown leaves the backlog item in flight" +} + +test_environment_selected_adapter_is_not_forced_to_markdown() { + local case_dir id out + id=fm-env-adapter-b15 + case_dir=$(make_home env-adapter "$id") + configure_env_backend_tasks_axi "$case_dir" + + out=$(TASKS_AXI_BACKEND=beads run_ship_spawn "$case_dir" "$id") \ + || fail "environment-selected adapter spawn failed: $out" + [ "$(cat "$case_dir/env-backend-start")" = "start $id" ] \ + || fail "environment-selected adapter received legacy markdown arguments" + pass "environment-selected adapters bypass the legacy markdown file override" +} + test_manual_backend_home_dispatches_and_completes_without_touching_the_backlog() { local case_dir id data data_resolved out id=atomic-manual-b12 case_dir=$(make_home manual-backend "$id") printf '%s\n' manual > "$(home_of "$case_dir")/config/backlog-backend" + pin_markdown_backend "$case_dir" data="$case_dir/manual-data" mv "$(home_of "$case_dir")/data" "$data" data_resolved=$(cd "$data" && pwd -P) @@ -2687,8 +3063,13 @@ test_retiring_a_persistent_secondmate_needs_no_backlog_item() { pass "retiring a persistent secondmate needs no backlog item and creates none" } +test_backend_resolution_preserves_config_errors +test_backend_resolution_preserves_precedence_and_defaults +test_backlog_callers_refuse_unreadable_backend_config +test_captain_hold_preserves_relocated_backlog_on_backend_error test_dispatch_moves_the_item_in_flight_in_the_same_run test_dispatch_omits_the_file_for_a_beads_show +test_a_leftover_markdown_symlink_does_not_brick_a_beads_home test_completion_omits_the_file_for_a_beads_done test_dispatch_refuses_a_pending_authoritative_close test_dispatch_refuses_a_held_row_before_creating_resources @@ -2772,6 +3153,13 @@ test_no_backlog_teardown_refuses_a_symlinked_task_record_at_entry test_teardown_rechecks_record_parent_after_lock_acquisition test_teardown_refuses_a_symlinked_state_directory_at_entry test_home_without_a_backlog_dispatches_and_completes +test_spawn_refuses_a_special_file_tasks_config +test_spawn_refuses_an_unsafe_tasks_config_before_exempting_a_missing_backlog +test_spawn_refuses_a_data_directory_symlinked_outside_the_home +test_configured_adapter_refuses_a_data_directory_outside_the_home +test_dispatch_and_completion_are_structural +test_refused_teardown_leaves_the_item_live +test_environment_selected_adapter_is_not_forced_to_markdown test_manual_backend_home_dispatches_and_completes_without_touching_the_backlog test_a_secondmate_home_keeps_its_own_books test_a_persistent_secondmate_is_never_a_backlog_item diff --git a/tests/fm-bearings-snapshot.test.sh b/tests/fm-bearings-snapshot.test.sh index 0ef7d34b41c..ac352923e79 100755 --- a/tests/fm-bearings-snapshot.test.sh +++ b/tests/fm-bearings-snapshot.test.sh @@ -14,6 +14,7 @@ set -u . "$ROOT/bin/fm-secondmate-registry-lib.sh" BEARINGS="$ROOT/bin/fm-bearings-snapshot.sh" +TASKS_AXI_BIN=$(command -v tasks-axi || true) TMP_ROOT=$(fm_test_tmproot fm-bearings) # Keep disposable homes outside the snapshot's fixture repo boundary even when # TMPDIR is inside an isolated source worktree. @@ -223,6 +224,14 @@ run() { # <home> <fakebin> <args...> PATH="$fakebin:$PATH" FM_HOME="$home" FM_BEARINGS_NOW=2026-07-11T18:00:00Z NET_LOG="$home/net.log" "$BEARINGS" "$@" } +run_captain() { # <home> <fakebin> <command args...> + local home=$1 fakebin=$2 + shift 2 + PATH="$fakebin:$PATH" REAL_TASKS_AXI="$TASKS_AXI_BIN" FM_ROOT_OVERRIDE="$ROOT" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-captain-hold.sh" "$@" +} + write_remote_home_summary() { # <remote-home> <generated-epoch> local home=$1 epoch=$2 mkdir -p "$home/state" @@ -315,12 +324,14 @@ SH run_remote_ledger_bearings() { # <parent-home> <fakebin> <epoch> local parent=$1 fakebin=$2 epoch=$3 + # Allow process startup on loaded hosts; the 30-second fake reads still + # exceed this shared deadline and must be cancelled. FM_HOME="$parent" FM_ROOT_OVERRIDE="$ROOT" FM_SSH_BIN="$fakebin/fake-ssh" \ FM_TEST_LEDGER_CALL_LOG="$parent/ledger-calls.log" \ FM_TEST_LEDGER_PID_LOG="$parent/ledger-pids.log" \ FM_TEST_LEDGER_ACTIVE_DIR="$parent/ledger-active" \ FM_SNAPSHOT_CACHE_DIR="$parent/state/summary-cache" \ - FM_SNAPSHOT_BUDGET=3 FM_SNAPSHOT_NOW_EPOCH="$epoch" \ + FM_SNAPSHOT_BUDGET=15 FM_SNAPSHOT_NOW_EPOCH="$epoch" \ FM_BEARINGS_NOW=2026-09-01T22:00:00Z "$BEARINGS" --json } @@ -1189,8 +1200,8 @@ test_queued_item_prose_never_hides_it() { # The collapsed captain-call contract: any due, unblocked captain-held task is # Captain's Call whatever its kind; a date-deferred hold is a dated gate until # due; deferral wording in the reason changes nothing, because only structured -# fields classify; and Recently Landed excludes only what closed while still -# held for the captain. +# fields classify; and Recently Landed applies the shared delivery selector to +# Done rows. test_collapsed_captain_call_deferral_and_landed() { local home fakebin json home=$(make_home collapsed-call) @@ -1665,6 +1676,238 @@ test_landed_includes_secondmate_home_merges() { pass "landed includes secondmate-managed merges alongside main-home merges" } +# Recently Landed accepts only the artifact owned by a row's task kind and +# completion path. Answered captain questions are the negative boundary. +test_landed_accepts_only_kind_owned_delivery_artifacts() { + local home fakebin json main_backlog report_path report_pr + local keyword_report shipping_report fleet_json created_kind failures='' + [ -n "$TASKS_AXI_BIN" ] || fail "tasks-axi is required for the landed-selector regression" + home=$(make_home kind-owned-landed) + write_fixture "$home" + fakebin=$(make_fakebin "$home") + main_backlog="$home/data/backlog.md" + + "$TASKS_AXI_BIN" add answered-question \ + "Decide whether https://github.com/o/r/pull/7 may merge" --kind ship \ + --repo firstmate --file "$main_backlog" >/dev/null \ + || fail "could not create the answered captain question" + run_captain "$home" "$fakebin" hold answered-question \ + --reason "captain route choice pending" >/dev/null \ + || fail "could not hold the answered captain question" + printf 'Choose route north.\n' > "$home/question-answer.txt" + run_captain "$home" "$fakebin" answer answered-question \ + --decision-file "$home/question-answer.txt" >/dev/null \ + || fail "could not close the answered captain question" + + run_captain "$home" "$fakebin" hold created-local-question \ + --title "Choose local main" --reason "captain local route pending" >/dev/null \ + || fail "could not create the local-only captain question" + created_kind=$("$TASKS_AXI_BIN" show created-local-question --full --file "$main_backlog" \ + | sed -n 's/^ kind: *//p' | head -1) + run_captain "$home" "$fakebin" answer created-local-question \ + --decision-file "$home/question-answer.txt" >/dev/null \ + || fail "could not close the local-only captain question" + + "$TASKS_AXI_BIN" add legacy-local-question "Choose local main" \ + --repo firstmate --file "$main_backlog" >/dev/null \ + || fail "could not create the legacy kindless captain question" + run_captain "$home" "$fakebin" hold legacy-local-question \ + --reason "captain legacy local route pending" >/dev/null \ + || fail "could not hold the legacy kindless captain question" + run_captain "$home" "$fakebin" answer legacy-local-question \ + --decision-file "$home/question-answer.txt" >/dev/null \ + || fail "could not close the legacy kindless captain question" + + keyword_report="data/keyword-scout/report.md" + shipping_report="data/shipping-scout/report.md" + mkdir -p "$home/data/keyword-scout" "$home/data/shipping-scout" + printf '# Keyword scout\n' > "$home/$keyword_report" + printf '# Shipping scout\n' > "$home/$shipping_report" + "$TASKS_AXI_BIN" add keyword-scout "SCOUT parser keywords" --kind scout \ + --repo firstmate --start --file "$main_backlog" >/dev/null \ + || fail "could not create the canonical keyword scout" + "$TASKS_AXI_BIN" 'done' keyword-scout --report "$keyword_report" \ + --file "$main_backlog" >/dev/null \ + || fail "could not complete the canonical keyword scout" + "$TASKS_AXI_BIN" add keyword-local "SHIP keyword local main" --kind ship \ + --repo firstmate --start --file "$main_backlog" >/dev/null \ + || fail "could not create the canonical keyword local delivery" + "$TASKS_AXI_BIN" 'done' keyword-local --note "local main" \ + --file "$main_backlog" >/dev/null \ + || fail "could not complete the canonical keyword local delivery" + "$TASKS_AXI_BIN" add noted-local "Land the local-only change" --kind ship \ + --repo firstmate --start --file "$main_backlog" >/dev/null \ + || fail "could not create the recorded-note local delivery" + "$TASKS_AXI_BIN" 'done' noted-local --note "local main" \ + --file "$main_backlog" >/dev/null \ + || fail "could not complete the recorded-note local delivery" + "$TASKS_AXI_BIN" add legacy-noted-local "Complete the legacy work" \ + --repo firstmate --start --file "$main_backlog" >/dev/null \ + || fail "could not create the kindless local delivery" + "$TASKS_AXI_BIN" 'done' legacy-noted-local --note "local main" \ + --file "$main_backlog" >/dev/null \ + || fail "could not complete the kindless local delivery" + "$TASKS_AXI_BIN" add shipping-scout "SHIPPING parser boundary" --kind scout \ + --repo firstmate --start --file "$main_backlog" >/dev/null \ + || fail "could not create the longer-word scout" + "$TASKS_AXI_BIN" 'done' shipping-scout --report "$shipping_report" \ + --file "$main_backlog" >/dev/null \ + || fail "could not complete the longer-word scout" + + report_path="data/reported-scout/report.md" + report_pr="https://github.com/o/r/pull/8" + mkdir -p "$home/data/reported-scout" + printf '# Reported scout\n' > "$home/$report_path" + cat >> "$main_backlog" <<EOF +- [x] done-pr-nondelivery - Closed without merge https://github.com/o/r/pull/5 (repo: firstmate) (kind: ship) (done 2026-07-12) +- [x] scout-pr-no-report - Scout without report https://github.com/o/r/pull/6 (repo: firstmate) (kind: scout) (merged 2026-07-12) +- [x] ship-reported-path - Ship naming data/ship-reported-path/report.md (repo: firstmate) (kind: ship) (reported 2026-07-12) +- [x] reported-scout - Report with $report_pr context $report_path (repo: firstmate) (kind: scout) (reported 2026-07-12) +- [x] local-delivery - Local with https://github.com/o/r/pull/9 context local main (repo: firstmate) (kind: ship) (done 2026-07-12) +EOF + + : > "$home/net.log" + fleet_json=$(PATH="$fakebin:$PATH" FM_HOME="$home" \ + FM_SNAPSHOT_NOW=2026-07-11T18:00:00Z NET_LOG="$home/net.log" \ + "$ROOT/bin/fm-fleet-snapshot.sh" --json) \ + || fail "Fleet snapshot failed for canonical kind keywords" + printf '%s' "$fleet_json" | jq -e ' + (.backlog.records | any(.id == "keyword-scout" and .kind == "scout")) + and (.backlog.records | any(.id == "keyword-local" and .kind == "ship")) + and (.backlog.records | any(.id == "shipping-scout" and .kind == "scout")) + ' >/dev/null || failures="${failures}canonical or explicit task kind was lost; " + json=$(run "$home" "$fakebin" --json --all-landed) \ + || fail "Bearings failed for captain-approved deliveries" + [ "$created_kind" = captain ] \ + || failures="${failures}wrapper-created call kind was ${created_kind:-absent}; " + printf '%s' "$json" | jq -e '.landed | any(.id == "answered-question") | not' >/dev/null \ + || failures="${failures}PR-naming captain answer was listed; " + printf '%s' "$json" | jq -e '.landed | any(.id == "created-local-question") | not' >/dev/null \ + || failures="${failures}wrapper-created local answer was listed; " + printf '%s' "$json" | jq -e '.landed | any(.id == "legacy-local-question") | not' >/dev/null \ + || failures="${failures}legacy kindless local answer was listed; " + # At the base commit the generic Done-row selector publishes this PR-naming + # row, so this exclusion assertion fails when the selector fix is absent. + printf '%s' "$json" | jq -e '.landed | any(.id == "done-pr-nondelivery") | not' >/dev/null \ + || failures="${failures}non-merged PR row was listed; " + # At the base commit a scout can fall through to its pull request, so this + # exclusion assertion fails when report ownership is not enforced. + printf '%s' "$json" | jq -e '.landed | any(.id == "scout-pr-no-report") | not' >/dev/null \ + || failures="${failures}reportless scout PR was listed; " + # At the base commit a ship can fall through to a report path, so this + # exclusion assertion fails when artifact-kind ownership is not enforced. + printf '%s' "$json" | jq -e '.landed | any(.id == "ship-reported-path") | not' >/dev/null \ + || failures="${failures}reported ship row was listed; " + # At the base commit pull-request links render ahead of report links, so this + # artifact assertion fails by receiving report_pr instead of report_path. + printf '%s' "$json" | jq -e --arg report_path "$report_path" \ + '.landed | any(.id == "reported-scout" and .artifact == $report_path)' >/dev/null \ + || failures="${failures}reported scout artifact was missing; " + printf '%s' "$json" | jq -e \ + '.landed | any(.id == "local-delivery" and .artifact == "local main")' >/dev/null \ + || failures="${failures}local-only artifact was missing; " + printf '%s' "$json" | jq -e --arg report "$keyword_report" \ + '.landed | any(.id == "keyword-scout" and .artifact == $report)' >/dev/null \ + || failures="${failures}canonical keyword scout was missing; " + printf '%s' "$json" | jq -e \ + '.landed | any(.id == "keyword-local" and .artifact == "local main")' >/dev/null \ + || failures="${failures}canonical keyword local delivery was missing; " + # tasks-axi records `done --note` as an indented body line, so this artifact + # assertion fails whenever the note is read from the row title alone. + printf '%s' "$json" | jq -e \ + '.landed | any(.id == "noted-local" and .artifact == "local main")' >/dev/null \ + || failures="${failures}recorded-note local delivery artifact was missing; " + # A kindless row records the same landing, so reading the note from the body + # must not cost it the section it reached while that note went unparsed. + printf '%s' "$json" | jq -e \ + '.landed | any(.id == "legacy-noted-local" and .artifact == "local main")' >/dev/null \ + || failures="${failures}kindless local delivery was missing; " + printf '%s' "$json" | jq -e --arg report "$shipping_report" \ + '.landed | any(.id == "shipping-scout" and .artifact == $report)' >/dev/null \ + || failures="${failures}longer-word explicit scout kind was lost; " + [ -z "$failures" ] || fail "$failures$json" + [ ! -s "$home/net.log" ] \ + || fail "kind-owned landed selection made a network call: $(cat "$home/net.log")" + pass "landed accepts only kind-owned delivery artifacts while answered questions stay out" +} + +test_kind_fallback_matches_tasks_axi_word_boundaries() { + local home fakebin id title kind producer_kind fleet_json json + [ -n "$TASKS_AXI_BIN" ] || fail "tasks-axi is required for the kind-boundary regression" + home=$(make_home kind-word-boundaries) + fakebin=$(make_fakebin "$home") + : > "$home/net.log" + : > "$home/expected.jsonl" + while IFS='|' read -r id title kind; do + "$TASKS_AXI_BIN" add "$id" "$title" --start --file "$home/data/backlog.md" >/dev/null \ + || fail "could not create keyword fixture $id" + "$TASKS_AXI_BIN" 'done' "$id" --report "data/$id/report.md" \ + --file "$home/data/backlog.md" >/dev/null || fail "could not complete keyword fixture $id" + producer_kind=$("$TASKS_AXI_BIN" show "$id" --full --file "$home/data/backlog.md" \ + | sed -n 's/^ kind: *//p' | head -1) + [ "$producer_kind" = "${kind/-/task}" ] || fail "tasks-axi kind differs for $title: $producer_kind" + jq -cn --arg id "$id" --arg kind "$kind" \ + '{id:$id,kind:(if $kind == "-" then null else $kind end)}' >> "$home/expected.jsonl" + done <<'EOF' +scout-colon|SCOUT: investigate regression|scout +scout-unicode|SCOUTé investigate|scout +scout-longer|SCOUTING investigate|- +scout-underscore|SCOUT_investigate|- +scout-digit|SCOUT7 investigate|- +scout-lowercase|scout: investigate|- +scout-nonleading|Investigate SCOUT: regression|- +ship-colon|SHIP: implement|ship +ship-unicode|SHIPé implement|ship +ship-longer|SHIPPING implement|- +EOF + fleet_json=$(PATH="$fakebin:$PATH" FM_HOME="$home" NET_LOG="$home/net.log" \ + "$ROOT/bin/fm-fleet-snapshot.sh" --json) || fail "keyword fleet snapshot failed" + printf '%s' "$fleet_json" | jq -e --slurpfile expected "$home/expected.jsonl" \ + '(.backlog.records | length) == ($expected | length)' >/dev/null \ + || fail "producer fixture rows were archived before comparison" + printf '%s' "$fleet_json" | jq -e --slurpfile expected "$home/expected.jsonl" ' + (.backlog.records | map({id,kind}) | sort_by(.id)) == ($expected | sort_by(.id)) + ' >/dev/null || fail "snapshot kinds differ from tasks-axi word boundaries: $fleet_json" + json=$(run "$home" "$fakebin" --json --all-landed) || fail "keyword bearings failed" + printf '%s' "$json" | jq -e --slurpfile expected "$home/expected.jsonl" ' + (.landed | map({id,artifact}) | sort_by(.id)) == + ($expected | map(select(.kind == "scout") | {id,artifact:("data/" + .id + "/report.md")}) | sort_by(.id)) + ' >/dev/null || fail "keyword scout deliveries differ from producer kinds: $json" + [ ! -s "$home/net.log" ] || fail "keyword projection made a network call" + pass "kind fallback matches tasks-axi word boundaries and preserves scout deliveries" +} + +test_landed_preserves_kindless_v1_summary_reports() { + local parent fakebin remote_home json freshness epoch=1100 + parent=$(make_home kindless-v1-reports) + make_remote_ledger_fleet "$parent" 1 + remote_home="$TMP_ROOT/remote-ledger-home-1" + fakebin=$(make_remote_ledger_ssh "$parent/remote-ssh") + jq ' + .landed = [ + {id:"legacy-report",title:"Scout report",report_path:"data/scout/report.md",completion:{verb:"reported",date:"2026-09-01"}}, + {id:"legacy-pr",title:"Merged change",report_path:"data/scout/report.md",pr_url:"https://github.com/o/r/pull/1",completion:{verb:"merged",date:"2026-09-01"}}, + {id:"legacy-local",title:"Local delivery",report_path:"data/scout/report.md",local_note:"local main",completion:{verb:"done",date:"2026-09-01"}} + ] | .counts.landed = (.landed | length) + ' "$remote_home/state/home-summary.json" > "$remote_home/state/legacy-summary.json" + mv "$remote_home/state/legacy-summary.json" "$remote_home/state/home-summary.json" + for freshness in fresh cached; do + json=$(run_remote_ledger_bearings "$parent" "$fakebin" "$epoch") \ + || fail "kindless v1 summary bearings failed" + printf '%s' "$json" | jq -e --arg freshness "$freshness" ' + (.secondmates | any(.id == "ledger-1" and .freshness == $freshness)) + and (.landed | map({id,artifact,owner}) | sort_by(.id)) == [ + {id:"legacy-local",artifact:"local main",owner:"ledger-1"}, + {id:"legacy-pr",artifact:"https://github.com/o/r/pull/1",owner:"ledger-1"}, + {id:"legacy-report",artifact:"data/scout/report.md",owner:"ledger-1"} + ] + ' >/dev/null || fail "kindless v1 $freshness artifacts were lost: $json" + rm -f "$remote_home/state/home-summary.json" + epoch=1200 + done + pass "kindless v1 summaries retain report artifacts from fresh and cached ledgers" +} + test_landed_default_balances_dominant_and_sparse_homes() { local home dominant sparse_a sparse_b sparse_c fakebin json i actual expected home=$(make_home landed-balanced-default) @@ -2881,7 +3124,7 @@ SH } test_remote_ledgers_share_one_concurrent_budget_and_fall_back_to_cache() { - local parent fakebin json i remote_home pid collector_pid sleeper_pid duplicate_base cache_file candidate tmp + local parent fakebin json i remote_home pid collector_pid sleeper_pid duplicate_base cache_file candidate tmp approved_pr parent=$(make_home concurrent-remote-ledgers) make_remote_ledger_fleet "$parent" 5 fakebin=$(make_remote_ledger_ssh "$parent/remote-ssh") @@ -2909,6 +3152,17 @@ test_remote_ledgers_share_one_concurrent_budget_and_fall_back_to_cache() { fi done [ -n "$cache_file" ] || fail "healthy remote read did not populate its summary cache" + approved_pr="https://github.com/acme/remote/pull/1368" + mkdir -p "$remote_home/data" + cat > "$remote_home/data/backlog.md" <<EOF +## In flight + +## Queued + +## Done +- [x] remote-approved - Captain-approved delivery $approved_pr (repo: firstmate) (kind: ship) (hold-kind: captain) (merged 2026-09-03) +EOF + write_remote_home_summary "$remote_home" 1000 tmp="$cache_file.tmp" jq 'del(.hold_classifier_schema)' \ "$cache_file" > "$tmp" && mv "$tmp" "$cache_file" @@ -2977,7 +3231,7 @@ test_remote_ledgers_share_one_concurrent_budget_and_fall_back_to_cache() { rm -f "$parent/ledger-active/overlap-proved" "$parent/ledger-active"/collector-* json=$(run_remote_ledger_bearings "$parent" "$fakebin" 2000) [ -f "$parent/ledger-active/overlap-proved" ] \ - || fail "five wedged remote reads never overlapped within the shared three-second budget" + || fail "five wedged remote reads never overlapped within the shared fifteen-second budget" printf '%s' "$json" | jq -e ' (.secondmates | length) == 5 and all(.secondmates[]; .freshness == "cached" and .age_seconds == 1000 @@ -3103,6 +3357,9 @@ test_current_landed_baseline_is_repeatable_and_prior_report_independent test_default_is_bounded_and_local_only test_toon_json_parity test_landed_includes_secondmate_home_merges +test_landed_accepts_only_kind_owned_delivery_artifacts +test_kind_fallback_matches_tasks_axi_word_boundaries +test_landed_preserves_kindless_v1_summary_reports test_landed_default_balances_dominant_and_sparse_homes test_landed_default_refills_capacity_after_sparse_homes_exhaust test_landed_default_uses_deterministic_home_order_when_homes_exceed_cap diff --git a/tests/fm-bootstrap.test.sh b/tests/fm-bootstrap.test.sh index 77bea60fd4c..c5d8a16d413 100755 --- a/tests/fm-bootstrap.test.sh +++ b/tests/fm-bootstrap.test.sh @@ -2003,6 +2003,12 @@ gemini array profile is accepted^{"default":[{"harness":"gemini"},{"harness":"co unsupported codex max effort is flagged^{"rules":[{"when":"big feature","use":{"harness":"codex","model":"gpt-5","effort":"max"}}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: codex:max unsupported grok max effort is flagged^{"rules":[{"when":"deep current work","use":{"harness":"grok","model":"grok-4","effort":"max"}}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: grok:max unsupported grok xhigh effort is flagged^{"rules":[{"when":"deep current work","use":{"harness":"grok","model":"grok-4","effort":"xhigh"}}]}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: grok:xhigh +native pi ultra is accepted^{"rules":[],"default":{"harness":"pi","model":"codex-native/gpt-6-astra","effort":"ultra"}}^empty^ +native signed pi ultra is accepted^{"rules":[{"when":"native reasoning","use":{"harness":"pi-signed","model":"codex-native/gpt-6-astra","effort":"ultra"}}]}^empty^ +ordinary pi ultra is refused^{"default":{"harness":"pi","model":"openai-codex/gpt-6-astra","effort":"ultra"}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: pi:ultra +missing native model ultra is refused^{"default":{"harness":"pi","effort":"ultra"}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: pi:ultra +empty native model ultra is refused^{"default":{"harness":"pi","model":"codex-native/","effort":"ultra"}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: pi:ultra +codex harness ultra is refused^{"default":{"harness":"codex","model":"codex-native/gpt-6-astra","effort":"ultra"}}^exact^CREW_DISPATCH: invalid config/crew-dispatch.json - invalid effort: codex:ultra pi max effort is accepted^{"rules":[{"when":"deep coding","use":{"harness":"pi","model":"openai-codex/gpt-5.6-sol","effort":"max"}}]}^empty^ pi-signed max effort is accepted^{"rules":[{"when":"signed coding","use":{"harness":"pi-signed","model":"openai-codex/gpt-5.6-sol","effort":"max"}}]}^empty^ muse shared efforts are accepted^{"rules":[{"when":"muse low","use":{"harness":"muse","effort":"low"}},{"when":"muse medium","use":{"harness":"muse","effort":"medium"}},{"when":"muse high","use":{"harness":"muse","effort":"high"}},{"when":"muse xhigh","use":{"harness":"muse","effort":"xhigh"}},{"when":"muse max","use":{"harness":"muse","effort":"max"}}]}^empty^ diff --git a/tests/fm-busy-adapter-wiring.test.sh b/tests/fm-busy-adapter-wiring.test.sh index bbd1af21bc5..c8967b4256d 100755 --- a/tests/fm-busy-adapter-wiring.test.sh +++ b/tests/fm-busy-adapter-wiring.test.sh @@ -59,7 +59,7 @@ drive_pi_ext() { import { pathToFileURL } from "node:url"; const mod = await import(pathToFileURL(process.env.EXT_PATH).href); const handlers = {}; -mod.default({ on: (name, fn) => { handlers[name] = fn; } }); +mod.default({ on: (name, fn) => { handlers[name] = fn; }, events: { on: (name, fn) => { handlers[name] = fn; } } }); const ctx = { isIdle: () => process.env.MODE !== "settle-continuing" }; switch (process.env.MODE) { case "agent-start": await handlers["agent_start"]({}, ctx); break; @@ -70,9 +70,10 @@ switch (process.env.MODE) { await handlers["agent_start"]({}, ctx); break; case "turn-end": await handlers["turn_end"]({}, ctx); break; + case "progress": await handlers["codex-native:progress"]({ type: "commandExecution", phase: "completed" }); break; default: throw new Error("unknown mode " + process.env.MODE); } -if (process.env.MODE === "turn-end") { +if (["turn-end", "progress"].includes(process.env.MODE)) { await new Promise((resolve) => setTimeout(resolve, 200)); } EOF @@ -92,6 +93,11 @@ test_pi_extension_semantic_lifecycle() { [ "$out" = "busy fm-spawn" ] || fail "seed after spawn must be 'busy fm-spawn', got '$out'" rm -f "$state/$id.turn-ended" + out=$(drive_pi_ext "$ext" progress) || fail "native progress drive failed: $out" + [ -f "$state/$id.progress" ] || fail "native progress did not write its separate marker" + [ ! -e "$state/$id.turn-ended" ] || fail "native progress fabricated a completed turn" + out=$(classify pi "$id" "$state") + [ "$out" = "busy fm-spawn" ] || fail "native progress changed semantic state: $out" out=$(drive_pi_ext "$ext" turn-end) || fail "turn_end drive failed: $out" [ -f "$state/$id.turn-ended" ] || fail "turn_end no longer touches the notification marker" out=$(classify pi "$id" "$state") @@ -144,6 +150,8 @@ test_pi_extension_stale_incarnation_rejected() { out=$(drive_pi_ext "$ext" settle-idle) || fail "stale settle drive failed: $out" out=$(classify pi "$id" "$state") [ "$out" = "busy fm-spawn" ] || fail "a stale extension event must not change state, got '$out'" + out=$(drive_pi_ext "$ext" progress) || fail "stale progress drive failed: $out" + [ ! -e "$state/$id.progress" ] || fail "stale native progress refreshed the new incarnation" pass "pi extension events from a superseded incarnation are rejected as stale" } diff --git a/tests/fm-busy-state.test.sh b/tests/fm-busy-state.test.sh index e7e823f76b9..eec684b0f20 100755 --- a/tests/fm-busy-state.test.sh +++ b/tests/fm-busy-state.test.sh @@ -484,6 +484,27 @@ test_boolean_view_never_promotes_unknown() { pass "the boolean view reports busy only on an exact busy verdict" } +test_progress_is_generation_bound_and_not_semantic_state() { + local state gen replacement before + state=$(new_state_dir native-progress) + gen=$("$EV" arm "$state" t1) + before=$(cat "$state/t1.busy-state") + "$EV" progress "$state" t1 --gen "$gen" || fail "current progress was refused" + [ -f "$state/t1.progress" ] || fail "progress marker missing" + [ ! -e "$state/t1.turn-ended" ] || fail "progress emitted a completed turn" + [ "$(cat "$state/t1.busy-state")" = "$before" ] || fail "progress changed semantic state" + replacement=$("$EV" arm "$state" t1) + [ ! -e "$state/t1.progress" ] || fail "arm retained the previous incarnation's progress" + if "$EV" progress "$state" t1 --gen "$gen" 2>/dev/null; then fail "stale progress was accepted"; fi + [ ! -e "$state/t1.progress" ] || fail "stale progress wrote a marker" + "$EV" progress "$state" t1 --gen "$replacement" || fail "replacement progress was refused" + "$EV" retire "$state" t1 --gen "$replacement" || fail "retire failed" + [ ! -e "$state/t1.progress" ] || fail "retire retained progress" + pass "native progress is generation-bound, separately recorded, and cleared on arm and retire" +} + +test_progress_is_generation_bound_and_not_semantic_state + test_arm_seeds_busy_spawn test_apply_advances_seq_and_source test_apply_current_gen_reset diff --git a/tests/fm-calm-pi-extension.test.sh b/tests/fm-calm-pi-extension.test.sh index 433c8ae1b13..73ce85d6906 100755 --- a/tests/fm-calm-pi-extension.test.sh +++ b/tests/fm-calm-pi-extension.test.sh @@ -759,6 +759,7 @@ test_rendering_and_session_lifecycle() { cp "$ROOT/.pi/extensions/lib/fm-pi-loaded-marker.ts" "$fixture/lib/fm-pi-loaded-marker.ts" cp "$ROOT/.pi/extensions/lib/fm-pi-prompt-delivery.ts" "$fixture/lib/fm-pi-prompt-delivery.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$fixture/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$fixture/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$fixture/lib/fm-async-exec.ts" cp "$WATCH_EXT" "$fixture/fm-primary-pi-watch.ts" ln -s "$PI_PACKAGE_DIR" "$fixture/node_modules/@earendil-works/pi-coding-agent" @@ -3465,6 +3466,7 @@ test_interactive_terminal_e2e() { cp "$ROOT/.pi/extensions/lib/fm-pi-loaded-marker.ts" "$project/.pi/extensions/lib/fm-pi-loaded-marker.ts" cp "$ROOT/.pi/extensions/lib/fm-pi-prompt-delivery.ts" "$project/.pi/extensions/lib/fm-pi-prompt-delivery.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$project/.pi/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$project/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$project/.pi/extensions/lib/fm-async-exec.ts" cp "$WATCH_EXT" "$project/.pi/extensions/fm-primary-pi-watch.ts" cp "$ROOT/.pi/extensions/fm-primary-turnend-guard.ts" "$project/.pi/extensions/fm-primary-turnend-guard.ts" diff --git a/tests/fm-captain-hold-lifecycle.test.sh b/tests/fm-captain-hold-lifecycle.test.sh index fc82b04b1eb..7d8f1dd8b74 100755 --- a/tests/fm-captain-hold-lifecycle.test.sh +++ b/tests/fm-captain-hold-lifecycle.test.sh @@ -8,6 +8,8 @@ set -u # shellcheck source=tests/lib.sh # shellcheck disable=SC1091 . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=bin/fm-timeout-lib.sh +. "$ROOT/bin/fm-timeout-lib.sh" TEARDOWN="$ROOT/bin/fm-teardown.sh" BEARINGS="$ROOT/bin/fm-bearings-snapshot.sh" @@ -45,10 +47,11 @@ run_lavish() { # <home> <command args...> "$ROOT/bin/fm-procevent-lavish.sh" "$@" } -run_bearings() { # <home> +run_bearings() { # <home> [extra args] local home=$1 + shift PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_BEARINGS_NOW=2026-07-14T12:00:00Z \ - "$BEARINGS" --json + "$BEARINGS" --json "$@" } run_teardown() { # <home> <id> @@ -81,6 +84,80 @@ request_reconciles() { # <home> <source-id> <task-id>... --source "captured board result" >/dev/null } +configure_merged_github() { # <home> + local home=$1 + cat > "$home/fakebin/gh" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$FM_TEST_GH_LOG" +case "${1:-} ${2:-}" in + "pr view") printf '%s\n' 1111111111111111111111111111111111111111 ;; + "api graphql") + printf '%s\n' 'state=MERGED' 'merged=true' 'queued=false' 'base=main' + ;; +esac +SH + cat > "$home/fakebin/gh-axi" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$FM_TEST_GH_AXI_LOG" +case "${1:-} ${2:-}" in + "pr merge") printf 'merged:\n number: %s\n status: ok\n' "${3:-}" ;; + "pr view") printf 'pull_request:\n number: %s\n state: merged\n' "${3:-}" ;; +esac +SH + chmod +x "$home/fakebin/gh" "$home/fakebin/gh-axi" + : > "$home/gh.log" + : > "$home/gh-axi.log" +} + +run_pr_merge() { # <home> <id> <url> + local home=$1 + shift + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" FM_TEST_GH_LOG="$home/gh.log" \ + FM_TEST_GH_AXI_LOG="$home/gh-axi.log" "$ROOT/bin/fm-pr-merge.sh" "$@" +} + +wait_for_test_file() { # <path> <pid> + local path=$1 pid=$2 i=0 + while [ "$i" -lt 500 ]; do + [ -e "$path" ] && return 0 + kill -0 "$pid" 2>/dev/null || return 1 + sleep 0.01 + i=$((i + 1)) + done + return 1 +} + +install_reused_task_barriers() { # <home> + local home=$1 + cat > "$home/fakebin/perl" <<'SH' +#!/usr/bin/env bash +if [ "${FM_TEST_REUSE_TEARDOWN:-}" = 1 ] \ + && [ ! -e "${FM_TEST_REUSE_TEARDOWN_ONCE:-}" ]; then + : > "$FM_TEST_REUSE_TEARDOWN_ONCE" + : > "$FM_TEST_REUSE_TEARDOWN_READY" + while [ ! -e "$FM_TEST_REUSE_TEARDOWN_RELEASE" ]; do + "$FM_TEST_REAL_SLEEP" 0.01 + done +fi +exec "$FM_TEST_REAL_PERL" "$@" +SH + cat > "$home/fakebin/sleep" <<'SH' +#!/usr/bin/env bash +if [ "${FM_TEST_REUSE_MERGE:-}" = 1 ] && [ "${1:-}" = 0.1 ] \ + && [ ! -e "${FM_TEST_REUSE_MERGE_ONCE:-}" ]; then + : > "$FM_TEST_REUSE_MERGE_ONCE" + : > "$FM_TEST_REUSE_MERGE_READY" + while [ ! -e "$FM_TEST_REUSE_MERGE_RELEASE" ]; do + "$FM_TEST_REAL_SLEEP" 0.01 + done +fi +exec "$FM_TEST_REAL_SLEEP" "$@" +SH + chmod +x "$home/fakebin/perl" "$home/fakebin/sleep" +} + # The retired command surface, kept for one release as a shim; in-flight # pre-collapse work still drives the lifecycle through these spellings. run_shim() { # <home> <command args...> @@ -2457,6 +2534,258 @@ test_teardown_never_closes_a_captain_held_task() { pass "cleanup leaves a captain-held work item open with its deliverable, and only an answer closes it" } +test_retained_row_artifacts_survive_captain_answers() { + local home retained_id precedence_id rejected_id rejected_local_id report_question_id + local approved_id released_id local_id answered_id reportless_scout_id legacy_id + local repo wt + local local_repo local_wt + local precedence_pr rejected_pr approved_pr released_pr json show + home=$(make_home retained-row-artifacts) + perl -0pi -e 's/done_keep = 10/done_keep = 20/' "$home/.tasks.toml" + retained_id=sample-retained-report + mkdir -p "$home/data/$retained_id" + tasks_in "$home" add "$retained_id" "Investigate retained report evidence" --kind scout \ + --repo sample --start >/dev/null || fail "could not create the retained report fixture" + write_origin_meta "$home" "$retained_id" + printf 'done: report complete\n' > "$home/state/$retained_id.status" + printf '# Retained report\n\nThe captain must choose the follow-up.\n' \ + > "$home/data/$retained_id/report.md" + run_captain "$home" hold "$retained_id" --reason "captain must choose the report follow-up" \ + >/dev/null || fail "could not hold the retained report" + run_captain "$home" complete "$retained_id" "$retained_id" >/dev/null \ + || fail "completion gate failed for the retained report" + run_teardown "$home" "$retained_id" > "$home/retained-teardown.out" \ + 2> "$home/report-teardown.err" \ + || fail "retained report cleanup failed: $(cat "$home/report-teardown.err")" + printf 'Proceed with the report follow-up.\n' > "$home/report-answer.txt" + run_captain "$home" answer "$retained_id" --decision-file "$home/report-answer.txt" >/dev/null \ + || fail "could not answer the retained report call" + + precedence_id=sample-retained-report-pr-title + precedence_pr=https://github.com/sample/sample/pull/21 + mkdir -p "$home/data/$precedence_id" + tasks_in "$home" add "$precedence_id" "Investigate $precedence_pr regression" --kind scout \ + --repo sample --start >/dev/null || fail "could not create the report precedence fixture" + write_origin_meta "$home" "$precedence_id" + printf 'done: report complete\n' > "$home/state/$precedence_id.status" + printf '# Retained report with pull request context\n' \ + > "$home/data/$precedence_id/report.md" + run_captain "$home" hold "$precedence_id" \ + --reason "captain must choose the report follow-up" >/dev/null \ + || fail "could not hold the report precedence fixture" + run_captain "$home" complete "$precedence_id" "$precedence_id" >/dev/null \ + || fail "completion gate failed for the report precedence fixture" + run_teardown "$home" "$precedence_id" > "$home/precedence-teardown.out" \ + 2> "$home/precedence-teardown.err" \ + || fail "report precedence cleanup failed: $(cat "$home/precedence-teardown.err")" + printf 'Proceed with the pull-request regression report.\n' \ + > "$home/precedence-answer.txt" + run_captain "$home" answer "$precedence_id" \ + --decision-file "$home/precedence-answer.txt" >/dev/null \ + || fail "could not answer the report precedence call" + show=$(tasks_in "$home" show "$precedence_id" --full) \ + || fail "the report precedence row disappeared" + assert_contains "$show" "state: done" "the report precedence answer did not close its call" + assert_contains "$show" "hold_kind: captain" \ + "the report precedence row lost its retained-scout evidence" + + rejected_id=sample-rejected-merge + rejected_pr="https://github.com/sample/sample/pull/22" + tasks_in "$home" add "$rejected_id" "Decide whether $rejected_pr may merge" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the rejected merge fixture" + run_captain "$home" hold "$rejected_id" --reason "captain merge approval pending" \ + >/dev/null || fail "could not hold the rejected merge" + printf 'Do not merge this pull request.\n' > "$home/rejected-answer.txt" + run_captain "$home" answer "$rejected_id" --decision-file "$home/rejected-answer.txt" \ + >/dev/null || fail "could not record the rejected merge" + show=$(tasks_in "$home" show "$rejected_id" --full) || fail "the rejected merge disappeared" + assert_contains "$show" "state: done" "the rejected merge answer did not close its call" + assert_contains "$show" "hold_kind: captain" "the rejected merge lost its non-release evidence" + + rejected_local_id=sample-rejected-local + tasks_in "$home" add "$rejected_local_id" "Decide whether to land local main" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the rejected local fixture" + run_captain "$home" hold "$rejected_local_id" --reason "captain local merge approval pending" \ + >/dev/null || fail "could not hold the rejected local merge" + printf 'Do not land this change locally.\n' > "$home/rejected-local-answer.txt" + run_captain "$home" answer "$rejected_local_id" \ + --decision-file "$home/rejected-local-answer.txt" >/dev/null \ + || fail "could not record the rejected local merge" + show=$(tasks_in "$home" show "$rejected_local_id" --full) \ + || fail "the rejected local merge disappeared" + assert_contains "$show" "state: done" "the rejected local answer did not close its call" + assert_contains "$show" "hold_kind: captain" \ + "the rejected local merge lost its non-release evidence" + + report_question_id=sample-report-path-question + tasks_in "$home" add "$report_question_id" \ + "Decide whether data/$report_question_id/report.md should be published" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the report-path question fixture" + run_captain "$home" hold "$report_question_id" \ + --reason "captain report publication decision pending" >/dev/null \ + || fail "could not hold the report-path question" + printf 'Do not publish this report.\n' > "$home/report-question-answer.txt" + run_captain "$home" answer "$report_question_id" \ + --decision-file "$home/report-question-answer.txt" >/dev/null \ + || fail "could not record the report-path answer" + show=$(tasks_in "$home" show "$report_question_id" --full) \ + || fail "the report-path question disappeared" + assert_contains "$show" "state: done" "the report-path answer did not close its call" + assert_contains "$show" "hold_kind: captain" \ + "the report-path question lost its non-release evidence" + + approved_id=sample-approved-merge + approved_pr="https://github.com/sample/sample/pull/23" + repo="$home/projects/sample-approved" + wt="$home/projects/$approved_id" + fm_git_worktree "$repo" "$wt" fm/approved-merge + tasks_in "$home" add "$approved_id" "Ship the approved pull request $approved_pr" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the approved merge fixture" + fm_write_meta "$home/state/$approved_id.meta" \ + "window=firstmate:fm-$approved_id" "endpoint_task_id=$approved_id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "pr=$approved_pr" "spawn_gen=fixture-$approved_id" + printf 'done: PR %s merged\n' "$approved_pr" > "$home/state/$approved_id.status" + run_captain "$home" hold "$approved_id" --reason "captain merge approval pending" \ + >/dev/null || fail "could not hold the approved merge" + printf 'Merge the approved pull request.\n' > "$home/approved-answer.txt" + run_captain "$home" answer "$approved_id" --release \ + --decision-file "$home/approved-answer.txt" >/dev/null \ + || fail "could not release the approved merge" + show=$(tasks_in "$home" show "$approved_id" --full) || fail "the approved merge disappeared" + assert_not_contains "$show" "hold_kind: captain" "merge approval retained its captain hold kind" + run_teardown "$home" "$approved_id" > "$home/approved-teardown.out" \ + 2> "$home/approved-teardown.err" \ + || fail "approved merge cleanup failed: $(cat "$home/approved-teardown.err")" + + local_id=sample-released-local + local_repo="$home/projects/sample-local" + local_wt="$home/projects/$local_id" + fm_git_worktree "$local_repo" "$local_wt" "fm/$local_id" + printf 'landed locally\n' > "$local_wt/local.txt" + git -C "$local_wt" add local.txt + git -C "$local_wt" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm 'local delivery' + tasks_in "$home" add "$local_id" "Land the approved local-only change" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the released local fixture" + fm_write_meta "$home/state/$local_id.meta" \ + "window=firstmate:fm-$local_id" "endpoint_task_id=$local_id" "worktree=$local_wt" \ + "project=$local_repo" "harness=codex" "kind=ship" "mode=local-only" \ + "spawn_gen=fixture-$local_id" + printf 'done: local merge ready\n' > "$home/state/$local_id.status" + run_captain "$home" hold "$local_id" --reason "captain local merge approval pending" \ + >/dev/null || fail "could not hold the released local merge" + printf 'Land the approved change locally.\n' > "$home/local-answer.txt" + run_captain "$home" answer "$local_id" --release \ + --decision-file "$home/local-answer.txt" >/dev/null \ + || fail "could not release the local merge" + show=$(tasks_in "$home" show "$local_id" --full) || fail "the released local merge disappeared" + assert_not_contains "$show" "hold_kind: captain" \ + "local merge approval retained its captain hold kind" + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" "$ROOT/bin/fm-merge-local.sh" "$local_id" \ + > "$home/local-merge.out" 2> "$home/local-merge.err" \ + || fail "approved local merge failed: $(cat "$home/local-merge.err")" + run_teardown "$home" "$local_id" > "$home/local-teardown.out" \ + 2> "$home/local-teardown.err" \ + || fail "released local cleanup failed: $(cat "$home/local-teardown.err")" + + released_id=sample-released-report + released_pr=https://github.com/sample/sample/pull/24 + mkdir -p "$home/data/$released_id" + tasks_in "$home" add "$released_id" "Investigate $released_pr report evidence" \ + --kind scout --repo sample --start >/dev/null \ + || fail "could not create the released report fixture" + write_origin_meta "$home" "$released_id" + printf 'done: report complete\n' > "$home/state/$released_id.status" + printf '# Released report\n' > "$home/data/$released_id/report.md" + run_captain "$home" hold "$released_id" --reason "captain report release pending" \ + >/dev/null || fail "could not hold the released report" + run_captain "$home" complete "$released_id" "$released_id" >/dev/null \ + || fail "completion gate failed for the released report" + printf 'Release the completed report.\n' > "$home/released-answer.txt" + run_captain "$home" answer "$released_id" --release \ + --decision-file "$home/released-answer.txt" >/dev/null \ + || fail "could not release the completed report" + run_teardown "$home" "$released_id" > "$home/released-teardown.out" \ + 2> "$home/released-teardown.err" \ + || fail "released report cleanup failed: $(cat "$home/released-teardown.err")" + + answered_id=sample-artifactless-captain-answer + tasks_in "$home" add "$answered_id" "Choose the artifactless route" --kind captain \ + --repo sample --start >/dev/null || fail "could not create the artifactless captain call" + run_captain "$home" hold "$answered_id" --reason "captain route choice pending" \ + >/dev/null || fail "could not hold the artifactless captain call" + printf 'Continue without a delivery artifact.\n' > "$home/artifactless-answer.txt" + run_captain "$home" answer "$answered_id" --release \ + --decision-file "$home/artifactless-answer.txt" >/dev/null \ + || fail "could not release the artifactless captain call" + tasks_in "$home" 'done' "$answered_id" >/dev/null \ + || fail "could not complete the released artifactless captain call" + + decision_local_id=sample-released-local-worded-decision + tasks_in "$home" add "$decision_local_id" "Land the reviewed change" --kind ship \ + --repo sample --start >/dev/null \ + || fail "could not create the local-worded decision fixture" + run_captain "$home" hold "$decision_local_id" --reason "captain landing route pending" \ + >/dev/null || fail "could not hold the local-worded decision fixture" + printf 'local main\n' > "$home/local-worded-answer.txt" + run_captain "$home" answer "$decision_local_id" --release \ + --decision-file "$home/local-worded-answer.txt" >/dev/null \ + || fail "could not release the local-worded decision fixture" + tasks_in "$home" 'done' "$decision_local_id" >/dev/null \ + || fail "could not complete the local-worded decision fixture" + + reportless_scout_id=sample-reportless-scout + tasks_in "$home" add "$reportless_scout_id" "Investigate without a report" --kind scout \ + --repo sample --start >/dev/null || fail "could not create the reportless scout" + tasks_in "$home" 'done' "$reportless_scout_id" >/dev/null \ + || fail "could not complete the reportless scout" + + legacy_id=sample-kindless-legacy-delivery + tasks_in "$home" add "$legacy_id" "Complete the legacy work" --repo sample --start \ + >/dev/null || fail "could not create the kindless legacy delivery" + tasks_in "$home" 'done' "$legacy_id" >/dev/null \ + || fail "could not complete the kindless legacy delivery" + + json=$(run_bearings "$home" --all-landed) \ + || fail "Bearings failed after retained delivery answers" + printf '%s' "$json" | jq -e \ + --arg retained_id "$retained_id" --arg retained "data/$retained_id/report.md" \ + --arg precedence_id "$precedence_id" \ + --arg precedence "data/$precedence_id/report.md" \ + --arg rejected_id "$rejected_id" --arg rejected_local_id "$rejected_local_id" \ + --arg report_question_id "$report_question_id" \ + --arg approved_id "$approved_id" --arg local_id "$local_id" \ + --arg approved_pr "$approved_pr" --arg released_id "$released_id" \ + --arg answered_id "$answered_id" --arg reportless_scout_id "$reportless_scout_id" \ + --arg legacy_id "$legacy_id" --arg decision_local_id "$decision_local_id" \ + --arg released "data/$released_id/report.md" ' + (.landed | any(.id == $retained_id and .artifact == $retained)) + and (.landed | any(.id == $precedence_id and .artifact == $precedence)) + and (.landed | any(.id == $rejected_id) | not) + and (.landed | any(.id == $rejected_local_id) | not) + and (.landed | any(.id == $report_question_id) | not) + and (.landed | any(.id == $approved_id and .artifact == $approved_pr)) + and (.landed | any(.id == $local_id)) + and (.landed | any(.id == $released_id and .artifact == $released)) + # Without the explicit captain-kind boundary, the artifactless answered + # call falls through the compatibility path and this assertion fails. + and (.landed | any(.id == $answered_id) | not) + # Without the explicit scout-kind boundary, this row is rendered with + # an empty artifact even though a scout has no delivery without a report. + and (.landed | any(.id == $reportless_scout_id) | not) + # Requiring a present non-captain kind would also remove this older + # artifactless delivery, so the compatibility boundary stays observable. + and (.landed | any(.id == $legacy_id)) + # The captain worded this decision "local main" and the work closed + # with no artifact, so reading that prose as a recorded note would + # publish a local-only landing that never happened. + and (.landed | any(.id == $decision_local_id and .artifact == "-")) + ' >/dev/null || fail "released, retained, or rejected deliveries were misclassified: $json" + pass "release and scout report retention distinguish deliveries from rejected merge answers" +} + # Retention happens after destructive cleanup, through the same pending record # an ordinary close stages first. A cleanup that fails part-way therefore leaves # the row exactly as it was, and the next session start finishes the retention @@ -2521,6 +2850,185 @@ SH pass "an interrupted cleanup keeps the captain call recoverable and session start retains it" } +test_answer_before_cleanup_replay_preserves_the_retained_report() { + local home id wt rc bootstrap json + home=$(make_home answer-before-cleanup-replay) + id=sample-answer-before-cleanup-replay + wt="$home/projects/$id" + mkdir -p "$home/data/$id" "$wt" "$home/projects/sample" + tasks_in "$home" add "$id" "Investigate answer before cleanup replay" --kind scout \ + --repo sample --start >/dev/null || fail "could not create the answer-before-replay fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "worktree=$wt" "project=$home/projects/sample" \ + "harness=codex" "kind=scout" "mode=scout" "spawn_gen=fixture-$id" + printf 'done: report complete\n' > "$home/state/$id.status" + printf '# Interrupted cleanup\n\nThe captain call remains open.\n' > "$home/data/$id/report.md" + run_captain "$home" hold "$id" --reason "captain must choose after interrupted cleanup" \ + >/dev/null || fail "could not hold the answer-before-replay fixture" + run_captain "$home" complete "$id" "$id" >/dev/null \ + || fail "completion gate failed for the answer-before-replay fixture" + cat > "$home/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +exit 1 +SH + chmod +x "$home/fakebin/treehouse" + + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" --force \ + > "$home/teardown.out" 2> "$home/teardown.err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "cleanup succeeded despite the failed worktree return" + assert_present "$home/state/$id.backlog-close" \ + "the interrupted cleanup lost its retained-artifact record" + + printf 'Proceed with the reported result.\n' > "$home/answer.txt" + run_captain "$home" answer "$id" --decision-file "$home/answer.txt" >/dev/null \ + || fail "the captain could not answer before cleanup replay" + fm_fake_exit0 "$home/fakebin" treehouse + bootstrap=$(PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" FM_BOOTSTRAP_NETWORK=skip \ + "$ROOT/bin/fm-bootstrap.sh" 2>&1) \ + || fail "session start could not replay cleanup after the answer: $bootstrap" + assert_absent "$home/state/$id.meta" "session start left the interrupted task record behind" + assert_absent "$home/state/$id.backlog-close" "session start left the pending record behind" + json=$(run_bearings "$home") || fail "Bearings failed after the answer-before-replay lifecycle" + printf '%s' "$json" | jq -e \ + --arg id "$id" --arg report "data/$id/report.md" \ + '.landed | any(.id == $id and .artifact == $report)' >/dev/null \ + || fail "the retained report disappeared when the captain answered before replay: $json" + pass "an answer before cleanup replay preserves the retained report" +} + +test_unusable_pending_close_record_names_its_reason() { + local home id wt rc err marker + home=$(make_home unusable-pending-close-reason) + id=sample-unusable-pending-close + wt="$home/projects/$id" + marker="$home/state/$id.backlog-close" + mkdir -p "$home/data/$id" "$wt" "$home/projects/sample" "$home/elsewhere" + tasks_in "$home" add "$id" "Investigate the unusable pending close" --kind scout \ + --repo sample --start >/dev/null || fail "could not create the unusable pending-close fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "worktree=$wt" "project=$home/projects/sample" \ + "harness=codex" "kind=scout" "mode=scout" "spawn_gen=fixture-$id" + printf 'done: report complete\n' > "$home/state/$id.status" + printf '# Unusable pending close\n\nThe captain call remains open.\n' > "$home/data/$id/report.md" + run_captain "$home" hold "$id" --reason "captain must choose after interrupted cleanup" \ + >/dev/null || fail "could not hold the unusable pending-close fixture" + run_captain "$home" complete "$id" "$id" >/dev/null \ + || fail "completion gate failed for the unusable pending-close fixture" + cat > "$home/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +exit 1 +SH + chmod +x "$home/fakebin/treehouse" + + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" --force \ + > "$home/teardown.out" 2> "$home/teardown.err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "cleanup succeeded despite the failed worktree return" + assert_present "$marker" "the interrupted cleanup lost its retained-artifact record" + sed "s|^data=.*$|data=$home/elsewhere|" "$marker" > "$marker.rewritten" \ + || fail "could not rewrite the pending-close record" + mv "$marker.rewritten" "$marker" + + printf 'Proceed with the reported result.\n' > "$home/answer.txt" + set +e + err=$(run_captain "$home" answer "$id" --decision-file "$home/answer.txt" 2>&1 >/dev/null) + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "the unusable pending-close record was answered as if it were valid" + assert_contains "$err" "$marker" \ + "the refusal did not name the pending-close record the captain must repair" + assert_contains "$err" "foreign data directory" \ + "the refusal did not name why the pending-close record could not be used" + pass "an unusable pending-close record names its reason instead of a bare refusal" +} + +test_relocated_report_does_not_wedge_an_answer_before_replay() { + local home data id wt rc show bootstrap json + home=$(make_home relocated-answer-before-replay) + data="$home/données" + mv "$home/data" "$data" + id=sample-relocated-answer-before-replay + wt="$home/projects/$id" + mkdir -p "$home/data" "$data/$id" "$wt" "$home/projects/sample" + cat > "$home/data/backlog.md" <<'EOF' +## In flight + +## Queued + +## Done +EOF + (cd "$home" && tasks-axi add "$id" "Investigate relocated answer replay" --kind scout \ + --repo sample --start --file "$data/backlog.md" >/dev/null) \ + || fail "could not create the relocated answer-before-replay fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "worktree=$wt" "project=$home/projects/sample" \ + "harness=codex" "kind=scout" "mode=scout" "spawn_gen=fixture-$id" + printf 'done: report complete\n' > "$home/state/$id.status" + printf '# Relocated interrupted cleanup\n\nThe captain call remains open.\n' > "$data/$id/report.md" + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-captain-hold.sh" hold "$id" \ + --reason "captain must choose after relocated interrupted cleanup" >/dev/null \ + || fail "could not hold the relocated answer-before-replay fixture" + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-captain-hold.sh" complete "$id" "$id" >/dev/null \ + || fail "completion gate failed for the relocated answer-before-replay fixture" + cat > "$home/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +exit 1 +SH + chmod +x "$home/fakebin/treehouse" + + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" --force \ + > "$home/teardown.out" 2> "$home/teardown.err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "relocated cleanup succeeded despite the failed worktree return" + assert_present "$home/state/$id.backlog-close" \ + "the interrupted relocated cleanup lost its pending record" + + printf 'Proceed despite the reporting limitation.\n' > "$home/answer.txt" + PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$home/config" \ + "$ROOT/bin/fm-captain-hold.sh" answer "$id" --decision-file "$home/answer.txt" \ + >/dev/null || fail "the unsupported relocated report wedged the captain's answer" + show=$(cd "$home" && tasks-axi show "$id" --full --file "$data/backlog.md") \ + || fail "the answered relocated row disappeared" + assert_contains "$show" "state: done" "the relocated report kept the answered call open" + assert_contains "$show" "held: no" "the relocated report kept the answered call held" + + fm_fake_exit0 "$home/fakebin" treehouse + bootstrap=$(PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$data" \ + FM_CONFIG_OVERRIDE="$home/config" FM_BOOTSTRAP_NETWORK=skip \ + "$ROOT/bin/fm-bootstrap.sh" 2>&1) \ + || fail "session start could not replay relocated cleanup after the answer: $bootstrap" + assert_absent "$home/state/$id.meta" "session start left the relocated task record behind" + assert_absent "$home/state/$id.backlog-close" "session start left the relocated pending record behind" + json=$(PATH="$home/fakebin:$PATH" FM_HOME="$home" FM_DATA_OVERRIDE="$data" \ + FM_BEARINGS_NOW=2026-07-14T12:00:00Z "$BEARINGS" --json) \ + || fail "Bearings failed after the relocated answer-before-replay lifecycle" + printf '%s' "$json" | jq -e --arg id "$id" \ + '.landed | any(.id == $id) | not' >/dev/null \ + || fail "the unsupported relocated report was published as a landed delivery: $json" + pass "an unsupported relocated report does not wedge the captain's answer" +} + # A home whose data directory is relocated keeps one backlog; the predicate and # the retention must address it the way teardown does, not FM_HOME/data. test_teardown_retains_captain_calls_in_a_relocated_backlog() { @@ -2573,6 +3081,670 @@ EOF pass "cleanup retains captain calls in the configured backlog" } +test_merge_approval_releases_before_zero_done_retention() { + local home id archive repo wt pr show + home=$(make_home zero-done-retention) + id=sample-zero-retention-merge + archive="$home/data/done-archive.md" + repo="$home/projects/sample" + wt="$home/projects/$id" + pr="https://github.com/sample/sample/pull/19" + printf '%s\n' 'backend = "markdown"' '' '[markdown]' \ + 'path = "data/backlog.md"' 'archive = "data/done-archive.md"' \ + 'done_keep = 0' > "$home/.tasks.toml" + fm_git_worktree "$repo" "$wt" fm/zero-retention-merge + tasks_in "$home" add "$id" "Ship zero-retention merge $pr" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the zero-retention fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "pr=$pr" "spawn_gen=fixture-$id" + printf 'done: merge ready\n' > "$home/state/$id.status" + run_captain "$home" hold "$id" --reason "captain merge approval pending" >/dev/null \ + || fail "could not hold the zero-retention merge" + printf 'Merge the approved change.\n' > "$home/merge-answer.txt" + run_captain "$home" answer "$id" --release \ + --decision-file "$home/merge-answer.txt" >/dev/null \ + || fail "could not release the approved zero-retention merge" + show=$(tasks_in "$home" show "$id" --full) || fail "the released merge row disappeared" + assert_contains "$show" "state: in_flight" \ + "merge approval completed the zero-retention row before landing" + assert_contains "$show" "Resolution mode: released" \ + "merge approval did not record the existing release mode" + run_teardown "$home" "$id" > "$home/teardown.out" 2> "$home/teardown.err" \ + || fail "zero-retention cleanup failed: $(cat "$home/teardown.err")" + assert_no_grep "$id" "$home/data/backlog.md" \ + "zero-retention cleanup kept the completed row in the active backlog" + assert_grep "$id" "$archive" "zero-retention cleanup did not archive the completed row" + assert_grep "$pr" "$archive" "zero-retention archival lost the merged pull request" + assert_grep "Merge the approved change." "$archive" \ + "zero-retention archival lost the recorded merge approval" + assert_absent "$home/state/$id.meta" "zero-retention cleanup retained task metadata" + assert_absent "$home/state/$id.backlog-close" \ + "zero-retention cleanup retained its pending close record" + pass "merge approval releases before zero-retention cleanup records completion" +} + +test_pr_merge_entrypoint_refuses_a_captain_held_task() { + local home pr_id pr repo wt rc + home=$(make_home held-merge-entrypoints) + configure_merged_github "$home" + + pr_id=sample-held-pr-entrypoint + pr=https://github.com/sample/sample/pull/31 + repo="$home/projects/sample-pr" + wt="$home/projects/$pr_id" + fm_git_worktree "$repo" "$wt" "fm/$pr_id" + tasks_in "$home" add "$pr_id" "Ship the held pull request" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the held PR fixture" + fm_write_meta "$home/state/$pr_id.meta" \ + "window=firstmate:fm-$pr_id" "endpoint_task_id=$pr_id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "pr=$pr" "spawn_gen=fixture-$pr_id" + run_captain "$home" hold "$pr_id" --reason "captain merge approval pending" >/dev/null \ + || fail "could not hold the PR entrypoint fixture" + + # Without the entrypoint guard, this run reaches gh-axi and returns success + # even though the task is still held for the captain. + set +e + run_pr_merge "$home" "$pr_id" "$pr" > "$home/pr.out" 2> "$home/pr.err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "the PR merge entrypoint accepted a still-held task" + assert_no_grep 'pr merge 31 ' "$home/gh-axi.log" \ + "the PR merge entrypoint reached the irreversible forge call for a held task" + assert_grep "$pr_id is still held for the captain" "$home/pr.err" \ + "the PR merge refusal did not name the held task" + assert_absent "$home/state/.control-$pr_id.lock" \ + "the refused PR merge left its task control lock held" + pass "the PR merge entrypoint refuses a captain-held task before merging" +} + +test_local_merge_entrypoint_refuses_a_captain_held_task() { + local home local_id local_repo local_wt before after rc + home=$(make_home held-local-merge-entrypoint) + local_id=sample-held-local-entrypoint + local_repo="$home/projects/sample-local" + local_wt="$home/projects/$local_id" + fm_git_worktree "$local_repo" "$local_wt" "fm/$local_id" + printf 'held local delivery\n' > "$local_wt/local.txt" + git -C "$local_wt" add local.txt + git -C "$local_wt" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm 'held local delivery' + tasks_in "$home" add "$local_id" "Ship the held local change" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the held local fixture" + fm_write_meta "$home/state/$local_id.meta" \ + "window=firstmate:fm-$local_id" "endpoint_task_id=$local_id" "worktree=$local_wt" \ + "project=$local_repo" "harness=codex" "kind=ship" "mode=local-only" \ + "spawn_gen=fixture-$local_id" + run_captain "$home" hold "$local_id" --reason "captain local merge approval pending" \ + >/dev/null || fail "could not hold the local entrypoint fixture" + before=$(git -C "$local_repo" rev-parse main) + + # Without the entrypoint guard, this run fast-forwards main while the task + # still carries the captain hold. + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-merge-local.sh" "$local_id" \ + > "$home/local.out" 2> "$home/local.err" + rc=$? + set -e + after=$(git -C "$local_repo" rev-parse main) + [ "$rc" -ne 0 ] || fail "the local merge entrypoint accepted a still-held task" + [ "$after" = "$before" ] || fail "the local merge entrypoint moved main for a held task" + assert_grep "$local_id is still held for the captain" "$home/local.err" \ + "the local merge refusal did not name the held task" + assert_absent "$home/state/.control-$local_id.lock" \ + "the refused local merge left its task control lock held" + pass "the local merge entrypoint refuses a captain-held task before merging" +} + +test_pr_merge_entrypoint_separates_an_unreadable_record_from_an_absent_one() { + local home id pr rc merge_count + home=$(make_home missing-pr-authority-record) + configure_merged_github "$home" + id=sample-missing-pr-authority + pr=https://github.com/sample/sample/pull/43 + write_origin_meta "$home" "$id" ship + + # A backlog that exists but cannot be read may hide a live captain hold, so + # the merge must refuse without reaching the forge. + chmod 000 "$home/data/backlog.md" + set +e + run_pr_merge "$home" "$id" "$pr" > "$home/missing-pr.out" 2> "$home/missing-pr.err" + rc=$? + set -e + chmod 644 "$home/data/backlog.md" + [ "$rc" -ne 0 ] || fail "the PR merge entrypoint accepted an unreadable captain-hold authority record" + assert_grep "could not determine whether task $id is still held for the captain" "$home/missing-pr.err" \ + "the PR merge refusal did not name its unreadable authority record" + assert_no_grep 'pr merge 43 ' "$home/gh-axi.log" \ + "the PR merge entrypoint reached the forge without a readable authority record" + + # A home with no backlog at all records no captain calls, so nothing can be + # held and the merge proceeds. + rm "$home/data/backlog.md" + run_pr_merge "$home" "$id" "$pr" > "$home/absent-pr.out" 2> "$home/absent-pr.err" \ + || fail "the PR merge entrypoint refused a home carrying no backlog" + merge_count=$(grep -c 'pr merge 43 ' "$home/gh-axi.log" || true) + [ "$merge_count" -eq 1 ] || fail "the absent backlog did not permit exactly one PR merge" + pass "the PR merge entrypoint separates an unreadable authority record from an absent one" +} + +test_local_merge_entrypoint_separates_an_unreadable_record_from_an_absent_one() { + local home id repo wt before after rc + home=$(make_home missing-local-authority-record) + id=sample-missing-local-authority + repo="$home/projects/sample-local" + wt="$home/projects/$id" + fm_git_worktree "$repo" "$wt" "fm/$id" + printf 'untracked local delivery\n' > "$wt/local.txt" + git -C "$wt" add local.txt + git -C "$wt" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm 'untracked local delivery' + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=local-only" \ + "spawn_gen=fixture-$id" + before=$(git -C "$repo" rev-parse main) + + # Unreadable authority record: refuse, and leave the default branch where it was. + chmod 000 "$home/data/backlog.md" + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-merge-local.sh" "$id" \ + > "$home/missing-local.out" 2> "$home/missing-local.err" + rc=$? + set -e + chmod 644 "$home/data/backlog.md" + after=$(git -C "$repo" rev-parse main) + [ "$rc" -ne 0 ] || fail "the local merge entrypoint accepted an unreadable captain-hold authority record" + [ "$after" = "$before" ] || fail "the local merge entrypoint moved main without a readable authority record" + assert_grep "could not determine whether task $id is still held for the captain" "$home/missing-local.err" \ + "the local merge refusal did not name its unreadable authority record" + + # No backlog at all: nothing can be held, so the landing proceeds. + rm "$home/data/backlog.md" + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$ROOT/bin/fm-merge-local.sh" "$id" \ + > "$home/absent-local.out" 2> "$home/absent-local.err" \ + || fail "the local merge entrypoint refused a home carrying no backlog" + after=$(git -C "$repo" rev-parse main) + [ "$after" != "$before" ] || fail "the absent backlog did not permit the local merge" + pass "the local merge entrypoint separates an unreadable authority record from an absent one" +} + +test_merge_entrypoints_validate_identity_and_state_before_locking() { + local home pr_state local_state bad_id rc + home=$(make_home invalid-merge-entrypoint-inputs) + configure_merged_github "$home" + + pr_state="$home/missing-pr-state" + set +e + fm_run_timed 2 env PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$pr_state" \ + "$ROOT/bin/fm-pr-merge.sh" sample-missing-pr-state \ + https://github.com/sample/sample/pull/41 \ + > "$home/missing-pr-state.out" 2> "$home/missing-pr-state.err" + rc=$? + set -e + # Without pre-lock state validation, the lock library creates the missing + # directory before the entrypoint discovers that no task record exists. + [ "$rc" -ne 124 ] || fail "the PR merge waited forever for a missing state directory" + [ "$rc" -ne 0 ] || fail "the PR merge accepted a missing state directory" + assert_absent "$pr_state" "the PR merge created a missing state directory while refusing" + assert_grep "state directory is not a real directory" "$home/missing-pr-state.err" \ + "the PR merge did not identify its missing state directory" + + local_state="$home/missing-local-state" + set +e + fm_run_timed 2 env PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$local_state" \ + "$ROOT/bin/fm-merge-local.sh" sample-missing-local-state \ + > "$home/missing-local-state.out" 2> "$home/missing-local-state.err" + rc=$? + set -e + # Without pre-lock state validation, the lock library creates the missing + # directory before the entrypoint discovers that no task record exists. + [ "$rc" -ne 124 ] || fail "the local merge waited forever for a missing state directory" + [ "$rc" -ne 0 ] || fail "the local merge accepted a missing state directory" + assert_absent "$local_state" "the local merge created a missing state directory while refusing" + assert_grep "state directory is not a real directory" "$home/missing-local-state.err" \ + "the local merge did not identify its missing state directory" + + bad_id=sample/bad-local-id + set +e + fm_run_timed 2 env PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" \ + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" \ + "$ROOT/bin/fm-merge-local.sh" "$bad_id" \ + > "$home/bad-local-id.out" 2> "$home/bad-local-id.err" + rc=$? + set -e + # Without canonical ID validation, the slash creates a nested lock path whose + # absent parent makes the blocking acquisition retry until the bound expires. + [ "$rc" -ne 124 ] || fail "the local merge waited forever on a slash-containing task id" + [ "$rc" -eq 2 ] || fail "the local merge returned $rc instead of rejecting the unsafe task id" + assert_grep "invalid local merge request" "$home/bad-local-id.err" \ + "the local merge did not identify the unsafe task id" + assert_absent "$home/state/.control-sample" \ + "the unsafe task id constructed a nested task control path" + pass "merge entrypoints reject unsafe identities and absent state before locking" +} + +test_merge_entrypoints_refuse_a_reused_task_incarnation() { + local home id pr old_repo old_wt new_repo new_wt teardown_pid merge_pid + local teardown_ready teardown_release merge_ready merge_release teardown_rc merge_rc + local local_home local_id local_old_repo local_old_wt local_new_repo local_new_wt + local local_teardown_pid local_merge_pid local_teardown_ready local_teardown_release + local local_merge_ready local_merge_release local_teardown_rc local_merge_rc before after + local real_perl real_sleep + real_perl=$(command -v perl) + real_sleep=$(command -v sleep) + + home=$(make_home reused-pr-incarnation) + configure_merged_github "$home" + install_reused_task_barriers "$home" + id=sample-reused-pr-incarnation + pr=https://github.com/sample/sample/pull/42 + old_repo="$home/projects/sample-reused-pr-old" + old_wt="$home/projects/$id" + fm_git_worktree "$old_repo" "$old_wt" "fm/$id" + tasks_in "$home" add "$id" "Ship the original pull request" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the original PR task" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$old_wt" \ + "project=$old_repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "spawn_gen=original-$id" + printf 'done: merge ready\n' > "$home/state/$id.status" + + teardown_ready="$home/reuse-teardown-ready" + teardown_release="$home/reuse-teardown-release" + merge_ready="$home/reuse-merge-ready" + merge_release="$home/reuse-merge-release" + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" FM_TEST_REUSE_TEARDOWN=1 \ + FM_TEST_REUSE_TEARDOWN_ONCE="$home/reuse-teardown-once" \ + FM_TEST_REUSE_TEARDOWN_READY="$teardown_ready" \ + FM_TEST_REUSE_TEARDOWN_RELEASE="$teardown_release" \ + FM_TEST_REAL_PERL="$real_perl" FM_TEST_REAL_SLEEP="$real_sleep" \ + "$TEARDOWN" "$id" --force > "$home/reuse-teardown.out" \ + 2> "$home/reuse-teardown.err" & + teardown_pid=$! + if ! wait_for_test_file "$teardown_ready" "$teardown_pid"; then + : > "$teardown_release" + wait "$teardown_pid" 2>/dev/null || true + fail "forced PR cleanup did not reach its task-locked synchronization point" + fi + + FM_TEST_REUSE_MERGE=1 FM_TEST_REUSE_MERGE_ONCE="$home/reuse-merge-once" \ + FM_TEST_REUSE_MERGE_READY="$merge_ready" FM_TEST_REUSE_MERGE_RELEASE="$merge_release" \ + FM_TEST_REAL_PERL="$real_perl" FM_TEST_REAL_SLEEP="$real_sleep" \ + run_pr_merge "$home" "$id" "$pr" > "$home/reuse-merge.out" \ + 2> "$home/reuse-merge.err" & + merge_pid=$! + if ! wait_for_test_file "$merge_ready" "$merge_pid"; then + : > "$teardown_release" + : > "$merge_release" + wait "$teardown_pid" 2>/dev/null || true + wait "$merge_pid" 2>/dev/null || true + fail "the PR merge did not wait behind forced cleanup" + fi + + : > "$teardown_release" + set +e + wait "$teardown_pid" + teardown_rc=$? + set -e + if [ "$teardown_rc" -ne 0 ]; then + : > "$merge_release" + wait "$merge_pid" 2>/dev/null || true + fail "forced PR cleanup failed before the task could be reused: $(cat "$home/reuse-teardown.err")" + fi + + new_repo="$home/projects/sample-reused-pr-new" + new_wt="$home/projects/reused-$id" + fm_git_worktree "$new_repo" "$new_wt" "fm/$id" + tasks_in "$home" reopen "$id" >/dev/null || fail "could not reopen the reused PR task" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$new_wt" \ + "project=$new_repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "spawn_gen=replacement-$id" + tasks_in "$home" start "$id" >/dev/null || fail "could not start the reused PR task" + : > "$merge_release" + set +e + wait "$merge_pid" + merge_rc=$? + set -e + + # Without the pre-wait generation capture and locked comparison, the waiter + # records and merges pull request 42 against the replacement task record. + [ "$merge_rc" -ne 0 ] || fail "the PR merge accepted a replacement task incarnation" + assert_no_grep 'pr merge 42 ' "$home/gh-axi.log" \ + "the PR merge reached the forge for a replacement task incarnation" + assert_grep "changed incarnation while waiting to merge" "$home/reuse-merge.err" \ + "the PR merge did not identify the replacement task incarnation" + assert_grep "spawn_gen=replacement-$id" "$home/state/$id.meta" \ + "the refused PR merge damaged the replacement task record" + assert_absent "$home/state/.control-$id.lock" \ + "the reused-incarnation PR refusal left its task control lock held" + + local_home=$(make_home reused-local-incarnation) + install_reused_task_barriers "$local_home" + local_id=sample-reused-local-incarnation + local_old_repo="$local_home/projects/sample-reused-local-old" + local_old_wt="$local_home/projects/$local_id" + fm_git_worktree "$local_old_repo" "$local_old_wt" "fm/$local_id" + printf 'original local delivery\n' > "$local_old_wt/original.txt" + git -C "$local_old_wt" add original.txt + git -C "$local_old_wt" -c user.name='Firstmate Tests' \ + -c user.email='tests@example.invalid' commit -qm 'original local delivery' + tasks_in "$local_home" add "$local_id" "Ship the original local change" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the original local task" + fm_write_meta "$local_home/state/$local_id.meta" \ + "window=firstmate:fm-$local_id" "endpoint_task_id=$local_id" \ + "worktree=$local_old_wt" "project=$local_old_repo" "harness=codex" \ + "kind=ship" "mode=local-only" "spawn_gen=original-$local_id" + printf 'done: local merge ready\n' > "$local_home/state/$local_id.status" + + local_teardown_ready="$local_home/reuse-teardown-ready" + local_teardown_release="$local_home/reuse-teardown-release" + local_merge_ready="$local_home/reuse-merge-ready" + local_merge_release="$local_home/reuse-merge-release" + PATH="$local_home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$local_home" \ + FM_STATE_OVERRIDE="$local_home/state" FM_DATA_OVERRIDE="$local_home/data" \ + FM_CONFIG_OVERRIDE="$local_home/config" FM_TEST_REUSE_TEARDOWN=1 \ + FM_TEST_REUSE_TEARDOWN_ONCE="$local_home/reuse-teardown-once" \ + FM_TEST_REUSE_TEARDOWN_READY="$local_teardown_ready" \ + FM_TEST_REUSE_TEARDOWN_RELEASE="$local_teardown_release" \ + FM_TEST_REAL_PERL="$real_perl" FM_TEST_REAL_SLEEP="$real_sleep" \ + "$TEARDOWN" "$local_id" --force > "$local_home/reuse-teardown.out" \ + 2> "$local_home/reuse-teardown.err" & + local_teardown_pid=$! + if ! wait_for_test_file "$local_teardown_ready" "$local_teardown_pid"; then + : > "$local_teardown_release" + wait "$local_teardown_pid" 2>/dev/null || true + fail "forced local cleanup did not reach its task-locked synchronization point" + fi + + PATH="$local_home/fakebin:$PATH" FM_TEST_REUSE_MERGE=1 \ + FM_TEST_REUSE_MERGE_ONCE="$local_home/reuse-merge-once" \ + FM_TEST_REUSE_MERGE_READY="$local_merge_ready" \ + FM_TEST_REUSE_MERGE_RELEASE="$local_merge_release" \ + FM_TEST_REAL_PERL="$real_perl" FM_TEST_REAL_SLEEP="$real_sleep" \ + FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$local_home" \ + FM_STATE_OVERRIDE="$local_home/state" FM_DATA_OVERRIDE="$local_home/data" \ + FM_CONFIG_OVERRIDE="$local_home/config" \ + "$ROOT/bin/fm-merge-local.sh" "$local_id" > "$local_home/reuse-merge.out" \ + 2> "$local_home/reuse-merge.err" & + local_merge_pid=$! + if ! wait_for_test_file "$local_merge_ready" "$local_merge_pid"; then + : > "$local_teardown_release" + : > "$local_merge_release" + wait "$local_teardown_pid" 2>/dev/null || true + wait "$local_merge_pid" 2>/dev/null || true + fail "the local merge did not wait behind forced cleanup" + fi + + : > "$local_teardown_release" + set +e + wait "$local_teardown_pid" + local_teardown_rc=$? + set -e + if [ "$local_teardown_rc" -ne 0 ]; then + : > "$local_merge_release" + wait "$local_merge_pid" 2>/dev/null || true + fail "forced local cleanup failed before the task could be reused: $(cat "$local_home/reuse-teardown.err")" + fi + + local_new_repo="$local_home/projects/sample-reused-local-new" + local_new_wt="$local_home/projects/reused-$local_id" + fm_git_worktree "$local_new_repo" "$local_new_wt" "fm/$local_id" + printf 'replacement local delivery\n' > "$local_new_wt/replacement.txt" + git -C "$local_new_wt" add replacement.txt + git -C "$local_new_wt" -c user.name='Firstmate Tests' \ + -c user.email='tests@example.invalid' commit -qm 'replacement local delivery' + before=$(git -C "$local_new_repo" rev-parse main) + tasks_in "$local_home" reopen "$local_id" >/dev/null \ + || fail "could not reopen the reused local task" + fm_write_meta "$local_home/state/$local_id.meta" \ + "window=firstmate:fm-$local_id" "endpoint_task_id=$local_id" \ + "worktree=$local_new_wt" "project=$local_new_repo" "harness=codex" \ + "kind=ship" "mode=local-only" "spawn_gen=replacement-$local_id" + tasks_in "$local_home" start "$local_id" >/dev/null \ + || fail "could not start the reused local task" + : > "$local_merge_release" + set +e + wait "$local_merge_pid" + local_merge_rc=$? + set -e + after=$(git -C "$local_new_repo" rev-parse main) + + # Without the pre-wait generation capture and locked comparison, the waiter + # fast-forwards the replacement task's branch despite never approving it. + [ "$local_merge_rc" -ne 0 ] || fail "the local merge accepted a replacement task incarnation" + [ "$after" = "$before" ] || fail "the local merge moved main for a replacement task incarnation" + assert_grep "changed incarnation while waiting to merge" "$local_home/reuse-merge.err" \ + "the local merge did not identify the replacement task incarnation" + assert_grep "spawn_gen=replacement-$local_id" "$local_home/state/$local_id.meta" \ + "the refused local merge damaged the replacement task record" + assert_absent "$local_home/state/.control-$local_id.lock" \ + "the reused-incarnation local refusal left its task control lock held" + pass "merge entrypoints refuse a replacement incarnation after waiting for cleanup" +} + +test_merge_entrypoints_serialize_forced_teardown_before_task_reads() { + local home id pr repo wt ready release merge_pid teardown_rc merge_rc real_grep + local local_home local_id local_repo local_wt local_ready local_release local_pid + local local_teardown_rc local_merge_rc real_git before after i + + home=$(make_home teardown-race-pr-entrypoint) + configure_merged_github "$home" + id=sample-teardown-race-pr + pr=https://github.com/sample/sample/pull/33 + repo="$home/projects/sample-pr-race" + wt="$home/projects/$id" + fm_git_worktree "$repo" "$wt" "fm/$id" + tasks_in "$home" add "$id" "Ship the released pull request" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the PR teardown-race fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "spawn_gen=fixture-$id" + printf 'done: merge ready\n' > "$home/state/$id.status" + run_captain "$home" hold "$id" --reason "captain merge approval pending" >/dev/null \ + || fail "could not hold the PR teardown-race fixture" + printf 'Merge the released pull request.\n' > "$home/race-answer.txt" + run_captain "$home" answer "$id" --release --decision-file "$home/race-answer.txt" \ + >/dev/null || fail "could not release the PR teardown-race fixture" + + real_grep=$(command -v grep) + ready="$home/pr-metadata-read-ready" + release="$home/pr-metadata-read-release" + cat > "$home/fakebin/grep" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = -qxF ] && [ "${2:-}" = "pr=${FM_TEST_RACE_PR_URL:-}" ] \ + && [ "${3:-}" = "${FM_TEST_RACE_PR_META:-}" ]; then + "$FM_TEST_REAL_GREP" "$@" || exit $? + : > "$FM_TEST_RACE_READY" + while [ ! -e "$FM_TEST_RACE_RELEASE" ]; do sleep 0.01; done + exit 0 +fi +exec "$FM_TEST_REAL_GREP" "$@" +SH + chmod +x "$home/fakebin/grep" + FM_TEST_REAL_GREP="$real_grep" FM_TEST_RACE_PR_URL="$pr" \ + FM_TEST_RACE_PR_META="$home/state/$id.meta" FM_TEST_RACE_READY="$ready" \ + FM_TEST_RACE_RELEASE="$release" run_pr_merge "$home" "$id" "$pr" \ + > "$home/race-pr.out" 2> "$home/race-pr.err" & + merge_pid=$! + i=0 + while [ "$i" -lt 500 ]; do + [ -e "$ready" ] && break + kill -0 "$merge_pid" 2>/dev/null || break + sleep 0.01 + i=$((i + 1)) + done + if [ ! -e "$ready" ]; then + : > "$release" + wait "$merge_pid" 2>/dev/null || true + fail "the PR merge did not reach the post-metadata synchronization point" + fi + set +e + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" --force \ + > "$home/race-pr-teardown.out" 2> "$home/race-pr-teardown.err" + teardown_rc=$? + set -e + : > "$release" + wait "$merge_pid" + merge_rc=$? + + # Without early lock ownership, forced cleanup succeeds after metadata is + # recorded, and the resumed forge call merges work whose task was retired. + [ "$teardown_rc" -ne 0 ] \ + || fail "forced cleanup retired the task between PR metadata recording and merge" + assert_grep "another lifecycle action is already running for task $id" \ + "$home/race-pr-teardown.err" \ + "PR cleanup was not refused by the merge's task control lock" + [ "$merge_rc" -eq 0 ] || fail "the serialized PR merge failed after cleanup was refused" + assert_present "$home/state/$id.meta" "the refused PR cleanup removed task metadata" + assert_grep 'pr merge 33 ' "$home/gh-axi.log" \ + "the serialized PR merge did not reach the forge after cleanup was refused" + + local_home=$(make_home teardown-race-local-entrypoint) + local_id=sample-teardown-race-local + local_repo="$local_home/projects/sample-local-race" + local_wt="$local_home/projects/$local_id" + fm_git_worktree "$local_repo" "$local_wt" "fm/$local_id" + printf 'serialized local delivery\n' > "$local_wt/local.txt" + git -C "$local_wt" add local.txt + git -C "$local_wt" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm 'serialized local delivery' + tasks_in "$local_home" add "$local_id" "Ship the released local change" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the local teardown-race fixture" + fm_write_meta "$local_home/state/$local_id.meta" \ + "window=firstmate:fm-$local_id" "endpoint_task_id=$local_id" "worktree=$local_wt" \ + "project=$local_repo" "harness=codex" "kind=ship" "mode=local-only" \ + "spawn_gen=fixture-$local_id" + printf 'done: local merge ready\n' > "$local_home/state/$local_id.status" + run_captain "$local_home" hold "$local_id" \ + --reason "captain local merge approval pending" >/dev/null \ + || fail "could not hold the local teardown-race fixture" + printf 'Land the released local change.\n' > "$local_home/race-answer.txt" + run_captain "$local_home" answer "$local_id" --release \ + --decision-file "$local_home/race-answer.txt" >/dev/null \ + || fail "could not release the local teardown-race fixture" + + real_git=$(command -v git) + local_ready="$local_home/local-validation-ready" + local_release="$local_home/local-validation-release" + cat > "$local_home/fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "$*" = "-C ${FM_TEST_RACE_REPO:-} rev-parse --short main" ]; then + output=$("$FM_TEST_REAL_GIT" "$@") || exit $? + : > "$FM_TEST_RACE_READY" + while [ ! -e "$FM_TEST_RACE_RELEASE" ]; do sleep 0.01; done + printf '%s\n' "$output" + exit 0 +fi +exec "$FM_TEST_REAL_GIT" "$@" +SH + chmod +x "$local_home/fakebin/git" + before=$(git -C "$local_repo" rev-parse main) + PATH="$local_home/fakebin:$PATH" FM_TEST_REAL_GIT="$real_git" \ + FM_TEST_RACE_REPO="$local_repo" FM_TEST_RACE_READY="$local_ready" \ + FM_TEST_RACE_RELEASE="$local_release" FM_ROOT_OVERRIDE="$ROOT" \ + FM_HOME="$local_home" FM_STATE_OVERRIDE="$local_home/state" \ + FM_DATA_OVERRIDE="$local_home/data" FM_CONFIG_OVERRIDE="$local_home/config" \ + "$ROOT/bin/fm-merge-local.sh" "$local_id" \ + > "$local_home/race-local.out" 2> "$local_home/race-local.err" & + local_pid=$! + i=0 + while [ "$i" -lt 500 ]; do + [ -e "$local_ready" ] && break + kill -0 "$local_pid" 2>/dev/null || break + sleep 0.01 + i=$((i + 1)) + done + if [ ! -e "$local_ready" ]; then + : > "$local_release" + wait "$local_pid" 2>/dev/null || true + fail "the local merge did not reach the post-validation synchronization point" + fi + set +e + PATH="$local_home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$local_home" \ + FM_STATE_OVERRIDE="$local_home/state" FM_DATA_OVERRIDE="$local_home/data" \ + FM_CONFIG_OVERRIDE="$local_home/config" "$TEARDOWN" "$local_id" --force \ + > "$local_home/race-local-teardown.out" 2> "$local_home/race-local-teardown.err" + local_teardown_rc=$? + set -e + : > "$local_release" + wait "$local_pid" + local_merge_rc=$? + after=$(git -C "$local_repo" rev-parse main) + + # Without early lock ownership, forced cleanup succeeds after validation and + # the resumed fast-forward lands work whose task was already retired. + [ "$local_teardown_rc" -ne 0 ] \ + || fail "forced cleanup retired the task between local validation and merge" + assert_grep "another lifecycle action is already running for task $local_id" \ + "$local_home/race-local-teardown.err" \ + "local cleanup was not refused by the merge's task control lock" + [ "$local_merge_rc" -eq 0 ] || fail "the serialized local merge failed after cleanup was refused" + [ "$after" != "$before" ] || fail "the serialized local merge did not fast-forward main" + assert_present "$local_home/state/$local_id.meta" \ + "the refused local cleanup removed task metadata" + pass "merge entrypoints own task state before forced cleanup can retire it" +} + +# No regression covers a re-hold after a merge lands and before cleanup because +# that accepted window spans two separate lifecycle owners. +# Queued forge merges are also uncovered because they land asynchronously after +# the local merge command and its task control lock have returned. +test_released_merge_passes_the_entrypoint_and_lands() { + local home id pr repo wt show json + home=$(make_home released-merge-entrypoint) + configure_merged_github "$home" + id=sample-released-merge + pr=https://github.com/sample/sample/pull/32 + repo="$home/projects/sample-released" + wt="$home/projects/$id" + fm_git_worktree "$repo" "$wt" "fm/$id" + tasks_in "$home" add "$id" "Ship the approved pull request" --kind ship \ + --repo sample --start >/dev/null || fail "could not create the released merge fixture" + fm_write_meta "$home/state/$id.meta" \ + "window=firstmate:fm-$id" "endpoint_task_id=$id" "worktree=$wt" \ + "project=$repo" "harness=codex" "kind=ship" "mode=no-mistakes" \ + "pr=$pr" "spawn_gen=fixture-$id" + printf 'done: merge ready\n' > "$home/state/$id.status" + run_captain "$home" hold "$id" --reason "captain merge approval pending" >/dev/null \ + || fail "could not hold the released merge fixture" + printf 'Merge the approved pull request.\n' > "$home/merge-answer.txt" + run_captain "$home" answer "$id" --release \ + --decision-file "$home/merge-answer.txt" >/dev/null \ + || fail "could not release the approved merge" + show=$(tasks_in "$home" show "$id" --full) || fail "the released merge task disappeared" + assert_not_contains "$show" "hold_kind: captain" \ + "the approved merge remained captain-held after its release" + run_pr_merge "$home" "$id" "$pr" > "$home/merge.out" 2> "$home/merge.err" \ + || fail "the released merge was refused: $(cat "$home/merge.err")" + PATH="$home/fakebin:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_CONFIG_OVERRIDE="$home/config" "$TEARDOWN" "$id" --force \ + > "$home/teardown.out" 2> "$home/teardown.err" \ + || fail "the released merge cleanup failed: $(cat "$home/teardown.err")" + json=$(run_bearings "$home") || fail "Bearings failed after the released merge lifecycle" + printf '%s' "$json" | jq -e --arg id "$id" --arg pr "$pr" \ + '.landed | any(.id == $id and .artifact == $pr)' >/dev/null \ + || fail "the released merge was absent from Recently Landed: $json" + pass "a released merge passes the guarded entrypoint and remains recently landed" +} + # "Cannot tell" is not permission to close. A ship row has no separate # inventory gate ahead of the close, so the predicate itself must refuse before # any destructive step when the hold cannot be read. @@ -2646,8 +3818,21 @@ test_origin_slug_validation_precedes_path_construction test_status_resolution_over_an_open_hold_is_signalled test_legitimate_holds_produce_no_divergence_signal test_teardown_never_closes_a_captain_held_task +test_retained_row_artifacts_survive_captain_answers test_interrupted_cleanup_keeps_the_captain_call_recoverable +test_answer_before_cleanup_replay_preserves_the_retained_report +test_unusable_pending_close_record_names_its_reason +test_relocated_report_does_not_wedge_an_answer_before_replay test_teardown_retains_captain_calls_in_a_relocated_backlog +test_merge_approval_releases_before_zero_done_retention +test_pr_merge_entrypoint_refuses_a_captain_held_task +test_local_merge_entrypoint_refuses_a_captain_held_task +test_pr_merge_entrypoint_separates_an_unreadable_record_from_an_absent_one +test_local_merge_entrypoint_separates_an_unreadable_record_from_an_absent_one +test_merge_entrypoints_validate_identity_and_state_before_locking +test_merge_entrypoints_refuse_a_reused_task_incarnation +test_merge_entrypoints_serialize_forced_teardown_before_task_reads +test_released_merge_passes_the_entrypoint_and_lands test_teardown_refuses_a_ship_when_the_captain_hold_cannot_be_read test_verify_resolves_a_hold_migrated_to_beads_notes test_verify_resolves_a_hold_migrated_under_the_configured_prefix diff --git a/tests/fm-claude-stop-autoarm.test.sh b/tests/fm-claude-stop-autoarm.test.sh index 066ecd070b0..5cea32ca562 100755 --- a/tests/fm-claude-stop-autoarm.test.sh +++ b/tests/fm-claude-stop-autoarm.test.sh @@ -87,6 +87,8 @@ write_arm_fixture() { cat > "$dir/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash echo "$$" >> "$FM_HOME/state/arm-ran" +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +touch "$FM_HOME/state/.last-watcher-beat" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" printf 'stale: fixture-win actionable\n' exit 0 @@ -131,6 +133,8 @@ SH #!/usr/bin/env bash echo "$$" >> "$FM_HOME/state/arm-ran" sleep 2 +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +touch "$FM_HOME/state/.last-watcher-beat" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" printf 'signal: task.status done: slow fixture\n' exit 0 @@ -141,6 +145,8 @@ SH #!/usr/bin/env bash echo "$$" >> "$FM_HOME/state/arm-ran" sleep 6 +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +touch "$FM_HOME/state/.last-watcher-beat" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" printf 'stale: fixture-win actionable\n' exit 0 @@ -161,6 +167,8 @@ SH #!/usr/bin/env bash echo "$$" >> "$FM_HOME/state/arm-ran" rm -f "$FM_HOME/state/task.meta" +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +touch "$FM_HOME/state/.last-watcher-beat" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" printf 'signal: task.status done: fixture\n' exit 0 @@ -171,6 +179,8 @@ SH #!/usr/bin/env bash echo "$$" >> "$FM_HOME/state/arm-ran" : > "$FM_HOME/state/.afk" +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +touch "$FM_HOME/state/.last-watcher-beat" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" printf 'stale: fixture-win actionable\n' exit 0 @@ -395,6 +405,10 @@ test_actionable_close_rewakes_with_reason() { assert_contains "$out" "bin/fm-wake-drain.sh" "rewake must direct the drain-first protocol" assert_contains "$out" "do NOT run bin/fm-watch-arm.sh" "rewake must forbid a duplicate model re-arm" [ "$(epoch_outcome "$dir")" = rewake ] || fail "epoch must record outcome=rewake, got: $(epoch_outcome "$dir")" + [ "$(epoch_field "$dir" session_pid)" = "$(cat "$dir/state/.lock")" ] \ + || fail "rewake epoch must bind the lock-owning Claude session" + [ "$(epoch_field "$dir" recovery_generation)" = fixture-generation ] \ + || fail "rewake epoch must bind the watcher recovery generation" [ ! -e "$dir/state/.claude-autoarm.lock" ] || fail "owner lock must be released after the cycle" [ -e "$dir/state/arm-ran" ] || fail "hook never foregrounded the arm wrapper" pass "auto-arm: actionable close translates to exactly one exit-2 rewake with reason" diff --git a/tests/fm-control-herdr-smoke.test.sh b/tests/fm-control-herdr-smoke.test.sh index 5d861ba7490..29799fd7bf1 100755 --- a/tests/fm-control-herdr-smoke.test.sh +++ b/tests/fm-control-herdr-smoke.test.sh @@ -52,7 +52,14 @@ SCRATCH=$(mktemp -d "${TMPDIR:-/tmp}/fm-control-herdr.XXXXXX") SCRATCH=$(cd "$SCRATCH" && pwd) HOME_DIR="$SCRATCH/home" mkdir -p "$HOME_DIR/state" "$HOME_DIR/data/hsmoke" -printf '# brief\n' > "$HOME_DIR/data/hsmoke/brief.md" +cat > "$HOME_DIR/data/hsmoke/brief.md" <<'EOF' +# Task +## Captain's intent +Exercise Herdr lifecycle control safely. + +## Firstmate spec +Keep the isolated endpoint and worktree intact. +EOF # A real git worktree so the control plane's checkpoint has a real local copy. PROJ="$SCRATCH/proj" @@ -63,6 +70,8 @@ printf '# proj\n' > "$PROJ/README.md" git -C "$PROJ" add README.md git -C "$PROJ" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' commit -qm initial git -C "$PROJ" worktree add --quiet -b hsmoke "$WT" +PROJ_REAL=$(cd "$PROJ" && pwd -P) +WT_REAL=$(cd "$WT" && pwd -P) # shellcheck source=/dev/null . "$ROOT/bin/fm-backend.sh" @@ -98,7 +107,7 @@ EOF } > "$HOME_DIR/state/hsmoke.meta" run_control() { - env FM_HOME="$HOME_DIR" HERDR_SESSION="$SESSION" \ + env FM_HOME="$HOME_DIR" HERDR_SESSION="$SESSION" FM_SPAWN_NO_GUARD=1 \ FM_CONTROL_POLL=0.2 FM_CONTROL_EXIT_WAIT=2 \ "$ROOT/bin/fm-control.sh" "$@" 2>&1 } @@ -112,6 +121,83 @@ case "$OUT" in esac pass "real herdr: exit on a pane with no registered agent is idempotent success" +# --- the recovery-grade read, against the real binary ------------------------ +# +# The classification that decides whether a task can be recovered at all is read +# out of what herdr actually answers, so a stub can only confirm the assumption +# already written into the stub. Its logic is pinned portably in +# tests/fm-backend-herdr.test.sh; this is the check that notices when the real +# client stops answering the way that logic expects, and it names the version so +# a release change is attributed rather than mysterious. +HERDR_VERSION=$(herdr --version 2>&1 | head -1) +HERDR_VERSION=${HERDR_VERSION#herdr } +version_fail() { # <message> + fail "$1 [herdr $HERDR_VERSION]" +} + +STATE=$(fm_backend_agent_state herdr "$SESSION:$PANE_ID") +[ "$STATE" = dead ] \ + || version_fail "a real, present, agent-free pane reads '$STATE' rather than 'dead'; every relaunch would be refused" + +# `status --json` is the second signal, and the only one that answers for a +# session whose operational calls cannot be reached at all. A release that drops +# or renames `.server.running` would silently make every gone endpoint +# unrecoverable again, so it is asserted by name on both a live and an absent +# session. +[ "$(fm_backend_herdr_server_running_state "$SESSION")" = running ] \ + || version_fail "this run's own live lab session does not report .server.running=true through status --json" +[ "$(fm_backend_herdr_server_running_state "fm-lab-never-started-$$")" = stopped ] \ + || version_fail "a session with no server does not report .server.running=false, so authoritative absence can no longer be told from an unreadable read" + +# Issue #4091's exact stranding shape: an endpoint recorded in a session whose +# server is not running used to read `unreadable` and block recovery. +[ "$(fm_backend_agent_state herdr "fm-lab-never-started-$$:w1:p2")" = missing ] \ + || version_fail "an endpoint in a session with no running server is not classified as recoverable" + +# And the safety direction: an uninterpretable read must never license recovery. +[ "$(fm_backend_agent_state herdr "no-separator-here")" = unreadable ] \ + || version_fail "a malformed endpoint target does not stay unreadable" +pass "real herdr $HERDR_VERSION: a gone session reads recoverable while a live pane and a malformed target do not" + +FAKEBIN="$SCRATCH/fakebin" +mkdir -p "$FAKEBIN" +cat > "$FAKEBIN/codex" <<EOF +#!/usr/bin/env bash +: > "$SCRATCH/codex-launched" +EOF +chmod +x "$FAKEBIN/codex" +printf -v FAKEBIN_Q '%q' "$FAKEBIN" +printf -v PROJ_Q '%q' "$PROJ" +fm_backend_herdr_send_text_line "$SESSION:$PANE_ID" "export PATH=$FAKEBIN_Q:\$PATH" \ + || fail "could not put the inert test harness on the pane PATH" +fm_backend_herdr_send_text_line "$SESSION:$PANE_ID" "cd -- $PROJ_Q" \ + || fail "could not move the agent-free pane out of its recorded worktree" +for _ in $(seq 1 20); do + [ "$(fm_backend_herdr_current_path "$SESSION:$PANE_ID" 2>/dev/null || true)" != "$PROJ_REAL" ] || break + sleep 0.1 +done +[ "$(fm_backend_herdr_current_path "$SESSION:$PANE_ID" 2>/dev/null || true)" = "$PROJ_REAL" ] \ + || fail "the real Herdr pane did not drift out of its recorded worktree" + +OUT=$(env FM_HOME="$HOME_DIR" HERDR_SESSION="$SESSION" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" hsmoke --relaunch --harness codex) \ + || fail "a drifted, agent-free Herdr pane should be re-homed and relaunched: $OUT" +for _ in $(seq 1 20); do + [ ! -e "$SCRATCH/codex-launched" ] || break + sleep 0.1 +done +[ -e "$SCRATCH/codex-launched" ] || fail "the replacement harness was not launched" +[ "$(fm_backend_herdr_current_path "$SESSION:$PANE_ID" 2>/dev/null || true)" = "$WT_REAL" ] \ + || fail "the relaunched Herdr shell did not end up in its recorded worktree" +[ "$(sed -n 's/^window=//p' "$HOME_DIR/state/hsmoke.meta" | tail -1)" = "$SESSION:$PANE_ID" ] \ + || fail "the Herdr relaunch replaced its endpoint instead of reusing it" +herdr pane get "$PANE_ID" --session "$SESSION" >/dev/null 2>&1 \ + || fail "the Herdr relaunch removed the endpoint it was required to reuse" +awk -F= '$1 == "harness" {$0="harness=claude"} {print}' "$HOME_DIR/state/hsmoke.meta" \ + > "$HOME_DIR/state/hsmoke.meta.tmp" +mv "$HOME_DIR/state/hsmoke.meta.tmp" "$HOME_DIR/state/hsmoke.meta" +pass "real herdr: a drifted agent-free shell returns to its worktree and reuses the same endpoint" + if OUT=$(run_control hsmoke interrupt 2>&1); then fail "interrupt should refuse when herdr reports no agent on the pane: $OUT" fi diff --git a/tests/fm-control-relaunch.test.sh b/tests/fm-control-relaunch.test.sh index 5cba46a0f7a..babadfc9949 100755 --- a/tests/fm-control-relaunch.test.sh +++ b/tests/fm-control-relaunch.test.sh @@ -838,6 +838,31 @@ test_same_harness_relaunch_keeps_the_profile_axes() { pass "fm-control relaunch: a same-harness relaunch keeps the profile axes it was running with" } +test_native_ultra_relaunch_preserves_profile_and_rejects_before_stop() { + local dir out rc id=rl-ultra + dir=$(new_case native-ultra "$id") + add_ship_task "$dir" "$id" pi + printf pi > "$dir/fake/command" + printf pi > "$dir/fake/becomes" + printf '#!/usr/bin/env bash\nprintf "Options: --tui-mode\\n"\n' > "$dir/fakebin/pi" + chmod +x "$dir/fakebin/pi" + sed 's|^model=default$|model=codex-native/gpt-6-astra|; s/^effort=default$/effort=ultra/' \ + "$dir/home/state/$id.meta" > "$dir/home/state/$id.meta.tmp" + mv "$dir/home/state/$id.meta.tmp" "$dir/home/state/$id.meta" + out=$(run_control "$dir" "$id" relaunch --model openai-codex/gpt-6-astra --note "invalid native effort transfer"); rc=$? + expect_code 1 "$rc" "Ultra transferred to ordinary Pi" + assert_contains "$out" "ultra effort requires pi or pi-signed" "model-aware relaunch refusal missing" + [ "$(cat "$dir/fake/command")" = pi ] || fail "invalid Ultra relaunch stopped the running agent" + [ ! -s "$dir/fake/literal" ] || fail "invalid Ultra relaunch sent lifecycle input" + out=$(run_control "$dir" "$id" relaunch --note "preserve explicit native effort"); rc=$? + expect_code 0 "$rc" "native Ultra relaunch failed: $out" + [ "$(meta_field "$dir" "$id" effort)" = ultra ] || fail "relaunch lost Ultra metadata" + [ "$(meta_field "$dir" "$id" model)" = codex-native/gpt-6-astra ] || fail "relaunch lost native model" + assert_contains "$(cat "$dir/fake/literal")" "--codex-effort 'ultra'" "relaunch lost native flag" + assert_not_contains "$(cat "$dir/fake/literal")" "--thinking 'ultra'" "relaunch used an invalid Pi level" + pass "native Ultra relaunch preserves its profile and rejects an unsupported model before stopping" +} + test_explicit_model_wins_over_the_recorded_one() { local dir out rc dir=$(new_case explicit rl7) @@ -1739,7 +1764,8 @@ test_spawn_relaunch_refuses_a_pane_outside_the_worktree() { out=$(run_spawn "$dir" rl18 --relaunch --harness claude); rc=$? expect_code 1 "$rc" "a pane outside the worktree should refuse" assert_contains "$out" "not its recorded worktree" "the refusal should name the wrong location" - pass "fm-spawn --relaunch: refuses to start a replacement outside the copy holding the work" + [ ! -s "$dir/fake/keys" ] || fail "a refused tmux relaunch must send nothing to the pane" + pass "fm-spawn --relaunch: refuses to start a replacement outside the copy holding its work" } test_relaunch_reverifies_an_already_in_flight_item_instead_of_rewriting_it() { @@ -1795,6 +1821,7 @@ test_harness_switch_does_not_carry_the_old_profile_axes test_harness_switch_resolves_a_prefixed_recorded_harness test_prefixed_recorded_harness_requires_explicit_replacement test_same_harness_relaunch_keeps_the_profile_axes +test_native_ultra_relaunch_preserves_profile_and_rejects_before_stop test_explicit_model_wins_over_the_recorded_one test_relaunch_onto_an_unverified_harness_is_refused test_prior_harness_turnend_registry_entry_is_cleared diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index 0fc789d80ed..85e40e25d3f 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -22,6 +22,10 @@ # (d2) terminal failed run whose only failure is an orphaned ci monitor # after checks read green -> done # (e) cross-branch attribution: this branch's own run found via list lookup +# (e2) several runs bound to one worktree: the live one outranks the corpse +# (an unclassifiable status word keeps the ledger's newest-first order) +# (e3) the live sibling's head was never fetched into the task copy: it still +# outranks a terminal row sitting at the worktree's exact commit # (f) no run + semantic busy -> pane # (g) no run + semantic idle falls to the status-log verb -> status-log # (h) dead pane: no run -> unknown/none; with a run -> run-step (not the shell) @@ -131,8 +135,8 @@ case "${1:-}" in esac ;; runs) - [ "${FM_FAKE_RUNS_RC:-0}" = 0 ] || exit "${FM_FAKE_RUNS_RC}" - printf '%s\n' "${FM_FAKE_RUNS_LIST:-}" ;; + printf '%s\n' "${FM_FAKE_RUNS_LIST:-}" + exit "${FM_FAKE_RUNS_RC:-0}" ;; daemon) # FM_FAKE_DAEMON_DOWN: the explicit down-probe fails, as the real # `no-mistakes daemon status` does when the daemon is not running. @@ -2046,6 +2050,197 @@ EOF pass "coarse branch-head identity preserves observed merge evidence" } +# Live-over-terminal selection (bin/fm-nm-run-lib.sh). Reproduces the proven +# 2026-08 case: a crashed validation daemon left a FAILED run at the worktree's +# exact commit, while the live run that replaced it validates a descendant +# commit on the same branch. Both bind - the corpse by the equal-commit rule, +# the live run by the ancestor rule - and bare `axi status` answers with the +# corpse, so every recomputation read a healthy task as failed. +test_terminal_corpse_loses_to_live_run_on_same_branch() { + reset_fakes + local d base_head live_head short_base short_live out + d=$(new_case live-beats-corpse) + make_repo_on_branch "$d/wt" fm/feat-corpse + base_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" commit -q --allow-empty -m 'live run advanced the tip' + live_head=$(git -C "$d/wt" rev-parse HEAD) + # Worktree stays at the commit the dead run recorded; the live run is ahead. + git -C "$d/wt" reset -q --hard "$base_head" + short_base=$(git -C "$d/wt" rev-parse --short=7 "$base_head") + short_live=$(git -C "$d/wt" rev-parse --short=7 "$live_head") + [ "$short_base" != "$short_live" ] || fail "live run head did not advance past the worktree" + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/corpse.meta" "window=fm:fm-corpse" "worktree=$d/wt" "kind=ship" + # The corpse is the most-recently-touched run, so it is what `axi status` + # reports, at this worktree's own commit. + FM_FAKE_RUN_HEAD="$base_head" + FM_FAKE_AXI_STATUS="$(run_failed fm/feat-corpse)" + # It is also the newest row in the listing (the crash marked it after the + # live run started), so row order alone still selects the corpse. + FM_FAKE_RUNS_LIST="$(cat <<EOF + failed fm/feat-corpse ${short_base} 2026-08-05 11:20 + running fm/feat-corpse ${short_live} 2026-08-05 10:05 +EOF +)" + out=$(run_crew_state "$d" corpse) + assert_contains "$out" "state: working" "the live run outranks the terminal corpse bound to the same worktree" + assert_contains "$out" "source: run-step" "the live run is still an attributed run-step verdict" + assert_not_contains "$out" "state: failed" "a dead run at the worktree commit must not report a healthy task as failed" + pass "a live run outranks a terminal run bound to the same worktree" +} + +# The same preference on the runs-list path itself: `axi status` answers for +# another crew's branch, and this branch's newest row is terminal while an older +# row is still live. +test_runs_list_live_row_outranks_newer_terminal_row() { + reset_fakes + local d base_head live_head short_base short_live out + d=$(new_case live-row-beats-terminal-row) + make_repo_on_branch "$d/wt" fm/feat-liverow + base_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" commit -q --allow-empty -m 'live run advanced the tip' + live_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" reset -q --hard "$base_head" + short_base=$(git -C "$d/wt" rev-parse --short=7 "$base_head") + short_live=$(git -C "$d/wt" rev-parse --short=7 "$live_head") + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/liverow.meta" "window=fm:fm-liverow" "worktree=$d/wt" "kind=ship" + FM_FAKE_AXI_STATUS="$(run_running fm/other-crew)" + FM_FAKE_RUNS_LIST="$(cat <<EOF + running fm/other-crew aaaaaaa 2026-08-05 11:30 + failed fm/feat-liverow ${short_base} 2026-08-05 11:20 + running fm/feat-liverow ${short_live} 2026-08-05 10:05 +EOF +)" + out=$(run_crew_state "$d" liverow) + assert_contains "$out" "state: working" "an older live row outranks the branch's newest terminal row" + assert_not_contains "$out" "state: failed" "the terminal row must not win while a live row binds" + pass "runs-list selection prefers a live row over a newer terminal one" +} + +# The routine production shape of the same case: the live run's fix-round +# commits live only in the gate repo, so its head is not a git object in the +# task copy and can never bind by the head rule. The terminal row sitting at +# the worktree's EXACT commit is the anchor that proves the unfetched live row +# is this worktree's own continuation, so the live run still wins. +test_unfetched_live_sibling_outranks_terminal_row_at_exact_head() { + reset_fakes + local d base_head short_base unfetched out + d=$(new_case unfetched-live-sibling) + make_repo_on_branch "$d/wt" fm/feat-unfetched + base_head=$(git -C "$d/wt" rev-parse HEAD) + short_base=$(git -C "$d/wt" rev-parse --short=7 "$base_head") + unfetched=0123abc + git -C "$d/wt" rev-parse --verify --quiet "${unfetched}^{commit}" >/dev/null 2>&1 \ + && fail "the unfetched head must not resolve in the task copy" + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/unfetched.meta" "window=fm:fm-unfetched" "worktree=$d/wt" "kind=ship" + FM_FAKE_RUN_HEAD="$base_head" + FM_FAKE_AXI_STATUS="$(run_failed fm/feat-unfetched)" + FM_FAKE_RUNS_LIST="$(cat <<EOF + failed fm/feat-unfetched ${short_base} 2026-08-05 11:20 + running fm/feat-unfetched ${unfetched} 2026-08-05 10:05 +EOF +)" + out=$(run_crew_state "$d" unfetched) + assert_contains "$out" "state: working" "an unfetched live row anchored by the exact-head terminal row outranks it" + assert_not_contains "$out" "state: failed" "the corpse at the worktree commit must not report a healthy task as failed" + pass "an unfetched live sibling outranks a terminal row at the worktree's exact commit" +} + +# The preference must not widen: candidates of the SAME liveness class keep the +# listing's existing newest-first precedence, so two terminal rows still resolve +# to the newer one rather than to whichever the scan happens to reach last. +test_only_terminal_rows_keep_newest_first_precedence() { + reset_fakes + local d base_head older_head short_base short_older out + d=$(new_case only-terminal-rows) + make_repo_on_branch "$d/wt" fm/feat-allterminal + base_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" commit -q --allow-empty -m 'an earlier terminal run advanced the tip' + older_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" reset -q --hard "$base_head" + short_base=$(git -C "$d/wt" rev-parse --short=7 "$base_head") + short_older=$(git -C "$d/wt" rev-parse --short=7 "$older_head") + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/allterminal.meta" "window=fm:fm-allterminal" "worktree=$d/wt" "kind=ship" + FM_FAKE_AXI_STATUS="$(run_running fm/other-crew)" + FM_FAKE_RUNS_LIST="$(cat <<EOF + running fm/other-crew aaaaaaa 2026-08-05 11:30 + cancelled fm/feat-allterminal ${short_base} 2026-08-05 11:20 + completed fm/feat-allterminal ${short_older} 2026-08-05 10:05 +EOF +)" + out=$(run_crew_state "$d" allterminal) + assert_contains "$out" "state: failed" "the newest terminal row still wins when no live row binds" + assert_contains "$out" "run cancelled" "the newer cancelled row, not the older completed one" + pass "two terminal rows keep the existing newest-first precedence" +} + +# An unclassifiable status word keeps the ledger's own newest-first precedence: +# the live-over-terminal preference only ever reorders rows whose liveness is +# known, so an unexpected newest row is answered as-is instead of being +# displaced by an older running row and reported as working. +test_unknown_status_row_keeps_newest_first_precedence() { + reset_fakes + local d base_head live_head short_base short_live out + d=$(new_case unknown-status-row) + make_repo_on_branch "$d/wt" fm/feat-unknownrow + base_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" commit -q --allow-empty -m 'an older run advanced the tip' + live_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" reset -q --hard "$base_head" + short_base=$(git -C "$d/wt" rev-parse --short=7 "$base_head") + short_live=$(git -C "$d/wt" rev-parse --short=7 "$live_head") + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/unknownrow.meta" "window=fm:fm-unknownrow" "worktree=$d/wt" "kind=ship" + FM_FAKE_AXI_STATUS="$(run_running fm/other-crew)" + FM_FAKE_RUNS_LIST="$(cat <<EOF + running fm/other-crew aaaaaaa 2026-08-05 11:30 + quarantined fm/feat-unknownrow ${short_base} 2026-08-05 11:20 + running fm/feat-unknownrow ${short_live} 2026-08-05 10:05 +EOF +)" + out=$(run_crew_state "$d" unknownrow) + assert_contains "$out" "runs list status: quarantined" "the newest row's unclassifiable status is answered as-is" + assert_not_contains "$out" "state: working" "an older live row must not displace an unclassifiable newer row" + pass "an unclassifiable status row keeps the ledger's newest-first precedence" +} + +# The other half of the no-widening criterion: a terminal `axi status` run with +# no live sibling on this worktree keeps reporting its own terminal outcome, in +# full run-step detail rather than degraded to the coarse listing. +test_terminal_run_without_live_sibling_is_unchanged() { + reset_fakes + local d base_head other_head short_base short_other out + d=$(new_case terminal-no-live-sibling) + make_repo_on_branch "$d/wt" fm/feat-nosibling + base_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" commit -q --allow-empty -m 'a second terminal run advanced the tip' + other_head=$(git -C "$d/wt" rev-parse HEAD) + git -C "$d/wt" reset -q --hard "$base_head" + short_base=$(git -C "$d/wt" rev-parse --short=7 "$base_head") + short_other=$(git -C "$d/wt" rev-parse --short=7 "$other_head") + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/nosibling.meta" "window=fm:fm-nosibling" "worktree=$d/wt" "kind=ship" + FM_FAKE_RUN_HEAD="$base_head" + FM_FAKE_AXI_STATUS="$(run_failed fm/feat-nosibling)" + FM_FAKE_RUNS_LIST="$(cat <<EOF + failed fm/feat-nosibling ${short_base} 2026-08-05 11:20 + completed fm/feat-nosibling ${short_other} 2026-08-05 10:05 +EOF +)" + out=$(run_crew_state "$d" nosibling) + assert_contains "$out" "state: failed" "a terminal run with no live sibling still reports its outcome" + assert_contains "$out" "source: run-step" "terminal outcome stays an attributed run-step verdict" + assert_contains "$out" "run failed" "the full axi-status detail is kept, not degraded to the listing" + FM_FAKE_RUNS_LIST="running fm/feat-nosibling ${short_base} 2026-08-05 11:20" + FM_FAKE_RUNS_RC=124 + out=$(run_crew_state "$d" nosibling) + assert_contains "$out" "run failed" "partial output from a failed ledger probe must not replace full terminal evidence" + pass "a terminal run with no live sibling is unchanged" +} + test_coarse_run_does_not_probe_other_branch_ci_log_for_ready_status() { reset_fakes local d short; d=$(new_case coarse-ready-other-log) @@ -3835,6 +4030,12 @@ test_full_and_coarse_paths_share_run_identity test_coarse_abandoned_run_keeps_terminal_identity test_coarse_completed_preserves_observed_merge test_coarse_same_run_reuses_branch_head_merge_evidence +test_terminal_corpse_loses_to_live_run_on_same_branch +test_runs_list_live_row_outranks_newer_terminal_row +test_unfetched_live_sibling_outranks_terminal_row_at_exact_head +test_only_terminal_rows_keep_newest_first_precedence +test_unknown_status_row_keeps_newest_first_precedence +test_terminal_run_without_live_sibling_is_unchanged test_coarse_run_does_not_probe_other_branch_ci_log_for_ready_status test_other_branch_run_ignored test_no_run_busy_pane diff --git a/tests/fm-daemon.test.sh b/tests/fm-daemon.test.sh index b6af1fa4aa8..b87b07fec46 100755 --- a/tests/fm-daemon.test.sh +++ b/tests/fm-daemon.test.sh @@ -1,10 +1,11 @@ #!/usr/bin/env bash # tests/fm-daemon.test.sh - supervise-daemon classifiers, the captain-relevant -# status-phrase matrix (a product contract), escalation batching/dedupe, afk -# presence-gating, and the injection-hardening units that an e2e cannot -# deterministically reach (persistent-Enter-swallow, max-defer wedge alarms, -# fm-send typed-plane swallow reporting, composer-pending ANSI parsing). The operator-visible -# inject flow lives in fm-afk-inject-e2e and fm-wake-daemon-lifecycle-e2e. +# status-phrase matrix (a product contract), escalation batching/dedupe, +# decision-owned queued-row suppression, afk presence-gating, and the +# injection-hardening units that an e2e cannot deterministically reach +# (persistent-Enter-swallow, max-defer wedge alarms, fm-send typed-plane swallow +# reporting, composer-pending ANSI parsing). The operator-visible inject flow +# lives in fm-afk-inject-e2e and fm-wake-daemon-lifecycle-e2e. set -u # shellcheck source=tests/wake-helpers.sh @@ -976,6 +977,8 @@ test_housekeeping_captain_held_resurfaces_and_resets() { local dir state fakebin win pane key age dir=$(make_supercase captain-held-resurface) state="$dir/state"; fakebin="$dir/fakebin" + FM_HOME="$dir" FM_STATE_OVERRIDE="$state" "$ROOT/bin/fm-afk-contract.sh" propose >/dev/null || fail "could not propose away posture" + FM_HOME="$dir" FM_STATE_OVERRIDE="$state" "$ROOT/bin/fm-afk-contract.sh" confirm >/dev/null || fail "could not confirm away posture" win="sess:fm-held-w11h"; pane="$dir/pane.txt" printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$state/held-w11h.status" printf 'idle prompt $\n' > "$pane" @@ -1106,6 +1109,55 @@ test_housekeeping_busy_declared_wait_matures_its_window() { pass "housekeeping matures a busy pane's declared-wait window into exactly one recheck per window" } +test_housekeeping_declared_time_controls_pause_recheck() { + local dir state fakebin task win pane key now future distant past escalations + dir=$(make_supercase pause-until-cadence) + state="$dir/state"; fakebin="$dir/fakebin" + task='held-until'; win="sess:fm-$task"; pane="$dir/pane.txt" + printf 'idle prompt $\n' > "$pane" + fm_write_meta "$state/$task.meta" "window=$win" "worktree=$dir/wt" "kind=ship" "harness=pi" + key=$(printf '%s' "$task" | tr ':/.' '___') + now=$(date +%s) + if [ "$(uname)" = Darwin ]; then + future=$(date -u -r "$((now + 120))" +%Y-%m-%dT%H:%M:%SZ) + distant=$(date -u -r "$((now + 31536000))" +%Y-%m-%dT%H:%M:%SZ) + past=$(date -u -r "$((now - 120))" +%Y-%m-%dT%H:%M:%SZ) + else + future=$(date -u -d "@$((now + 120))" +%Y-%m-%dT%H:%M:%SZ) + distant=$(date -u -d "@$((now + 31536000))" +%Y-%m-%dT%H:%M:%SZ) + past=$(date -u -d "@$((now - 120))" +%Y-%m-%dT%H:%M:%SZ) + fi + printf 'paused: waiting for release until %s\n' "$future" > "$state/$task.status" + echo $((now - 60)) > "$state/.subsuper-paused-$key" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + [ ! -s "$state/.subsuper-escalations" ] \ + || fail "a near-future declared time was rechecked before that time" + + printf 'paused: waiting for release until %s\n' "$distant" > "$state/$task.status" + echo $((now - 300)) > "$state/.subsuper-paused-$key" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + escalations=$(wc -l < "$state/.subsuper-escalations" | tr -d ' ') + [ "$escalations" -eq 1 ] || fail "a wrong-year declared time silenced daemon housekeeping beyond the cadence" + grep -F 'declared time is beyond the recheck cadence' "$state/.subsuper-escalations" >/dev/null \ + || fail "the bounded daemon recheck gave the wrong reason: $(cat "$state/.subsuper-escalations")" + grep -F 'declared clearing time has passed' "$state/.subsuper-escalations" >/dev/null \ + && fail "the bounded daemon recheck falsely claimed the future declared time passed" + + printf 'paused: waiting for release until %s\n' "$past" > "$state/$task.status" + date +%s > "$state/.subsuper-paused-$key" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + escalations=$(wc -l < "$state/.subsuper-escalations" | tr -d ' ') + [ "$escalations" -eq 2 ] || fail "a reached declared time did not trigger an immediate recheck" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ + FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + escalations=$(wc -l < "$state/.subsuper-escalations" | tr -d ' ') + [ "$escalations" -eq 2 ] || fail "a reached declared time bypassed the reset pause cadence" + pass "housekeeping bounds a distant declared time, defers to a near one, and rechecks a passed one at once" +} + # A pane still idle but whose status is no longer a pause (the crew changed state # without becoming busy) drops the marker - the signal path owns the new state, so # the pause recheck must not re-surface a stale pause reason. @@ -1494,6 +1546,122 @@ test_handle_wake_routes_self_and_escalate() { pass "handle_wake routes routine->self and captain->escalate" } +# Decision-owned queued rows are marked needs-decision:<files> by the watcher. +# The away-mode daemon must classify that payload the same way it classifies an +# ordinary signal: escalate once as the decision, suppress an unchanged repeat +# (queued-row and catch-all alike), and re-escalate when the status log grows. +# https://github.com/kunchenguid/firstmate/issues/4096 +test_needs_decision_queued_row_escalates_once_as_the_decision() { + local dir state fakebin status_file payload out + dir=$(make_supercase needs-decision-queued-row) + state="$dir/state" + fakebin="$dir/daemon-bin" + mkdir -p "$fakebin" + status_file="$state/decision-task.status" + printf 'working: setup\nneeds-decision [key=release]: pick A or B\n' > "$status_file" + payload="needs-decision: $status_file" + cat > "$fakebin/fm-wake-drain.sh" <<EOF +#!/usr/bin/env bash +if [ "\${1:-}" = --ack-through ]; then printf '%s\n' ack >> "$dir/acked"; exit 0; fi +printf '1\t1\tsignal\tdecision-task.status\t%s\n' "$payload" +printf 'WAKE_ACK_REQUIRED: retry --ack-through 1 --recovery-generation gen\n' >&2 +EOF + chmod +x "$fakebin/fm-wake-drain.sh" + + FM_DAEMON_DIR="$fakebin" FM_STATE_OVERRIDE="$state" FM_ESCALATE_BATCH_SECS=999 \ + handle_durable_wakes fallback "$state" \ + || fail "the first decision-owned queued row was not handled" + out=$(cat "$state/.subsuper-escalations" 2>/dev/null || true) + case "$out" in + *"unknown wake:"*) fail "a decision-owned wake was labelled unknown: $out" ;; + esac + case "$out" in + *"needs-decision [key=release]: pick A or B"*) ;; + *) fail "the first decision-owned wake was not presented as the decision: $out" ;; + esac + [ "$(wc -l < "$state/.subsuper-escalations" | tr -d ' ')" = 1 ] \ + || fail "the first decision-owned wake did not escalate exactly once: $out" + + : > "$state/.subsuper-escalations" + FM_DAEMON_DIR="$fakebin" FM_STATE_OVERRIDE="$state" FM_ESCALATE_BATCH_SECS=999 \ + handle_durable_wakes fallback "$state" \ + || fail "an unchanged decision-owned repeat was not handled" + [ ! -s "$state/.subsuper-escalations" ] \ + || fail "an unchanged open decision re-escalated: $(cat "$state/.subsuper-escalations")" + + rm -f "$state/.subsuper-last-scan" + FM_STATE_OVERRIDE="$state" housekeeping "$state" + [ ! -s "$state/.subsuper-escalations" ] \ + || fail "the catch-all scan re-escalated an already-surfaced open decision: $(cat "$state/.subsuper-escalations")" + + printf 'needs-decision [key=release]: pick A, C, or D\n' >> "$status_file" + : > "$state/.subsuper-escalations" + FM_DAEMON_DIR="$fakebin" FM_STATE_OVERRIDE="$state" FM_ESCALATE_BATCH_SECS=999 \ + handle_durable_wakes fallback "$state" \ + || fail "a changed decision-owned wake was not handled" + out=$(cat "$state/.subsuper-escalations" 2>/dev/null || true) + case "$out" in + *"unknown wake:"*) fail "a changed decision-owned wake was labelled unknown: $out" ;; + esac + case "$out" in + *"needs-decision [key=release]: pick A, C, or D"*) ;; + *) fail "a later status change did not re-escalate the decision: $out" ;; + esac + + : > "$state/.subsuper-escalations" + printf 'working: still going\n' > "$state/ordinary-routine.status" + FM_STATE_OVERRIDE="$state" handle_wake "signal: $state/ordinary-routine.status" "$state" + [ -s "$state/.subsuper-escalations" ] \ + && fail "an ordinary routine signal was escalated: $(cat "$state/.subsuper-escalations")" + printf 'done: PR https://example.test/pull/1\n' > "$state/ordinary-done.status" + FM_STATE_OVERRIDE="$state" handle_wake "signal: $state/ordinary-done.status" "$state" + out=$(cat "$state/.subsuper-escalations" 2>/dev/null || true) + case "$out" in + *"done: PR https://example.test/pull/1"*) ;; + *) fail "an ordinary captain-relevant signal lost its escalate behaviour: $out" ;; + esac + case "$out" in + *"unknown wake:"*) fail "an ordinary signal was labelled unknown: $out" ;; + esac + + pass "a decision-owned queued row escalates once as the decision, then suppresses until the status changes" +} + +# The watcher also marks a row decision-owned when its only new line is a +# captain-held transfer. fm-captain-hold.sh complete writes that line through the +# self-announced append and the hold stays durable in the backlog, so the daemon +# self-handles the row like any other captain-held line, now and on repeat. +test_captain_held_decision_owned_row_is_self_handled() { + local dir state fakebin status_file + dir=$(make_supercase captain-held-decision-owned-row) + state="$dir/state" + fakebin="$dir/daemon-bin" + mkdir -p "$fakebin" + status_file="$state/held-task.status" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$status_file" + cat > "$fakebin/fm-wake-drain.sh" <<EOF +#!/usr/bin/env bash +if [ "\${1:-}" = --ack-through ]; then exit 0; fi +printf '1\t1\tsignal\theld-task.status\t%s\n' "needs-decision: $status_file" +printf 'WAKE_ACK_REQUIRED: retry --ack-through 1 --recovery-generation gen\n' >&2 +EOF + chmod +x "$fakebin/fm-wake-drain.sh" + + FM_DAEMON_DIR="$fakebin" FM_STATE_OVERRIDE="$state" FM_ESCALATE_BATCH_SECS=999 \ + handle_durable_wakes fallback "$state" \ + || fail "the captain-held decision-owned row was not handled" + [ ! -s "$state/.subsuper-escalations" ] \ + || fail "a captain-held decision-owned row escalated: $(cat "$state/.subsuper-escalations")" + + FM_DAEMON_DIR="$fakebin" FM_STATE_OVERRIDE="$state" FM_ESCALATE_BATCH_SECS=999 \ + handle_durable_wakes fallback "$state" \ + || fail "an unchanged captain-held decision-owned repeat was not handled" + [ ! -s "$state/.subsuper-escalations" ] \ + || fail "an unchanged captain-held decision-owned repeat escalated: $(cat "$state/.subsuper-escalations")" + + pass "a captain-held decision-owned row is self-handled without escalation, now and on repeat" +} + test_inject_skip_forces_self() { local dir state dir=$(make_supercase skip) @@ -2823,6 +2991,7 @@ test_housekeeping_paused_resurfaces_and_resets test_housekeeping_captain_held_resurfaces_and_resets test_housekeeping_paused_resumed_cleared test_housekeeping_busy_declared_wait_matures_its_window +test_housekeeping_declared_time_controls_pause_recheck test_housekeeping_paused_unpaused_cleared test_housekeeping_replaced_wait_recheck_uses_the_base_width test_housekeeping_paused_recheck_survives_status_churn @@ -2838,6 +3007,8 @@ test_escalate_batches_into_one_digest test_escalate_batch_age_uses_first_append test_heartbeat_scan_dedup test_handle_wake_routes_self_and_escalate +test_needs_decision_queued_row_escalates_once_as_the_decision +test_captain_held_decision_owned_row_is_self_handled test_inject_skip_forces_self test_is_wake_reason_distinguishes_status_stdout test_terminal_stale_escalate_leaves_no_marker diff --git a/tests/fm-dashboard-events.test.sh b/tests/fm-dashboard-events.test.sh index ef4f43e8a39..4a2e54b359f 100755 --- a/tests/fm-dashboard-events.test.sh +++ b/tests/fm-dashboard-events.test.sh @@ -766,6 +766,7 @@ make_autoarm_primary() { # <dir> <close-kind> cat > "$dir/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash printf 'watcher: started pid=%s (beacon fresh)\n' "$$" +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" printf 'stale: fixture-win actionable\n' exit 0 SH @@ -801,12 +802,14 @@ run_autoarm() { # <dir> <stderr-file> # the only fields two separate runs cannot share - the identity records the # claiming process's own start time and its home-scoped command line - so those # three are normalized and everything else, the epoch sequence, the recorded -# outcome, and the presence of each marker, has to match. Normalizing the -# identity here would hide a claim that recorded none, so the caller asserts the +# outcome, recovery generation, and marker presence must match. +# The session pid is also process-specific and is checked before normalization. +# Normalizing the identity would hide a claim that recorded none, so the caller asserts the # line is present before comparing. autoarm_ledger() { # <dir> local marker sed -e '1s/owner_pid=[0-9]*/owner_pid=PID/' -e '1s/updated_at=[0-9]*/updated_at=AT/' \ + -e '1s/session_pid=[0-9]*/session_pid=SESSION/' \ -e '2s/^..*$/identity=RECORDED/' \ "$1/state/.claude-autoarm-epoch" 2>/dev/null || printf 'epoch absent\n' for marker in .claude-autoarm-failure-notified .claude-autoarm-failure-alarmed; do @@ -861,6 +864,12 @@ test_instrumentation_cannot_change_what_the_stop_autoarm_decides() { || fail "the $kind-close auto-arm's ledger changed when the emitter ran beside it" done + for home in "$root/autoarm-actionable-bare" "$root/autoarm-actionable-with"; do + grep -q "session_pid=$(cat "$home/state/.lock") recovery_generation=fixture-generation" \ + "$home/state/.claude-autoarm-epoch" \ + || fail "the actionable close did not bind its actual session and recovery generation" + done + # Both closes have to be the ones this test believes it exercised, or the # comparison above is two identical no-ops. grep -q 'outcome=rewake' "$root/autoarm-actionable-bare/state/.claude-autoarm-epoch" \ diff --git a/tests/fm-guard-stale-banner.test.sh b/tests/fm-guard-stale-banner.test.sh index 0da0d9c8337..0ee4baac045 100755 --- a/tests/fm-guard-stale-banner.test.sh +++ b/tests/fm-guard-stale-banner.test.sh @@ -74,6 +74,23 @@ run_guard_case_autoarm() { "$ROOT/bin/fm-guard.sh" 2>&1 } +# Age the rewake ledger into the past so a passing long-turn case cannot rest on +# epoch freshness: a handling turn that has already outrun grace still has this +# shape, which is the false alarm this suite now pins. +record_aged_rewake_epoch() { + local home=$1 session_pid=$2 recovery=${3:-guard-test-generation} + printf 'epoch=7 owner_pid=1 outcome=rewake updated_at=1 session_pid=%s recovery_generation=%s\n' \ + "$session_pid" "$recovery" > "$home/state/.claude-autoarm-epoch" + printf 'acked:handling:%s\n' "$recovery" > "$home/state/.watcher-down" + touch -t 201901010000 "$home/state/.last-watcher-beat" + touch -t 202001010000 "$home/state/.claude-autoarm-epoch" +} + +record_session_lock_pid() { + local home=$1 pid=$2 + printf '%s\n' "$pid" > "$home/state/.lock" +} + # The Pi extension model: .pi/extensions/fm-primary-pi-watch.ts tears the watcher # down on every actionable wake and spawns the replacement itself, so the lock is # legitimately unheld during a hand-off. @@ -472,6 +489,133 @@ test_autoarm_stale_episode_is_stable() { pass "fm-guard stale banner: auto-arm stale episode stays one episode across calls" } +# The send-time false alarm on a long Claude handling turn: the between-turns +# watcher has already exited, the beacon is older than grace, and the auto-arm +# ledger still shows a healthy rewake with no failure markers while the session +# lock names a live pid. Turn-end will re-arm, so the pull guard must stay silent. +test_autoarm_long_handling_turn_stays_silent() { + local dir home out pid + dir=$(make_guard_case autoarm-long-turn) + home=$(case_home "$dir") + sleep 60 & + pid=$! + record_aged_rewake_epoch "$home" "$pid" + record_session_lock_pid "$home" "$pid" + out=$(run_guard_case_autoarm "$dir") + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + [ -z "$out" ] \ + || fail "a long auto-arm handling turn past grace must stay silent, got: $out" + assert_absent "$home/state/.guard-watcher-stale-banner" \ + "a healthy long handling turn must not open a down-episode" + pass "fm-guard stale banner: auto-arm long handling turn with a healthy rewake stays silent" +} + +# Drive the long-turn signals apart on the same stale beacon. Losing any one +# healthy-generation signal must restore the banner; the stale beacon alone +# is not enough to stay quiet, and adding a failure marker is not either. +test_autoarm_long_turn_requires_every_healthy_signal() { + local dir home out pid replacement_pid='' case_name + for case_name in no-epoch failed-outcome failure-notified failure-alarmed dead-lock missing-lock changed-lock moved-recovery later-beacon; do + dir=$(make_guard_case "autoarm-long-turn-$case_name") + home=$(case_home "$dir") + sleep 60 & + pid=$! + record_aged_rewake_epoch "$home" "$pid" + record_session_lock_pid "$home" "$pid" + case "$case_name" in + no-epoch) + rm -f "$home/state/.claude-autoarm-epoch" + ;; + failed-outcome) + printf 'epoch=7 owner_pid=1 outcome=failed updated_at=1\n' > "$home/state/.claude-autoarm-epoch" + touch -t 202001010000 "$home/state/.claude-autoarm-epoch" + ;; + failure-notified) + : > "$home/state/.claude-autoarm-failure-notified" + ;; + failure-alarmed) + : > "$home/state/.claude-autoarm-failure-alarmed" + ;; + dead-lock) + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + pid= + ;; + missing-lock) + rm -f "$home/state/.lock" + ;; + changed-lock) + sleep 60 & + replacement_pid=$! + record_session_lock_pid "$home" "$replacement_pid" + ;; + moved-recovery) + printf 'pending:handling:later-turn-generation\n' > "$home/state/.watcher-down" + ;; + later-beacon) + touch -t 202101010000 "$home/state/.last-watcher-beat" + ;; + esac + out=$(run_guard_case_autoarm "$dir") + [ -z "$pid" ] || kill "$pid" 2>/dev/null || true + [ -z "$pid" ] || wait "$pid" 2>/dev/null || true + [ -z "$replacement_pid" ] || kill "$replacement_pid" 2>/dev/null || true + [ -z "$replacement_pid" ] || wait "$replacement_pid" 2>/dev/null || true + replacement_pid= + [ "$(count_text "$out" "WATCHER DOWN - SUPERVISION IS OFF")" -eq 1 ] \ + || fail "auto-arm long-turn health must not survive $case_name; guard output: $out" + assert_contains "$out" "no watcher has a fresh beacon" \ + "a genuine auto-arm lapse with $case_name must still name the stale beacon" + done + pass "fm-guard stale banner: every auto-arm long-turn healthy signal is load-bearing" +} + +# An open arming claim is between-turn startup, not evidence that the current +# handling turn came from a healthy rewake. +test_autoarm_open_claim_does_not_explain_stale_beacon() { + local dir home out pid identity + dir=$(make_guard_case autoarm-open-claim) + home=$(case_home "$dir") + sleep 60 & + pid=$! + identity=$(fm_test_pid_identity "$pid") \ + || fail "could not compute the open-claim owner identity" + [ -n "$identity" ] || fail "open-claim owner identity was empty" + printf 'epoch=3 owner_pid=%s outcome=arming updated_at=%s\n%s\n' \ + "$pid" "$(date +%s)" "$identity" > "$home/state/.claude-autoarm-epoch" + out=$(run_guard_case_autoarm "$dir") + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + [ "$(count_text "$out" "WATCHER DOWN - SUPERVISION IS OFF")" -eq 1 ] \ + || fail "an open auto-arm claim must not suppress a stale-beacon alarm: $out" + pass "fm-guard stale banner: an open auto-arm claim does not explain a stale beacon" +} + +# The long-turn tolerance is a Claude auto-arm carve-out. The same leftover +# rewake ledger must not silence Pi/extension or persistent primaries. +test_autoarm_long_turn_does_not_silence_other_models() { + local dir home out pid model + for model in persistent extension; do + dir=$(make_guard_case "autoarm-carveout-$model") + home=$(case_home "$dir") + sleep 60 & + pid=$! + record_aged_rewake_epoch "$home" "$pid" + record_session_lock_pid "$home" "$pid" + if [ "$model" = extension ]; then + out=$(run_guard_case_extension "$dir") + else + out=$(run_guard_case "$dir") + fi + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + [ "$(count_text "$out" "WATCHER DOWN - SUPERVISION IS OFF")" -eq 1 ] \ + || fail "a $model primary must still alarm on a stale beacon despite a leftover rewake epoch: $out" + done + pass "fm-guard stale banner: auto-arm long-turn health does not leak to other models" +} + test_persistent_no_watcher_banner_names_missing_process() { local dir out dir=$(make_guard_case persistent-no-watcher-reason) @@ -818,6 +962,10 @@ test_extension_live_watcher_is_healthy_without_ownership_evidence test_autoarm_fresh_beacon_without_watcher_is_healthy test_autoarm_stale_beacon_alarms_with_correct_reason test_autoarm_stale_episode_is_stable +test_autoarm_long_handling_turn_stays_silent +test_autoarm_long_turn_requires_every_healthy_signal +test_autoarm_open_claim_does_not_explain_stale_beacon +test_autoarm_long_turn_does_not_silence_other_models test_persistent_no_watcher_banner_names_missing_process test_persistent_no_watcher_episode_survives_beacon_touch test_fresh_beacon_without_live_watcher_stays_alarm diff --git a/tests/fm-issue-writeback.test.sh b/tests/fm-issue-writeback.test.sh index f310ee19d44..82eca83391e 100755 --- a/tests/fm-issue-writeback.test.sh +++ b/tests/fm-issue-writeback.test.sh @@ -1509,7 +1509,9 @@ test_an_unknown_milestone_is_a_usage_error() { test_the_merge_path_posts_its_own_milestones() { local dir out rc body dir=$(board_case mergepath) - mkdir -p "$dir/wt" "$dir/projects/widget" + mkdir -p "$dir/wt" "$dir/projects/widget" "$dir/data" + # The merge guard must resolve a real home before proving no delivery hold. + cp "$ROOT/.tasks.toml" "$dir/.tasks.toml" # `gh api` is the fake GitHub; every other `gh` call fm-pr-check.sh makes # answers as the PR-head lookup, and gh-axi records the merge. mv "$dir/fakebin/gh" "$dir/fakebin/gh-api-fake" @@ -1563,7 +1565,9 @@ SH test_a_refusing_tracker_never_makes_a_completed_merge_look_retryable() { local dir out rc dir=$(board_case mergepath-refused) - mkdir -p "$dir/wt" "$dir/projects/widget" + mkdir -p "$dir/wt" "$dir/projects/widget" "$dir/data" + # The merge guard must resolve a real home before proving no delivery hold. + cp "$ROOT/.tasks.toml" "$dir/.tasks.toml" mv "$dir/fakebin/gh" "$dir/fakebin/gh-api-fake" cat > "$dir/fakebin/gh" <<'SH' #!/usr/bin/env bash diff --git a/tests/fm-launch-lib.test.sh b/tests/fm-launch-lib.test.sh index c3e4e3fd61e..a2a435556dd 100755 --- a/tests/fm-launch-lib.test.sh +++ b/tests/fm-launch-lib.test.sh @@ -288,6 +288,24 @@ test_adapter_bindings_are_single_pass() { pass "adapter path bindings render once and reject duplicate or incomplete bindings" } +test_pi_native_effort_uses_provider_flag() { + local harness model output + for harness in pi pi-signed; do + assert_eq "$(fm_launch_effort_flag "$harness" ultra codex-native/gpt-6)" "--codex-effort 'ultra' " \ + "$harness native ultra must reach the provider flag" + assert_eq "$(fm_launch_effort_flag "$harness" high codex-native/gpt-6)" "--thinking 'high' " \ + "$harness ordinary effort must retain the thinking flag" + for model in '' default openai-codex/gpt-6 codex-native/; do + if output=$(fm_launch_effort_flag "$harness" ultra "$model" 2>&1); then + fail "$harness accepted native ultra without an explicit native model: $model ($output)" + fi + done + done + assert_eq "$(fm_launch_effort_flag omp ultra codex-native/gpt-6)" "" \ + "OMP must not acquire Pi native provider effort" + pass "Pi native ultra requires an explicit provider model and preserves ordinary effort rendering" +} + test_rovo_override_keeps_paths_and_effort_together() { local data state flag parsed effort data="$TMP_LAUNCH_ROOT/rovo data'quote"; state="$TMP_LAUNCH_ROOT/rovo state" @@ -310,6 +328,7 @@ test_rovo_override_keeps_paths_and_effort_together() { } test_adapter_bindings_are_single_pass +test_pi_native_effort_uses_provider_flag test_rovo_override_keeps_paths_and_effort_together test_render_substitutes_operational_input test_render_rejects_unknown_placeholder diff --git a/tests/fm-lint.test.sh b/tests/fm-lint.test.sh index 338ec7df49b..57c30df3497 100755 --- a/tests/fm-lint.test.sh +++ b/tests/fm-lint.test.sh @@ -530,7 +530,7 @@ test_changed_mode_drops_external_sources_and_excludes_cross_file_codes() { "changed-mode local lint did not disclose dropped source following" assert_grep $'analysis_mode\tlocal' "$telemetry" \ "telemetry did not record local analysis mode" - assert_grep $'source_directives\t3' "$telemetry" \ + assert_grep $'source_directives\t4' "$telemetry" \ "telemetry did not count the changed root's source directives" assert_grep $'source_followed_directives\t0' "$telemetry" \ "telemetry reported followed sources in no-external-sources mode" @@ -1142,6 +1142,80 @@ SH pass "fm-lint.sh catches a real lint defect the old no-op gate passed" } +test_rejects_direct_beads_cli_invocations() { + local tmp fakebin log lint_copy invocation out rc + tmp=$(fm_test_tmproot fm-lint-backend-purity) + fakebin=$(fm_fakebin "$tmp") + log="$tmp/shellcheck.log" + mkdir -p "$tmp/repo/bin/backends" "$tmp/repo/tests" + lint_copy="$tmp/repo/bin/fm-lint.sh" + cp "$LINT" "$lint_copy" + cat > "$tmp/repo/bin/fm-lint-workflows.sh" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + cat > "$tmp/repo/bin/backends/noop.sh" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + cat > "$tmp/repo/tests/noop.test.sh" <<'SH' +#!/usr/bin/env bash +exit 0 +SH + chmod +x "$lint_copy" "$tmp/repo/bin/fm-lint-workflows.sh" + fm_lint_stub_shellcheck "$fakebin" "$log" + + for invocation in \ + 'bd update fm-example --status in_progress' \ + 'BD_ACTOR=firstmate bd update fm-example --status closed' \ + 'env bd close fm-example' \ + 'env -i BD_ACTOR=firstmate bd close fm-example' \ + 'env -u BD_ACTOR bd close fm-example' \ + 'env -- bd close fm-example' \ + '/usr/local/bin/bd close fm-example' \ + '"/usr/local/bin/bd" close fm-example' \ + "'/usr/local/bin/bd' close fm-example" \ + "b'd' close fm-example" \ + "/usr/local/bin/b'd' close fm-example" \ + "\$'bd' close fm-example" \ + '$"bd" close fm-example' \ + "\$'\\x62\\x64' close fm-example" \ + "\$'\\142\\144' close fm-example" \ + "b\$'\\x64' close fm-example" + do + printf '#!/usr/bin/env bash\n%s\n' "$invocation" > "$tmp/repo/bin/direct-beads.sh" + rc=0 + out=$(cd "$tmp/repo" && CI=true PATH="$fakebin:$PATH" "$lint_copy" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "lint accepted a direct Beads CLI invocation: $invocation" + assert_contains "$out" "direct Beads CLI invocation bypasses tasks-axi" \ + "lint did not identify the backend-boundary violation: $invocation" + done + pass "fm-lint.sh rejects direct Beads CLI invocations in firstmate core" +} + +test_rejects_direct_beads_cli_in_explicit_core_path() { + local tmp fakebin log lint_copy target spelling out rc + tmp=$(fm_test_tmproot fm-lint-explicit-backend-purity) + fakebin=$(fm_fakebin "$tmp") + log="$tmp/shellcheck.log" + mkdir -p "$tmp/repo/bin/backends" + lint_copy="$tmp/repo/bin/fm-lint.sh" + target="$tmp/repo/bin/direct-beads.sh" + cp "$LINT" "$lint_copy" + printf '#!/usr/bin/env bash\nbd close fm-example\n' > "$target" + chmod +x "$lint_copy" + fm_lint_stub_shellcheck "$fakebin" "$log" + + for spelling in bin/direct-beads.sh bin/../bin/direct-beads.sh; do + rc=0 + out=$(cd "$tmp/repo" && PATH="$fakebin:$PATH" "$lint_copy" "$spelling" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "explicit core path bypassed backend-purity lint: $spelling" + assert_contains "$out" "direct Beads CLI invocation bypasses tasks-axi" \ + "explicit core path did not report the backend-boundary violation: $spelling" + done + pass "fm-lint.sh enforces backend purity for explicit core paths" +} + test_ignores_ambient_shellcheck_opts() { if ! pinned_ready; then pass "SKIP (ShellCheck $REQUIRED not resolved): ambient options regression check" @@ -1431,6 +1505,8 @@ test_installer_rejects_unsupported_platform test_missing_shellcheck_fails_closed test_rejects_wrong_shellcheck_version test_catches_a_real_lint_defect +test_rejects_direct_beads_cli_invocations +test_rejects_direct_beads_cli_in_explicit_core_path test_ignores_ambient_shellcheck_opts test_clean_fixture_passes test_jobs_are_deterministic_and_complete diff --git a/tests/fm-mail-check.test.sh b/tests/fm-mail-check.test.sh new file mode 100644 index 00000000000..36152594924 --- /dev/null +++ b/tests/fm-mail-check.test.sh @@ -0,0 +1,423 @@ +#!/usr/bin/env bash +# Behavior tests for bin/fm-mail-check.sh, the standing received-mail check. +# +# Two surfaces are exercised through their executable interfaces: +# +# * arming/disarming state/mail.check.sh with its trust binding, including +# the refusal paths (symlink at the shim path, missing mail plane); +# +# * the `check` action itself, which runs the real fm-mail.sh poll against a +# scratch home whose .env and fake python3 decide the outcome. The cases +# that matter are the reporting contract: a successful poll that surfaces +# new mail emits one wake line (the poll still also surfaces new mail as +# durable wakes), a failing poll reports one line, a proven no-op stays +# silent, and fail-closed-after-queue or timeout still doorbells. +# +# No case ever contacts a real IMAP or SMTP server. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +CHECK="$ROOT/bin/fm-mail-check.sh" +TMP_ROOT=$(fm_test_tmproot fm-mail-check) + +# Pin the watcher bound and clear any ambient mail config so the poll's +# readiness decision comes from the fixture .env alone. The check binary is an +# explicit argument so the missing-mail-plane case runs a copy without fm-mail.sh. +run_check() { + local home=$1 out=$2 check=$3 + shift 3 + local status=0 + env -u FM_MAIL_USER -u FM_MAIL_PASS -u FM_IMAP_HOST -u FM_SMTP_HOST \ + -u FM_MAIL_CHECK_BUDGET \ + FM_CHECK_TIMEOUT=30 \ + "$@" FM_HOME="$home" PATH="$FAKEBIN:$PATH" \ + "$check" check >"$out" 2>&1 || status=$? + expect_code 0 "$status" "check exit" +} + +# make_home <name>: a scratch home without a mail plane yet. +make_home() { + local name=$1 home + home="$TMP_ROOT/$name" + mkdir -p "$home/state" + printf '%s\n' "$home" +} + +# write_env <home>: the four FM_MAIL_* values the poll requires. +write_env() { + local home=$1 + printf '%s\n' \ + 'FM_MAIL_USER=test@example.invalid' \ + 'FM_MAIL_PASS=test-pass' \ + 'FM_IMAP_HOST=imap.test.invalid' \ + 'FM_SMTP_HOST=smtp.test.invalid' > "$home/.env" +} + +# enter_mailbox <home> <mailbox-generator-command...>: wires the scratch home's +# bin to the real wake library and FAKEBIN to a python3 running <cmd> so a poll +# against this home can surface and durably record a wake. +enter_mailbox() { + local home=$1 generator=$2 + mkdir -p "$home/bin" "$FAKEBIN" + [ -e "$home/bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$home/bin/fm-wake-lib.sh" + printf '%s\n' "$generator" > "$FAKEBIN/python3" + chmod +x "$FAKEBIN/python3" +} + +FAKEBIN="$TMP_ROOT/fakebin" + +test_help_and_usage() { + local out rc=0 + out=$("$CHECK" --help 2>&1) || rc=$? + expect_code 0 "$rc" "--help must exit 0" + assert_contains "$out" "check" "--help lists the check action" + assert_contains "$out" "arm" "--help lists the arm action" + assert_contains "$out" "disarm" "--help lists the disarm action" + rc=0 + out=$("$CHECK" bogus 2>&1) || rc=$? + expect_code 2 "$rc" "unknown action must exit 2" + assert_contains "$out" "unknown action" "unknown action is refused loudly" + pass "fm-mail-check: help and usage plumbing" +} + +test_arm_writes_and_binds_the_check_and_disarm_removes_it() { + local home out + home=$(make_home arm) + write_env "$home" + out=$(FM_HOME="$home" "$CHECK" arm 2>&1) || fail "arm must succeed: $out" + assert_contains "$out" "armed: state/mail.check.sh" "arm names the shim it wrote" + assert_present "$home/state/mail.check.sh" "arm writes the check shim" + assert_present "$home/state/mail.check-trust" "arm binds the shim for the watcher" + assert_contains "$(cat "$home/state/mail.check.sh")" "fm-mail-check.sh check" "shim dispatches the check action" + assert_contains "$(cat "$home/state/mail.check.sh")" "FM_HOME=$home" "shim pins the absolute home" + + out=$(FM_HOME="$home" "$CHECK" arm 2>&1) || fail "re-arm must succeed: $out" + assert_contains "$out" "armed" "re-arm stays armed" + + out=$(FM_HOME="$home" "$CHECK" disarm 2>&1) || fail "disarm must succeed: $out" + assert_absent "$home/state/mail.check.sh" "disarm removes the check shim" + assert_absent "$home/state/mail.check-trust" "disarm removes the trust binding" + assert_absent "$home/state/.mail-check" "disarm removes the report record" + pass "fm-mail-check: arm writes and binds, re-arm is idempotent, disarm removes" +} + +test_arm_resolves_a_relative_home_into_the_shim() { + local home rel out + home=$(make_home relative) + write_env "$home" + rel="$(basename "$home")" + out=$(cd "$TMP_ROOT" && env FM_HOME="$rel" "$CHECK" arm 2>&1) || fail "arm with a relative FM_HOME must succeed: $out" + assert_contains "$(cat "$home/state/mail.check.sh")" "export FM_HOME=$home" "the shim pins the resolved absolute home, not the relative spelling" + pass "fm-mail-check: arm resolves a relative home into the shim" +} + +test_arm_refuses_a_symlink_at_the_shim_path() { + local home target out rc=0 + home=$(make_home symlink) + write_env "$home" + target="$TMP_ROOT/outside" + mkdir -p "$target" + printf '#!/usr/bin/env bash\n' > "$target/mail.check.sh" + ln -s "$target/mail.check.sh" "$home/state/mail.check.sh" + out=$(FM_HOME="$home" "$CHECK" arm 2>&1) || rc=$? + expect_code 1 "$rc" "arm must refuse a symlink at the shim path" + assert_contains "$out" "could not write" "arm reports the shim write failure" + assert_absent "$home/state/mail.check-trust" "no trust binding is left behind by a refused arm" + pass "fm-mail-check: arm refuses a symlink at the shim path" +} + +test_arm_refuses_without_the_mail_plane() { + local tmpbin home out rc=0 + # A copy of the check tool with no fm-mail.sh beside it is a home whose mail + # plane has not landed yet: arming must refuse instead of delegating silence. + tmpbin="$TMP_ROOT/plane/bin" + home="$TMP_ROOT/plane/home" + mkdir -p "$tmpbin" "$home/state" + cp "$ROOT/bin/fm-mail-check.sh" "$tmpbin/" + for lib in fm-timeout-lib.sh fm-pr-lib.sh fm-line-cap-lib.sh fm-check-lib.sh; do + [ -e "$tmpbin/$lib" ] || ln -s "$ROOT/bin/$lib" "$tmpbin/$lib" + done + out=$(FM_HOME="$home" "$tmpbin/fm-mail-check.sh" arm 2>&1) || rc=$? + expect_code 1 "$rc" "arm must refuse when the mail plane is missing" + assert_contains "$out" "mail plane is missing" "arm names the missing plane" + assert_absent "$home/state/mail.check.sh" "a refused arm writes no shim" + pass "fm-mail-check: arm refuses without the mail plane" +} + +test_successful_poll_with_new_mail_emits_one_wake_line() { + local home out wakeq + home=$(make_home success) + write_env "$home" + enter_mailbox "$home" \ + 'printf "uidvalidity\\t20002\\n" +printf "42\\t2026-09-05T00:00:00Z\\talice@example.com\\tHello\\n"' + out="$home/out.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: new mail: woke for 42" "a successful poll that surfaces new mail emits one wake line" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "a successful new-mail poll reports exactly one line: $(cat "$out")" + assert_present "$home/state/.mail-check" "a successful poll records its outcome" + assert_contains "$(cat "$home/state/.mail-check")" "fm-mail-check-v1" "the record carries its schema" + assert_contains "$(cat "$home/state/.mail-check")" "reported=new mail: woke for 42" "the record carries the reported new-mail finding" + wakeq="$home/state/.wake-queue" + assert_contains "$(cat "$wakeq" 2>/dev/null)" "mail from alice@example.com" "the check-run poll still surfaces new mail as a durable wake" + assert_contains "$(cat "$home/state/.mail-seen" 2>/dev/null)" "42" "the check-run poll still advances the inbox cursor" + pass "fm-mail-check: a successful poll that surfaces new mail emits one wake line" +} + +test_failure_is_reported_once_until_it_changes() { + local home out + home=$(make_home failure) + write_env "$home" + enter_mailbox "$home" \ + 'printf "fm-mail poll error: connection refused\\n" >&2 +exit 1' + + out="$home/out.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: fm-mail poll error: connection refused" "a failing poll reports its cause in one line" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "a failing poll reports exactly one line: $(cat "$out")" + + out="$home/out2.txt" + run_check "$home" "$out" "$CHECK" + [ ! -s "$out" ] || fail "the same failure must not be reported again: $(cat "$out")" + assert_contains "$(cat "$home/state/.mail-check")" "reported=fm-mail poll error: connection refused" "the record carries the reported finding" + + # A healthy poll clears the record, so the next failure is news again. + enter_mailbox "$home" \ + 'printf "uidvalidity\\t20003\\n"' + out="$home/out3.txt" + run_check "$home" "$out" "$CHECK" + [ ! -s "$out" ] || fail "a recovered poll must stay silent: $(cat "$out")" + + enter_mailbox "$home" \ + 'printf "fm-mail poll error: connection refused\\n" >&2 +exit 1' + out="$home/out4.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: fm-mail poll error: connection refused" "a failure after a healthy poll is news again" + pass "fm-mail-check: a poll failure is reported once and re-reported after recovery" +} + +test_unconfigured_home_is_reported_once() { + local home out + home=$(make_home unconfigured) + # No .env: the poll names the missing value, and the check turns that into + # its one line instead of leaving an armed channel quiet. + out="$home/out.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: missing required" "an unconfigured home reports the missing setup" + assert_contains "$(cat "$out")" "FM_MAIL_USER" "the report names the missing variable" + out="$home/out2.txt" + run_check "$home" "$out" "$CHECK" + [ ! -s "$out" ] || fail "the unconfigured state must not repeat: $(cat "$out")" + pass "fm-mail-check: an unconfigured home is reported once, not every poll" +} + +test_slow_poll_times_out_and_is_reported() { + local home out + home=$(make_home slow) + write_env "$home" + enter_mailbox "$home" \ + 'sleep 6' + out="$home/out.txt" + run_check "$home" "$out" "$CHECK" FM_MAIL_CHECK_BUDGET=5 + assert_contains "$(cat "$out")" "mail: poll did not finish within the 5s budget" "a poll past its budget is reported, not ignored" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "a poll timeout reports exactly one line: $(cat "$out")" + pass "fm-mail-check: a slow poll times out into a one-line report" +} + +test_repeated_failure_that_queued_new_mail_still_wakes() { + # A poll can append durable mail wakes and then fail with the same cause as + # the last check. Difference-record silence would leave those wakes queued + # and unacted; the check must print again so the watcher wakes firstmate. + local tmpbin home out check_bin + tmpbin="$TMP_ROOT/repeat-wake/bin" + home="$TMP_ROOT/repeat-wake/home" + mkdir -p "$tmpbin" "$home/state" + check_bin="$tmpbin/fm-mail-check.sh" + cp "$ROOT/bin/fm-mail-check.sh" "$tmpbin/" + for lib in fm-timeout-lib.sh fm-pr-lib.sh fm-line-cap-lib.sh fm-check-lib.sh; do + [ -e "$tmpbin/$lib" ] || ln -s "$ROOT/bin/$lib" "$tmpbin/$lib" + done + printf '%s\n' '#!/usr/bin/env bash' 'echo "fm-mail: woke for 42"' 'echo "fm-mail: connection refused" >&2' 'exit 1' > "$tmpbin/fm-mail.sh" + chmod +x "$tmpbin/fm-mail.sh" + + out="$home/out1.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "mail: connection refused" "the first failed poll that queued new mail reports the failure" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "the first report is exactly one line: $(cat "$out")" + + out="$home/out2.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "mail: connection refused" "the same failure must still print when that poll queued new mail" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "the repeat report is exactly one line: $(cat "$out")" + pass "fm-mail-check: a repeated failure that queued new mail still wakes" +} + +test_repeated_timeout_still_wakes() { + # A timeout can kill the poll after wake_for queued mail and before the + # woke-for line is printed. Difference-record silence would then leave that + # mail undrained; a timeout always prints so the watcher wakes. + local tmpbin home out check_bin + tmpbin="$TMP_ROOT/repeat-timeout/bin" + home="$TMP_ROOT/repeat-timeout/home" + mkdir -p "$tmpbin" "$home/state" + check_bin="$tmpbin/fm-mail-check.sh" + cp "$ROOT/bin/fm-mail-check.sh" "$tmpbin/" + for lib in fm-timeout-lib.sh fm-pr-lib.sh fm-line-cap-lib.sh fm-check-lib.sh; do + [ -e "$tmpbin/$lib" ] || ln -s "$ROOT/bin/$lib" "$tmpbin/$lib" + done + printf '%s\n' '#!/usr/bin/env bash' 'exit 124' > "$tmpbin/fm-mail.sh" + chmod +x "$tmpbin/fm-mail.sh" + + out="$home/out1.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "mail: poll did not finish within" "the first timeout reports" + + out="$home/out2.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "mail: poll did not finish within" "a repeated timeout must still print so queued mail is not stranded" + pass "fm-mail-check: a repeated timeout still wakes" +} + +test_fail_closed_poll_after_wake_reports_the_failure() { + # A poll can publish a wake and then fail closed (stale retry still on disk + # and unwritable). The standing check must report that failure, not the + # earlier success-wake line, or the news key hides the real condition. + local home out + home=$(make_home fail-closed-after-wake) + write_env "$home" + enter_mailbox "$home" \ + 'printf "uidvalidity\\t90009\\n" +printf "77\\t2026-09-05T00:00:00Z\\tfrom@x\\tHello\\tok\\n"' + printf 'uidvalidity=90009\n' > "$home/state/.mail-seen" + printf '77\n' > "$home/state/.mail-retry" + chmod 0000 "$home/state/.mail-retry" + out="$home/out.txt" + run_check "$home" "$out" "$CHECK" + chmod 0600 "$home/state/.mail-retry" + assert_contains "$(cat "$out")" "mail: could not clear retry for recovered 77 after publish" "a fail-closed poll after a wake reports the failure" + assert_not_contains "$(cat "$out")" "woke for 77" "the standing check must not treat the success-wake line as the failure" + assert_contains "$(cat "$home/state/.mail-check")" "reported=could not clear retry for recovered 77 after publish" "the news key is the failure, not the wake" + pass "fm-mail-check: a fail-closed poll after a wake reports the failure, not the wake" +} + +test_repeated_status4_fail_closed_still_wakes() { + # Status 4: wake_for published the row, then retry-clear failed, so the poll + # returns 1 without printing woke-for. A second identical poll must still + # print so the watcher drains the queued check: mail <uid> row. + local home out wakeq + home=$(make_home repeat-status4) + write_env "$home" + enter_mailbox "$home" \ + 'printf "uidvalidity\\t90009\\n" +printf "77\\t2026-09-05T00:00:00Z\\tfrom@x\\tHello\\tretry\\n"' + printf 'uidvalidity=90009\n77\n' > "$home/state/.mail-seen" + printf '77\n' > "$home/state/.mail-retry" + chmod 0000 "$home/state/.mail-retry" + + out="$home/out1.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: could not clear retry for recovered 77 after publish" "the first status-4 poll reports the failure" + assert_not_contains "$(cat "$out")" "woke for 77" "status 4 does not print woke-for" + wakeq="$home/state/.wake-queue" + assert_contains "$(cat "$wakeq" 2>/dev/null)" "check: mail 77" "status 4 leaves the recovery wake queued" + + out="$home/out2.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: could not clear retry for recovered 77 after publish" "a repeated status-4 poll still prints so the queued mail is not stranded" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "the repeat report is exactly one line: $(cat "$out")" + assert_contains "$(cat "$wakeq" 2>/dev/null)" "check: mail 77" "the queued recovery wake is still present" + chmod 0600 "$home/state/.mail-retry" + pass "fm-mail-check: a repeated status-4 fail-closed poll still wakes" +} + +test_repeated_status2_stays_queued_still_wakes() { + # Status 2: the wake stays queued with no durable record and no woke-for line. + # A second identical diagnostic must still print. + local tmpbin home out check_bin + tmpbin="$TMP_ROOT/repeat-status2/bin" + home="$TMP_ROOT/repeat-status2/home" + mkdir -p "$tmpbin" "$home/state" + check_bin="$tmpbin/fm-mail-check.sh" + cp "$ROOT/bin/fm-mail-check.sh" "$tmpbin/" + for lib in fm-timeout-lib.sh fm-pr-lib.sh fm-line-cap-lib.sh fm-check-lib.sh; do + [ -e "$tmpbin/$lib" ] || ln -s "$ROOT/bin/$lib" "$tmpbin/$lib" + done + cat > "$tmpbin/fm-mail.sh" <<EOF +#!/usr/bin/env bash +printf '1\t1\tcheck\tmail:9\tcheck: mail 9 - stays queued\\n' >> "\$FM_HOME/state/.wake-queue" +echo "fm-mail: wake for 9 could not be rolled back or durably recorded; the wake stays queued and the next poll heals it - a possible duplicate, never a lost mail" >&2 +exit 1 +EOF + chmod +x "$tmpbin/fm-mail.sh" + + out="$home/out1.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "the wake stays queued" "the first status-2 poll reports that the wake stays queued" + + out="$home/out2.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "the wake stays queued" "a repeated status-2 poll still prints so the queued mail is not stranded" + [ "$(wc -l < "$out" | tr -d '[:space:]')" = 1 ] || fail "the repeat report is exactly one line: $(cat "$out")" + pass "fm-mail-check: a repeated status-2 stays-queued poll still wakes" +} + +test_repeated_heal_failure_stays_silent() { + # Heal failure happens before wake_for: no queued mail and no .mail-woken + # growth. The first standing check reports it; a repeated identical + # pre-wake failure must stay silent. + local home out + home=$(make_home heal-fail) + write_env "$home" + enter_mailbox "$home" \ + 'printf "uidvalidity\\t90009\\n"' + printf 'uidvalidity=90009\n' > "$home/state/.mail-seen" + printf '%s\t%s\n' '90009' '55' > "$home/state/.mail-woken" + chmod 0400 "$home/state/.mail-seen" + + out="$home/out1.txt" + run_check "$home" "$out" "$CHECK" + assert_contains "$(cat "$out")" "mail: heal could not record a uid" "the first heal failure reports" + + out="$home/out2.txt" + run_check "$home" "$out" "$CHECK" + [ ! -s "$out" ] || fail "a repeated pre-wake heal failure must stay silent: $(cat "$out")" + chmod 0600 "$home/state/.mail-seen" + pass "fm-mail-check: a repeated pre-wake heal failure stays silent" +} + +test_missing_mail_plane_is_reported() { + local tmpbin home out check_bin + tmpbin="$TMP_ROOT/plane2/bin" + home="$TMP_ROOT/plane2/home" + mkdir -p "$tmpbin" "$home/state" + check_bin="$tmpbin/fm-mail-check.sh" + cp "$ROOT/bin/fm-mail-check.sh" "$tmpbin/" + for lib in fm-timeout-lib.sh fm-pr-lib.sh fm-line-cap-lib.sh fm-check-lib.sh; do + [ -e "$tmpbin/$lib" ] || ln -s "$ROOT/bin/$lib" "$tmpbin/$lib" + done + out="$home/out.txt" + run_check "$home" "$out" "$check_bin" + assert_contains "$(cat "$out")" "mail: fm-mail.sh is missing next to this check" "a home lacking the mail plane reports it" + pass "fm-mail-check: a missing mail plane is reported, not assumed" +} + +test_help_and_usage +test_arm_writes_and_binds_the_check_and_disarm_removes_it +test_arm_resolves_a_relative_home_into_the_shim +test_arm_refuses_a_symlink_at_the_shim_path +test_arm_refuses_without_the_mail_plane +test_successful_poll_with_new_mail_emits_one_wake_line +test_failure_is_reported_once_until_it_changes +test_unconfigured_home_is_reported_once +test_slow_poll_times_out_and_is_reported +test_fail_closed_poll_after_wake_reports_the_failure +test_repeated_status4_fail_closed_still_wakes +test_repeated_status2_stays_queued_still_wakes +test_repeated_failure_that_queued_new_mail_still_wakes +test_repeated_timeout_still_wakes +test_repeated_heal_failure_stays_silent +test_missing_mail_plane_is_reported \ No newline at end of file diff --git a/tests/fm-mail.test.sh b/tests/fm-mail.test.sh new file mode 100644 index 00000000000..518844c9d22 --- /dev/null +++ b/tests/fm-mail.test.sh @@ -0,0 +1,2659 @@ +#!/usr/bin/env bash +# Behavior tests for bin/fm-mail.sh. +# +# fm-mail.sh is a network mail client, so these tests exercise only the paths +# that need no real IMAP/SMTP connection: the config-validation dry run, the +# read-only `status` surface, and the CLI usage/help plumbing. All of them go +# through the executable public interface of bin/fm-mail.sh and never assert +# internal source bytes. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +MAIL="$ROOT/bin/fm-mail.sh" +TMP_ROOT=$(fm_test_tmproot fm-mail) +HOME_DIR="$TMP_ROOT/home" +mkdir -p "$HOME_DIR" + +test_missing_secret_fails_cleanly() { + local out rc + env -u FM_MAIL_USER -u FM_MAIL_PASS -u FM_IMAP_HOST -u FM_SMTP_HOST \ + FM_HOME="$HOME_DIR" "$MAIL" status >"$TMP_ROOT/out" 2>"$TMP_ROOT/err" + rc=$? + expect_code 1 "$rc" "status without configuration must fail" + out=$(cat "$TMP_ROOT/err") + assert_contains "$out" "FM_MAIL_USER" "missing-config error names the missing variable" + assert_contains "$out" "FM_MAIL_*" "missing-config error names the configuration family" + assert_not_contains "$out" "test-pass" "missing-config error never leaks a secret" + pass "fm-mail: missing required configuration fails cleanly naming the variable" +} + +test_env_overrides_env_file() { + local env_home out + env_home="$TMP_ROOT/envfile-home" + mkdir -p "$env_home" + cat > "$env_home/.env" <<'EOF' +FM_MAIL_USER=fromfile@example.com +FM_MAIL_PASS=filepass +FM_IMAP_HOST=imap.file.invalid +FM_SMTP_HOST=smtp.file.invalid +EOF + # No environment: .env supplies the configuration. + out=$(FM_HOME="$env_home" "$MAIL" status 2>&1) + assert_contains "$out" "mail account: fromfile@example.com" "status uses .env when environment is unset" + # A single environment value wins for that key; the other keys still come + # from .env, matching the Relay/FMX "env wins over .env" contract. + out=$(FM_MAIL_USER=fromenv@example.com FM_HOME="$env_home" "$MAIL" status 2>&1) + assert_contains "$out" "mail account: fromenv@example.com" "environment overrides .env for a direct invocation" + pass "fm-mail: environment values override the .env file" +} + +test_status_without_network() { + local out rc + out=$(FM_MAIL_USER="test@example.com" FM_MAIL_PASS="test-pass" \ + FM_IMAP_HOST="imap.test.invalid" FM_SMTP_HOST="smtp.test.invalid" \ + FM_HOME="$HOME_DIR" "$MAIL" status 2>&1) + rc=$? + expect_code 0 "$rc" "status with configuration must succeed without network" + assert_contains "$out" "mail account: test@example.com" "status prints the configured account" + assert_contains "$out" "imap.test.invalid:993" "status prints the configured imap endpoint" + assert_contains "$out" "smtp.test.invalid:465" "status prints the configured smtp endpoint" + assert_contains "$out" "cursor:" "status prints the cursor line" + pass "fm-mail: status succeeds without network and prints configuration" +} + +test_help_plumbing() { + local out rc + out=$(FM_MAIL_USER="test@example.com" FM_MAIL_PASS="test-pass" \ + FM_IMAP_HOST="imap.test.invalid" FM_SMTP_HOST="smtp.test.invalid" \ + FM_HOME="$HOME_DIR" "$MAIL" --help 2>&1) + rc=$? + expect_code 0 "$rc" "--help must exit 0" + assert_contains "$out" "read" "--help lists the read subcommand" + assert_contains "$out" "send" "--help lists the send subcommand" + assert_contains "$out" "poll" "--help lists the poll subcommand" + assert_contains "$out" "status" "--help lists the status subcommand" + pass "fm-mail: --help prints usage for every subcommand" +} + +test_unknown_subcommand_prints_usage() { + local out rc + out=$(FM_MAIL_USER="test@example.com" FM_MAIL_PASS="test-pass" \ + FM_IMAP_HOST="imap.test.invalid" FM_SMTP_HOST="smtp.test.invalid" \ + FM_HOME="$HOME_DIR" "$MAIL" bogus 2>&1) + rc=$? + expect_code 1 "$rc" "unknown subcommand must exit 1" + assert_contains "$out" "read" "unknown subcommand prints usage" + assert_contains "$out" "status" "unknown subcommand prints usage" + pass "fm-mail: unknown subcommand prints usage and exits non-zero" +} + +test_no_secret_leaked_to_status() { + local out + out=$(FM_MAIL_USER="test@example.com" FM_MAIL_PASS="test-pass" \ + FM_IMAP_HOST="imap.test.invalid" FM_SMTP_HOST="smtp.test.invalid" \ + FM_HOME="$HOME_DIR" "$MAIL" status 2>&1) + assert_not_contains "$out" "test-pass" "status must never print the password" + pass "fm-mail: status never prints the password" +} + +test_send_passes_body() { + local fakebin body_file stdin_file + fakebin=$(fm_fakebin "$TMP_ROOT") + stdin_file="$TMP_ROOT/stdin_capture.txt" + + # Fake python3 that reads stdin (the body pipe) and writes it to a file. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +cat > "${FM_MAIL_TEST_STDIN_FILE:-/dev/null}" +exit 0 +SH + chmod +x "$fakebin/python3" + + body_file="$TMP_ROOT/body.txt" + printf '%s' "hello world" > "$body_file" + export FM_MAIL_TEST_STDIN_FILE="$stdin_file" + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=h FM_SMTP_HOST=h \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" send to@example.com subj "hello world" 2>&1) || rc=$? + expect_code 0 "$rc" "send with body must succeed" + local captured + captured=$(cat "$stdin_file" 2>/dev/null || echo "") + assert_contains "$captured" "hello world" "send passes body through stdin to python3" + pass "fm-mail: send passes body not empty through stdin" +} + +test_poll_error_propagates() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # Fake python3 that exits with an error (simulating IMAP failure). + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +echo "fm-mail poll error: connection refused" >&2 +exit 1 +SH + chmod +x "$fakebin/python3" + + local out rc=0 + out=$(env -u FM_MAIL_USER -u FM_MAIL_PASS -u FM_IMAP_HOST -u FM_SMTP_HOST \ + FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must propagate python3 errors" + assert_contains "$out" "connection refused" "poll error message is visible" + pass "fm-mail: poll propagates errors instead of swallowing them" +} + +test_poll_dedupes_surfaces_by_uid() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + + # Fake python3 that emits the mailbox generation guard then one UID'd + # poll_list line (uidvalidity, uid \t date \t from \t subj). + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t10001\n' +printf '42\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + # Provide the real wake lib under the temp home so wake_for can append wakes + # into the temp home's state (never the repo's). + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "poll must succeed when python3 lists mail" + assert_contains "$out" "woke for 42" "first poll wakes the new uid" + local wakeq="$HOME_DIR/state/.wake-queue" + assert_contains "$(cat "$wakeq" 2>/dev/null)" "mail from alice@example.com" "wake queue names the sender" + assert_contains "$(cat "$HOME_DIR/state/.mail-seen" 2>/dev/null)" "42" "cursor records the surfaced uid" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "second poll must succeed" + assert_not_contains "$out" "woke for 42" "re-polling the same uid must not re-wake" + assert_contains "$out" "no new mail" "second poll reports no new mail" + assert_not_contains "$(cat "$HOME_DIR/state/.mail-seen" 2>/dev/null)" $'\t' \ + "heal must record only the uid, not a tagged journal field, into the cursor" + pass "fm-mail: poll surfaces each new uid exactly once" +} + +test_poll_resurfaces_uid_after_generation_change() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # First mailbox generation surfaces uid 77 under uidvalidity 30003. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t30003\n' +printf '77\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first-generation poll must succeed" + assert_contains "$out" "woke for 77" "first generation wakes uid 77" + + # Recreated mailbox: same numeric uid 77 under a new UIDVALIDITY. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t40004\n' +printf '77\t2026-09-06T00:00:00Z\tbob@example.com\tAgain\n' +SH + chmod +x "$fakebin/python3" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "second-generation poll must succeed" + assert_contains "$out" "woke for 77" "reused uid wakes again under a new generation" + pass "fm-mail: generation change prevents a reused uid from being suppressed" +} + +test_poll_heals_wake_without_cursor_record() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t60006\n' +printf '99\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + # First poll wakes 99 and records it, proving the normal path. + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first poll must succeed" + assert_contains "$out" "woke for 99" "first poll wakes uid 99" + + # Simulate a poll interrupted after its wake append but before its cursor + # write: remove the uid from the cursor while its wake stays queued. + printf 'uidvalidity=60006\n' > "$HOME_DIR/state/.mail-seen" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "healing poll must succeed" + assert_not_contains "$out" "woke for 99" "healing poll must not re-wake the queued mail" + assert_contains "$(cat "$HOME_DIR/state/.mail-seen" 2>/dev/null)" "99" "healing poll restores the cursor record" + local wakeq + wakeq=$(grep -c "check: mail 99" "$HOME_DIR/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "queued wake is still appended exactly once" + pass "fm-mail: poll heals a wake whose cursor record was interrupted" +} + +test_poll_serializes_overlapping_invocations() { +local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # Fresh-generation fake python3 that pauses so two concurrently started + # polls genuinely overlap and contend on the cursor. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +sleep 0.2 +printf 'uidvalidity\t50005\n' +printf '88\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + local combined woke_count + FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll >"$TMP_ROOT/poll-a.out" 2>&1 & + FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll >"$TMP_ROOT/poll-b.out" 2>&1 & + wait + + combined="$(cat "$TMP_ROOT/poll-a.out" "$TMP_ROOT/poll-b.out")" + woke_count=$(printf '%s' "$combined" | grep -c "woke for 88" || true) + expect_code 1 "$woke_count" "overlapping polls surface uid 88 exactly once" + local wakeq + wakeq=$(grep -c "check: mail 88" "$HOME_DIR/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "overlapping polls append exactly one wake for uid 88" + pass "fm-mail: the poll lock serializes overlapping polls so mail wakes exactly once" +} + +test_poll_recovers_journaled_wake_after_ack() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t70007\n' +printf '55\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first poll must succeed" + assert_contains "$out" "woke for 55" "first poll wakes uid 55" + + # Simulate a poll killed between wake append and cursor record, then the + # fleet drain acknowledging and consuming that wake: the wake is removed + # from the queue and the uid is absent from the cursor, but the journal + # survives. + printf 'uidvalidity=70007\n' > "$HOME_DIR/state/.mail-seen" + printf '%s\t%s\n' '70007' '55' > "$HOME_DIR/state/.mail-woken" + grep -v "check: mail 55" "$HOME_DIR/state/.wake-queue" > "$TMP_ROOT/wakeq.acked" 2>/dev/null || true + mv "$TMP_ROOT/wakeq.acked" "$HOME_DIR/state/.wake-queue" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "recovery poll must succeed" + assert_not_contains "$out" "woke for 55" "recovery must not re-wake the acked mail" + assert_contains "$(cat "$HOME_DIR/state/.mail-seen" 2>/dev/null)" "55" "journal heal restores the cursor record" + local wakeq + wakeq=$(grep -c "check: mail 55" "$HOME_DIR/state/.wake-queue" 2>/dev/null || true) + expect_code 0 "$wakeq" "recovery must not append a second wake for uid 55" + pass "fm-mail: journal recovers a wake the drain already acknowledged" +} + +test_poll_duplicate_wakes_on_interrupted_poll() { + # A poll killed after the wake row was appended but before the journal or + # cursor was written leaves the uid only in the durable queue. The next poll + # must record the uid from that queued key, not surface the mail again. + local fakebin homedir_bin interrupted_home out rc=0 wakeq + fakebin=$(fm_fakebin "$TMP_ROOT") + interrupted_home="$TMP_ROOT/interrupted-home" + homedir_bin="$interrupted_home/bin" + mkdir -p "$homedir_bin" "$interrupted_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity 80008\n' +printf '33\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + # First poll appends the wake and writes the evidence. + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$interrupted_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first poll must succeed" + assert_contains "$out" "woke for 33" "first poll wakes uid 33" + assert_contains "$(cat "$interrupted_home/state/.mail-woken" 2>/dev/null)" "80008" \ + "first poll writes the journal generation" + assert_contains "$(cat "$interrupted_home/state/.mail-woken" 2>/dev/null)" "33" \ + "first poll writes the journal uid" + + # Simulate a kill between the queue append and the evidence writes: keep the + # queued wake row, but drop both journal and cursor records. + printf 'uidvalidity=80008\n' > "$interrupted_home/state/.mail-seen" + : > "$interrupted_home/state/.mail-woken" + wakeq=$(grep -c "check: mail 33" "$interrupted_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "the wake row survived the simulated interruption" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$interrupted_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "healing poll must succeed" + assert_not_contains "$out" "woke for 33" "healing poll must not duplicate the queued mail" + assert_contains "$(cat "$interrupted_home/state/.mail-seen" 2>/dev/null)" "33" \ + "healing poll records the uid from the queued wake key" + wakeq=$(grep -c "check: mail 33" "$interrupted_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "the queued wake row stays appended exactly once" + pass "fm-mail: a poll interrupted before evidence writes does not duplicate on recovery" +} + +test_poll_acknowledged_wake_evading_recovery() { + # A poll killed after the journal was written but before the cursor, followed + # by the drain acknowledging the wake, leaves the uid only in the journal. + # The next poll must record the uid from the journal and clear the journal, + # never re-waking the mail. + local fakebin homedir_bin acked_home out rc=0 wakeq + fakebin=$(fm_fakebin "$TMP_ROOT") + acked_home="$TMP_ROOT/acked-home" + homedir_bin="$acked_home/bin" + mkdir -p "$homedir_bin" "$acked_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '44\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$acked_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first poll must succeed" + assert_contains "$out" "woke for 44" "first poll wakes uid 44" + + # Simulate: journal survived, cursor did not, and the drain consumed the wake. + printf 'uidvalidity=90009\n' > "$acked_home/state/.mail-seen" + printf '%s\t%s\n' '90009' '44' > "$acked_home/state/.mail-woken" + grep -v "check: mail 44" "$acked_home/state/.wake-queue" > "$TMP_ROOT/wakeq.acked" 2>/dev/null || true + mv "$TMP_ROOT/wakeq.acked" "$acked_home/state/.wake-queue" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$acked_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "recovery poll must succeed" + assert_not_contains "$out" "woke for 44" "recovery must not re-wake the acknowledged mail" + assert_contains "$(cat "$acked_home/state/.mail-seen" 2>/dev/null)" "44" \ + "journal heal records the acknowledged uid in the cursor" + assert_equals "" "$(cat "$acked_home/state/.mail-woken" 2>/dev/null)" \ + "journal is cleared once every uid is durably recorded" + wakeq=$(grep -c "check: mail 44" "$acked_home/state/.wake-queue" 2>/dev/null || true) + expect_code 0 "$wakeq" "recovery must not append a second wake for uid 44" + pass "fm-mail: an acknowledged wake whose cursor record was lost is recovered from the journal" +} + +test_poll_legacy_wake_does_not_leak_into_generation() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # Fresh mailbox generation (uidvalidity 90009) whose uid 42 is currently + # unseen. A legacy generation-less wake `mail:42` from an earlier era is + # still queued. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '42\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + + # Seed a legacy wake key (no generation) directly in the wake queue. + printf '0\t9001\tcheck\tmail:42\tcheck: mail 42 - legacy\n' >> "$HOME_DIR/state/.wake-queue" + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "poll with a queued legacy wake must succeed" + assert_contains "$out" "woke for 42" "a reused uid must still wake under the current generation" + pass "fm-mail: a legacy generation-less wake never marks a reused uid surfaced" +} + +test_poll_missing_wake_lib_does_not_suppress() { + local fakebin miss_home miss_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + miss_home="$TMP_ROOT/misslib-home" + miss_bin="$TMP_ROOT/misslib-bin" + mkdir -p "$miss_home" "$miss_bin" + # Hide the script-relative wake library: poll sources fm-wake-lib.sh from + # next to fm-mail.sh, not from $FM_HOME/bin. Copy only the plane scripts. + cp "$ROOT/bin/fm-mail.sh" "$miss_bin/fm-mail.sh" + cp "$ROOT/bin/fm-mail.py" "$miss_bin/fm-mail.py" + chmod +x "$miss_bin/fm-mail.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t10010\n' +printf '33\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$miss_home" PATH="$fakebin:$PATH" \ + "$miss_bin/fm-mail.sh" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must stop when the wake library is missing" + assert_contains "$out" "fm-wake-lib.sh missing" "missing-lib error names the wake library" + assert_not_contains "$(cat "$miss_home/state/.mail-seen" 2>/dev/null)" "33" "a failed wake must never be committed to the cursor" + pass "fm-mail: a missing wake library fails the poll instead of suppressing mail" +} + +test_poll_rolls_back_wake_without_durable_record() { + local fakebin homedir_bin roll_home + fakebin=$(fm_fakebin "$TMP_ROOT") + roll_home="$TMP_ROOT/rollback-home" + mkdir -p "$roll_home" + homedir_bin="$roll_home/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '66\t2026-09-05T00:00:00Z\nalice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + # Make both durable evidence writes fail. The wake append still succeeds (the + # queue is a different, writable file), but the journal and cursor cannot be + # recorded. The rollback must remove the queued wake so nothing ackable + # survives without a durable record. + mkdir -p "$roll_home/state" + printf 'uidvalidity=90009\n' > "$roll_home/state/.mail-seen" + : > "$roll_home/state/.mail-woken" + chmod 0400 "$roll_home/state/.mail-seen" "$roll_home/state/.mail-woken" + [ -w "$roll_home/state/.mail-seen" ] && { echo "fixture unexpected: cursor still writable"; return 1; } + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$roll_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when no durable record can be written" + assert_contains "$out" "rolled back" "poll reports the wake was rolled back" + local wakeq + wakeq=$(grep -c "check: mail 66" "$roll_home/state/.wake-queue" 2>/dev/null || true) + expect_code 0 "$wakeq" "rolled-back wake must not stay queued without a durable record" + + # Restore write access: the next poll must surface the mail fresh, exactly + # once, as if the interrupted attempt never happened. + chmod 0600 "$roll_home/state/.mail-seen" "$roll_home/state/.mail-woken" + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$roll_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "retry poll must succeed" + assert_contains "$out" "woke for 66" "retry poll surfaces the mail exactly once" + wakeq=$(grep -c "check: mail 66" "$roll_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "retry poll appends exactly one wake for uid 66" + pass "fm-mail: a wake with no durable record is rolled back, not left ackable" +} + +test_poll_rollback_failure_never_leaves_unrecorded_ackable_wake() { + local fakebin roll_home + fakebin=$(fm_fakebin "$TMP_ROOT") + roll_home="$TMP_ROOT/rollback-failure-home" + mkdir -p "$roll_home/bin" "$roll_home/state" + [ -e "$roll_home/bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$roll_home/bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '88\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + + # Triple-fault fixture: the journal and cursor cannot be written (read-only), + # and the queue file is write-only so the wake append succeeds but the + # rollback's awk rewrite cannot read the queue and must fail. No durable + # record and no queue rewrite can remove the wake row, so the poll must fail + # closed with an honest report and leave the row for the next poll to heal. + printf 'uidvalidity=90009\n' > "$roll_home/state/.mail-seen" + : > "$roll_home/state/.mail-woken" + : > "$roll_home/state/.wake-queue" + chmod 0400 "$roll_home/state/.mail-seen" "$roll_home/state/.mail-woken" + chmod 0200 "$roll_home/state/.wake-queue" + [ -w "$roll_home/state/.mail-seen" ] && { echo "fixture unexpected: cursor still writable"; return 1; } + [ -w "$roll_home/state/.wake-queue" ] || { echo "fixture unexpected: queue not appendable"; return 1; } + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$roll_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when the wake can be neither recorded nor rolled back" + assert_not_contains "$out" "rolled back (journal and cursor writes failed)" "poll must not report a rollback it did not achieve" + assert_contains "$out" "could not be rolled back or durably recorded" "poll reports the honest rollback-failure outcome" + assert_not_contains "$(cat "$roll_home/state/.mail-seen" 2>/dev/null)" "88" "a failed wake must never be committed to the cursor" + + # Restore access: the still-queued wake must be healed without re-waking, so + # the mail surfaces exactly once from the retained row and never duplicates. + chmod 0600 "$roll_home/state/.mail-seen" "$roll_home/state/.mail-woken" + chmod 0644 "$roll_home/state/.wake-queue" + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$roll_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "retry poll must succeed after access is restored" + assert_contains "$out" "no new mail" "retry poll heals the retained wake without re-waking" + assert_not_contains "$out" "woke for 88" "retry poll must not surface the mail a second time" + assert_contains "$(cat "$roll_home/state/.mail-seen")" "88" "retry poll records the retained wake's uid in the cursor" + local wakeq + wakeq=$(grep -c "check: mail 88" "$roll_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "the retained wake row stays queued for the drain exactly once" + pass "fm-mail: a rollback failure never releases a wake the drain could acknowledge without a durable record" +} + +test_poll_retry_surfaces_under_new_mail_flood() { + local harness out + harness="$TMP_ROOT/retry-budget-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_POLL_MAX_WAKES': '4', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 41']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[3]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # cap=4 reserves retry_budget = max(1, 4//4) = 1. Twenty new uids (51-70) + # exceed the new window (max(4*4, 4+10) = 16) and would fill the cap alone; + # the recovering retry uid 41, sliced separately, must still take its slot. + { + printf 'uidvalidity=90009\n' + printf '41\n' + } > "$HOME_DIR/state/.mail-seen" + printf '41\n' > "$HOME_DIR/state/.mail-retry" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" "$ROOT/bin/fm-mail.py" 2>&1) + assert_contains "$out" $'41\t\ta@b.c\tgood\tretry' "recovered metadata surfaces despite the new-mail flood" + pass "fm-mail: the reserved retry budget survives a new-mail flood" +} + +test_poll_cap_one_alternates_new_and_retry() { + local harness out1 out2 rc1=0 rc2=0 + harness="$TMP_ROOT/cap-one-turn-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_TURN': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'90 100']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # uid 90 is cursor-recorded and in the retry set (recovered degraded mail); + # uid 100 is new unseen mail. cap=1 leaves one contended slot, so the poll + # alternates: the first poll surfaces new mail and the next surfaces the + # recovered retry metadata, and neither class can starve the other. + { + printf 'uidvalidity=90009\n' + printf '90\n' + } > "$HOME_DIR/state/.mail-seen" + printf '90\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-turn" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$ROOT/bin/fm-mail.py" 2>&1) || rc1=$? + expect_code 0 "$rc1" "first contended poll must succeed" + assert_contains "$out1" $'100\t\ta@b.c\tgood\tok' "the first contended slot surfaces new mail" + assert_not_contains "$out1" $'90\t' "the retry recovery waits its turn" + + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$ROOT/bin/fm-mail.py" 2>&1) || rc2=$? + expect_code 0 "$rc2" "second contended poll must succeed" + assert_contains "$out2" $'90\t\ta@b.c\tgood\tretry' "the second contended slot surfaces the recovered retry metadata" + assert_not_contains "$out2" $'100\t' "new mail waits its turn" + pass "fm-mail: a single contended slot alternates between new mail and retry recovery" +} + +test_poll_cap_one_does_not_advance_unexamined_retry_window() { + local harness out1 out2 pos1 pos2 rc1=0 rc2=0 + harness="$TMP_ROOT/cap-one-retry-pos-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_TURN': sys.argv[3], + 'FM_MAIL_RETRY_POS': sys.argv[4], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'71 72 73 74 75 76 77 78 79 80 81 82 100']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[5]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # 12 retry uids plus one new uid, cap=1. The first contended poll spends the + # slot on new mail (retry_budget=0) and must leave the retry-scan position + # unchanged so the next poll still examines retries 71-81 instead of wrapping + # to 82. + { + printf 'uidvalidity=90009\n' + for u in 71 72 73 74 75 76 77 78 79 80 81 82; do + printf '%s\n' "$u" + done + } > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.mail-retry" + for u in 71 72 73 74 75 76 77 78 79 80 81 82; do + printf '%s\n' "$u" >> "$HOME_DIR/state/.mail-retry" + done + : > "$HOME_DIR/state/.mail-turn" + : > "$HOME_DIR/state/.mail-retry-pos" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc1=$? + expect_code 0 "$rc1" "first contended poll must succeed" + assert_contains "$out1" $'100\t\ta@b.c\tgood\tok' "the first contended slot surfaces new mail" + assert_not_contains "$out1" $'71\t' "retry recovery waits its turn" + pos1=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "" "$pos1" "retry_budget 0 must not advance the retry-scan position" + + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc2=$? + expect_code 0 "$rc2" "second contended poll must succeed" + assert_contains "$out2" $'71\t\ta@b.c\tgood\tretry' "the unexamined retry window is scanned from the start" + assert_not_contains "$out2" $'82\t' "the scan must not wrap over the unexamined window" + pos2=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "" "$pos2" "position does not advance while a retry row is emitted" + pass "fm-mail: a zero retry budget leaves the retry-scan position unchanged" +} + +test_poll_cap_one_never_suppresses_new_mail() { + local harness out + harness="$TMP_ROOT/cap-one-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'61 41']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[3]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # cap=1 with a retry present would compute new_budget=0; the fix guarantees + # new mail keeps at least one slot, so uid 61 surfaces and the retry waits. + { + printf 'uidvalidity=90009\n' + printf '41\n' + } > "$HOME_DIR/state/.mail-seen" + printf '41\n' > "$HOME_DIR/state/.mail-retry" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" "$ROOT/bin/fm-mail.py" 2>&1) + assert_contains "$out" $'61\t\ta@b.c\tgood\tok' "new mail keeps its slot when the cap is one" + pass "fm-mail: a cap of one never suppresses new mail while retries exist" +} + +test_poll_restores_retry_when_recovered_wake_cannot_append() { + local fakebin homedir_bin out rc=0 + fakebin=$(fm_fakebin "$TMP_ROOT") + mkdir -p "$HOME_DIR/bin" + [ -e "$HOME_DIR/bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$HOME_DIR/bin/fm-wake-lib.sh" + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '77\t\tfrom@x\tRe: hi\tretry\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n77\n' > "$HOME_DIR/state/.mail-seen" + rm -f "$HOME_DIR/state/.mail-retry" + printf '77\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.wake-queue.seq" + chmod 0000 "$HOME_DIR/state/.wake-queue.seq" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when the recovered wake cannot be appended" + assert_contains "$(cat "$HOME_DIR/state/.mail-retry" 2>/dev/null)" "77" "the retry record is restored so the recovered metadata can be re-fetched" + chmod 0600 "$HOME_DIR/state/.wake-queue.seq" + pass "fm-mail: a recovered wake that cannot append restores the retry instead of stranding the metadata" +} + +test_poll_death_between_retry_remove_and_publish_does_not_strand() { + # Old ordering: retry_remove ran BEFORE wake_for, so a kill after the remove + # but before the publish left the uid cursor-recorded from the degraded wake + # but no longer retry-eligible. The retry clear is now inside wake_for and + # runs only after a successful publish, so the same kill window cannot strand + # metadata. This test proves the invariant by forcing publish to fail and + # verifying the retry entry survives, then that a follow-up poll recovers it. + local fakebin homedir_bin test_home out rc=0 wakeq + fakebin=$(fm_fakebin "$TMP_ROOT") + test_home="$TMP_ROOT/retry-survives-failed-publish-home" + homedir_bin="$test_home/bin" + mkdir -p "$homedir_bin" "$test_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity 90009\n' +printf '77\t2026-09-05T00:00:00Z\tfrom@x\tRe: hi\tretry\n' +SH + chmod +x "$fakebin/python3" + + # Cursor records the uid from the earlier degraded wake; retry set exists. + printf 'uidvalidity=90009\n77\n' > "$test_home/state/.mail-seen" + printf '77\n' > "$test_home/state/.mail-retry" + + # Make the wake queue unwritable so the recovered wake cannot append. The + # retry record must NOT be cleared in this case. + : > "$test_home/state/.wake-queue.seq" + chmod 0000 "$test_home/state/.wake-queue.seq" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$test_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail closed when the recovered wake cannot be published" + assert_contains "$out" "wake append failed for 77" "poll reports the failed wake append" + assert_grep "77" "$test_home/state/.mail-retry" "retry entry survives a failed publish" + assert_not_contains "$(cat "$test_home/state/.wake-queue" 2>/dev/null)" "check: mail 77" \ + "no wake is queued when publish fails" + + # Restore writable state: the next poll must recover the metadata. + rm -f "$test_home/state/.wake-queue.seq" + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$test_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "follow-up poll must succeed and recover the metadata" + assert_contains "$out" "woke for 77" "follow-up poll re-surfaces the recovered uid" + assert_contains "$(cat "$test_home/state/.mail-seen" 2>/dev/null)" "77" \ + "follow-up poll records the uid in the cursor" + wakeq=$(grep -c "check: mail 77" "$test_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "exactly one recovery wake is queued" + pass "fm-mail: a death between retry remove and wake publish cannot strand recovered metadata" +} + +test_poll_fails_closed_when_poll_list_fails() { + local fakebin out rc=0 + fakebin=$(fm_fakebin "$TMP_ROOT") + mkdir -p "$HOME_DIR/bin" + [ -e "$HOME_DIR/bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$HOME_DIR/bin/fm-wake-lib.sh" + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +if [ -f "$FM_POLL_FAIL_MARKER" ]; then + echo 'fm-mail poll error: simulated failure' >&2 + exit 1 +fi +printf 'uidvalidity\t90009\n' +printf '7\t\tfrom@x\tnew mail\tok\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + touch "$HOME_DIR/fail.marker" + + out=$(FM_POLL_FAIL_MARKER="$HOME_DIR/fail.marker" FM_MAIL_USER=test FM_MAIL_PASS=pass \ + FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when poll_list fails" + assert_not_contains "$out" "woke for 7" "nothing is woken from a failed poll_list" + assert_not_contains "$out" "no new mail" "a failed poll is not reported as no new mail" + + rm -f "$HOME_DIR/fail.marker" + rc=0 + out=$(FM_POLL_FAIL_MARKER="$HOME_DIR/fail.marker" FM_MAIL_USER=test FM_MAIL_PASS=pass \ + FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "a later poll must succeed, proving the mail-seen lock was released" + assert_contains "$out" "woke for 7" "the later poll wakes the new mail" + pass "fm-mail: a failed poll_list fails closed, wakes nothing, and releases the lock" +} + +test_poll_fails_closed_when_retry_clear_fails() { + # The retry record is now cleared inside wake_for, AFTER the wake is durably + # published. A failed clear therefore leaves the wake in the queue while the + # poll fails closed; later polls retry the clear without appending another wake. + local fakebin homedir_bin out rc=0 test_home wakeq + fakebin=$(fm_fakebin "$TMP_ROOT") + test_home="$TMP_ROOT/retry-clear-fail-home" + homedir_bin="$test_home/bin" + mkdir -p "$homedir_bin" "$test_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '77\t\tfrom@x\tRe: hi\tretry\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n77\n' > "$test_home/state/.mail-seen" + printf '77\n' > "$test_home/state/.mail-retry" + chmod 0000 "$test_home/state/.mail-retry" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$test_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when the retry record cannot be cleared" + assert_contains "$out" "could not clear retry for recovered 77 after publish" "failure names the post-publish retry cleanup" + assert_grep "check: mail 77" "$test_home/state/.wake-queue" "the recovery wake was already published before the cleanup failed" + wakeq=$(grep -c "check: mail 77" "$test_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "exactly one recovery wake is queued after the failed clear" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$test_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "a later poll must still fail closed while the retry record cannot be cleared" + assert_not_contains "$out" "woke for 77" "a later poll must not re-append a recovery wake" + wakeq=$(grep -c "check: mail 77" "$test_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "the queued recovery wake is not duplicated while the retry clear keeps failing" + + chmod 0600 "$test_home/state/.mail-retry" + assert_grep "77" "$test_home/state/.mail-retry" "the retry entry remains for the next poll to clear" + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$test_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "poll must succeed once the retry record can be cleared" + assert_not_contains "$out" "woke for 77" "clearing the retry record must not re-wake the uid" + assert_not_contains "$(cat "$test_home/state/.mail-retry" 2>/dev/null || true)" "77" \ + "the retry entry is cleared without a duplicate wake" + wakeq=$(grep -c "check: mail 77" "$test_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "exactly one recovery wake remains after a successful retry clear" + pass "fm-mail: a failed retry clear fails the poll; the published wake stays and the retry entry remains" +} + +test_poll_fails_closed_when_stale_retry_clear_fails() { + # After a successful non-degraded wake, a stale retry entry must be cleared + # fail-closed: swallowing that failure would leave the uid eligible for a + # duplicate recovery wake on the next poll. + local fakebin homedir_bin out rc=0 test_home + fakebin=$(fm_fakebin "$TMP_ROOT") + test_home="$TMP_ROOT/stale-retry-clear-fail-home" + homedir_bin="$test_home/bin" + mkdir -p "$homedir_bin" "$test_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '77\t2026-09-05T00:00:00Z\tfrom@x\tHello\tok\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$test_home/state/.mail-seen" + printf '77\n' > "$test_home/state/.mail-retry" + chmod 0000 "$test_home/state/.mail-retry" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$test_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when a stale retry cannot be cleared after wake" + assert_contains "$out" "could not clear retry for recovered 77 after publish" "failure names the post-publish retry cleanup" + assert_grep "check: mail 77" "$test_home/state/.wake-queue" "the wake already landed before the cleanup failure" + chmod 0600 "$test_home/state/.mail-retry" + assert_grep "77" "$test_home/state/.mail-retry" "the stale retry entry remains for the next poll to clear" + pass "fm-mail: a failed stale-retry clear fails the poll instead of silently leaving a duplicate-wake entry" +} + +test_assert_equals_rejects_mismatch() { + # Guard against a silent false pass: assert_equals must be defined and must + # abort on a mismatch (the retry-scan cursor test depends on it). + if ( assert_equals "11" "10" "deliberate mismatch" ) >/dev/null 2>&1; then + fail "assert_equals must fail when expected and actual differ" + fi + assert_equals "11" "11" "matching values must pass" + pass "tests/lib: assert_equals fails on mismatch so cursor assertions cannot false-pass" +} + +test_poll_fails_closed_when_retry_unwritable() { + local fakebin homedir_bin out rc=0 + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '77\t\t(no header)\tunfetchable header - see fm-mail read\tdegraded\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.mail-retry" + chmod 0400 "$HOME_DIR/state/.mail-retry" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when the retry record cannot be written" + assert_not_contains "$out" "woke for 77" "no wake is emitted before the retry record" + assert_not_contains "$(cat "$HOME_DIR/state/.mail-seen" 2>/dev/null)" "77" "a failed retry write must not cursor-record the uid" + chmod 0600 "$HOME_DIR/state/.mail-retry" + pass "fm-mail: a failed retry write fails the poll instead of losing recovery" +} + +test_poll_journal_failure_never_cursor_records() { + # A mail must never be marked surfaced in the cursor without the journal + # recording its wake. If the journal write fails, the cursor is NOT written + # and the poll fails closed, so the next poll re-wakes the mail instead of + # silently suppressing it (cursor-without-journal suppression). + local fakebin homedir_bin out rc=0 + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '78\t2026-09-06T00:00:00Z\tfrom@x\tHello\tok\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.mail-woken" + chmod 0400 "$HOME_DIR/state/.mail-woken" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when the journal cannot be written" + assert_not_contains "$out" "woke for 78" "no wake is emitted before the journal commits" + assert_not_contains "$(cat "$HOME_DIR/state/.mail-seen" 2>/dev/null)" "78" \ + "a failed journal write must not cursor-record the uid (no cursor-without-journal suppression)" + chmod 0600 "$HOME_DIR/state/.mail-woken" + pass "fm-mail: a journal write failure never cursor-records a suppressed mail" +} + +test_poll_resurfaces_degraded_uid_whose_wake_never_recorded() { + local harness out rc=0 + harness="$TMP_ROOT/retry-not-seen-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_POLL_MAX_WAKES': '2', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'90']) + if cmd == 'fetch': + return ('NO', None) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[3]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # uid 90 is in the retry set (a prior degraded wake) but NOT in the cursor + # (that wake failed and was rolled back). It must be re-surfaced as degraded + # on this poll - a retry uid the cursor does not record must never be + # silently dropped as a pure retry. + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + printf '90\n' > "$HOME_DIR/state/.mail-retry" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "poll must succeed" + assert_contains "$out" $'90\t\t(no header)\tunfetchable header - see fm-mail read\tdegraded' "a retry uid the cursor never recorded is re-surfaced degraded" + pass "fm-mail: a retry uid whose degraded wake never recorded is re-surfaced, not dropped" +} + +test_poll_fetch_raise_does_not_abort_the_scan() { + local harness out rc=0 + harness="$TMP_ROOT/fetch-raise-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_POLL_MAX_WAKES': '2', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'1 2 3']) + if cmd == 'fetch': + if args[0] == b'1': + raise RuntimeError('simulated imap fetch failure') + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[2]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "a raised fetch must not abort the scan" + assert_contains "$out" $'1\t\t(no header)\tunfetchable header - see fm-mail read\tdegraded' "the raising uid is surfaced degraded" + assert_contains "$out" $'2\t\ta@b.c\tgood\tok' "the scan advances past the raising uid" + pass "fm-mail: a raised FETCH surfaces that uid degraded and advances" +} + +test_poll_retry_cursor_advances_past_failures() { + local harness out1 out2 pos1 pos2 rc1=0 rc2=0 + harness="$TMP_ROOT/retry-cursor-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[5], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +FAILING = set(sys.argv[3].split(',')) if len(sys.argv) > 3 else set() +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'71 72 73 74 75 76 77 78 79 80 81 82']) + if cmd == 'fetch': + if args[0].decode() in FAILING: + return ('NO', None) + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # All 12 uids are cursor-recorded (degraded wakes) and listed in the retry + # set; 71..81 keep failing, 82 recovered. window = max(1*4, 1+10) = 11, so a + # single poll examines a bounded 11-uid window starting at the durable retry + # position. Position 0 scans 71..81 first, then the stored position advances + # to 11 so the next poll wraps and reaches 82 - a recovered uid can never be + # stranded behind the persistent-failure prefix. + { + printf 'uidvalidity=90009\n' + for u in 71 72 73 74 75 76 77 78 79 80 81 82; do + printf '%s\n' "$u" + done + } > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.mail-retry" + for u in 71 72 73 74 75 76 77 78 79 80 81 82; do + printf '%s\n' "$u" >> "$HOME_DIR/state/.mail-retry" + done + : > "$HOME_DIR/state/.mail-retry-pos" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "71,72,73,74,75,76,77,78,79,80,81" "$ROOT/bin/fm-mail.py" "$HOME_DIR/state/.mail-retry-pos" 2>&1) || rc1=$? + expect_code 0 "$rc1" "first retry poll must succeed" + assert_not_contains "$out1" $'82\t' "the recovered uid is not reached while failures hold the window" + pos1=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "11" "$pos1" "the retry-scan position advances past the scanned window" + + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "71,72,73,74,75,76,77,78,79,80,81" "$ROOT/bin/fm-mail.py" "$HOME_DIR/state/.mail-retry-pos" 2>&1) || rc2=$? + expect_code 0 "$rc2" "second retry poll must succeed" + assert_contains "$out2" $'82\t\ta@b.c\tgood\tretry' "the cursor wraps and the recovered uid surfaces on the next poll" + pos2=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "11" "$pos2" "position does not advance while a recovered row is emitted (bash removes it from the retry set)" + + # Simulate the bash layer removing the recovered uid from the retry set + # after a successful publish; the next poll scans the remaining uids from + # the same numeric start, so an unfetchable-only window still advances. + grep -vx -e '82' "$HOME_DIR/state/.mail-retry" > "$HOME_DIR/state/.mail-retry.tmp" || true + mv -f -- "$HOME_DIR/state/.mail-retry.tmp" "$HOME_DIR/state/.mail-retry" + + rc2=0 + _out3=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "71,72,73,74,75,76,77,78,79,80,81" "$ROOT/bin/fm-mail.py" "$HOME_DIR/state/.mail-retry-pos" 2>&1) || rc2=$? + expect_code 0 "$rc2" "third retry poll must succeed" + pos3=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "0" "$pos3" "the retry-scan position advances only past candidates examined within budget after the recovered uid is removed" + pass "fm-mail: the retry-scan cursor advances past persistent failures" +} + +test_poll_retry_position_does_not_advance_past_unpublished_wake() { + # The durable retry-scan position must never advance past a uid whose wake + # did not durably publish. cmd_poll_list cannot observe the bash wake_for + # result, so it leaves the position unchanged whenever a row is emitted; + # the bash layer removes recovered uids from the retry set, and the next + # poll's same numeric start scans the next remaining uid. + local harness out rc=0 pos + harness="$TMP_ROOT/retry-pos-unpublished-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'81 82']) + if cmd == 'fetch': + # 81 is still failing; 82 has just recovered real metadata. + if args[0] == b'81': + return ('NO', None) + return ('OK', [(b'', b'Subject: recovered\r\nFrom: bob@x.com\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # Both uids are already cursor-recorded from prior degraded wakes; 81 is + # still unfetchable, 82 has recovered. cap=1 emits only the recovered row. + { + printf 'uidvalidity=90009\n' + printf '81\n82\n' + } > "$HOME_DIR/state/.mail-seen" + printf '81\n82\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + # The recovered row is emitted, but the wake has not yet been published. + # The durable position must advance only up to the first emitted uid (index 1 + # because 81 is still unfetchable), never past it, so a failed publish would + # re-scan 82 on the next poll instead of dropping it until the cursor wraps. + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "poll must succeed while emitting the recovered row" + assert_contains "$out" $'82\t\tbob@x.com\trecovered\tretry' \ + "recovered row is emitted to the bash wake layer" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "1" "$pos" "cursor lands on the first emitted uid, not past it" + + # Simulate a failed wake publish: 82 is still in the retry set, and the next + # poll must start at 82 again rather than skipping it. + rc=0 + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "retry poll must re-scan the uid whose wake did not publish" + assert_contains "$out" $'82\t\tbob@x.com\trecovered\tretry' \ + "recovered uid is re-scanned while its wake is still unpublished" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "1" "$pos" "cursor stays on the unpublished uid" + + # Simulate the bash layer removing the recovered uid after a successful + # publish; the next poll scans the remaining uid from the same numeric start + # and advances only when no row is emitted. + printf '81\n' > "$HOME_DIR/state/.mail-retry" + rc=0 + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "retry poll must succeed after the recovered uid is removed" + assert_not_contains "$out" $'82\t' "recovered uid is no longer in the retry set" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "0" "$pos" "position advances only after an emission-free poll" + pass "fm-mail: retry-scan position does not advance past an unpublished wake" +} + +test_poll_retry_budget_advances_by_examined_not_window() { + # Greptile regression: when the retry window holds more uids than the + # reserved retry budget, the durable position must advance by the candidates + # actually examined within budget - never by the full window. Advancing by + # the full window while emitting only the budgeted prefix revisits the same + # prefix every poll and strands later recovered uids. + local harness out pos rc=0 + harness="$TMP_ROOT/retry-budget-cursor-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + # cap=4 -> retry_budget = max(1, 4//4) = 1, so only one retry uid may + # emit per poll while the window (max(4*4, 4+10) = 16) holds all 12. + 'FM_MAIL_POLL_MAX_WAKES': '4', +}) +FAILING = set(sys.argv[5].split(',')) if len(sys.argv) > 5 else set() +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'101 102 103 104 105 106 107 108 109 110 111 112']) + if cmd == 'fetch': + # The first three uids are still unfetchable; the rest have + # recovered. The durable position must advance by the examined + # prefix (3), never the full window (16), so the recovered uid + # behind the unfetchable prefix is reached promptly. + if args[0].decode() in FAILING: + return ('NO', None) + return ('OK', [(b'', b'Subject: r\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # All 12 uids are cursor-recorded and in the retry set; the first three are + # still unfetchable and the rest have recovered. The durable position must + # advance by the examined prefix (3), never the full window (16), so the + # recovered uid behind the unfetchable prefix is reached promptly. + mkdir -p "$HOME_DIR/state" + { + printf 'uidvalidity=90009\n' + for u in 101 102 103 104 105 106 107 108 109 110 111 112; do + printf '%s\n' "$u" + done + } > "$HOME_DIR/state/.mail-seen" + printf '101\n102\n103\n104\n105\n106\n107\n108\n109\n110\n111\n112\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" "101,102,103" 2>&1) || rc=$? + expect_code 0 "$rc" "first retry poll must succeed" + assert_contains "$out" $'104\t\ta@b.c\tr\tretry' \ + "the first recovered uid after the unfetchable prefix is emitted within budget" + assert_not_contains "$out" $'105\t' "only the budgeted retry uid is emitted" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "3" "$pos" \ + "position advances by the examined prefix, never the full window" + pass "fm-mail: retry budget advances the scan by examined candidates, not the window" +} + +test_poll_retry_window_of_unseen_never_stalls_cursor() { + # Greptile regression: when a retry scan window holds only uids absent from + # the seen cursor while an eligible retry uid lies beyond that window, the + # durable position must still advance so the later recovered uid is + # reachable. A window of unseen uids must never stall the cursor + # permanently (retry_candidates empty would otherwise keep budget 0 and the + # position would never move). + local harness out1 out2 rc1=0 rc2=0 pos + harness="$TMP_ROOT/retry-stall-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '4', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + # Nothing is unseen (all server-seen); only uid 320 is in our + # cursor and retry-eligible, beyond the first 16-uid window. + return ('OK', [b'']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: rec\r\nFrom: z@x.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + # Retry set 301..320; cursor records ONLY 320 (the recovered uid). The first + # scan window (301..316) therefore holds only unseen uids, yet the cursor + # must advance past it so 320 is reached on a later window. + mkdir -p "$HOME_DIR/state" + printf 'uidvalidity=90009\n320\n' > "$HOME_DIR/state/.mail-seen" + for u in $(seq 301 320); do printf '%s\n' "$u"; done > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc1=$? + expect_code 0 "$rc1" "first poll must succeed" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "16" "$pos" "the stale unseen window advances the cursor by the scanned window (cap=4 window=16)" + + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc2=$? + expect_code 0 "$rc2" "second poll must succeed" + assert_contains "$out2" $'320\t\tz@x.c\trec\tretry' \ + "the recovered uid beyond the stale window surfaces once the cursor advances" + pass "fm-mail: a retry window of unseen uids never stalls the cursor" +} + +test_poll_retry_unfetchable_window_advances_with_new_mail() { + # Greptile P1 / no-mistakes retry-pos-not-out-stalls-mixed: when every + # retry uid in the current window is unfetchable while new mail fills the + # poll output, the durable position must still advance. Gating persist on + # an empty `out` re-scans the same failed prefix forever and strands a + # recovered uid beyond the window. + local harness out1 out2 pos rc1=0 rc2=0 + harness="$TMP_ROOT/retry-newmail-unfetchable-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '4', +}) +# First window (cap=4 -> window=16) is 301..316; those stay unfetchable. +# 320 has recovered and sits beyond that window. 400 is sustained new mail. +FAILING = {str(u).encode() for u in range(301, 320)} +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'400']) + if cmd == 'fetch': + if args[0] in FAILING: + return ('NO', None) + return ('OK', [(b'', b'Subject: rec\r\nFrom: z@x.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + mkdir -p "$HOME_DIR/state" + { + printf 'uidvalidity=90009\n' + for u in $(seq 301 320); do printf '%s\n' "$u"; done + } > "$HOME_DIR/state/.mail-seen" + for u in $(seq 301 320); do printf '%s\n' "$u"; done > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc1=$? + expect_code 0 "$rc1" "first poll must succeed while new mail fills the output" + assert_contains "$out1" $'400\t\tz@x.c\trec\tok' \ + "sustained new mail is emitted on the first poll" + assert_not_contains "$out1" $'320\t' \ + "the recovered retry uid is still beyond the unfetchable window" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "16" "$pos" \ + "unfetchable retry window advances the cursor even while new mail fills out" + + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc2=$? + expect_code 0 "$rc2" "second poll must succeed" + assert_contains "$out2" $'320\t\tz@x.c\trec\tretry' \ + "the recovered uid beyond the failed prefix surfaces once the cursor advances" + pass "fm-mail: an unfetchable retry window still advances while new mail fills out" +} + +test_poll_retry_unseen_window_advances_with_new_mail() { + # Same stall as retry-pos-not-out-stalls-mixed on the all-unseen window + # branch: a retry window of uids absent from the seen cursor must still + # advance the durable position when new mail fills `out`, so a later + # eligible retry uid beyond that window remains reachable. + local harness out1 out2 pos rc1=0 rc2=0 + harness="$TMP_ROOT/retry-newmail-unseen-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '4', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'400']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: rec\r\nFrom: z@x.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + mkdir -p "$HOME_DIR/state" + printf 'uidvalidity=90009\n320\n' > "$HOME_DIR/state/.mail-seen" + for u in $(seq 301 320); do printf '%s\n' "$u"; done > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc1=$? + expect_code 0 "$rc1" "first poll must succeed while new mail fills the output" + assert_contains "$out1" $'400\t\tz@x.c\trec\tok' \ + "sustained new mail is emitted on the first poll" + assert_not_contains "$out1" $'320\t' \ + "the recovered retry uid is still beyond the unseen window" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "16" "$pos" \ + "unseen retry window advances the cursor even while new mail fills out" + + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc2=$? + expect_code 0 "$rc2" "second poll must succeed" + assert_contains "$out2" $'320\t\tz@x.c\trec\tretry' \ + "the recovered uid beyond the unseen window surfaces once the cursor advances" + pass "fm-mail: an unseen retry window still advances while new mail fills out" +} + +test_poll_retry_logout_before_emit_and_position_save() { + # A standing check can SIGKILL poll_list while IMAP logout is still blocked. + # Logout must finish before any emit or persist so that kill cannot advance + # the retry-scan position over rows bash never received. + local harness out rc=0 pos + harness="$TMP_ROOT/retry-logout-order-harness.py" + cat > "$harness" <<'PYEOF' +import os, signal, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'81 82']) + if cmd == 'fetch': + if args[0] == b'81': + return ('NO', None) + return ('OK', [(b'', b'Subject: recovered\r\nFrom: bob@x.com\r\n\r\n')]) + def logout(self): + os.kill(os.getpid(), signal.SIGKILL) +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + { + printf 'uidvalidity=90009\n' + printf '81\n82\n' + } > "$HOME_DIR/state/.mail-seen" + printf '81\n82\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "poll_list must not exit 0 when logout is killed" + assert_not_contains "$out" $'82\t' "a killed logout must not emit rows" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "" "$pos" "a killed logout must not advance the retry-scan position" + pass "fm-mail: hung logout cannot advance the retry-scan position" +} + +test_poll_list_logs_out_on_imap_error() { + # A standing check that fails after IMAP login must still log out so repeated + # select/search/fetch errors cannot leak sessions until the server times them + # out. + local harness out marker rc=0 + harness="$TMP_ROOT/poll-logout-on-error-harness.py" + marker="$TMP_ROOT/poll-logout-on-error.marker" + rm -f "$marker" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[2], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + raise RuntimeError('search failed') + return ('NO', None) + def logout(self): + open(sys.argv[1], 'w', encoding='utf-8').write('logged-out\n') +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[3]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + { + printf 'uidvalidity=90009\n' + } > "$HOME_DIR/state/.mail-seen" + + out=$(python3 "$harness" "$marker" "$HOME_DIR/state/.mail-seen" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 1 "$rc" "poll_list must fail when IMAP search raises" + assert_contains "$out" "search failed" "the IMAP error is reported on stderr" + [ -f "$marker" ] || fail "poll_list must log out after a post-login IMAP error" + assert_equals "logged-out" "$(cat "$marker")" "logout ran on the error path" + pass "fm-mail: poll_list logs out when IMAP work fails after login" +} + +test_poll_retry_position_not_saved_when_row_emitted() { + # When a retry row is emitted, cmd_poll_list must not persist the retry-scan + # position at all: a kill in the position-write window cannot drop a uid + # whose wake has not yet durably published. Wrapping save_retry_pos so that + # any call SIGKILLs the process proves the call is skipped when rows are + # emitted. + local harness out rc=0 pos + harness="$TMP_ROOT/retry-no-save-on-emit-harness.py" + cat > "$harness" <<'PYEOF' +import os, signal, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'81 82']) + if cmd == 'fetch': + # The first fetchable retry uid is emitted; the cursor does not + # advance when a row is emitted, so a save would write the same + # position. The kill wrapper proves save_retry_pos is still called + # and flushed stdout survives it. + return ('OK', [(b'', b'Subject: recovered\r\nFrom: bob@x.com\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +orig = mod.save_retry_pos +def save_then_kill(*a, **k): + orig(*a, **k) + os.kill(os.getpid(), signal.SIGKILL) +mod.save_retry_pos = save_then_kill +sys.exit(mod.cmd_poll_list()) +PYEOF + { + printf 'uidvalidity=90009\n' + printf '81\n82\n' + } > "$HOME_DIR/state/.mail-seen" + printf '81\n82\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "poll_list must not call save_retry_pos when a row is emitted" + assert_contains "$out" $'81\t\tbob@x.com\trecovered\tretry' \ + "the recovered row is emitted without persisting the position" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "" "$pos" "position must remain unchanged while a row is emitted" + pass "fm-mail: retry-scan position is not saved when a row is emitted" +} + +test_poll_retry_small_window_rotates_past_unfetchable_prefix() { + # Greptile regression: when the retry set fits inside the scan window, the + # durable position must still rotate the scan. A persistently unfetchable + # leading uid must not monopolize the retry budget and strand later + # recovered uids. + local harness out rc=0 pos + harness="$TMP_ROOT/retry-small-window-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_RETRY_POS': sys.argv[3], + # cap=4 -> window = max(4*4, 4+10) = 16, which is larger than the retry set. + 'FM_MAIL_POLL_MAX_WAKES': '4', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'101 102 103 104 105']) + if cmd == 'fetch': + # 101 is persistently unfetchable; every other uid has recovered. + if args[0] == b'101': + return ('NO', None) + return ('OK', [(b'', b'Subject: ok\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + mkdir -p "$HOME_DIR/state" + { + printf 'uidvalidity=90009\n' + for u in 101 102 103 104 105; do + printf '%s\n' "$u" + done + } > "$HOME_DIR/state/.mail-seen" + printf '101\n102\n103\n104\n105\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-retry-pos" + + # With the unfixed small-window branch, every poll would start at 101 and + # emit 102 repeatedly; with the fix each recovered uid surfaces in turn. + # The durable cursor advances only up to the first emitted uid, so the first + # poll moves from 101 to 102 (index 1) and then stays there while recovered + # uids are removed by the bash layer. + for expected in 102 103 104 105; do + local want + rc=0 + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "poll for $expected must succeed" + want=$(printf '%s\t\ta@b.c\tok\tretry' "$expected") + assert_contains "$out" "$want" "recovered uid $expected surfaces after the unfetchable prefix" + assert_not_contains "$out" $'101\t' "unfetchable uid 101 is not emitted as retry" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "1" "$pos" "cursor advances only up to the first emitted retry uid" + # Simulate the bash layer removing the just-recovered uid from the retry set. + grep -vx -e "$expected" "$HOME_DIR/state/.mail-retry" > "$HOME_DIR/state/.mail-retry.tmp" || true + mv -f -- "$HOME_DIR/state/.mail-retry.tmp" "$HOME_DIR/state/.mail-retry" + done + + # Only the unfetchable uid remains; an emission-free poll advances the + # cursor so the scan does not stall. + rc=0 + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "poll with only the unfetchable uid must succeed" + assert_not_contains "$out" $'101\t' "unfetchable uid still is not emitted" + pos=$(cat "$HOME_DIR/state/.mail-retry-pos" 2>/dev/null || printf '') + assert_equals "0" "$pos" "emission-free poll advances the position past the small window" + + # A non-zero starting position must rotate the small scan too: starting at + # position 4 puts uid 105 first in the rotated order. + printf '101\n102\n103\n104\n105\n' > "$HOME_DIR/state/.mail-retry" + printf '4\n' > "$HOME_DIR/state/.mail-retry-pos" + rc=0 + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "wrap poll must succeed" + assert_contains "$out" $'105\t\ta@b.c\tok\tretry' \ + "a non-zero starting position wraps within the small retry set" + + pass "fm-mail: small retry window rotates past an unfetchable prefix" +} + +test_poll_cap_one_turn_not_saved_before_emit() { + # At cap==1 with both new and retry candidates, the alternating turn must be + # persisted only after the rows are emitted and flushed. A kill in the + # logout/emit window must not advance the turn, so the next poll still gets + # the new-mail turn it was owed. + local harness out rc=0 turn + harness="$TMP_ROOT/cap-one-turn-kill-harness.py" + cat > "$harness" <<'PYEOF' +import os, signal, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_TURN': sys.argv[3], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'90 100']) + if cmd == 'fetch': + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + # Kill the process between the fetch decision and the emit/flush, + # before the turn can be persisted. + os.kill(os.getpid(), signal.SIGKILL) +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[4]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + { + printf 'uidvalidity=90009\n' + printf '90\n' + } > "$HOME_DIR/state/.mail-seen" + printf '90\n' > "$HOME_DIR/state/.mail-retry" + : > "$HOME_DIR/state/.mail-turn" + + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "poll_list must not exit 0 when logout is killed before emit" + assert_not_contains "$out" $'100\t' "a killed logout must not emit rows" + assert_not_contains "$out" $'90\t' "a killed logout must not emit rows" + turn=$(cat "$HOME_DIR/state/.mail-turn" 2>/dev/null || printf '') + assert_equals "" "$turn" "the alternating turn must not advance when emit is killed" + pass "fm-mail: cap-one turn is not saved before emit/flush" +} + +test_poll_cap_one_turn_not_saved_when_retry_pos_write_fails() { + # At cap=1 on the retry turn, 81 is unfetchable and 82 is recovered, so the + # durable retry-scan position must advance. If that write fails closed, the + # poll must not spend the retry turn: the next successful poll still emits 82 + # instead of handing the slot to new mail. + local harness out1 out2 turn rc1=0 rc2=0 + harness="$TMP_ROOT/cap-one-turn-pos-fail-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_RETRY': sys.argv[2], + 'FM_MAIL_TURN': sys.argv[3], + 'FM_MAIL_RETRY_POS': sys.argv[4], + 'FM_MAIL_POLL_MAX_WAKES': '1', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'81 82 100']) + if cmd == 'fetch': + if args[0] == b'81': + return ('NO', None) + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[5]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + { + printf 'uidvalidity=90009\n' + printf '81\n82\n' + } > "$HOME_DIR/state/.mail-seen" + printf '81\n82\n' > "$HOME_DIR/state/.mail-retry" + printf '1\n' > "$HOME_DIR/state/.mail-turn" + rm -f "$HOME_DIR/state/.mail-retry-pos" + mkdir -p "$HOME_DIR/state/.mail-retry-pos" + + out1=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc1=$? + [ "$rc1" -ne 0 ] || fail "poll_list must fail closed when retry-pos cannot be written" + turn=$(cat "$HOME_DIR/state/.mail-turn" 2>/dev/null || printf '') + assert_equals "1" "$turn" "a failed retry-pos write must not spend the retry turn" + + rmdir "$HOME_DIR/state/.mail-retry-pos" + : > "$HOME_DIR/state/.mail-retry-pos" + out2=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$HOME_DIR/state/.mail-retry" \ + "$HOME_DIR/state/.mail-turn" "$HOME_DIR/state/.mail-retry-pos" "$ROOT/bin/fm-mail.py" 2>&1) || rc2=$? + expect_code 0 "$rc2" "the next poll after a fail-closed position write must succeed" + assert_contains "$out2" $'82\t\ta@b.c\tgood\tretry' "the unspent retry turn still surfaces recovered uid 82" + assert_not_contains "$out2" $'100\t' "new mail must not take the unspent retry turn" + pass "fm-mail: a failed retry-pos write does not spend the cap-one retry turn" +} + +test_poll_skips_unfetchable_uid_but_keeps_progress() { + local harness out rc=0 + harness="$TMP_ROOT/poll-window-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', + 'FM_MAIL_CURSOR': sys.argv[1], + 'FM_MAIL_POLL_MAX_WAKES': '2', +}) +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'1 2 3']) + if cmd == 'fetch': + if args[0] == b'1': + return ('NO', None) # persistently unfetchable message + return ('OK', [(b'', b'Subject: good\r\nFrom: a@b.c\r\n\r\n')]) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[2]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_poll_list()) +PYEOF + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + out=$(python3 "$harness" "$HOME_DIR/state/.mail-seen" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "window poll must succeed" + assert_contains "$out" "uidvalidity 90009" "poll emits the generation guard" + assert_contains "$out" $'1\t\t(no header)\tunfetchable header - see fm-mail read\tdegraded' \ + "the unfetchable uid is still surfaced degraded, not missed" + assert_contains "$out" $'2\t\ta@b.c\tgood\tok' \ + "a later uid surfaces as ok despite the earlier failure" + pass "fm-mail: an unfetchable uid is surfaced degraded and cannot starve later mail" +} + +test_poll_retries_transient_fetch_and_surfaces_real_metadata() { + local fakebin homedir_bin real_py harness retry_home control out rc=0 wakeq + fakebin=$(fm_fakebin "$TMP_ROOT") + retry_home="$TMP_ROOT/retry-home" + homedir_bin="$retry_home/bin" + mkdir -p "$homedir_bin" "$retry_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + real_py=$(command -v python3) + harness="$TMP_ROOT/retry-poll-harness.py" + control="$TMP_ROOT/retry-fetch-count" + printf '0' > "$control" + + # Drive the real poll_list through a stubbed IMAP connection whose header + # fetch fails on the first two polls and succeeds on the third, proving a + # transient failure surfaces degraded once, does not re-wake while still + # failing, then recovers the real From/Subject and leaves the retry set. + cat > "$harness" <<'PYEOF' +import imaplib, importlib.util, os, sys + +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'41']) + if cmd == 'fetch': + n = int(open(os.environ['FM_MAIL_TEST_FETCH_COUNT']).read() or '0') + if n < 3: + return ('NO', None) + return ('OK', [(b'', b'Subject: Hello captain\r\nFrom: alice@example.com\r\nDate: 5 Sep 2026 00:00:00 +0000\r\n\r\n')]) + return ('NO', None) + def logout(self): + pass + +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +if len(sys.argv) > 2 and sys.argv[2] == 'poll_list': + path = os.environ['FM_MAIL_TEST_FETCH_COUNT'] + n = int(open(path).read() or '0') + open(path, 'w').write(str(n + 1)) + sys.exit(mod.cmd_poll_list()) +sys.exit(1) +PYEOF + cat > "$fakebin/python3" <<EOF +#!/bin/bash +exec "$real_py" "$harness" "\$@" +EOF + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$retry_home/state/.mail-seen" + : > "$retry_home/state/.wake-queue" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$retry_home" PATH="$fakebin:$PATH" \ + FM_MAIL_TEST_FETCH_COUNT="$control" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first poll of a failing fetch must succeed" + assert_contains "$out" "woke for 41" "failed fetch still wakes once, never missed" + assert_contains "$(cat "$retry_home/state/.wake-queue")" "(no header)" \ + "first wake uses the degraded sender placeholder" + assert_contains "$(cat "$retry_home/state/.wake-queue")" "unfetchable header" \ + "first wake uses the degraded subject placeholder" + assert_contains "$(cat "$retry_home/state/.mail-retry" 2>/dev/null)" "41" \ + "the uid is recorded for retry after the degraded wake" + assert_contains "$(cat "$retry_home/state/.mail-seen")" "41" \ + "the degraded surfacing is still cursor-recorded" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$retry_home" PATH="$fakebin:$PATH" \ + FM_MAIL_TEST_FETCH_COUNT="$control" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "still-failing retry poll must succeed" + assert_not_contains "$out" "woke for 41" "a still-unfetchable retry uid must not re-wake" + assert_contains "$out" "no new mail" "a still-unfetchable retry poll reports no new mail" + assert_contains "$(cat "$retry_home/state/.mail-retry" 2>/dev/null)" "41" \ + "the uid stays in the retry set while the fetch keeps failing" + wakeq=$(grep -c "check: mail" "$retry_home/state/.wake-queue" 2>/dev/null || true) + expect_code 1 "$wakeq" "still-failing retry must not append a second wake" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$retry_home" PATH="$fakebin:$PATH" \ + FM_MAIL_TEST_FETCH_COUNT="$control" \ + "$MAIL" poll 2>&1) || rc=$? + if [ "$rc" != 0 ]; then + printf 'poll output on failure:\n%s\n' "$out" >&2 + fail "recovered fetch poll must succeed: expected exit 0, got $rc" + fi + assert_contains "$out" "woke for 41" "recovered fetch wakes with the real metadata" + assert_contains "$(cat "$retry_home/state/.wake-queue")" "alice@example.com" \ + "recovered wake names the real sender" + assert_contains "$(cat "$retry_home/state/.wake-queue")" "Hello captain" \ + "recovered wake names the real subject" + assert_not_contains "$(cat "$retry_home/state/.mail-retry" 2>/dev/null || true)" "41" \ + "the uid leaves the retry set after a successful fetch" + wakeq=$(grep -c "check: mail" "$retry_home/state/.wake-queue" 2>/dev/null || true) + expect_code 2 "$wakeq" "degraded then recovered metadata are two wakes" + pass "fm-mail: a transient fetch failure recovers real metadata on a later poll" +} + +test_poll_keeps_journal_when_heal_cannot_record() { + local fakebin homedir_bin out rc=0 + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # No unseen mail; the poll only heals the seeded journal entry. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + printf '%s\t%s\n' '90009' '55' > "$HOME_DIR/state/.mail-woken" + chmod 0400 "$HOME_DIR/state/.mail-seen" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "poll must fail when the heal cannot record a uid" + assert_contains "$out" "heal could not record a uid" "poll reports the unrecordable heal" + assert_contains "$(cat "$HOME_DIR/state/.mail-woken" 2>/dev/null)" "55" "journal evidence survives an unrecordable heal" + chmod 0600 "$HOME_DIR/state/.mail-seen" + pass "fm-mail: the journal survives when the heal cannot commit a uid" +} + +test_poll_heal_failure_does_not_rewake_unseen_mail() { + local fakebin homedir_bin heal_home out rc=0 + fakebin=$(fm_fakebin "$TMP_ROOT") + heal_home="$TMP_ROOT/heal-fail-home" + homedir_bin="$heal_home/bin" + mkdir -p "$homedir_bin" "$heal_home/state" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # Journal names uid 55; the cursor cannot be appended to; IMAP still lists + # 55 as UNSEEN. The poll must fail closed before the wake loop so the uid + # is not surfaced a second time. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '55\t2026-09-05T00:00:00Z\talice@example.com\tHello\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$heal_home/state/.mail-seen" + printf '%s\t%s\n' '90009' '55' > "$heal_home/state/.mail-woken" + : > "$heal_home/state/.wake-queue" + chmod 0400 "$heal_home/state/.mail-seen" + + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$heal_home" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 1 "$rc" "heal failure with unseen mail must fail the poll" + assert_not_contains "$out" "woke for 55" "heal failure must not re-wake a journaled uid" + assert_contains "$(cat "$heal_home/state/.mail-woken" 2>/dev/null)" "55" "journal evidence is kept" + local wakeq + wakeq=$(grep -c "check: mail 55" "$heal_home/state/.wake-queue" 2>/dev/null || true) + expect_code 0 "$wakeq" "heal failure must not append a second wake" + chmod 0600 "$heal_home/state/.mail-seen" + pass "fm-mail: heal failure with unseen mail fails the poll instead of re-waking" +} + +test_body_preview_falls_back_from_empty_plain() { + local harness out + harness="$TMP_ROOT/body-preview-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({'FM_MAIL_USER':'t','FM_MAIL_PASS':'p','FM_IMAP_HOST':'h','FM_IMAP_PORT':'993','FM_SMTP_HOST':'s','FM_SMTP_PORT':'465'}) +import email, importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +msg = email.message_from_string( + "Content-Type: multipart/alternative; boundary=b\r\n\r\n" + "--b\r\nContent-Type: text/plain\r\n\r\n\r\n" + "--b\r\nContent-Type: text/html\r\n\r\n<p>Hello</p>\r\n" + "--b--\r\n") +print(mod.body_preview(msg)) +PYEOF + out=$(python3 "$harness" "$ROOT/bin/fm-mail.py") + assert_contains "$out" "Hello" "empty plain-text alternative falls back to the html preview" + pass "fm-mail: an empty plain-text alternative falls back to the html preview" +} + +test_body_preview_tolerates_none_payload() { + local harness out rc=0 + harness="$TMP_ROOT/none-payload-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({'FM_MAIL_USER':'t','FM_MAIL_PASS':'p','FM_IMAP_HOST':'h','FM_IMAP_PORT':'993','FM_SMTP_HOST':'s','FM_SMTP_PORT':'465'}) +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) + +class NonePart: + def walk(self): + yield self + def get_content_type(self): + return 'text/plain' + def get_payload(self, decode=True): + return None + +print(repr(mod.body_preview(NonePart()))) +PYEOF + out=$(python3 "$harness" "$ROOT/bin/fm-mail.py") || rc=$? + expect_code 0 "$rc" "None payload must not crash body_preview" + assert_contains "$out" "''" "None payload yields an empty preview" + pass "fm-mail: body_preview does not crash on a None payload" +} + +test_read_surfaces_unfetchable_uid() { + local harness out rc=0 + harness="$TMP_ROOT/read-unfetchable-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', +}) +class FakeConn: + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'7 8']) + if cmd == 'fetch': + if args[0] == b'7': + return ('NO', None) + return ('OK', [(b'', b'From: a@b.c\r\nSubject: ok\r\n\r\nplain body\r\n')]) + return ('NO', None) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_read()) +PYEOF + out=$(python3 "$harness" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "read must succeed when one uid is unfetchable" + assert_contains "$out" "Uid: 7" "unfetchable uid is still named" + assert_contains "$out" "unfetchable body" "unfetchable uid is reported degraded" + assert_contains "$out" "Body: (body unavailable)" "degraded fetch row always prints a Body line" + assert_contains "$out" "Subj: ok" "a later fetchable uid is still shown" + pass "fm-mail: read does not silently hide an unfetchable uid" +} + +test_read_tolerates_none_payload() { + local harness out rc=0 + harness="$TMP_ROOT/read-none-payload-harness.py" + cat > "$harness" <<'PYEOF' +import os, sys +os.environ.update({ + 'FM_MAIL_USER': 't', 'FM_MAIL_PASS': 'p', + 'FM_IMAP_HOST': 'imap.test', 'FM_IMAP_PORT': '993', + 'FM_SMTP_HOST': 'smtp.test', 'FM_SMTP_PORT': '465', +}) +class FakeConn: + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'9']) + if cmd == 'fetch': + # imaplib may return a tuple whose payload bytes are None. + return ('OK', [(b'', None)]) + return ('NO', None) + def logout(self): + pass +import imaplib +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +import importlib.util +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +sys.exit(mod.cmd_read()) +PYEOF + out=$(python3 "$harness" "$ROOT/bin/fm-mail.py" 2>&1) || rc=$? + expect_code 0 "$rc" "read must succeed when the fetch payload is None" + assert_contains "$out" "Uid: 9" "None-payload uid is still named" + assert_contains "$out" "Body: (body unavailable)" "None-payload degraded row prints a Body line" + pass "fm-mail: read tolerates a None payload without crashing" +} + +test_invalid_port_fails_cleanly() { + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=h FM_SMTP_HOST=h \ + FM_IMAP_PORT=abc FM_HOME="$HOME_DIR" "$MAIL" status 2>&1) || rc=$? + expect_code 1 "$rc" "a non-numeric IMAP port must fail" + assert_contains "$out" "FM_IMAP_PORT" "invalid IMAP port names the variable" + assert_not_contains "$out" "ValueError" "invalid port must not leak a python traceback" + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=h FM_SMTP_HOST=h \ + FM_SMTP_PORT=abc FM_HOME="$HOME_DIR" "$MAIL" status 2>&1) || rc=$? + expect_code 1 "$rc" "a non-numeric SMTP port must fail" + assert_contains "$out" "FM_SMTP_PORT" "invalid SMTP port names the variable" + pass "fm-mail: a non-numeric port fails cleanly in bash" +} + +test_poll_caps_wakes_per_run() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # Three unseen messages with a per-poll cap of two: exactly two wakes this + # poll, and the third stays unseen so the next poll surfaces it. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +printf 'uidvalidity\t90009\n' +printf '71\t2026-09-05T00:00:00Z\talice@example.com\tA\n' +printf '72\t2026-09-05T00:00:00Z\talice@example.com\tB\n' +printf '73\t2026-09-05T00:00:00Z\talice@example.com\tC\n' +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.wake-queue" + + local out rc=0 wakeq + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" FM_MAIL_POLL_MAX_WAKES=2 \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "capped poll must succeed" + assert_contains "$out" "woke for 71" "first message wakes within the cap" + assert_contains "$out" "woke for 72" "second message wakes within the cap" + assert_not_contains "$out" "woke for 73" "third message must not wake in a capped poll" + assert_contains "$out" "per-poll wake cap" "poll reports the cap" + wakeq=$(grep -c "check: mail" "$HOME_DIR/state/.wake-queue" 2>/dev/null || true) + expect_code 2 "$wakeq" "the durable wake queue holds exactly the capped wakes" + + # The third message is still unseen: the next poll surfaces it. + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" FM_MAIL_POLL_MAX_WAKES=2 \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "follow-up poll must succeed" + assert_contains "$out" "woke for 73" "deferred message wakes on the next poll" + pass "fm-mail: per-poll wake cap bounds the durable queue without missing mail" +} + +test_poll_sanitizes_header_fields() { + local fakebin homedir_bin real_py harness + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + real_py=$(command -v python3) + harness="$TMP_ROOT/sanitize-poll-harness.py" + + # Drive the real poll_list/clean path through a stubbed IMAP connection so a + # tab in Subject and an RFC-2047-encoded newline cannot split the TSV or + # inject a forged uid for the bash wake loop. + cat > "$harness" <<'PYEOF' +import imaplib, importlib.util, sys + +class FakeConn: + untagged_responses = {'UIDVALIDITY': [b'90009']} + def __init__(self, *a, **k): + pass + def login(self, *a): + pass + def select(self, *a): + return ('OK', []) + def uid(self, cmd, *args): + if cmd == 'search': + return ('OK', [b'60 61']) + if cmd == 'fetch': + if args[0] == b'60': + return ('OK', [(b'', b'Subject: Tab\there\r\nFrom: alice@example.com\r\nDate: 5 Sep 2026 00:00:00 +0000\r\n\r\n')]) + # RFC-2047 payload of "Line\n99\tfake" so decode keeps the newline + # and tab; clean() must collapse them or bash would wake forged uid 99. + raw = b'Subject: =?utf-8?b?TGluZQo5OQlmYWtl?=\r\nFrom: alice@example.com\r\nDate: 5 Sep 2026 00:00:00 +0000\r\n\r\n' + return ('OK', [(b'', raw)]) + return ('NO', None) + def logout(self): + pass + +imaplib.IMAP4_SSL = lambda *a, **k: FakeConn() +spec = importlib.util.spec_from_file_location('fm_mail', sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +if len(sys.argv) > 2 and sys.argv[2] == 'poll_list': + sys.exit(mod.cmd_poll_list()) +sys.exit(1) +PYEOF + cat > "$fakebin/python3" <<EOF +#!/bin/bash +exec "$real_py" "$harness" "\$@" +EOF + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.wake-queue" + + local out rc=0 wakeq + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "sanitizing poll must succeed" + assert_contains "$out" "woke for 60" "tab-bearing subject still wakes once" + assert_contains "$out" "woke for 61" "newline-bearing subject still wakes once" + assert_not_contains "$out" "woke for 99" "a newline in Subject must not inject a forged uid" + assert_not_contains "$out" "woke for fake" "a tab in Subject must not inject a forged uid" + wakeq=$(grep -c "check: mail" "$HOME_DIR/state/.wake-queue" 2>/dev/null || true) + expect_code 2 "$wakeq" "exactly the two real uids wake" + assert_contains "$(cat "$HOME_DIR/state/.wake-queue")" "mail:90009/60" "wake key is the real uid 60" + assert_contains "$(cat "$HOME_DIR/state/.wake-queue")" "mail:90009/61" "wake key is the real uid 61" + pass "fm-mail: poll sanitizes tabs and newlines in header fields" +} + +test_poll_bounded_fetch_progresses_large_backlog() { + local fakebin homedir_bin + fakebin=$(fm_fakebin "$TMP_ROOT") + homedir_bin="$HOME_DIR/bin" + mkdir -p "$homedir_bin" + [ -e "$homedir_bin/fm-wake-lib.sh" ] || ln -s "$ROOT/bin/fm-wake-lib.sh" "$homedir_bin/fm-wake-lib.sh" + + # Fake python3 emulating the bounded poll_list: read FM_MAIL_CURSOR, return + # only uids not already recorded in the cursor, capped at the poll cap. Five + # unseen messages with a cap of two must progress two per poll and finish + # cleanly, never stalling on the already-surfaced head of a large backlog. + cat > "$fakebin/python3" <<'SH' +#!/usr/bin/env bash +cursor="$FM_MAIL_CURSOR" +cap="${FM_MAIL_POLL_MAX_WAKES:-20}" +seen="" +[ -f "$cursor" ] && seen="$(grep -v '^uidvalidity=' "$cursor" 2>/dev/null || true)" +printf 'uidvalidity\t90009\n' +count=0 +for u in 91 92 93 94 95; do + if printf '%s\n' "$seen" | grep -Fqx "$u"; then continue; fi + [ "$count" -ge "$cap" ] && break + printf '%s\t2026-09-05T00:00:00Z\talice@example.com\tM\n' "$u" + count=$((count + 1)) +done +SH + chmod +x "$fakebin/python3" + printf 'uidvalidity=90009\n' > "$HOME_DIR/state/.mail-seen" + : > "$HOME_DIR/state/.wake-queue" + + local out rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" FM_MAIL_POLL_MAX_WAKES=2 \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "first bounded poll must succeed" + assert_contains "$out" "woke for 91" "first batch surfaces 91" + assert_contains "$out" "woke for 92" "first batch surfaces 92" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" FM_MAIL_POLL_MAX_WAKES=2 \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "second bounded poll must succeed" + assert_contains "$out" "woke for 93" "second batch surfaces 93" + assert_contains "$out" "woke for 94" "second batch surfaces 94" + + rc=0 + out=$(FM_MAIL_USER=test FM_MAIL_PASS=pass FM_IMAP_HOST=imap.test FM_SMTP_HOST=smtp.test \ + FM_HOME="$HOME_DIR" PATH="$fakebin:$PATH" FM_MAIL_POLL_MAX_WAKES=2 \ + "$MAIL" poll 2>&1) || rc=$? + expect_code 0 "$rc" "third bounded poll must succeed" + assert_contains "$out" "woke for 95" "tail batch surfaces 95" + assert_not_contains "$out" "woke for 91" "already-surfaced mail never re-wakes" + pass "fm-mail: bounded fetch makes progress through a large backlog without re-surfacing mail" +} + +test_missing_secret_fails_cleanly +test_env_overrides_env_file +test_status_without_network +test_help_plumbing +test_unknown_subcommand_prints_usage +test_no_secret_leaked_to_status +test_send_passes_body +test_poll_error_propagates +test_poll_dedupes_surfaces_by_uid +test_poll_resurfaces_uid_after_generation_change +test_poll_heals_wake_without_cursor_record +test_poll_duplicate_wakes_on_interrupted_poll +test_poll_serializes_overlapping_invocations +test_poll_recovers_journaled_wake_after_ack +test_poll_acknowledged_wake_evading_recovery +test_poll_legacy_wake_does_not_leak_into_generation +test_poll_missing_wake_lib_does_not_suppress +test_poll_rolls_back_wake_without_durable_record +test_poll_rollback_failure_never_leaves_unrecorded_ackable_wake +test_poll_caps_wakes_per_run +test_poll_sanitizes_header_fields +test_poll_bounded_fetch_progresses_large_backlog +test_poll_skips_unfetchable_uid_but_keeps_progress +test_poll_retries_transient_fetch_and_surfaces_real_metadata +test_poll_retry_cursor_advances_past_failures +test_poll_retry_position_does_not_advance_past_unpublished_wake +test_poll_retry_budget_advances_by_examined_not_window +test_poll_retry_window_of_unseen_never_stalls_cursor +test_poll_retry_unfetchable_window_advances_with_new_mail +test_poll_retry_unseen_window_advances_with_new_mail +test_poll_retry_logout_before_emit_and_position_save +test_poll_list_logs_out_on_imap_error +test_poll_retry_position_not_saved_when_row_emitted +test_poll_retry_small_window_rotates_past_unfetchable_prefix +test_poll_cap_one_turn_not_saved_before_emit +test_poll_cap_one_turn_not_saved_when_retry_pos_write_fails +test_poll_retry_surfaces_under_new_mail_flood +test_poll_resurfaces_degraded_uid_whose_wake_never_recorded +test_poll_cap_one_never_suppresses_new_mail +test_poll_cap_one_alternates_new_and_retry +test_poll_cap_one_does_not_advance_unexamined_retry_window +test_poll_fails_closed_when_retry_unwritable +test_poll_journal_failure_never_cursor_records +test_poll_fails_closed_when_retry_clear_fails +test_poll_fails_closed_when_stale_retry_clear_fails +test_assert_equals_rejects_mismatch +test_poll_fails_closed_when_poll_list_fails +test_poll_restores_retry_when_recovered_wake_cannot_append +test_poll_death_between_retry_remove_and_publish_does_not_strand +test_poll_fetch_raise_does_not_abort_the_scan +test_poll_keeps_journal_when_heal_cannot_record +test_poll_heal_failure_does_not_rewake_unseen_mail +test_body_preview_falls_back_from_empty_plain +test_body_preview_tolerates_none_payload +test_read_tolerates_none_payload +test_read_surfaces_unfetchable_uid +test_invalid_port_fails_cleanly diff --git a/tests/fm-merge-local.test.sh b/tests/fm-merge-local.test.sh index ba6e127a612..34fc8b08c42 100755 --- a/tests/fm-merge-local.test.sh +++ b/tests/fm-merge-local.test.sh @@ -14,7 +14,8 @@ test_recorded_continued_branch_lands_locally() { project="$case_dir/project" wt="$case_dir/wt" branch=feature/existing-local - mkdir -p "$case_dir/state" "$project" + mkdir -p "$case_dir/state" "$case_dir/data" "$project" + cp "$ROOT/.tasks.toml" "$case_dir/.tasks.toml" git init -q "$project" git -C "$project" symbolic-ref HEAD refs/heads/main printf 'base\n' > "$project/base.txt" @@ -29,8 +30,9 @@ test_recorded_continued_branch_lands_locally() { fm_write_meta "$case_dir/state/task-x1.meta" \ "project=$project" "mode=local-only" "branch=$branch" - out=$(FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$case_dir/state" \ - "$MERGE_LOCAL" task-x1 2> "$case_dir/stderr") + out=$(FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$case_dir" FM_STATE_OVERRIDE="$case_dir/state" \ + "$MERGE_LOCAL" task-x1 2> "$case_dir/stderr") \ + || fail "local merge refused the fixture: $(cat "$case_dir/stderr")" [ "$(git -C "$project" rev-parse main)" = "$after" ] \ || fail "local landing ignored the recorded continued branch" diff --git a/tests/fm-on.test.sh b/tests/fm-on.test.sh index c4068cbd15f..6f82310a53b 100755 --- a/tests/fm-on.test.sh +++ b/tests/fm-on.test.sh @@ -55,7 +55,7 @@ case "\${1:-}:\${2:-}" in esac SH cp "$ROOT/bin/fm-remote-doctor.sh" "$ROOT/bin/fm-tasks-axi-lib.sh" \ - "$ROOT/bin/fm-backend.sh" "$REMOTE_ROOT/bin/" + "$ROOT/bin/fm-remote-herdr-owner-lib.sh" "$ROOT/bin/fm-backend.sh" "$REMOTE_ROOT/bin/" mkdir -p "$REMOTE_ROOT/bin/backends" cp "$ROOT/bin/backends/herdr.sh" "$REMOTE_ROOT/bin/backends/herdr.sh" cat > "$REMOTE_ROOT/bin/fm-mutate.sh" <<'SH' diff --git a/tests/fm-pending-reply.test.sh b/tests/fm-pending-reply.test.sh index b752c3945ca..1c1353050db 100755 --- a/tests/fm-pending-reply.test.sh +++ b/tests/fm-pending-reply.test.sh @@ -27,6 +27,8 @@ # escalating as a false miss # 14. The mechanical helper writes the parent channel from (verb, corr, note) # 15. Remote parent-replies.status is not classified as wrong-home +# 16. An escalated correlation stays retryable while undelivered, is never reset +# once delivered, and its delivery-unknown decision still closes on resolve set -u # shellcheck source=tests/lib.sh @@ -1504,6 +1506,70 @@ test_failed_send_discards_undelivered_expectation() { pass "failed transport discards undelivered expectation only" } +test_escalated_undelivered_correlation_stays_retryable() { + local home state corr rec marker delivered_corr delivered_rec open + home=$(setup_parent escalated-retry) + state="$home/state" + export FM_PENDING_REPLY_NOW=9600 + corr=$(fm_pending_reply_create "$home" "$state" "hibit" "wake after lost transport") + rec=$(fm_pending_reply_path "$state" "$corr") + marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr") + # The owner prepared the delivery, the remote transport was lost, and the + # watcher escalated the unknown delivery before any resend ran. + fm_pending_reply_prepare_delivery "$state" "$corr" || fail "prepare delivery failed" + fm_pending_reply_mark_delivery_unknown "$state" "$corr" || fail "mark delivery unknown failed" + fm_pending_reply_maybe_escalate "$state" "$corr" || fail "delivery-unknown escalation should fire" + [ "$(phase_of "$state" "$corr")" = escalated ] || fail "phase should be escalated" + [ -z "$(fm_pending_reply_get "$rec" delivered_epoch)" ] || fail "escalation must not invent delivery" + [ "$(grep -cF "blocked [key=pending-reply-$corr]:" "$state/hibit.status")" = 1 ] \ + || fail "delivery-unknown escalation should publish once" + fm_pending_reply_corr_reusable "$state" "$corr" hibit \ + || fail "an escalated undelivered correlation must stay reusable by its owner" + if fm_pending_reply_corr_reusable "$state" "$corr" other 2>/dev/null; then + fail "an escalated correlation must not be reusable for another task" + fi + fm_pending_reply_reset_known_undelivered "$state" "$corr" \ + || fail "an escalated undelivered correlation must reset for its resend" + [ "$(phase_of "$state" "$corr")" = awaiting_report ] \ + || fail "reset should return the undelivered escalation to awaiting_report" + [ ! -e "$marker" ] || fail "reset should drop the stale attempted marker" + [ -n "$(fm_pending_reply_get "$rec" escalated_epoch)" ] \ + || fail "reset must keep the escalation history so its decision can still close" + open=$(status_open_decisions "$state/hibit.status" | cut -f1) + [ "$open" = "pending-reply-$corr" ] \ + || fail "the published escalation must stay open until the record resolves, got '$open'" + # The resend lands and the mate reports: the ordinary resolve closes the + # delivery-unknown decision the retry left open. + export FM_PENDING_REPLY_NOW=9601 + fm_pending_reply_prepare_delivery "$state" "$corr" || fail "resend prepare failed" + fm_pending_reply_confirm_delivery "$state" "$corr" || fail "resend confirm failed" + [ -n "$(fm_pending_reply_get "$rec" delivered_epoch)" ] || fail "resend should confirm delivery" + printf 'done [corr=%s]: routed work picked up\n' "$corr" >> "$state/hibit.status" + fm_pending_reply_try_resolve "$state" "$corr" || fail "correlated report should resolve" + [ "$(phase_of "$state" "$corr")" = resolved ] || fail "phase should be resolved" + open=$(status_open_decisions "$state/hibit.status") + [ -z "$open" ] || fail "resolution left the delivery-unknown decision open: $open" + # A delivered record escalated for a genuine missed report is never reset. + export FM_PENDING_REPLY_NOW=9700 + delivered_corr=$(fm_pending_reply_create "$home" "$state" "hibit" "delivered then missed") + delivered_rec=$(fm_pending_reply_path "$state" "$delivered_corr") + fm_pending_reply_mark_delivered "$state" "$delivered_corr" || fail "mark delivered failed" + fm_pending_reply_set "$delivered_rec" phase escalated + fm_pending_reply_set "$delivered_rec" escalated_epoch 9700 + if fm_pending_reply_corr_reusable "$state" "$delivered_corr" hibit 2>/dev/null; then + fail "a delivered escalated correlation must not be reusable" + fi + if fm_pending_reply_reset_known_undelivered "$state" "$delivered_corr" 2>/dev/null; then + fail "a delivered escalated correlation must never be reset" + fi + [ "$(phase_of "$state" "$delivered_corr")" = escalated ] \ + || fail "refused reset must leave the delivered escalation untouched" + [ "$(fm_pending_reply_get "$delivered_rec" delivered_epoch)" = 9700 ] \ + || fail "refused reset must keep the confirmed delivery" + unset FM_PENDING_REPLY_NOW + pass "an escalated correlation stays retryable only while undelivered" +} + # --- run -------------------------------------------------------------------- test_normal_correlated_reply_resolves_once @@ -1544,5 +1610,6 @@ test_child_status_wrong_home_is_not_copied test_mechanical_helper_writes_parent_channel test_remote_parent_replies_is_not_wrong_home test_local_parent_replies_is_wrong_home_evidence +test_escalated_undelivered_correlation_stays_retryable printf 'ok - all pending-reply tests passed\n' diff --git a/tests/fm-pi-branch-extension.test.sh b/tests/fm-pi-branch-extension.test.sh index 19e569d4d8c..0b9d9e7851c 100644 --- a/tests/fm-pi-branch-extension.test.sh +++ b/tests/fm-pi-branch-extension.test.sh @@ -54,6 +54,7 @@ install_pi_branch_extension_fixture() { "$repo/node_modules/typebox" cp "$EXT" "$repo/.pi/extensions/fm-branch-supervision.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$repo/.pi/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$repo/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$repo/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-model-picker.ts" "$repo/.pi/extensions/lib/fm-branch-model-picker.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$repo/.pi/extensions/lib/fm-calm-visibility.ts" @@ -1244,8 +1245,8 @@ test_captain_outcome_processing_turn_is_sequence_keyed_and_re_presented() { PLUGIN="$repo/.pi/extensions/fm-branch-supervision.ts" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ DRIVER_PRELUDE="$DRIVER_PRELUDE" node --input-type=module > "$TMP_ROOT/node-output" 2>&1 <<'EOF' const prelude = process.env.DRIVER_PRELUDE; -await eval(`(async () => { ${prelude}; globalThis.__t = { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home }; })()`); -const { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home } = globalThis.__t; +await eval(`(async () => { ${prelude}; globalThis.__t = { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home, bus }; })()`); +const { fire, dispatch, settle, sentToMain, mainEntries, mainTools, outcomeScript, defaultSessionCtx, home, bus } = globalThis.__t; import { readFileSync, writeFileSync } from "node:fs"; const requests = () => sentToMain.filter((sent) => sent.message.customType === "fm-branch-process"); @@ -1331,7 +1332,18 @@ if (mainEntries.filter((entry) => entry.customType === "fm-branch-visible-outcom } // Only the sequence-bound acknowledgement closes it. -const processed = mainTools.find((tool) => tool.name === "fm_branch_processed"); +const nativeTools = new Map(); +const messageTypes = new Set(); +bus.emit("firstmate:native-tools", { + register: (tool) => nativeTools.set(tool.name, tool), + allowMessageType: (type) => messageTypes.add(type), +}); +if ([...nativeTools.keys()].sort().join(",") !== "fm_branch_outcomes,fm_branch_processed") throw new Error("native discovery exposed unrelated tools"); +for (const tool of mainTools) { + if (nativeTools.get(tool.name)?.execute !== tool.execute) throw new Error("native controls lost the original guards"); +} +if ([...messageTypes].sort().join(",") !== "firstmate-sessionstart-nudge,fm-branch-merge,fm-branch-process") throw new Error("operational message allowlist changed"); +const processed = nativeTools.get("fm_branch_processed"); if (!processed) throw new Error("main did not receive its acknowledgement tool"); const routineAck = await processed.execute("ack-routine", { through: routineSeq }, undefined, undefined, {}); if (!routineAck.isError || !routineAck.content.some((item) => item.type === "text" && item.text.includes("not an unprocessed captain outcome"))) { @@ -2630,6 +2642,16 @@ if (cleared.options.model?.id === "cheap-1") { if (cleared.options.model?.provider !== "anthropic" || cleared.options.model?.id !== "main-model") { throw new Error(`clearing the pin did not return the branch to main's model: ${JSON.stringify(cleared.options.model)}`); } +// A native main must use an explicit independent ordinary-Pi branch. +registryModels.push({ provider: "openai-codex", id: "gpt-6-astra" }); +await fire("session_shutdown", {}); +await fire("session_start", {}, makeCtx({ model: { provider: "codex-native", id: "gpt-6-astra" } })); +dispatch("signal: native main ordinary branch"); +await settle(() => (globalThis.__fmSessions ?? []).length === 6, "native-main branch build"); +const nativeBranch = globalThis.__fmSessions[5].options.model; +if (nativeBranch?.provider !== "openai-codex" || nativeBranch?.id !== "gpt-6-astra") { + throw new Error(`native main inherited an unsafe branch runtime: ${JSON.stringify(nativeBranch)}`); +} process.exit(0); EOF status=$? @@ -2839,6 +2861,52 @@ if ( ) { throw new Error(`post-clear resolution failure was not reported honestly: ${JSON.stringify(clearFailureNotices)}`); } + +// Under a codex-native main, Follow main reports the ordinary openai-codex +// model the next build actually runs, and the picker never offers the main +// native provider itself. +registryModels.push({ provider: "codex-native", id: "gpt-6-astra" }, { provider: "openai-codex", id: "gpt-6-astra" }); +const nativeCtx = makeCtx({ model: { provider: "codex-native", id: "gpt-6-astra" } }); +const nativePromptCount = uiPrompts.length; +const nativeNoticeCount = notices.length; +uiSelections.push("Follow main (codex-native/gpt-6-astra)"); +await command.handler("", nativeCtx); +const nativeOffer = uiPrompts[nativePromptCount]; +if (nativeOffer.options[0] !== "Follow main (codex-native/gpt-6-astra)" || nativeOffer.options.includes("codex-native/gpt-6-astra")) { + throw new Error(`the picker must offer following a native main without offering its native provider: ${JSON.stringify(nativeOffer.options)}`); +} +const nativeNotices = notices.slice(nativeNoticeCount); +if (nativeNotices.length !== 1 || nativeNotices[0].type !== "info" || !nativeNotices[0].message.includes("openai-codex/gpt-6-astra")) { + throw new Error(`following a native main did not report the ordinary Pi model the build uses: ${JSON.stringify(nativeNotices)}`); +} +dispatch("signal: native follow"); +await settle(() => (globalThis.__fmSessions ?? []).length === 6, "native-main follow build"); +const nativeFollowed = globalThis.__fmSessions[5].options.model; +if (nativeFollowed?.provider !== "openai-codex" || nativeFollowed?.id !== "gpt-6-astra") { + throw new Error(`the build did not run the model the picker reported: ${JSON.stringify(nativeFollowed)}`); +} + +// When that ordinary model is unavailable, the picker reports the refusal the +// next build enforces instead of claiming the branch keeps a recorded model. +registryModels.splice(registryModels.findIndex((model) => model.provider === "openai-codex" && model.id === "gpt-6-astra"), 1); +const refusalNoticeCount = notices.length; +uiSelections.push("Follow main (codex-native/gpt-6-astra)"); +await command.handler("", nativeCtx); +const refusalNotices = notices.slice(refusalNoticeCount); +if ( + refusalNotices.length !== 1 || + refusalNotices[0].type !== "warning" || + !refusalNotices[0].message.includes("refuses to build") || + refusalNotices[0].message.includes("keeps the model its own session recorded") +) { + throw new Error(`following an unavailable native main did not report the build refusal: ${JSON.stringify(refusalNotices)}`); +} +const refusedOffer = dispatch("signal: native follow refused"); +const refusal = await refusedOffer.settlement.then(() => null, (error) => error); +if (!(refusal instanceof Error) || !refusal.message.includes("refuses to build")) { + throw new Error(`the build did not refuse as the picker reported: ${String(refusal)}`); +} +if (globalThis.__fmSessions.length !== 6) throw new Error("a refused native follow still built a branch"); process.exit(0); EOF status=$? @@ -3376,6 +3444,18 @@ const unparseable = globalThis.__fmSessions[0].options.model; if (unparseable?.provider !== "anthropic" || unparseable?.id !== "main-model") { throw new Error(`an unparseable pin must be treated as no pin and follow main: ${JSON.stringify(unparseable)}`); } +// Even a registered native provider cannot be selected by the independent +// supervision session: its persistent native thread belongs to main. +registryModels.push({ provider: "codex-native", id: "gpt-6-astra" }); +writeFileSync(`${home}/config/supervision-branch-model`, "codex-native/gpt-6-astra\n"); +await fire("session_shutdown", {}); +await fire("session_start", {}, makeCtx()); +const nativeOffer = dispatch("signal: native branch pin refused"); +const nativeFailure = await nativeOffer.settlement.then(() => null, (error) => error); +if (!(nativeFailure instanceof Error) || !nativeFailure.message.includes("ordinary Pi provider")) { + throw new Error(`native branch pin was not explicitly refused: ${String(nativeFailure)}`); +} +if (globalThis.__fmSessions.length !== 1) throw new Error("native pin built a shared native branch"); process.exit(0); EOF status=$? @@ -3726,6 +3806,7 @@ test_branch_dispatch_classifies_main_only_rows_and_writes_the_eligible_snapshot( home="$TMP_ROOT/dispatch-classify-home" mkdir -p "$repo/.pi/extensions/lib" "$home/state" "$home/projects/approved" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$repo/.pi/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$repo/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$repo/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-model-picker.ts" "$repo/.pi/extensions/lib/fm-branch-model-picker.ts" printf 'project=%s/projects/approved\nwindow=fm-window\n' "$home" > "$home/state/task-a.meta" @@ -4172,6 +4253,7 @@ test_outcomes_tool_uses_stock_execution_and_export_consumers() { mkdir -p "$fixture/.pi/extensions/lib" "$fixture/node_modules/@earendil-works" cp "$EXT" "$fixture/.pi/extensions/fm-branch-supervision.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$fixture/.pi/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$fixture/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$fixture/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-model-picker.ts" "$fixture/.pi/extensions/lib/fm-branch-model-picker.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$fixture/.pi/extensions/lib/fm-calm-visibility.ts" diff --git a/tests/fm-pi-branch-live-e2e.test.sh b/tests/fm-pi-branch-live-e2e.test.sh index 4941eab24ba..e909f63002a 100644 --- a/tests/fm-pi-branch-live-e2e.test.sh +++ b/tests/fm-pi-branch-live-e2e.test.sh @@ -55,6 +55,7 @@ mkdir -p "$repo/.pi/extensions/lib" "$repo/node_modules/@earendil-works" \ cp "$ROOT/.pi/extensions/fm-branch-supervision.ts" "$repo/.pi/extensions/fm-branch-supervision.ts" cp "$ROOT/.pi/extensions/fm-primary-pi-watch.ts" "$repo/.pi/extensions/fm-primary-pi-watch.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$repo/.pi/extensions/lib/fm-branch-dispatch.ts" +cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$repo/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$repo/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-model-picker.ts" "$repo/.pi/extensions/lib/fm-branch-model-picker.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$repo/.pi/extensions/lib/fm-calm-visibility.ts" diff --git a/tests/fm-pi-branch-responsiveness-live-e2e.test.sh b/tests/fm-pi-branch-responsiveness-live-e2e.test.sh index bd2645378b5..01257dd6b62 100755 --- a/tests/fm-pi-branch-responsiveness-live-e2e.test.sh +++ b/tests/fm-pi-branch-responsiveness-live-e2e.test.sh @@ -53,7 +53,7 @@ cleanup() { trap cleanup EXIT cp "$ROOT/.pi/extensions/fm-branch-supervision.ts" "$PROJECT/.pi/extensions/fm-branch-supervision.ts" -for lib in fm-async-exec fm-branch-dispatch fm-branch-model-picker fm-calm-visibility fm-operational-input; do +for lib in fm-async-exec fm-branch-dispatch fm-branch-model-picker fm-calm-visibility fm-native-contract fm-operational-input; do cp "$ROOT/.pi/extensions/lib/$lib.ts" "$PROJECT/.pi/extensions/lib/$lib.ts" done diff --git a/tests/fm-pi-codex-native.test.sh b/tests/fm-pi-codex-native.test.sh new file mode 100755 index 00000000000..137bd62d18f --- /dev/null +++ b/tests/fm-pi-codex-native.test.sh @@ -0,0 +1,362 @@ +#!/usr/bin/env bash +# Native Codex-through-Pi primary compatibility guard. Runs installed Pi and +# the adapter package with actual FirstMate extensions and durable scripts. +# A local protocol peer and watcher close stand in for model calls and a live +# fleet. No provider request leaves the machine and no live fleet is touched. +# PI_CODEX_NATIVE_PACKAGE selects the installed adapter package; FM_PI_BIN +# selects Pi. FM_NATIVE_TEST_KEEP=1 preserves the isolated fixture for diagnosis. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +fm_live_gate default-on FM_PI_CODEX_NATIVE_LIVE node "${FM_PI_BIN:-pi}" +PI_CODEX_NATIVE_PACKAGE=${PI_CODEX_NATIVE_PACKAGE:-"$HOME/.pi/agent/packages/pi-codex-native"} +if [ ! -f "$PI_CODEX_NATIVE_PACKAGE/index.ts" ]; then + if [ "${FM_PI_CODEX_NATIVE_LIVE:-${FM_LIVE:-0}}" = 1 ]; then + fail "native Pi adapter missing: $PI_CODEX_NATIVE_PACKAGE" + fi + echo "skip: native Pi adapter not installed (PI_CODEX_NATIVE_PACKAGE to select it)" + exit 0 +fi +export PI_CODEX_NATIVE_PACKAGE +FM_NATIVE_TEST_ROOT="$ROOT" node --input-type=module <<'JS' +import fs from "node:fs"; +import path from "node:path"; +import assert from "node:assert/strict"; +import { spawn, execFileSync } from "node:child_process"; +import { homedir, tmpdir } from "node:os"; +const root = process.env.FM_NATIVE_TEST_ROOT; +const nativePackage = + process.env.PI_CODEX_NATIVE_PACKAGE || + path.join(homedir(), ".pi/agent/packages/pi-codex-native"); +const piVersion = execFileSync(process.env.FM_PI_BIN || "pi", ["--version"], { + encoding: "utf8", timeout: 10000, +}).trim(); +let adapterVersion = "unknown"; +try { + adapterVersion = JSON.parse(fs.readFileSync(path.join(nativePackage, "package.json"), "utf8")).version || "unknown"; +} catch { /* A source-only adapter may omit package metadata. */ } +console.log(`Native Codex guard: Pi ${piVersion}, pi-codex-native ${adapterVersion}`); +const fixture = fs.mkdtempSync(path.join(tmpdir(), "fm-native-primary.")); +const repo = path.join(fixture, "repo"), + home = path.join(fixture, "home"), + state = path.join(home, "state"); +fs.mkdirSync(path.join(repo, "bin"), { recursive: true }); +fs.mkdirSync(state, { recursive: true }); +fs.mkdirSync(path.join(home, "config")); +fs.cpSync( + path.join(root, ".pi/extensions"), + path.join(repo, ".pi/extensions"), + { recursive: true }, +); +for (const item of fs.readdirSync(path.join(root, "bin"))) + fs.symlinkSync(path.join(root, "bin", item), path.join(repo, "bin", item)); +function script(name, text) { + const file = path.join(repo, "bin", name); + fs.unlinkSync(file); + fs.writeFileSync(file, text, { mode: 0o755 }); +} +script( + "fm-sessionstart-run.sh", + '#!/usr/bin/env bash\nprintf "NATIVE_PRIMARY_STARTUP_SENTINEL\\n"\n', +); +script("fm-turnend-guard.sh", "#!/usr/bin/env bash\nexit 0\n"); +script( + "fm-watch-arm.sh", + `#!/usr/bin/env bash +if [ "\${1:-}" = --handling-delivered ]; then printf 'confirmed\\n' >> "$FM_HOME/state/arm.log"; exit 0; fi +printf 'arm\\n' >> "$FM_HOME/state/arm.log" +printf 'watcher: started pid=%s (beacon fresh) recovery-generation=native-smoke\\n' "$$" +trap 'exit 0' TERM INT +while :; do + if [ -f "$FM_HOME/state/trigger" ]; then rm -f "$FM_HOME/state/trigger"; printf 'check: native-worker-complete\\n'; exit 0; fi + sleep 0.05 +done +`, +); +const own = path.join(fixture, "own-lock.ts"); +fs.writeFileSync( + own, + `import {writeFileSync,appendFileSync} from 'node:fs'; export default function(pi){pi.on('session_start',()=>writeFileSync(process.env.FM_HOME+'/state/.lock',String(process.pid)+'\\n'));pi.events.on('codex-native:progress',event=>appendFileSync(process.env.FM_HOME+'/state/progress-events',JSON.stringify(event)+'\\n'));}`, +); +const peer = path.join(fixture, "native-peer.mjs"); +fs.writeFileSync( + peer, + `#!/usr/bin/env node +import fs from 'node:fs'; +let buffer='',counter=0,config; +const log=(data)=>fs.appendFileSync(process.env.FM_HOME+'/state/native.log',JSON.stringify(data)+'\\n'); +const emit=(x)=>process.stdout.write(JSON.stringify(x)+'\\n'); +async function control(name,args={}){ + const response=await fetch(config.url,{method:'POST',headers:{...config.http_headers,'Content-Type':'application/json',Accept:'application/json, text/event-stream'},body:JSON.stringify({jsonrpc:'2.0',id:++counter,method:'tools/call',params:{name,arguments:args}})}); + const value=await response.json(); if(value.error) throw Error(JSON.stringify(value));log({kind:'control',name,args,result:value.result});return value.result; +} +async function handle(q){ + const reply=(result)=>emit({id:q.id,result});const thread={id:'fm-native-primary-thread',cwd:process.cwd(),turns:[],status:{type:'idle'}}; + if(q.method==='initialize')return reply({userAgent:'native-primary-smoke/1'}); + if(q.method==='model/list')return reply({data:[{id:'gpt-6-astra',model:'gpt-6-astra',displayName:'Astra',supportedReasoningEfforts:['high','ultra'].map(reasoningEffort=>({reasoningEffort,description:reasoningEffort})),defaultReasoningEffort:'high',inputModalities:['text'],isDefault:true}],nextCursor:null}); + if(q.method==='thread/start'||q.method==='thread/resume'){config=q.params.config.mcp_servers.pi_firstmate;log({kind:'thread',method:q.method});return reply({thread,model:'gpt-6-astra',modelProvider:'openai',cwd:process.cwd(),approvalPolicy:'never',sandbox:{type:'dangerFullAccess'}});} + if(q.method==='turn/start'){ + const id='native-turn-'+ ++counter;const text=JSON.stringify(q.params.input);log({kind:'input',text,effort:q.params.effort});reply({turn:{id,status:'inProgress',items:[]}});emit({method:'turn/started',params:{threadId:thread.id,turn:{id,status:'inProgress',items:[]}}}); + // Yield once so the adapter has accepted the turn and opened its MCP guard. + await new Promise(r=>setTimeout(r,150)); + if(text.includes('ARM_PRIMARY'))await control('fm_watch_arm_pi'); + const seq=text.match(/\\[seq (\\d+)\\]/); + if(seq){await control('fm_branch_outcomes',{recent:1});await control('fm_branch_processed',{through:Number(seq[1])});await control('fm_branch_processed',{through:Number(seq[1])});} + const answer=seq?'NATIVE_OUTCOME_HANDLED':'NATIVE_PRIMARY_READY'; + emit({method:'item/agentMessage/delta',params:{threadId:thread.id,turnId:id,itemId:id+'-answer',delta:answer}}); + emit({method:'item/completed',params:{threadId:thread.id,turnId:id,item:{type:'agentMessage',id:id+'-answer',text:answer,phase:'final_answer'}}}); + emit({method:'turn/completed',params:{threadId:thread.id,turn:{id,status:'completed',items:[],error:null}}});return; + } + if(q.id!==undefined)reply({}); +} +process.stdin.setEncoding('utf8');process.stdin.on('data',data=>{buffer+=data;let i;while((i=buffer.indexOf('\\n'))>=0){const s=buffer.slice(0,i);buffer=buffer.slice(i+1);if(s.trim())void handle(JSON.parse(s)).catch(e=>{log({error:String(e)});process.exit(2)});}});process.stdin.on('end',()=>process.exit(0)); +`, + { mode: 0o755 }, +); +const delay = (ms) => new Promise((r) => setTimeout(r, ms)); +let child, + events = [], + next = 1; +async function wait(test, what, ms = 30000) { + const until = Date.now() + ms; + while (Date.now() < until) { + const result = test(); + if (result) return result; + if (child?.exitCode != null) + throw Error(`Pi exited while ${what}: ${child.err}`); + await delay(50); + } + throw Error( + `Timeout ${what}; ${child?.err}; ${JSON.stringify(events.slice(-5))}`, + ); +} +const log = () => { + try { + return fs + .readFileSync(path.join(state, "native.log"), "utf8") + .trim() + .split("\n") + .filter(Boolean) + .map(JSON.parse); + } catch { + return []; + } +}; +async function send(type, fields = {}) { + const id = String(next++); + child.stdin.write(JSON.stringify({ id, type, ...fields }) + "\n"); + const response = await wait( + () => events.find((e) => e.id === id), + "RPC " + type, + ); + assert(response.success, JSON.stringify(response)); + return response; +} +async function start(resume) { + events = []; + const args = [ + "--mode", + "rpc", + "--offline", + "--no-extensions", + "--no-skills", + "--no-context-files", + "--approve", + "-e", + own, + "-e", + path.join(nativePackage, "index.ts"), + ...[ + "fm-primary-turnend-guard.ts", + "fm-primary-pi-watch.ts", + "fm-branch-supervision.ts", + ].flatMap((name) => ["-e", path.join(repo, ".pi/extensions", name)]), + "--model", + "codex-native/gpt-6-astra", + "--session-dir", + path.join(fixture, "sessions"), + ]; + if (resume) args.push("--session", resume); + else args.push("--codex-effort", "ultra"); + child = spawn(process.env.FM_PI_BIN || "pi", args, { + cwd: repo, + env: { + ...process.env, + FM_HOME: home, + FM_ROOT_OVERRIDE: repo, + PI_CODEX_NATIVE_BIN: peer, + PI_CODING_AGENT_DIR: path.join(fixture, "pi-config"), + PI_TELEMETRY: "false", + }, + stdio: ["pipe", "pipe", "pipe"], + }); + const ownedChild = child; + ownedChild.err = ""; + ownedChild.stderr.on("data", (d) => { + ownedChild.err += d; + fs.appendFileSync(path.join(fixture, "stderr.log"), d); + }); + let buffer = ""; + child.stdout.on("data", (d) => { + buffer += d; + let i; + while ((i = buffer.indexOf("\n")) >= 0) { + const line = buffer.slice(0, i); + buffer = buffer.slice(i + 1); + if (line.trim()) { + let e; + try { + e = JSON.parse(line); + } catch { + continue; + } + events.push(e); + fs.appendFileSync( + path.join(fixture, "events.jsonl"), + JSON.stringify(e) + "\n", + ); + } + } + }); + return (await send("get_state")).data.sessionFile; +} +async function stop() { + if (!child) return; + const c = child; + child = undefined; + c.kill("SIGTERM"); + await Promise.race([new Promise((r) => c.once("exit", r)), delay(3000)]); + if (c.exitCode === null && c.signalCode === null) c.kill("SIGKILL"); +} +let passed = false; +try { + const session = await start(); + await send("prompt", { message: "ARM_PRIMARY" }); + await wait( + () => events.some((e) => e.type === "agent_settled"), + "first settle", + ); + assert( + log().some( + (e) => + e.kind === "input" && + e.text.includes("NATIVE_PRIMARY_STARTUP_SENTINEL"), + ), + "startup missing", + ); + assert( + log().some( + (e) => + e.kind === "control" && + e.name === "fm_watch_arm_pi" && + !e.result.isError, + ), + "watch control missing", + ); + fs.writeFileSync(path.join(state, ".branch-outcomes-processed"), "0\n"); + execFileSync( + path.join(root, "bin/fm-branch-outcome.sh"), + [ + "append", + "--task", + "worker-smoke", + "--verdict", + "captain", + "--summary", + "WORKER_DONE_SENTINEL verified fixture completion", + ], + { env: { ...process.env, FM_HOME: home } }, + ); + fs.writeFileSync(path.join(state, "trigger"), "complete\n"); + await wait( + () => + fs.existsSync(path.join(state, ".branch-outcomes-processed")) && + fs + .readFileSync(path.join(state, ".branch-outcomes-processed"), "utf8") + .trim() === "1", + "native outcome acknowledgement", + ); + await wait( + () => log().filter((e) => e.name === "fm_branch_processed").length === 2, + "duplicate acknowledgement refusal", + ); + let calls = log().filter((e) => e.kind === "control"); + assert.equal( + calls.filter((e) => e.name === "fm_branch_processed" && !e.result.isError) + .length, + 1, + ); + assert.equal( + calls.filter((e) => e.name === "fm_branch_processed" && e.result.isError) + .length, + 1, + ); + assert( + calls + .find((e) => e.name === "fm_branch_outcomes") + .result.content.some((c) => c.text.includes("WORKER_DONE_SENTINEL")), + ); + await wait( + () => events.filter((e) => e.type === "agent_settled").length >= 3, + "completion settled", + ); + await stop(); + const prior = log().filter((e) => e.name === "fm_branch_processed").length; + await start(session); + await send("prompt", { message: "RESTART_OBSERVATION" }); + await wait( + () => events.some((e) => e.type === "agent_settled"), + "resume settle", + ); + assert.equal( + log().filter((e) => e.name === "fm_branch_processed").length, + prior, + "resume repeated processed outcome", + ); + const inputs = log().filter((entry) => entry.kind === "input"); + assert(inputs.length > 1 && inputs.every((entry) => entry.effort === "ultra"), + "native Ultra was lost on initial, operational, or resumed turns"); + const progress = fs.readFileSync(path.join(state, "progress-events"), "utf8") + .trim().split("\n").map(JSON.parse); + assert(progress.some((event) => event.threadId === "fm-native-primary-thread" && event.phase === "output"), + "installed native adapter did not emit streamed-output progress"); + passed = true; + console.log( + JSON.stringify( + { + result: "PASS", + piVersion, + adapterVersion, + ...(process.env.FM_NATIVE_TEST_KEEP === "1" ? { fixture } : {}), + checks: [ + "actual Pi runtime and native package", + "native Ultra preserved across operational turns and restart", + "installed native adapter emits observable output progress", + "actual FirstMate primary extensions", + "startup operational message forwarded", + "MCP watcher control", + "idle watcher notification opens native turn", + "durable completion visible/read/ack once", + "duplicate acknowledgement refused", + "restart does not reprocess acknowledged outcome", + ], + limits: [ + "native peer and watcher-close process are deterministic fixtures; no live model or backend tested", + ], + }, + null, + 2, + ), + ); +} catch (error) { + console.error(`Native Codex guard failed against Pi ${piVersion}, pi-codex-native ${adapterVersion}`); + throw error; +} finally { + await stop(); + if (passed && process.env.FM_NATIVE_TEST_KEEP !== "1") + fs.rmSync(fixture, { recursive: true, force: true }); + else console.error("Native FirstMate test fixture: " + fixture); +} + +JS diff --git a/tests/fm-pi-loaded-marker.test.sh b/tests/fm-pi-loaded-marker.test.sh index 1c70e5664e2..e6a303444ba 100755 --- a/tests/fm-pi-loaded-marker.test.sh +++ b/tests/fm-pi-loaded-marker.test.sh @@ -24,6 +24,7 @@ install_marker_fixture() { # <repo> "$repo/node_modules/typebox" cp "$ROOT/.pi/extensions/fm-primary-pi-watch.ts" "$ROOT/.pi/extensions/fm-primary-turnend-guard.ts" "$repo/.pi/extensions/" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$ROOT/.pi/extensions/lib/fm-async-exec.ts" \ + "$ROOT/.pi/extensions/lib/fm-native-contract.ts" \ "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$ROOT/.pi/extensions/lib/fm-operational-input.ts" \ "$ROOT/.pi/extensions/lib/fm-pi-loaded-marker.ts" "$ROOT/.pi/extensions/lib/fm-pi-prompt-delivery.ts" \ "$repo/.pi/extensions/lib/" diff --git a/tests/fm-pi-primary-live-e2e.test.sh b/tests/fm-pi-primary-live-e2e.test.sh index b54c15c0823..0a22ea29ee4 100755 --- a/tests/fm-pi-primary-live-e2e.test.sh +++ b/tests/fm-pi-primary-live-e2e.test.sh @@ -254,6 +254,7 @@ cp "$ROOT/.pi/extensions/lib/fm-calm-operational-user-layout.ts" "$PROJECT/.pi/e cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$PROJECT/.pi/extensions/lib/fm-calm-visibility.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-working-ship.ts" "$PROJECT/.pi/extensions/lib/fm-calm-working-ship.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$PROJECT/.pi/extensions/lib/fm-branch-dispatch.ts" +cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$PROJECT/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$PROJECT/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-operational-input.ts" "$PROJECT/.pi/extensions/lib/fm-operational-input.ts" cp "$ROOT/.pi/extensions/lib/fm-pi-loaded-marker.ts" "$PROJECT/.pi/extensions/lib/fm-pi-loaded-marker.ts" diff --git a/tests/fm-pi-primary-types.test.sh b/tests/fm-pi-primary-types.test.sh index efe28ce30ff..a46480fbcce 100755 --- a/tests/fm-pi-primary-types.test.sh +++ b/tests/fm-pi-primary-types.test.sh @@ -32,6 +32,7 @@ cp "$ROOT/.pi/extensions/fm-calm.ts" "$TMP_ROOT/fm-calm.ts" cp "$ROOT/.pi/extensions/fm-primary-pi-watch.ts" "$TMP_ROOT/fm-primary-pi-watch.ts" cp "$ROOT/.pi/extensions/fm-primary-turnend-guard.ts" "$TMP_ROOT/fm-primary-turnend-guard.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$TMP_ROOT/lib/fm-branch-dispatch.ts" +cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$TMP_ROOT/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$TMP_ROOT/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-model-picker.ts" "$TMP_ROOT/lib/fm-branch-model-picker.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-assistant-layout.ts" "$TMP_ROOT/lib/fm-calm-assistant-layout.ts" diff --git a/tests/fm-pi-watch-extension.test.sh b/tests/fm-pi-watch-extension.test.sh index 955c9dbb3a7..9f9ff22b641 100755 --- a/tests/fm-pi-watch-extension.test.sh +++ b/tests/fm-pi-watch-extension.test.sh @@ -33,6 +33,7 @@ install_pi_watch_extension_fixture() { "$repo/node_modules/typebox" cp "$EXT" "$repo/.pi/extensions/fm-primary-pi-watch.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$repo/.pi/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$repo/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$repo/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$repo/.pi/extensions/lib/fm-calm-visibility.ts" cp "$ROOT/.pi/extensions/lib/fm-operational-input.ts" "$repo/.pi/extensions/lib/fm-operational-input.ts" diff --git a/tests/fm-pr-merge.test.sh b/tests/fm-pr-merge.test.sh index 64ec7a0dd0b..da2c5e789ca 100755 --- a/tests/fm-pr-merge.test.sh +++ b/tests/fm-pr-merge.test.sh @@ -152,7 +152,10 @@ make_case() { local name=$1 case_dir fakebin case_dir="$TMP_ROOT/$name" fakebin="$case_dir/fakebin" - mkdir -p "$case_dir/state" "$fakebin" + mkdir -p "$case_dir/state" "$case_dir/home/data" "$case_dir/home/config" "$fakebin" + cp "$ROOT/.tasks.toml" "$case_dir/home/.tasks.toml" + printf '%s\n' '## In flight' '' '## Queued' '' '## Done' \ + > "$case_dir/home/data/backlog.md" fm_write_meta "$case_dir/state/task-x1.meta" \ "window=fm-task-x1" \ "worktree=$case_dir/wt" \ @@ -666,7 +669,7 @@ glab_merge_line() { run_pr_merge() { local case_dir=$1 rc; shift FM_ROOT_OVERRIDE="$ROOT" \ - FM_HOME="${FM_TEST_HOME:-$ROOT}" \ + FM_HOME="${FM_TEST_HOME:-$case_dir/home}" \ FM_STATE_OVERRIDE="$case_dir/state" \ FM_CONFIG_OVERRIDE="$case_dir/config" \ FM_TEST_GH_AXI_LOG="$case_dir/gh-axi.log" \ @@ -678,6 +681,7 @@ run_pr_merge() { FM_TEST_REAL_MV="$REAL_MV" \ FM_TEST_GLAB_LOG="$case_dir/glab.log" \ FM_TEST_GLAB_JSON="$case_dir/mr.json" \ + HOME="${FM_TEST_USER_HOME:-$case_dir/user-home}" \ PATH="$case_dir/fakebin:$PATH" \ "$PR_MERGE" "$@" rc=$? @@ -2976,6 +2980,183 @@ test_gitlab_stale_recorded_head_is_reported test_gitlab_unreadable_state_refuses test_gitlab_invalid_head_refuses test_gitlab_missing_tool_refuses_before_recording + +# The merge gate asks whether the task is still held for the captain. A home +# that carries no backlog records no captain calls at all, so nothing can be +# held and the merge must proceed; a backlog that EXISTS but cannot be read may +# hide a live hold, so that one must refuse. The two states are distinct and +# only the second is a refusal. +test_absent_backlog_still_merges() { + local case_dir rc + case_dir=$(make_case absent-backlog-merges) + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" 6161616161616161616161616161616161616161 + : > "$case_dir/gh-axi.log" + rm -f "$case_dir/home/data/backlog.md" + + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/61 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "absent-backlog-merges: a home with no backlog must still merge" + assert_no_grep 'held for the captain' "$case_dir/stderr" \ + "absent-backlog-merges: an absent backlog was read as a captain hold" + grep -qxF 'pr merge 61 --repo example/repo --squash' "$case_dir/gh-axi.log" \ + || fail "absent-backlog-merges: the merge was not attempted" + pass "fm-pr-merge proceeds when the home carries no backlog at all" +} + +test_unreadable_backlog_refuses_the_merge() { + local case_dir rc + case_dir=$(make_case unreadable-backlog-refuses) + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" 6262626262626262626262626262626262626262 + : > "$case_dir/gh-axi.log" + chmod 000 "$case_dir/home/data/backlog.md" + + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/62 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + chmod 644 "$case_dir/home/data/backlog.md" + + expect_code 1 "$rc" "unreadable-backlog-refuses: an unreadable authority record must refuse" + assert_grep 'refusing to merge' "$case_dir/stderr" \ + "unreadable-backlog-refuses: the refusal did not say it refused to merge" + [ ! -s "$case_dir/gh-axi.log" ] \ + || fail "unreadable-backlog-refuses: the forge was called despite an unreadable record" + pass "fm-pr-merge refuses when the backlog exists but cannot be read" +} + +test_unreadable_backend_config_refuses_the_merge() { + local case_dir rc + case_dir=$(make_case unreadable-backend-config-refuses) + mkdir -p "$case_dir/wt" + add_gh_mocks "$case_dir" 6363636363636363636363636363636363636363 + : > "$case_dir/gh-axi.log" + rm -f "$case_dir/home/data/backlog.md" + chmod 000 "$case_dir/home/.tasks.toml" + + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/63 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + chmod 644 "$case_dir/home/.tasks.toml" + + expect_code 1 "$rc" "unreadable-backend-config-refuses: an unreadable authority route must refuse" + assert_grep 'tasks-axi backend configuration cannot be read' "$case_dir/stderr" \ + "unreadable-backend-config-refuses: the unreadable authority route was not named" + [ ! -s "$case_dir/gh-axi.log" ] \ + || fail "unreadable-backend-config-refuses: the forge was called despite an unreadable authority route" + pass "fm-pr-merge refuses when its configured backend cannot be read" +} + +test_unreadable_user_backend_config_refuses_the_merge() { + local case_dir rc user_config + case_dir=$(make_case unreadable-user-backend-config-refuses) + user_config="$case_dir/user-home/.tasks-axi/config.toml" + mkdir -p "$case_dir/wt" "${user_config%/*}" + add_gh_mocks "$case_dir" 6464646464646464646464646464646464646464 + : > "$case_dir/gh-axi.log" + rm -f "$case_dir/home/.tasks.toml" "$case_dir/home/data/backlog.md" + printf '%s\n' 'backend = "beads"' > "$user_config" + chmod 000 "$user_config" + + set +e + FM_TEST_USER_HOME="$case_dir/user-home" \ + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/64 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + chmod 644 "$user_config" + + expect_code 1 "$rc" "unreadable-user-backend-config-refuses: an unreadable authority route must refuse" + assert_grep "tasks-axi backend configuration cannot be read at $user_config" "$case_dir/stderr" \ + "unreadable-user-backend-config-refuses: the unreadable authority route was not named" + [ ! -s "$case_dir/gh-axi.log" ] \ + || fail "unreadable-user-backend-config-refuses: the forge was called despite an unreadable authority route" + pass "fm-pr-merge refuses when its user backend configuration cannot be read" +} + +test_untraversable_user_backend_config_directory_refuses_the_merge() { + local case_dir rc user_config + case_dir=$(make_case untraversable-user-backend-config-directory-refuses) + user_config="$case_dir/user-home/.tasks-axi/config.toml" + mkdir -p "$case_dir/wt" "${user_config%/*}" + add_gh_mocks "$case_dir" 6666666666666666666666666666666666666666 + : > "$case_dir/gh-axi.log" + rm -f "$case_dir/home/.tasks.toml" "$case_dir/home/data/backlog.md" + printf '%s\n' 'backend = "beads"' > "$user_config" + chmod 000 "${user_config%/*}" + + set +e + FM_TEST_USER_HOME="$case_dir/user-home" \ + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/66 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + chmod 755 "${user_config%/*}" + + expect_code 1 "$rc" "untraversable-user-backend-config-directory-refuses: an unreadable authority route must refuse" + assert_grep "tasks-axi backend configuration cannot be read at $user_config" "$case_dir/stderr" \ + "untraversable-user-backend-config-directory-refuses: the unreadable authority route was not named" + [ ! -s "$case_dir/gh-axi.log" ] \ + || fail "untraversable-user-backend-config-directory-refuses: the forge was called despite an unreadable authority route" + pass "fm-pr-merge refuses when its user backend configuration directory cannot be traversed" +} + +test_absent_user_backend_config_directory_and_backlog_still_merge() { + local case_dir rc + case_dir=$(make_case absent-user-backend-config-directory-and-backlog-merges) + mkdir -p "$case_dir/wt" "$case_dir/user-home" + add_gh_mocks "$case_dir" 6767676767676767676767676767676767676767 + : > "$case_dir/gh-axi.log" + rm -f "$case_dir/home/.tasks.toml" "$case_dir/home/data/backlog.md" + + set +e + FM_TEST_USER_HOME="$case_dir/user-home" \ + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/67 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "absent-user-backend-config-directory-and-backlog-merges: sound defaults and no backlog must permit merging" + [ "$(grep -c '^pr merge ' "$case_dir/gh-axi.log")" -eq 1 ] \ + || fail "absent-user-backend-config-directory-and-backlog-merges: the forge must merge exactly once" + grep -qxF 'pr merge 67 --repo example/repo --squash' "$case_dir/gh-axi.log" \ + || fail "absent-user-backend-config-directory-and-backlog-merges: the expected merge was not attempted" + pass "fm-pr-merge proceeds once when its user configuration directory and backlog are genuinely absent" +} + +test_backend_override_bypasses_unreadable_user_config() { + local case_dir rc user_config + case_dir=$(make_case backend-override-bypasses-unreadable-user-config) + user_config="$case_dir/user-home/.tasks-axi/config.toml" + mkdir -p "$case_dir/wt" "${user_config%/*}" + add_gh_mocks "$case_dir" 6565656565656565656565656565656565656565 + : > "$case_dir/gh-axi.log" + rm -f "$case_dir/home/.tasks.toml" "$case_dir/home/data/backlog.md" + printf '%s\n' 'backend = "beads"' > "$user_config" + chmod 000 "$user_config" + + set +e + TASKS_AXI_BACKEND=markdown FM_TEST_USER_HOME="$case_dir/user-home" \ + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/65 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + chmod 644 "$user_config" + + expect_code 0 "$rc" "backend-override-bypasses-unreadable-user-config: an explicit backend must bypass config" + grep -qxF 'pr merge 65 --repo example/repo --squash' "$case_dir/gh-axi.log" \ + || fail "backend-override-bypasses-unreadable-user-config: the merge was not attempted" + pass "fm-pr-merge honors a backend override over an unreadable user configuration" +} + test_gitlab_head_override_args_refuse_before_recording test_github_still_forwards_sha_arg test_secondmate_merge_reports_upward_once @@ -2989,6 +3170,14 @@ test_queued_github_merge_leaves_the_poll_armed test_distinct_merged_prs_keep_distinct_wakes test_uncommitted_marker_retry_is_never_silent test_secondmate_without_parent_binding_is_loud +test_absent_backlog_still_merges +test_unreadable_backlog_refuses_the_merge +test_unreadable_backend_config_refuses_the_merge +test_unreadable_user_backend_config_refuses_the_merge +test_untraversable_user_backend_config_directory_refuses_the_merge +test_absent_user_backend_config_directory_and_backlog_still_merge +test_backend_override_bypasses_unreadable_user_config + # THE LANDED-MERGE INVARIANT. # # Asserted as a POSITIVE property rather than policed by inspection: EVERY path diff --git a/tests/fm-procevent.test.sh b/tests/fm-procevent.test.sh index 7b0c3bd1324..436b72449b7 100755 --- a/tests/fm-procevent.test.sh +++ b/tests/fm-procevent.test.sh @@ -914,7 +914,8 @@ pass "Lavish classification staging stays bounded while nonmatches stream" HW="$TMP_ROOT/hw"; new_home "$HW" TRIGW="$TMP_ROOT/trigger-restart-cut" pe_register "$HW" lavish restart-cut-src -- "$BLOCKER" "$TRIGW" "restart cut payload" >/dev/null -pe "$HW" reconcile >/dev/null +pe "$HW" start restart-cut-src > "$TMP_ROOT/restart-cut-start.log" 2>&1 & +restart_cut_start_pid=$! sleep 0.5 : > "$TRIGW" wait_for "$HW/state/.wake-queue" || fail "the restart-cut source published no event" @@ -927,7 +928,10 @@ assert_contains "$(wake_payloads "$HW")" "procevent lavish restart-cut-src 1" \ # capture a fresh generation; retiring leaves only the durable inbox and wake # state under test, matching the exact restart cut - the source side is done, # only the handling side is still open. -pe "$HW" retire restart-cut-src >/dev/null +# Publication precedes runner exit; wait for completion before retiring. +wait "$restart_cut_start_pid" || fail "the restart-cut source did not complete" +pe "$HW" retire restart-cut-src >/dev/null \ + || fail "the restart-cut registration was not retired" # Drain the wake without handling it: the end-user experience of a session # reading the wake queue at turn end without yet acting on this specific line. @@ -1287,8 +1291,12 @@ awk -v identity="$sr4_identity" 'NR == 4 { print identity; next } { print }' \ "$sr4_claim" > "$sr4_claim.tmp" && mv "$sr4_claim.tmp" "$sr4_claim" chmod 0600 "$sr4_claim" chmod 755 "$HSR4/state" -: > "$SR4_TRIGGER" -pe "$HSR4" retire reused-group-src >/dev/null +# Keep the child blocked so retirement alone ends the restored generation; +# releasing it races natural exit against the first signal's ownership check. +pe "$HSR4" retire reused-group-src >/dev/null \ + || fail "retirement failed after restoring the reused-group fixture's identity" +kill -0 -"$sr4_leader" 2>/dev/null \ + && fail "retirement left the restored reused-group fixture running" pass "a reused pid never makes its surviving process group reclaimable" HJ="$TMP_ROOT/hj"; new_home "$HJ" @@ -1530,63 +1538,113 @@ kill -0 "$noisy_child" 2>/dev/null && fail "TERM-resistant source child survived assert_absent "$staged" "retirement removes the tracked partial staging file" pass "live output stays bounded and retirement reaps the whole source group" -HPOST_TERM="$TMP_ROOT/post-term-reuse"; new_home "$HPOST_TERM" -POST_TERM_SOURCE="$TMP_ROOT/post-term-reuse-source.sh" -POST_TERM_PID="$TMP_ROOT/post-term-reuse.pid" -POST_TERM_MARKER="$TMP_ROOT/post-term-reuse.marker" -POST_TERM_COUNT="$TMP_ROOT/post-term-reuse.count" -cat > "$POST_TERM_SOURCE" <<'SH' +# When a fixture never records TERM, capture the claim, leader, group, and +# retirement output to help distinguish a blocked stop from a refused signal. +# Collect this evidence only on the failure path so passing cases stay quiet. +post_term_evidence() { # <case> <runner-pid> <claim> <signals> <started-epoch> <retire-output> + local case=$1 runner=$2 claim=$3 signals=$4 started=$5 out=$6 + { + printf 'post-TERM evidence (%s case)\n' "$case" + printf ' elapsed since retire started: %ss\n' "$(( $(date +%s) - started ))" + printf ' identity recorded at claim time: %s\n' "$(sed -n '4p' "$claim" 2>/dev/null || echo '<claim unreadable>')" + printf ' identity readable now (real ps): %s\n' "$(LC_ALL=C ps -p "$runner" -o lstart= 2>/dev/null || echo '<ps failed>')" + printf ' signals file: %s (%s bytes)\n' "$signals" "$(wc -c < "$signals" 2>/dev/null | tr -d ' ' || echo 0)" + printf ' leader state: %s\n' "$(ps -o pid=,ppid=,pgid=,stat= -p "$runner" 2>/dev/null || echo '<leader gone>')" + printf ' leader wchan: %s\n' "$(ps -o wchan= -p "$runner" 2>/dev/null || echo '<none>')" + printf ' live members of the runner group:\n' + ps -Ao pid,ppid,pgid,stat,wchan,command 2>/dev/null | awk -v g="$runner" 'NR==1 || $3==g' | sed 's/^/ /' + printf ' retire said: %s\n' "${out:-<no output>}" + } >&2 +} + +for post_term_case in mismatch unreadable unreadable-pgid nonleader; do + HPOST_TERM="$TMP_ROOT/post-term-$post_term_case"; new_home "$HPOST_TERM" + POST_TERM_SOURCE="$HPOST_TERM/source.sh" + POST_TERM_PID="$HPOST_TERM/child.pid" + POST_TERM_SIGNALS="$HPOST_TERM/child.signals" + cat > "$POST_TERM_SOURCE" <<'SH' #!/usr/bin/env bash -trap '' TERM +trap 'printf "signalled\n" >> "$2"' TERM printf '%s\n' "$$" > "$1" -while :; do sleep 1; done +while [ "$SECONDS" -lt "${FM_TEST_STUB_MAX_BLOCK_SECONDS:-120}" ]; do sleep 1; done SH -chmod +x "$POST_TERM_SOURCE" -POST_TERM_BIN=$(fm_fakebin "$TMP_ROOT/post-term-reuse-bin") -REAL_PS=$(command -v ps) || fail "the post-TERM reuse fixture requires ps" -cat > "$POST_TERM_BIN/ps" <<SH + chmod +x "$POST_TERM_SOURCE" + POST_TERM_BIN=$(fm_fakebin "$HPOST_TERM/tools") + REAL_PS=$(command -v ps) || fail "the post-TERM reuse fixture requires ps" + pe_register "$HPOST_TERM" lavish post-term-src -- \ + "$POST_TERM_SOURCE" "$POST_TERM_PID" "$POST_TERM_SIGNALS" >/dev/null + FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-post-term-proc" \ + FM_PROCEVENT_OWNER_CHECK_SECONDS=5 pe "$HPOST_TERM" reconcile >/dev/null + wait_for "$POST_TERM_PID" || fail "the post-TERM reuse fixture did not start" + wait_for "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim" \ + || fail "the post-TERM reuse fixture did not claim its source" + POST_TERM_RUNNER=$(sed -n '2p' "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim") + # The child can publish its PID before startup releases the source lock. + # Cross that boundary before suspending the runner, or retirement waits on + # a stopped lock owner instead of exercising the signal checks below. + FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-post-term-proc" pe "$HPOST_TERM" list >/dev/null \ + || fail "the post-TERM fixture never finished its source launch" + kill -STOP "$POST_TERM_RUNNER" || fail "the post-TERM fixture could not keep its leader alive" + cat > "$POST_TERM_BIN/ps" <<SH #!/usr/bin/env bash -if [ -e "$POST_TERM_MARKER" ] && [ "\${1-}" = -p ] \ +if [ "\${1-}" = -p ] && [ "\${2-}" = "$POST_TERM_RUNNER" ] \ && [ "\${3-}" = -o ] && [ "\${4-}" = lstart= ]; then - count=0 - [ ! -f "$POST_TERM_COUNT" ] || count=\$(cat "$POST_TERM_COUNT") - count=\$((count + 1)) - printf '%s\n' "\$count" > "$POST_TERM_COUNT" - if [ "\$count" -gt 1 ]; then - printf 'post-TERM reused identity\n' + if [ "$post_term_case" = mismatch ]; then + printf 'reused identity\n' exit 0 fi + [ ! -s "$POST_TERM_SIGNALS" ] || exit 1 +fi +if [ "\${1-}" = -o ] && [ "\${2-}" = pgid= ] \ + && [ "\${3-}" = -p ] && [ "\${4-}" = "$POST_TERM_RUNNER" ]; then + case "$post_term_case" in + unreadable-pgid) [ ! -s "$POST_TERM_SIGNALS" ] || exit 1 ;; + nonleader) printf '0\n'; exit 0 ;; + esac fi exec "$REAL_PS" "\$@" SH -chmod +x "$POST_TERM_BIN/ps" -pe_register "$HPOST_TERM" lavish post-term-src -- \ - "$POST_TERM_SOURCE" "$POST_TERM_PID" >/dev/null -FM_PROCEVENT_OWNER_CHECK_SECONDS=5 pe "$HPOST_TERM" reconcile >/dev/null -wait_for "$POST_TERM_PID" || fail "the post-TERM reuse fixture did not start" -wait_for "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim" \ - || fail "the post-TERM reuse fixture did not claim its source" -POST_TERM_RUNNER=$(sed -n '2p' "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim") -touch "$POST_TERM_MARKER" -post_term_status=0 -PATH="$POST_TERM_BIN:$PATH" FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-post-term-proc" \ - pe "$HPOST_TERM" retire post-term-src >/dev/null 2>&1 || post_term_status=$? -[ "$post_term_status" -ne 0 ] || fail "retirement escalated after runner identity became ambiguous" -# Which ambiguity the escalation meets here is platform-dependent, so this -# asserts the invariant both forms share rather than one form's internals. -# Where the runner leader keeps waiting on its TERM-ignoring source child the -# post-TERM check sees a live leader whose identity no longer matches, and -# where the leader dies promptly it sees a leaderless group carrying the same -# numeric id; fm_procevent_pid_state reaches the second verdict without -# consulting process identity at all, so counting identity lookups pins a -# timing- and platform-dependent internal rather than the behavior. -kill -0 -"$POST_TERM_RUNNER" 2>/dev/null \ - || fail "an ambiguous reused-PID group was killed during escalation" -kill -KILL -"$POST_TERM_RUNNER" 2>/dev/null || true -for _ in $(seq 1 50); do kill -0 -"$POST_TERM_RUNNER" 2>/dev/null || break; sleep 0.1; done -kill -0 -"$POST_TERM_RUNNER" 2>/dev/null && fail "could not clean up the post-TERM fixture group" -pe "$HPOST_TERM" retire post-term-src >/dev/null -pass "cleanup aborts escalation after runner identity becomes ambiguous" + chmod +x "$POST_TERM_BIN/ps" + post_term_status=0 + post_term_started=$(date +%s) + post_term_out=$(PATH="$POST_TERM_BIN:$PATH" FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-post-term-proc" \ + pe "$HPOST_TERM" retire post-term-src 2>&1) || post_term_status=$? + case "$post_term_case" in + mismatch|nonleader) + assert_absent "$POST_TERM_SIGNALS" "the first signal refuses $post_term_case evidence" + [ "$post_term_status" -ne 0 ] || fail "retirement escalated despite $post_term_case evidence" + assert_contains "$post_term_out" "cannot confirm runner identity" \ + "first-signal $post_term_case evidence refuses retirement" + kill -0 "$POST_TERM_RUNNER" 2>/dev/null \ + || fail "the post-TERM fixture lost its leader instead of exercising $post_term_case evidence" + kill -0 -"$POST_TERM_RUNNER" 2>/dev/null \ + || fail "a $post_term_case group was killed during escalation" + assert_present "$HPOST_TERM/state/procevent/post-term-src.source" \ + "first-signal $post_term_case evidence preserves registration" + assert_present "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim" \ + "first-signal $post_term_case evidence preserves its claim" + kill -KILL -"$POST_TERM_RUNNER" 2>/dev/null || true + ;; + *) + if [ ! -s "$POST_TERM_SIGNALS" ]; then + post_term_evidence "$post_term_case" "$POST_TERM_RUNNER" \ + "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim" "$POST_TERM_SIGNALS" \ + "$post_term_started" "$post_term_out" + fail "the post-TERM fixture never received TERM" + fi + [ "$post_term_status" -eq 0 ] \ + || fail "retirement abandoned a proved stop after $post_term_case identity: $post_term_out" + assert_absent "$HPOST_TERM/state/procevent/post-term-src.source" \ + "proved escalation retires the source after $post_term_case identity" + assert_absent "$FM_PROCEVENT_CLAIM_ROOT/post-term-src.claim" \ + "proved escalation releases its claim after $post_term_case identity" + ;; + esac + for _ in $(seq 1 50); do kill -0 -"$POST_TERM_RUNNER" 2>/dev/null || break; sleep 0.1; done + kill -0 -"$POST_TERM_RUNNER" 2>/dev/null && fail "the post-TERM fixture group survived: $post_term_case" + pe "$HPOST_TERM" retire post-term-src >/dev/null + pass "stop $post_term_case evidence preserves the proved-stop boundary" +done HBAD="$TMP_ROOT/hbad"; new_home "$HBAD" pe_register "$HBAD" lavish bad-limit -- /bin/true >/dev/null @@ -2521,10 +2579,36 @@ chmod +x "$QUIET_STUB" # Short enough to observe, and driven through the same environment a real home # uses, so the bound under test is the shipped one rather than a test-only path. +# One source of truth for the shortened lease and check these fixtures run under, +# so a case that derives a deadline from the guard's documented bound cannot +# silently diverge from the settings the guard is actually given. +PROOF_LEASE_SECONDS=2 +PROOF_CHECK_SECONDS=1 + +# The documented bound, derived here rather than restated as a flat number. +# +# The whole-second lease comparison is part of the bound, not slack: a lease of +# N is honoured until its age reads N+1, so the lease term is N+1. +PROOF_LEASE_BOUND=$((PROOF_LEASE_SECONDS + 1)) +# Detection is the lease plus ONE check interval. The guard still takes two +# consecutive failing reads before it acts - one unreadable read must not end a +# live runner - but they are spaced half an interval apart, so the pair fits +# inside the single interval this term budgets. +PROOF_DETECT_BOUND=$((PROOF_LEASE_BOUND + PROOF_CHECK_SECONDS)) +# The stop's own ceiling: two seconds for the ordinary signal, then two for the +# forced one. Only a group that outlives the ordinary signal spends it, so a +# case whose stub exits on that signal uses PROOF_PROMPT_STOP instead. +PROOF_STOP_CEILING=4 +PROOF_PROMPT_STOP=1 +# Additive scheduling slack shared by cleanup and timing cases. The strict +# timing case below owns and enforces its relation to BOUND_CHECK_SECONDS. +PROOF_LOAD_SLACK=2 + orphan_pe() { # <home> <command...> local home=$1 shift - FM_PROCEVENT_OWNER_LEASE_SECONDS=2 FM_PROCEVENT_OWNER_CHECK_SECONDS=1 \ + FM_PROCEVENT_OWNER_LEASE_SECONDS="$PROOF_LEASE_SECONDS" \ + FM_PROCEVENT_OWNER_CHECK_SECONDS="$PROOF_CHECK_SECONDS" \ FM_HOME="$home" "$ROOT/bin/fm-procevent.sh" "$@" } @@ -2571,13 +2655,21 @@ pass "a detached listener starts reparented, with a live descendant tree under i # owning session is the single difference between the two listeners. keep_owner_present() { orphan_pe "$HKEEP" reconcile >/dev/null 2>&1 || true; sleep 0.25; } -deadline=$((SECONDS + 40)) +# This stub exits on the ordinary signal, so the stop ceiling is not spent here. +# The deadline is DERIVED from the documented bound; the timing case below is +# the one that pins the bound's worst case, while this one asserts that the +# reaping happens at all and cannot quietly take an unbounded amount of time. +orphan_bound=$((PROOF_DETECT_BOUND + PROOF_PROMPT_STOP)) +deadline=$((SECONDS + orphan_bound + PROOF_LOAD_SLACK)) +orphan_started=$SECONDS while kill -0 -"$ORPHAN_PID" 2>/dev/null; do [ "$SECONDS" -lt "$deadline" ] \ - || fail "a listener whose owning session was gone kept its process group running" + || fail "a listener whose owning session was gone kept its process group running for $((SECONDS - orphan_started))s, against a documented bound of ${orphan_bound}s" keep_owner_present done -deadline=$((SECONDS + 20)) +# The descendant goes down with the same group signal, so it needs no bound of +# its own beyond the slack that covers a loaded host. +deadline=$((SECONDS + PROOF_LOAD_SLACK)) while kill -0 "$ORPHAN_DESCENDANT" 2>/dev/null; do [ "$SECONDS" -lt "$deadline" ] \ || fail "a listener whose owning session was gone left a descendant running" @@ -2672,4 +2764,506 @@ wait_gone "$RETRY_DESCENDANT" \ || fail "the guard stopped retrying before the expired runner's descendant was reaped" pass "a stop the guard cannot prove is retried until the expired runner is reaped" +# --- a stop reaches a child that does not die on the ordinary signal --------- +# +# Every reaper here sends the ordinary stop signal to the runner's process group +# and escalates only if the group outlives it. Both halves of that escalation +# were broken, in ways that hid each other: +# +# - The stop held the per-source lock across its wait while the runner's own +# exit cleanup waited for that same lock, so the runner outlived the ordinary +# signal every time and the forced kill silently became the normal path. +# - The escalation re-derived ownership from the leader, so once the leader did +# die to the stop's own signal it read that success as a leaderless group and +# refused to escalate at all. +# +# With only the first repaired, the second turned every stop of a signal-proof +# child into a refusal that left it running. They are asserted together because +# they only hold together. +# +# Earlier fixtures include TERM-resistant children and deliberately kept-alive +# leaders. The cases below also exercise escalation after TERM ends the leader. + +# Millisecond clock for supplementary retirement and stop-window measurements; +# the healthy-stop verdict below requires attached-start status 143 (TERM). +now_ms() { perl -MTime::HiRes=time -e 'printf "%d\n", time * 1000'; } + +SIGNAL_PROOF_STUB="$TMP_ROOT/signal-proof-stub.sh" +cat > "$SIGNAL_PROOF_STUB" <<'SH' +#!/usr/bin/env bash +# A blocking source whose child handles the ordinary stop signal and keeps +# waiting - the shape a poll client with its own shutdown handler presents while +# a request is still outstanding. Reaching it requires a real escalation. The +# signal log is what proves the child was signalled and survived, rather than +# never having been signalled at all. The wait stays bounded so an escaped stub +# cannot outlive the suite. +marker=$1 +trap 'printf "signalled\n" >> "$marker.signals"' TERM INT HUP +printf '%s\n' "$$" > "$marker.child" +while [ ! -e "$marker.trigger" ]; do + [ "$SECONDS" -lt "${FM_TEST_STUB_MAX_BLOCK_SECONDS:-120}" ] || exit 75 + sleep 0.1 & + wait $! +done +printf 'signal-proof payload\n' +SH +chmod +x "$SIGNAL_PROOF_STUB" + +trap '[ -z "${PROOF_RELEASE:-}" ] || touch "$PROOF_RELEASE"; fm_test_cleanup' EXIT +for proof_state in absent zombie; do + HPROOF="$TMP_ROOT/signal-proof-retire-$proof_state"; new_home "$HPROOF" + PROOF_MARKER="$HPROOF/poll" + pe_register "$HPROOF" lavish proof-src -- "$SIGNAL_PROOF_STUB" "$PROOF_MARKER" >/dev/null + PROOF_RELEASE= + if [ "$proof_state" = zombie ]; then + PROOF_RELEASE="$HPROOF/reap" + FM_HOME="$HPROOF" FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-proof-proc" \ + perl - "$PROOF_RELEASE" "$ROOT/bin/fm-procevent.sh" _start proof-src >"$HPROOF/start.log" 2>&1 <<'PL' & +my $release = shift @ARGV; +defined(my $pid = fork) or exit 125; +if ($pid == 0) { + setpgrp(0, 0) or exit 125; + $ENV{FM_PROCEVENT_RUNNER_GROUP} = $$; + exec @ARGV; + exit 125; +} +my $deadline = time + ($ENV{FM_TEST_STUB_MAX_BLOCK_SECONDS} // 120); +while (!-e $release && time < $deadline) { select undef, undef, undef, 0.05; } +waitpid($pid, 0) == $pid or exit 125; +PL + else + FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-proof-proc" \ + pe "$HPROOF" start proof-src >"$HPROOF/start.log" 2>&1 & + fi + PROOF_START=$! + wait_for "$HPROOF/state/procevent/proof-src.runner" \ + || fail "the signal-proof listener never recorded its runner" + PROOF_PID=$(cat "$HPROOF/state/procevent/proof-src.runner") + wait_for "$PROOF_MARKER.child" || fail "the signal-proof child never started" + PROOF_CHILD=$(cat "$PROOF_MARKER.child") + FM_PROC_ROOT_OVERRIDE="$TMP_ROOT/no-proof-proc" \ + pe "$HPROOF" retire proof-src >"$HPROOF/retire.log" 2>&1 & + PROOF_STOP=$! + proof_transition=0 + for _ in $(seq 1 100); do + if [ "$proof_state" = zombie ]; then + case "$(ps -o stat= -p "$PROOF_PID" 2>/dev/null | tr -d '[:space:]')" in + Z*) proof_transition=1; break ;; + esac + elif ! kill -0 "$PROOF_PID" 2>/dev/null; then + proof_transition=1 + break + fi + sleep 0.05 + done + proof_survivor=0 + kill -0 "$PROOF_CHILD" 2>/dev/null && proof_survivor=1 + proof_reaped=0 + for _ in $(seq 1 100); do + if ! kill -0 "$PROOF_CHILD" 2>/dev/null; then proof_reaped=1; break; fi + sleep 0.1 + done + [ -z "$PROOF_RELEASE" ] || touch "$PROOF_RELEASE" + PROOF_RELEASE= + proof_status=0 + wait "$PROOF_STOP" || proof_status=$? + [ "$proof_reaped" -eq 1 ] || kill -KILL -"$PROOF_PID" 2>/dev/null || true + wait "$PROOF_START" 2>/dev/null || true + [ "$proof_transition" -eq 1 ] || fail "the runner never became $proof_state after TERM" + [ "$proof_survivor" -eq 1 ] || fail "no child survived the $proof_state leader's TERM" + [ "$proof_reaped" -eq 1 ] || fail "escalation abandoned a child behind a $proof_state leader" + [ "$proof_status" -eq 0 ] || fail "retiring the $proof_state leader's group reported failure" + wait_gone "-$PROOF_PID" || fail "retirement left the $proof_state leader's group running" + [ -s "$PROOF_MARKER.signals" ] || fail "the signal-proof child never received TERM" + pass "retirement escalates after TERM leaves a surviving child ($proof_state leader)" +done +trap fm_test_cleanup EXIT + +# --- the owner guard reaps a signal-proof child too -------------------------- +# +# The guard is where the time bound on a leaked listener lives, so it is the half +# that matters most: a guard that signals, loses its leader to its own signal and +# then walks away leaves the survivor unreachable by anything at all - worse than +# no guard, because the leader it destroyed was the only proof of ownership left. + +HPGUARD="$TMP_ROOT/signal-proof-guard"; new_home "$HPGUARD" +fm_test_track_procevent_home "$HPGUARD" +orphan_pe "$HPGUARD" register lavish proof-guard-src \ + -- "$SIGNAL_PROOF_STUB" "$TMP_ROOT/proof-guard" >/dev/null +orphan_pe "$HPGUARD" reconcile >/dev/null +# The owner is kept present until the fixture is fully up, because the input +# under test is an owner that GOES AWAY, not a runner that never finished +# starting: on a loaded host the short lease here can otherwise expire while the +# runner is still between fork and its first recorded state. +deadline=$((SECONDS + 60)) +until [ -s "$HPGUARD/state/procevent/proof-guard-src.runner" ] \ + && [ -s "$TMP_ROOT/proof-guard.child" ]; do + [ "$SECONDS" -lt "$deadline" ] || fail "the guarded signal-proof listener never started" + orphan_pe "$HPGUARD" reconcile >/dev/null 2>&1 || true + sleep 0.25 +done +GUARD_PID=$(cat "$HPGUARD/state/procevent/proof-guard-src.runner") +GUARD_CHILD=$(cat "$TMP_ROOT/proof-guard.child") + +# Nothing refreshes this home's lease from here on, which is the whole input. +# +# The deadline is DERIVED from the bound this case exists to defend, not a flat +# wall-clock number. The documented bound is the lease term, plus ONE check +# interval for detection - the guard's two confirming reads are half an interval +# apart and both fit inside it - plus the stop's own grace, its ordinary signal +# window and then its forced one. THIS case does spend that grace, because its +# child ignores the ordinary signal; that is what separates its allowance from +# the ordinary-stop case above. +# +# This case bounds cleanup completion; the strict timing case below owns the +# phase and slack requirements that distinguish one check interval from two. +guard_bound=$((PROOF_DETECT_BOUND + PROOF_STOP_CEILING)) +deadline=$((SECONDS + guard_bound + PROOF_LOAD_SLACK)) +guard_started=$SECONDS +while kill -0 -"$GUARD_PID" 2>/dev/null; do + [ "$SECONDS" -lt "$deadline" ] \ + || fail "the guard exceeded its bound: still holding the group after $((SECONDS - guard_started))s, against a documented bound of ${guard_bound}s" + sleep 0.5 +done +wait_gone "$GUARD_CHILD" \ + || fail "the guard stopped at the leader and left the signal-proof child running" +[ -s "$TMP_ROOT/proof-guard.signals" ] \ + || fail "the guarded child was never signalled, so nothing about escalation was exercised" +pass "an expired runner's guard escalates past a signal-proof child" + +# --- the guard's bound is one check interval, not two ------------------------ +# +# The case above proves the guard reaps at all. This one measures HOW LONG it +# may take, because that is the number the operating contract states and the one +# a later change can quietly double. +# +# The bound: the lease term, plus ONE check interval. The guard still refuses to +# act on a single failed read - the debounce case below is what defends that - +# but its two confirming reads are spaced half an interval apart, so the pair +# fits inside the one interval budgeted here. A guard that put a whole interval +# between them would spend two, and this deadline is sized to catch exactly that. +# +# THE PHASE IS OBSERVED AND ENFORCED, NOT ASSUMED. Where the lease expiry falls +# relative to the guard's own check clock decides whether a run lands near the +# bound or well inside it, and a sampled phase would let a guard spending two +# intervals slip under this deadline on a lucky alignment. So the lease is +# synchronized to the guard's own FIRST observed lease read, every later real +# read is recorded, and the case then REFUSES unless one of those reads proves +# the required phase: fresh, before expiry, and late enough that two further +# full intervals could not finish before the deadline. +# +# Pinning the phase by construction instead - from an assumed startup time - is +# what an earlier version of this case did, and it is not enough: the day +# startup reaches two seconds it silently stops rejecting a two-interval guard +# and goes on passing. A bound that cannot fail for the reason it names is the +# defect this whole delivery exists to correct, so an unestablished precondition +# refuses here rather than proceeding on trust. +BOUND_LEASE_SECONDS=7 +BOUND_CHECK_SECONDS=6 +# The whole-second lease comparison is part of the bound, not slack: a lease of +# N is honoured until its age reads N+1. +bound_lease_term=$((BOUND_LEASE_SECONDS + 1)) +bound_detect=$((bound_lease_term + BOUND_CHECK_SECONDS)) +# This stub exits on the ordinary signal, so the stop's escalation ceiling is +# not spent here; one second covers signalling and exit against a measured +# ~0.4s for a whole retire command on this host. +bound_total=$((bound_detect + PROOF_PROMPT_STOP)) +# Additive load slack, under half a check interval for the reason above. The +# invariant is asserted rather than left to a comment, because a later widening +# is exactly what would disarm the deadline below. +bound_deadline_s=$((bound_total + PROOF_LOAD_SLACK)) +[ "$((PROOF_LOAD_SLACK * 2))" -lt "$BOUND_CHECK_SECONDS" ] \ + || fail "the bound fixture's load slack must stay below half a check interval" + +now_mono() { + perl -MTime::HiRes=clock_gettime,CLOCK_MONOTONIC -e \ + 'printf "%.3f\n", clock_gettime(CLOCK_MONOTONIC)' +} +mono_since() { # <monotonic-reference>: seconds elapsed, one decimal + perl -e 'printf "%.1f\n", $ARGV[0] - $ARGV[1]' "$(now_mono)" "$1" +} + +HBOUND="$TMP_ROOT/guard-bound"; new_home "$HBOUND" +fm_test_track_procevent_home "$HBOUND" +BOUND_STATE="$TMP_ROOT/guard-bound-state"; mkdir -p "$BOUND_STATE" +BOUND_BIN=$(fm_fakebin "$TMP_ROOT/guard-bound-bin") +REAL_PERL=$(command -v perl) || fail "this host has no perl to observe the guard's lease reads" +# Observes the real lease-age reads, identified by the lease-age program's own +# text, and changes nothing about what they return. The FIRST such read becomes +# the lease reference - that is the synchronization - and every later one is +# recorded with the value it read and the interval it spanned, which is the +# evidence the phase assertion below consumes. +cat > "$BOUND_BIN/perl" <<SH +#!/usr/bin/env bash +for arg in "\$@"; do + case \$arg in + *'int(\$now - \$value)'*) + started=\$("$REAL_PERL" -MTime::HiRes=clock_gettime,CLOCK_MONOTONIC -e \\ + 'printf "%.6f\\n", clock_gettime(CLOCK_MONOTONIC)') || exit 1 + age=\$("$REAL_PERL" "\$@") || exit \$? + finished=\$("$REAL_PERL" -MTime::HiRes=clock_gettime,CLOCK_MONOTONIC -e \\ + 'printf "%.6f\\n", clock_gettime(CLOCK_MONOTONIC)') || exit 1 + if [ ! -s "\$GUARD_BOUND_STATE/reference" ]; then + printf '%s\\n' "\$finished" > "\$FM_HOME/state/procevent/.owner-lease" || exit 1 + printf '%s\\n' "\$finished" > "\$GUARD_BOUND_STATE/reference" || exit 1 + else + printf '%s\\t%s\\t%s\\t%s\\n' "\$started" "\$finished" "\$age" "\${!#}" \\ + >> "\$GUARD_BOUND_STATE/reads" || exit 1 + fi + printf '%s\\n' "\$age" + exit 0 + ;; + esac +done +exec "$REAL_PERL" "\$@" +SH +chmod +x "$BOUND_BIN/perl" +bound_pe() { + PATH="$BOUND_BIN:$PATH" GUARD_BOUND_STATE="$BOUND_STATE" \ + FM_PROCEVENT_OWNER_LEASE_SECONDS="$BOUND_LEASE_SECONDS" \ + FM_PROCEVENT_OWNER_CHECK_SECONDS="$BOUND_CHECK_SECONDS" \ + FM_HOME="$HBOUND" "$ROOT/bin/fm-procevent.sh" "$@" +} +bound_pe register lavish bound-src -- "$QUIET_STUB" "$TMP_ROOT/guard-bound-marker" >/dev/null +bound_pe reconcile >/dev/null +wait_for "$HBOUND/state/procevent/bound-src.runner" \ + || fail "the bound fixture's listener never recorded its runner" +wait_for "$TMP_ROOT/guard-bound-marker.descendant" \ + || fail "the bound fixture's listener never spawned its descendant" +BOUND_PID=$(cat "$HBOUND/state/procevent/bound-src.runner") +BOUND_DESCENDANT=$(cat "$TMP_ROOT/guard-bound-marker.descendant") +# Elapsed is measured from the refresh the guard itself reads, not from a +# wall-clock moment near it, so the fixture's own startup cost cannot be +# mistaken for guard latency in either direction. +bound_reference=$(cat "$HBOUND/state/procevent/.owner-lease") \ + || fail "the bound fixture recorded no owner lease to measure against" +[ "$bound_reference" = "$(cat "$BOUND_STATE/reference" 2>/dev/null)" ] \ + || fail "the bound fixture did not synchronize its lease to an observed guard read" +while kill -0 -"$BOUND_PID" 2>/dev/null; do + [ "$(mono_since "$bound_reference" | cut -d. -f1)" -lt "$bound_deadline_s" ] \ + || fail "the guard exceeded its bound: group still running $(mono_since "$bound_reference")s after the last owner activity, against a documented bound of ${bound_total}s (lease term ${bound_lease_term}s + one ${BOUND_CHECK_SECONDS}s check interval + ${PROOF_PROMPT_STOP}s stop)" + sleep 0.2 +done +bound_elapsed=$(mono_since "$bound_reference") +# The loop above only ever checks the clock while the group is still alive, so a +# sampler descheduled past the deadline would see the group already gone and +# report success. Check the OBSERVED completion time too: a late observation +# must not certify timely completion. +[ "${bound_elapsed%%.*}" -lt "$bound_deadline_s" ] \ + || fail "the guard's completion was first observed ${bound_elapsed}s after the last owner activity, beyond its ${bound_deadline_s}s deadline" +# FAIL CLOSED ON THE PHASE. One recorded read must prove the run was in the part +# of the interval this deadline can actually judge: it read the synchronized +# reference, it was still fresh (pre-expiry), and it began late enough that two +# further FULL intervals could not finish before the deadline. Without such a +# read the case refuses - it does not pass on trust, however quickly the group +# happened to stop. +perl - "$BOUND_STATE/reads" "$bound_reference" "$BOUND_LEASE_SECONDS" \ + "$BOUND_CHECK_SECONDS" "$bound_deadline_s" <<'PL' \ + || fail "the bound fixture could not establish the required pre-expiry guard-read phase" +use strict; +use warnings; +my ($path, $reference, $lease, $check, $deadline) = @ARGV; +open my $reads, '<', $path or exit 1; +while (<$reads>) { + chomp; + my ($started, $finished, $age, $value) = split /\t/; + next unless defined $value && $value eq $reference && $age <= $lease; + next unless $started >= $reference && $finished >= $started; + next unless $finished < $reference + $lease + 1; + next unless $started + 2 * $check >= $reference + $deadline; + printf "guard phase: fresh read %.3f-%.3fs, expiry %ss, two full intervals could not finish before %.3fs (deadline %ss)\n", + $started - $reference, $finished - $reference, $lease + 1, + $started - $reference + 2 * $check, $deadline; + exit 0; +} +exit 1; +PL +wait_gone "$BOUND_DESCENDANT" \ + || fail "the guard stopped at the leader and left its descendant running" +printf 'guard bound: lease=%ss check=%ss reaped %ss after the last owner activity, documented bound %ss\n' \ + "$BOUND_LEASE_SECONDS" "$BOUND_CHECK_SECONDS" "$bound_elapsed" "$bound_total" +pass "an orphaned runner is reaped within the lease plus ONE check interval" + +# --- a zero-prefixed interval still starts a listener, and halves correctly --- +# +# OUR OWN REGRESSION, found in review before this change was published. The +# interval validator accepts a zero-prefixed value and `[` compares it as +# decimal, but the half-interval arithmetic introduced above reads `$(( ))`, +# which is octal for a leading zero: 010 halved to 4 instead of 5, and 08 was +# not a number at all, so the guard died before reporting ready and the runner +# failed closed and never listened. +# +# Asserted through the executable interface rather than by reading the source: +# a real listener is started at each value, and the guard's actual sleep +# argument is observed. Reading `10#` out of the script would prove nothing. +INTERVAL_BIN=$(fm_fakebin "$TMP_ROOT/decimal-interval-bin") +REAL_SLEEP=$(command -v sleep) || fail "this host has no sleep to observe guard intervals" +cat > "$INTERVAL_BIN/sleep" <<SH +#!/usr/bin/env bash +printf '%s\n' "\$*" >> "\$INTERVAL_SLEEP_LOG" +exec "$REAL_SLEEP" "\$@" +SH +chmod +x "$INTERVAL_BIN/sleep" +for interval in 08 010; do + case "$interval" in + 08) expected_half=4 ;; + 010) expected_half=5 ;; + esac + HINTERVAL="$TMP_ROOT/decimal-interval-$interval"; new_home "$HINTERVAL" + pe_register "$HINTERVAL" lavish "interval-$interval" \ + -- "$QUIET_STUB" "$HINTERVAL/poll" >/dev/null + PATH="$INTERVAL_BIN:$PATH" INTERVAL_SLEEP_LOG="$HINTERVAL/sleeps" \ + FM_PROCEVENT_OWNER_CHECK_SECONDS="$interval" \ + pe "$HINTERVAL" reconcile >/dev/null + wait_for "$HINTERVAL/poll.descendant" \ + || fail "a zero-prefixed decimal interval ($interval) prevented the listener from starting" + for _ in $(seq 1 100); do + grep -qx "$expected_half" "$HINTERVAL/sleeps" 2>/dev/null && break + sleep 0.1 + done + grep -qx "$expected_half" "$HINTERVAL/sleeps" \ + || fail "the guard did not sleep half of the decimal interval $interval (expected ${expected_half}s)" + pe "$HINTERVAL" retire "interval-$interval" >/dev/null \ + || fail "retiring the decimal-interval listener ($interval) reported failure" + printf 'decimal interval: %s halves to %ss and its listener started\n' "$interval" "$expected_half" +done +pass "a zero-prefixed decimal interval starts its listener and halves as decimal" + +# --- one unreadable read still does not end a live runner -------------------- +# +# The bound above was tightened by moving the guard's two reads closer together, +# NOT by dropping the second one. This is what that second read is for, asserted +# separately so the two cannot be traded for each other by accident: against a +# home that is still alive, an isolated failed read must not stop the runner. +# +# The failure is injected where the real path actually reads. ONE lease read +# fails, exactly once, identified by the lease-age program's own text so no +# other call in the runner is touched; every read before and after it is the +# real command, and the home's lease stays long and fresh throughout. The single +# failed read is therefore the only thing wrong that the guard can see. + +DEBOUNCE_HOME="$TMP_ROOT/lease-debounce"; new_home "$DEBOUNCE_HOME" +fm_test_track_procevent_home "$DEBOUNCE_HOME" +DEBOUNCE_STATE="$TMP_ROOT/lease-debounce-state"; mkdir -p "$DEBOUNCE_STATE" +DEBOUNCE_BIN=$(fm_fakebin "$TMP_ROOT/lease-debounce-bin") +REAL_PERL=$(command -v perl) || fail "this host has no perl to build the debounce fixture on" +cat > "$DEBOUNCE_BIN/perl" <<SH +#!/usr/bin/env bash +if [ -s "\$LEASE_DEBOUNCE_STATE/armed" ] && [ ! -s "\$LEASE_DEBOUNCE_STATE/spent" ]; then + for arg in "\$@"; do + case \$arg in + *'int(\$now - \$value)'*) + printf 'spent\n' > "\$LEASE_DEBOUNCE_STATE/spent" + exit 1 + ;; + esac + done +fi +exec "$REAL_PERL" "\$@" +SH +chmod +x "$DEBOUNCE_BIN/perl" + +# A long lease and a short check: many reads happen inside the observation +# window, and none of them can go stale on their own during it. +DEBOUNCE_LEASE_SECONDS=30 +DEBOUNCE_CHECK_SECONDS=1 +debounce_pe() { + PATH="$DEBOUNCE_BIN:$PATH" LEASE_DEBOUNCE_STATE="$DEBOUNCE_STATE" \ + FM_PROCEVENT_OWNER_LEASE_SECONDS="$DEBOUNCE_LEASE_SECONDS" \ + FM_PROCEVENT_OWNER_CHECK_SECONDS="$DEBOUNCE_CHECK_SECONDS" \ + FM_HOME="$DEBOUNCE_HOME" "$ROOT/bin/fm-procevent.sh" "$@" +} +debounce_pe register lavish debounce-src -- "$QUIET_STUB" "$TMP_ROOT/lease-debounce-marker" >/dev/null +debounce_pe reconcile >/dev/null +wait_for "$DEBOUNCE_HOME/state/procevent/debounce-src.runner" \ + || fail "the debounce fixture's listener never recorded its runner" +DEBOUNCE_PID=$(cat "$DEBOUNCE_HOME/state/procevent/debounce-src.runner") +# Armed only now. The guard proves the lease once before it reports ready, and +# failing THAT read would refuse the runner outright instead of exercising the +# debounce this case is about. +printf 'armed\n' > "$DEBOUNCE_STATE/armed" +wait_for "$DEBOUNCE_STATE/spent" \ + || fail "the single failed lease read this case injects never happened" +# Several further checks at the configured interval. A guard that acted on one +# failed read would have stopped the group during them. +sleep $((DEBOUNCE_CHECK_SECONDS * 4)) +kill -0 -"$DEBOUNCE_PID" 2>/dev/null \ + || fail "one unreadable lease read ended a runner whose home was still alive" +debounce_pe retire debounce-src >/dev/null \ + || fail "retiring the debounce fixture's source reported failure" +wait_gone "-$DEBOUNCE_PID" \ + || fail "retiring the debounce fixture left its process group running" +pass "one unreadable read does not end a live runner" + +# --- the ordinary stop signal is what stops a runner ------------------------ +# +# The forced kill is the backstop, not the normal path. When it carries every +# stop, it stops being able to report that anything went wrong - which is exactly +# how a listener that could not be stopped looked identical to one that could. + +HPROMPT="$TMP_ROOT/prompt-stop"; new_home "$HPROMPT" +pe_register "$HPROMPT" lavish prompt-src -- "$QUIET_STUB" "$TMP_ROOT/prompt-stop" >/dev/null +pe "$HPROMPT" start prompt-src >"$TMP_ROOT/prompt-start.log" 2>&1 & +PROMPT_START_PID=$! +wait_for "$HPROMPT/state/procevent/prompt-src.runner" \ + || fail "the promptly-stopping listener never recorded its runner" +PROMPT_PID=$(cat "$HPROMPT/state/procevent/prompt-src.runner") +wait_for "$TMP_ROOT/prompt-stop.descendant" \ + || fail "the promptly-stopping listener's child never spawned its own descendant" +stop_window_ms() { + local from to + from=$(now_ms) + for _ in $(seq 1 20); do sleep 0.1; done + to=$(now_ms) + printf '%s\n' "$((to - from))" +} +window_before=$(stop_window_ms) +start=$(now_ms) +pe "$HPROMPT" retire prompt-src >/dev/null || fail "retiring a healthy listener reported failure" +elapsed=$(( $(now_ms) - start )) +window_after=$(stop_window_ms) +prompt_status=0 +wait "$PROMPT_START_PID" || prompt_status=$? +wait_gone "-$PROMPT_PID" || fail "retiring a healthy listener left its process group running" +[ "$prompt_status" -eq 143 ] \ + || fail "the runner did not exit on TERM (start status=$prompt_status, retirement=${elapsed}ms, sampled windows=${window_before}/${window_after}ms)" +printf 'ordinary stop: start status=%s retirement=%sms sampled windows=%s/%sms\n' \ + "$prompt_status" "$elapsed" "$window_before" "$window_after" +pass "a runner exits on the ordinary stop signal instead of outliving it" + +# --- a crashed leader's group is still refused ------------------------------- +# +# The escalation above accepts a leaderless group in exactly one place: inside +# the stop that just proved and signalled that generation itself. Whether a group +# whose leader died to something ELSE may ever be signalled is a separate open +# question, and this pins that it stays refused - so the escalation cannot widen +# into an answer to it by accident. + +HCRASH="$TMP_ROOT/crashed-leader"; new_home "$HCRASH" +pe_register "$HCRASH" lavish crash-src -- "$QUIET_STUB" "$TMP_ROOT/crash-leader" >/dev/null +pe "$HCRASH" reconcile >/dev/null +wait_for "$HCRASH/state/procevent/crash-src.runner" \ + || fail "the crash-fixture listener never recorded its runner" +CRASH_PID=$(cat "$HCRASH/state/procevent/crash-src.runner") +wait_for "$TMP_ROOT/crash-leader.descendant" \ + || fail "the crash-fixture listener's child never spawned its own descendant" +kill -KILL "$CRASH_PID" 2>/dev/null || fail "the crash fixture could not stop its own leader" +deadline=$((SECONDS + 10)) +while kill -0 "$CRASH_PID" 2>/dev/null; do + [ "$SECONDS" -lt "$deadline" ] || fail "the crash fixture's leader never died" + sleep 0.1 +done +kill -0 -"$CRASH_PID" 2>/dev/null \ + || fail "the crash fixture left no surviving group, so nothing was refused" + +out=$(pe "$HCRASH" retire crash-src 2>&1) && fail "retirement claimed success on a crashed leader's group" +assert_contains "$out" "cannot confirm runner identity" \ + "a crashed leader's group is refused with its own diagnostic" +assert_present "$HCRASH/state/procevent/crash-src.source" \ + "a refused retirement leaves the source registered" +kill -0 -"$CRASH_PID" 2>/dev/null \ + || fail "a refused retirement signalled the leaderless group anyway" +pass "a group whose leader died to something else is still refused, not signalled" +kill -KILL -"$CRASH_PID" 2>/dev/null || true + printf '\nall procevent tests passed\n' diff --git a/tests/fm-remote-backlog-handoff.test.sh b/tests/fm-remote-backlog-handoff.test.sh index 061f245062c..e9e209d3434 100755 --- a/tests/fm-remote-backlog-handoff.test.sh +++ b/tests/fm-remote-backlog-handoff.test.sh @@ -311,11 +311,7 @@ pass "concurrent handoffs serialize staging through confirmed cleanup" # conservative procedure tasks-axi prints. write_backlog '- [ ] stale-lock-item - remote stale lock recovery (repo: alpha)' printf '999999:abandoned:0:1\n' > "$REMOTE/data/backlog.md.lock" -if [ "$(uname 2>/dev/null)" = Darwin ]; then - touch -t 202001010000 "$REMOTE/data/backlog.md.lock" -else - touch -d '2020-01-01 00:00:00' "$REMOTE/data/backlog.md.lock" -fi +touch -t 202001010000 "$REMOTE/data/backlog.md.lock" handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios stale-lock-item >/dev/null \ || fail "host-local stale lock recovery did not retry receipt" assert_grep 'stale-lock-item' "$REMOTE/data/backlog.md" "stale-lock receipt lost the item" @@ -339,21 +335,118 @@ handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending >/dev/null \ pass "bootstrap detects pending outbox handoffs without a journal" write_backlog '- [ ] remote-wake-fail - receiver failure stays recoverable (repo: alpha)' -set +e FM_FAKE_REMOTE_WAKE_RC=1 handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios remote-wake-fail \ - > "$TMP_ROOT/remote-wake-fail.out" 2>&1 -rc=$? -set -e -[ "$rc" -ne 0 ] || fail "remote handoff claimed success after its receiver wake failed" + > "$TMP_ROOT/remote-wake-fail.out" 2>&1 \ + || fail "durably received remote handoff failed only because its best-effort wake failed" assert_contains "$(cat "$TMP_ROOT/remote-wake-fail.out")" 'receiver wake failed' \ "remote receiver wake failure was not surfaced" -assert_present "$PARENT/data/handoff/ios.outbox.md" \ - "remote receiver wake failure discarded the recoverable outbox" +assert_absent "$PARENT/data/handoff/ios.outbox.md" \ + "remote receiver wake failure retained the durably received outbox" +assert_present "$PARENT/state/.backlog-handoff-ios.wake-pending" \ + "remote receiver wake failure was not tracked separately" handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending >/dev/null \ || fail "remote receiver wake failure did not recover through resume-pending" -assert_absent "$PARENT/data/handoff/ios.outbox.md" \ - "remote receiver wake recovery left its outbox pending" -pass "remote handoff wakes its supported endpoint or remains loudly recoverable" +assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" \ + "remote receiver wake recovery left separate wake state pending" +pass "remote handoff releases durable work and separately retries its receiver wake" + +# The receiver wake is a best-effort live nudge sent AFTER the backlog receipt +# is durable. A wake whose remote transport is lost leaves its correlation +# undelivered with delivery unknown, and the watcher's very next pending-reply +# tick escalates that correlation. That escalated-but-undelivered wake must +# stay retryable: the outbox otherwise jams every later handoff to this mate +# behind a correlation the resume refuses to resend forever. +write_backlog '- [ ] wake-escalated - escalated undelivered wake stays retryable (repo: alpha)' +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +FM_FAKE_REMOTE_WAKE_RC=255 handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios wake-escalated \ + > "$TMP_ROOT/wake-escalated.out" 2>&1 \ + || fail "durably received handoff failed only because its wake transport was lost" +assert_grep 'wake-escalated' "$REMOTE/data/backlog.md" "lost wake transport did not leave the backlog durably received" +assert_absent "$PARENT/data/handoff/ios.outbox.md" "lost wake transport retained the durably received outbox" +wake_marker=$(cat "$PARENT/state/.backlog-handoff-ios.wake-pending" 2>/dev/null || true) +case "$wake_marker" in + pending:*) escalated_corr=${wake_marker#pending:} ;; + *) fail "lost wake transport did not leave a pending correlated wake, got '$wake_marker'" ;; +esac +escalated_rec="$PARENT/state/pending-replies/$escalated_corr" +[ -f "$escalated_rec" ] || fail "lost wake transport left no pending-reply record for $escalated_corr" +[ "$(grep '^phase=' "$escalated_rec" | cut -d= -f2-)" = delivery_unknown ] \ + || fail "lost wake transport did not record delivery unknown" +# The watcher's pending-reply tick (bin/fm-watch.sh -> fm_pending_reply_tick) +# escalates an undelivered delivery-unknown correlation before any resume runs. +bash -c '. "$1"; fm_pending_reply_tick "$2"' _ "$ROOT/bin/fm-pending-reply-lib.sh" "$PARENT/state" \ + || fail "pending-reply tick failed on the undelivered wake" +[ "$(grep '^phase=' "$escalated_rec" | cut -d= -f2-)" = escalated ] \ + || fail "watcher tick did not escalate the undelivered wake, got $(grep '^phase=' "$escalated_rec")" +[ -z "$(grep '^delivered_epoch=' "$escalated_rec" | cut -d= -f2-)" ] \ + || fail "escalation must not invent a delivery for the undelivered wake" +[ "$(grep -cF "blocked [key=pending-reply-$escalated_corr]:" "$PARENT/state/ios.status")" -eq 1 ] \ + || fail "undelivered wake escalation was not published exactly once" +set +e +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending > "$TMP_ROOT/wake-escalated-resume.out" 2>&1 +rc=$? +set -e +if [ "$rc" -ne 0 ]; then + printf 'resume output:\n%s\n' "$(cat "$TMP_ROOT/wake-escalated-resume.out")" >&2 + fail "resume refused to retry the escalated undelivered wake (outbox deadlock)" +fi +assert_absent "$PARENT/data/handoff/ios.outbox.md" "escalated wake retry recreated a released outbox" +assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" "escalated wake retry left wake state behind" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq $((wakes_before + 3)) ] \ + || fail "escalated wake retry did not resend the wake exactly once after the two lost attempts" +[ -n "$(grep '^delivered_epoch=' "$escalated_rec" | cut -d= -f2-)" ] \ + || fail "successful wake retry did not confirm delivery on the same correlation" +[ "$(grep '^phase=' "$escalated_rec" | cut -d= -f2-)" = awaiting_report ] \ + || fail "delivered wake retry did not return the correlation to awaiting its report" +[ "$(grep -cF "blocked [key=pending-reply-$escalated_corr]:" "$PARENT/state/ios.status")" -eq 1 ] \ + || fail "wake retry duplicated the published escalation" +write_backlog '- [ ] after-escalated - next handoff flows once the escalated wake is retried (repo: alpha)' +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios after-escalated >/dev/null \ + || fail "handoff after the escalated wake retry did not flow" +[ "$(grep -cF after-escalated "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "handoff after the escalated wake retry was lost or duplicated" +assert_absent "$PARENT/data/handoff/ios.outbox.md" "handoff after the escalated wake retry left an outbox pending" +pass "an escalated undelivered receiver wake stays retryable instead of jamming the outbox" + +write_backlog '- [ ] wake-permanent-a - first handoff with permanently lost wake (repo: alpha)' +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +FM_FAKE_REMOTE_WAKE_RC=255 handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios wake-permanent-a \ + > "$TMP_ROOT/wake-permanent-a.out" 2>&1 \ + || fail "first durably received handoff was held hostage to a permanently lost wake" +assert_absent "$PARENT/data/handoff/ios.outbox.md" "permanently lost wake retained the first durable outbox" +[ "$(grep -cF wake-permanent-a "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "first handoff under a permanently lost wake was lost or duplicated" +permanent_marker=$(cat "$PARENT/state/.backlog-handoff-ios.wake-pending" 2>/dev/null || true) +case "$permanent_marker" in + pending:*) permanent_corr=${permanent_marker#pending:} ;; + *) fail "permanently lost wake was not separately tracked, got '$permanent_marker'" ;; +esac +wakes_after_first=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +[ "$wakes_after_first" -gt "$wakes_before" ] || fail "first permanently lost wake was not attempted" +set +e +FM_FAKE_REMOTE_WAKE_RC=255 handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending \ + > "$TMP_ROOT/wake-permanent-resume.out" 2>&1 +rc=$? +set -e +[ "$rc" -ne 0 ] || fail "resume claimed that the permanently lost wake was confirmed" +[ "$(cat "$PARENT/state/.backlog-handoff-ios.wake-pending")" = "pending:$permanent_corr" ] \ + || fail "failed wake resume did not preserve the original correlation" +wakes_after_resume=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +[ "$wakes_after_resume" -gt "$wakes_after_first" ] || fail "resume did not retry the separately pending wake" +write_backlog '- [ ] wake-permanent-b - second handoff despite permanently lost wake (repo: alpha)' +FM_FAKE_REMOTE_WAKE_RC=255 handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios wake-permanent-b \ + > "$TMP_ROOT/wake-permanent-b.out" 2>&1 \ + || fail "second durable handoff was blocked by the permanently lost wake" +assert_absent "$PARENT/data/handoff/ios.outbox.md" "permanently lost wake retained the second durable outbox" +[ "$(grep -cF wake-permanent-a "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "later handoff duplicated the first durably received item" +[ "$(grep -cF wake-permanent-b "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "later handoff was lost or duplicated behind the pending wake" +[ "$(cat "$PARENT/state/.backlog-handoff-ios.wake-pending")" = "pending:$permanent_corr" ] \ + || fail "later handoff did not retain the same pending wake correlation" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -gt "$wakes_after_resume" ] \ + || fail "later handoff did not retry the separately pending wake" +pass "a permanently unconfirmable wake never jams later durable handoffs" RM_FAKEBIN="$TMP_ROOT/rm-fakebin" mkdir -p "$RM_FAKEBIN" @@ -401,6 +494,153 @@ assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" \ "fresh handoff left confirmed wake state behind" pass "fresh remote work gets a new wake after confirmed cleanup recovery" +write_backlog '- [ ] confirmed-marker-stale - completed handoff ignores marker cleanup failure (repo: alpha)' +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +PATH="$RM_FAKEBIN:$PATH" FM_REAL_RM="$REAL_RM" \ + FM_FAIL_RM_PATH="$PARENT/state/.backlog-handoff-ios.wake-pending" \ + handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios confirmed-marker-stale \ + > "$TMP_ROOT/confirmed-marker-stale.out" 2>&1 \ + || fail "confirmed marker cleanup failure falsely failed a completed handoff" +assert_absent "$PARENT/data/handoff/ios.outbox.md" \ + "confirmed marker cleanup failure retained a completed outbox" +case "$(cat "$PARENT/state/.backlog-handoff-ios.wake-pending")" in + confirmed:*) ;; + *) fail "forced confirmed marker cleanup failure did not preserve confirmed state" ;; +esac +assert_contains "$(cat "$TMP_ROOT/confirmed-marker-stale.out")" \ + "stale confirmed wake marker remains at $PARENT/state/.backlog-handoff-ios.wake-pending" \ + "confirmed marker cleanup failure did not name the stale marker" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq $((wakes_before + 1)) ] \ + || fail "confirmed marker cleanup failure changed the completed wake count" +[ "$(grep -cF -- '- [ ] confirmed-marker-stale -' "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "confirmed marker cleanup failure lost or duplicated durable work" +rm -f -- "$PARENT/state/.backlog-handoff-ios.wake-pending" +pass "confirmed marker cleanup cannot fail a completed remote handoff" + +CONFIRM_MV_FAKEBIN="$TMP_ROOT/confirm-mv-fakebin" +mkdir -p "$CONFIRM_MV_FAKEBIN" +REAL_MV=$(command -v mv) +cat > "$CONFIRM_MV_FAKEBIN/mv" <<'SH' +#!/usr/bin/env bash +last=${!#} +if [ "$last" = "$FM_CONFIRM_MV_PATH" ]; then + count=$(cat "$FM_CONFIRM_MV_COUNT" 2>/dev/null || echo 0) + count=$((count + 1)) + printf '%s\n' "$count" > "$FM_CONFIRM_MV_COUNT" + [ "$count" -ne 2 ] || exit 1 +fi +exec "$FM_REAL_MV" "$@" +SH +chmod +x "$CONFIRM_MV_FAKEBIN/mv" +write_backlog '- [ ] delivered-pending-old - wake confirms before state promotion fails (repo: alpha)' +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +PATH="$CONFIRM_MV_FAKEBIN:$PATH" FM_REAL_MV="$REAL_MV" \ + FM_CONFIRM_MV_PATH="$PARENT/state/.backlog-handoff-ios.wake-pending" \ + FM_CONFIRM_MV_COUNT="$TMP_ROOT/confirm-mv.count" \ + handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios delivered-pending-old \ + > "$TMP_ROOT/delivered-pending-old.out" 2>&1 \ + || fail "confirmed wake state promotion failure failed durable work" +delivered_pending_marker=$(cat "$PARENT/state/.backlog-handoff-ios.wake-pending" 2>/dev/null || true) +case "$delivered_pending_marker" in + pending:*) delivered_pending_corr=${delivered_pending_marker#pending:} ;; + *) fail "confirmed wake state promotion failure did not leave pending correlation state" ;; +esac +delivered_pending_rec="$PARENT/state/pending-replies/$delivered_pending_corr" +[ -n "$(grep '^delivered_epoch=' "$delivered_pending_rec" | cut -d= -f2-)" ] \ + || fail "pending marker fixture did not retain confirmed delivery evidence" +write_backlog '- [ ] delivered-pending-new - new work after delivered pending marker (repo: alpha)' +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios delivered-pending-new >/dev/null \ + || fail "delivered pending marker blocked the next handoff" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq $((wakes_before + 2)) ] \ + || fail "delivered pending marker suppressed the new handoff wake" +[ "$(grep -cF -- '- [ ] delivered-pending-old -' "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "state promotion failure lost or duplicated the older item" +[ "$(grep -cF -- '- [ ] delivered-pending-new -' "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "handoff after delivered pending state was lost or duplicated" +assert_present "$delivered_pending_rec" \ + "clearing delivered pending state discarded its pending-reply record" +[ -n "$(grep '^delivered_epoch=' "$delivered_pending_rec" | cut -d= -f2-)" ] \ + || fail "clearing delivered pending state reset confirmed delivery" +assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" \ + "new handoff left delivered pending state behind" +pass "delivered pending state cannot suppress a new handoff wake" + +MV_FAKEBIN="$TMP_ROOT/mv-fakebin" +mkdir -p "$MV_FAKEBIN" +REAL_MV=$(command -v mv) +cat > "$MV_FAKEBIN/mv" <<'SH' +#!/usr/bin/env bash +last=${!#} +if [ "$last" = "$FM_FAIL_MV_PATH" ]; then + exit 1 +fi +exec "$FM_REAL_MV" "$@" +SH +chmod +x "$MV_FAKEBIN/mv" +write_backlog '- [ ] wake-state-drop - durable work survives lost wake state (repo: alpha)' +pending_records_before=$(find "$PARENT/state/pending-replies" -maxdepth 1 -type f | wc -l | tr -d ' ') +wakes_before=$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG") +PATH="$MV_FAKEBIN:$PATH" FM_REAL_MV="$REAL_MV" \ + FM_FAIL_MV_PATH="$PARENT/state/.backlog-handoff-ios.wake-pending" \ + handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios wake-state-drop \ + > "$TMP_ROOT/wake-state-drop.out" 2>&1 \ + || fail "wake-state write failure held the durable outbox hostage" +assert_contains "$(cat "$TMP_ROOT/wake-state-drop.out")" 'receiver wake state: DROPPED' \ + "wake-state write failure did not report an honest dropped state" +assert_contains "$(cat "$TMP_ROOT/wake-state-drop.out")" 'best-effort receiver wake was dropped' \ + "wake-state write failure did not log the dropped wake" +assert_absent "$PARENT/data/handoff/ios.outbox.md" \ + "wake-state write failure retained the durably received outbox" +assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" \ + "wake-state write failure left a marker claiming the wake was pending" +pending_records_after=$(find "$PARENT/state/pending-replies" -maxdepth 1 -type f | wc -l | tr -d ' ') +[ "$pending_records_after" -eq "$pending_records_before" ] \ + || fail "wake-state write failure left an unreferenced pending-reply record" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq "$wakes_before" ] \ + || fail "wake-state write failure attempted an untracked wake" +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending >/dev/null \ + || fail "resume treated the dropped wake as pending" +[ "$(grep -cF fm-remote-secondmate-control.sh "$WAKE_LOG")" -eq "$wakes_before" ] \ + || fail "resume retried a wake whose state was dropped" +write_backlog '- [ ] after-wake-state-drop - later handoff after dropped wake state (repo: alpha)' +handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios after-wake-state-drop >/dev/null \ + || fail "later handoff was blocked by dropped wake state" +[ "$(grep -cF -- '- [ ] wake-state-drop -' "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "wake-state failure lost or duplicated its durably received item" +[ "$(grep -cF -- '- [ ] after-wake-state-drop -' "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "later handoff after dropped wake state was lost or duplicated" +assert_absent "$PARENT/state/.backlog-handoff-ios.wake-pending" \ + "later successful handoff left stale wake state" +pass "unrecordable best-effort wake state drops without blocking later handoffs" + +stale_wake_marker="$PARENT/state/.backlog-handoff-ios.wake-pending" +printf 'invalid wake state\n' > "$stale_wake_marker" +PATH="$RM_FAKEBIN:$PATH" FM_REAL_RM="$REAL_RM" FM_FAIL_RM_PATH="$stale_wake_marker" \ + handoff_env "$ROOT/bin/fm-backlog-handoff.sh" --resume-pending \ + > "$TMP_ROOT/stale-wake-resume.out" 2>&1 \ + || fail "resume was blocked by an undeletable invalid wake marker" +assert_present "$stale_wake_marker" "invalid wake marker did not survive the forced removal failure" +assert_contains "$(cat "$TMP_ROOT/stale-wake-resume.out")" 'receiver wake state: DROPPED' \ + "resume did not report the invalid wake marker as dropped" +assert_contains "$(cat "$TMP_ROOT/stale-wake-resume.out")" "stale wake marker remains at $stale_wake_marker" \ + "resume did not name the surviving stale wake marker" +write_backlog '- [ ] after-stale-wake - later handoff ignores stale wake state (repo: alpha)' +PATH="$RM_FAKEBIN:$PATH" FM_REAL_RM="$REAL_RM" FM_FAIL_RM_PATH="$stale_wake_marker" \ + handoff_env "$ROOT/bin/fm-backlog-handoff.sh" ios after-stale-wake \ + > "$TMP_ROOT/after-stale-wake.out" 2>&1 \ + || fail "later handoff was blocked by an undeletable invalid wake marker" +assert_absent "$PARENT/data/handoff/ios.outbox.md" \ + "stale wake marker retained the later handoff outbox" +[ "$(grep -cF -- '- [ ] after-stale-wake -' "$REMOTE/data/backlog.md")" -eq 1 ] \ + || fail "handoff past a stale wake marker was lost or duplicated" +assert_contains "$(cat "$TMP_ROOT/after-stale-wake.out")" 'receiver wake state: DROPPED' \ + "later handoff did not report the stale wake as dropped" +assert_contains "$(cat "$TMP_ROOT/after-stale-wake.out")" "stale wake marker remains at $stale_wake_marker" \ + "later handoff did not name the surviving stale wake marker" +assert_present "$stale_wake_marker" "later handoff concealed the forced stale-marker removal failure" +rm -f -- "$stale_wake_marker" +pass "undeletable invalid wake state cannot block remote handoffs" + write_backlog '- [ ] route-race - remains dispatchable through retirement (repo: alpha)' registry_lock="$PARENT/state/.secondmate-registry.lock" handoff_lock="$PARENT/state/.backlog-handoff-ios.lock" diff --git a/tests/fm-remote-doctor.test.sh b/tests/fm-remote-doctor.test.sh index 60d02ade8b2..3d79e8e41d7 100755 --- a/tests/fm-remote-doctor.test.sh +++ b/tests/fm-remote-doctor.test.sh @@ -2,15 +2,19 @@ # tests/fm-remote-doctor.test.sh - the remote second-mate readiness gate. # # Drives the real bin/fm-remote-doctor.sh against a controlled account fixture: -# a private HOME, a fake launchctl backed by state files, a fake herdr CLI, and -# a fake uname that selects the platform under test. Nothing here touches the -# runner's own launch agents, login session, or herdr server. +# a private HOME, a fake launchctl backed by state files, a fake herdr CLI, a +# fake lsof that names a real holder process as the fm-remote socket owner, and +# a fake uname that selects the platform under test. The holders are real +# non-platform processes (jq blocked on a fifo) whose environment carries the +# birth markers bin/fm-remote-herdr-owner-lib.sh reads, so the Aqua-versus-SSH +# verdict is exercised for real. Nothing here touches the runner's own launch +# agents, login session, or herdr server. set -u # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" - command -v jq >/dev/null 2>&1 || { echo "skip: jq not found (the herdr adapter parses its JSON)"; exit 0; } +command -v python3 >/dev/null 2>&1 || { echo "skip: python3 not found (plistlib parses the owned launch-agent contract)"; exit 0; } TMP_ROOT=$(fm_test_tmproot fm-remote-doctor) LABEL=dev.firstmate.herdr.fm-remote @@ -20,7 +24,9 @@ TMP_ROOT=$(cd "$TMP_ROOT" && pwd -P) JOB_LABEL=dev.firstmate.remote-job CASE_N=0 DOCTOR_WORKER_PID= -trap 'if [ -n "$DOCTOR_WORKER_PID" ]; then kill "$DOCTOR_WORKER_PID" 2>/dev/null || true; fi; fm_test_cleanup || true' EXIT +HOLDER_PIDS=() +trap 'if [ -n "$DOCTOR_WORKER_PID" ]; then kill "$DOCTOR_WORKER_PID" 2>/dev/null || true; fi; if [ "${#HOLDER_PIDS[@]}" -gt 0 ]; then kill "${HOLDER_PIDS[@]}" 2>/dev/null || true; fi; fm_test_cleanup || true' EXIT +GUARD="$ROOT/bin/fm-remote-herdr-guard.sh" # A fixture must be able to present a host with NO herdr, so the doctor never # sees the runner's own PATH. Only the two required tools are re-exposed, by @@ -32,15 +38,47 @@ ln -sf "$(command -v jq)" "$TOOLS/jq" BASE_PATH="$TOOLS:/usr/bin:/bin:/usr/sbin:/sbin" SYSTEM_ID=$(command -v id) -# new_case <Darwin|Linux> [with-herdr] [gui] +# Real socket-owner holders for the Darwin birth check: jq blocked on a fifo +# this test keeps open, with exactly the marker environment each birth needs. +JQ=$(command -v jq) +HOLDER_FD=5 +hold() { # <marker-env...> -> HOLDER_PID + local fifo="$TMP_ROOT/holder-$HOLDER_FD.fifo" + mkfifo "$fifo" + env -i "$@" "$JQ" . "$fifo" & + HOLDER_PID=$! + HOLDER_PIDS+=("$HOLDER_PID") + eval "exec ${HOLDER_FD}>\"\$fifo\"" + HOLDER_FD=$((HOLDER_FD + 1)) +} +hold XPC_SERVICE_NAME=dev.firstmate.herdr.fm-remote +AQUA_HOLDER_PID=$HOLDER_PID +hold XPC_SERVICE_NAME=dev.firstmate.herdr.fm-remote +BACKGROUND_HOLDER_PID=$HOLDER_PID +hold XPC_SERVICE_NAME=0 +XPC_ZERO_HOLDER_PID=$HOLDER_PID +hold FM_REMOTE_JOB_ACTIVE=1 +WORKER_HOLDER_PID=$HOLDER_PID +hold SSH_CONNECTION='100.102.217.78 51234 100.100.1.2 22' SSH_CLIENT='100.102.217.78 51234 22' +SSH_HOLDER_PID=$HOLDER_PID + +# new_case <Darwin|Linux> [with-herdr] [gui] [login-shell] # Builds one isolated account fixture and points the module-level CASE_* # variables at it. "with-herdr" installs the fake herdr CLI; "gui" makes the -# fake launchctl report an existing Aqua login session. +# fake launchctl report an existing Aqua login session. login-shell is the +# Directory Services UserShell the fake dscl reports (default /bin/sh so the +# fixture is portable to hosts without /bin/zsh). new_case() { local platform=$1 want_herdr=${2:-with-herdr} want_gui=${3:-gui} unset CASE_REMOTE_JOB_ACTIVE unset CASE_PLATFORM_OVERRIDE + unset CASE_DSCL_FAIL + unset CASE_DSCL_HANG + unset CASE_SECOND_LOGIN_SHELL + unset CASE_ENV_SHELL + unset CASE_RESOLVE_DSCL CASE_N=$((CASE_N + 1)) + CASE_LOGIN_SHELL=${4:-/bin/sh} CASE_DIR="$TMP_ROOT/case$CASE_N" CASE_BIN="$CASE_DIR/bin" CASE_HOME="$CASE_DIR/home" @@ -83,6 +121,10 @@ loaded="$FM_FAKE_STATE/loaded-$label" case "${1:-}" in print) case "$domain" in + user/*/*) + [ -f "$FM_FAKE_STATE/user-loaded-$label" ] || exit 113 + cat "$FM_FAKE_STATE/user-loaded-$label" + ;; */dev.firstmate.herdr.fm-remote) [ -f "$loaded" ] || exit 113 cat "$loaded" @@ -125,18 +167,24 @@ EOF *) cat > "$loaded" <<EOF path = $FM_FAKE_PLIST -program = $FM_FAKE_HERDR_BIN +program = $FM_FAKE_LOGIN_SHELL arguments = { - $FM_FAKE_HERDR_BIN - server - --session - fm-remote + $FM_FAKE_LOGIN_SHELL + -l + -c + exec '$FM_FAKE_GUARD' '$FM_FAKE_HERDR_BIN' 'fm-remote' } stdout path = $FM_FAKE_LAUNCH_AGENT_LOG stderr path = $FM_FAKE_LAUNCH_AGENT_LOG -properties = keepalive | runatload | inferred program +semaphores = { + successful exit => 0 +} +properties = runatload | inferred program EOF - [ -f "$FM_FAKE_STATE/bootstrap-does-not-start" ] || printf 'true\n' > "$FM_FAKE_HERDR_RUNNING" + if [ ! -f "$FM_FAKE_STATE/bootstrap-does-not-start" ]; then + printf 'true\n' > "$FM_FAKE_HERDR_RUNNING" + printf '%s\n' "$FM_FAKE_AQUA_PID" > "$FM_FAKE_STATE/socket-owner" + fi ;; esac exit 0 @@ -146,6 +194,9 @@ EOF case "$label" in dev.firstmate.remote-job) : ;; *) + # The real job execs the guard, which stops a foreign server and + # becomes the Aqua-born owner; the fixture models that outcome. + printf '%s\n' "$FM_FAKE_AQUA_PID" > "$FM_FAKE_STATE/socket-owner" if [ -f "$FM_FAKE_STATE/kickstart-delay" ]; then cp "$FM_FAKE_STATE/kickstart-delay" "$FM_FAKE_STATE/herdr-delay" else @@ -171,6 +222,37 @@ SH chmod +x "$CASE_BIN/$forbidden" done + # The socket owner the birth check sees: the pid in socket-owner, or the + # Aqua holder when a case never chose one. + cat > "$CASE_BIN/lsof" <<'SH' +#!/usr/bin/env bash +pid=$(cat "$FM_FAKE_STATE/socket-owner" 2>/dev/null || printf '%s' "$FM_FAKE_AQUA_PID") +[ -n "$pid" ] || exit 0 +printf 'p%s\n' "$pid" +printf 'n%s\n' "$FM_FAKE_HERDR_SOCKET" +SH + chmod +x "$CASE_BIN/lsof" + + cat > "$CASE_BIN/dscl" <<'SH' +#!/usr/bin/env bash +set -u +[ "${FM_FAKE_DSCL_FAIL:-0}" != 1 ] || exit 1 +[ "${FM_FAKE_DSCL_HANG:-0}" != 1 ] || exec /bin/sleep 30 +if [ "${1:-}" = . ] && [ "${2:-}" = -read ] && [ "${4:-}" = UserShell ]; then + count_file="$FM_FAKE_STATE/dscl-count" + count=$(cat "$count_file" 2>/dev/null || printf 0) + count=$((count + 1)) + printf '%s\n' "$count" > "$count_file" + shell=${FM_FAKE_LOGIN_SHELL:-/bin/sh} + if [ "$count" -gt 1 ] && [ -n "${FM_FAKE_SECOND_LOGIN_SHELL:-}" ]; then + shell=$FM_FAKE_SECOND_LOGIN_SHELL + fi + printf 'UserShell: %s\n' "$shell" + exit 0 +fi +exit 1 +SH + if [ "$want_herdr" = with-herdr ]; then cat > "$CASE_BIN/herdr" <<'SH' #!/usr/bin/env bash @@ -189,7 +271,7 @@ case "${1:-} ${2:-}" in running=true fi fi - printf '{"client":{"version":"0.7.5","protocol":16},"server":{"running":%s}}\n' "$running" + printf '{"client":{"version":"0.7.5","protocol":16},"server":{"running":%s,"socket":"%s"}}\n' "$running" "$FM_FAKE_HERDR_SOCKET" ;; "server "*|"server ") printf 'true\n' > "$FM_FAKE_HERDR_RUNNING" @@ -215,7 +297,7 @@ SH #!/usr/bin/env bash exit 0 SH - chmod +x "$CASE_BIN/uname" "$CASE_BIN/id" "$CASE_BIN/launchctl" "$CASE_BIN/tasks-axi" "$CASE_BIN/treehouse" "$CASE_BIN/claude" + chmod +x "$CASE_BIN/dscl" "$CASE_BIN/uname" "$CASE_BIN/id" "$CASE_BIN/launchctl" "$CASE_BIN/tasks-axi" "$CASE_BIN/treehouse" "$CASE_BIN/claude" cat > "$CASE_BIN/sleep" <<'SH' #!/usr/bin/env bash exit 0 @@ -236,10 +318,19 @@ doctor() { FM_FAKE_FORBIDDEN_LOG="$CASE_FORBIDDEN_LOG" \ FM_FAKE_HERDR_RUNNING="$CASE_HERDR_RUNNING" \ FM_FAKE_HERDR_BIN="$CASE_BIN/herdr" \ + FM_FAKE_HERDR_SOCKET="$CASE_STATE/herdr.sock" \ + FM_FAKE_GUARD="$GUARD" \ + FM_FAKE_AQUA_PID="$AQUA_HOLDER_PID" \ FM_FAKE_PLIST="$CASE_PLIST" \ FM_FAKE_JOB_PLIST="$CASE_JOB_PLIST" \ FM_FAKE_JOB_WORKER="$ROOT/bin/fm-remote-job-worker.sh" \ FM_FAKE_LAUNCH_AGENT_LOG="$CASE_HOME/Library/Logs/$LABEL.log" \ + FM_FAKE_LOGIN_SHELL="${CASE_LOGIN_SHELL:-/bin/sh}" \ + FM_FAKE_SECOND_LOGIN_SHELL="${CASE_SECOND_LOGIN_SHELL:-}" \ + FM_FAKE_DSCL_FAIL="${CASE_DSCL_FAIL:-0}" \ + FM_FAKE_DSCL_HANG="${CASE_DSCL_HANG:-0}" \ + FM_LAUNCH_AGENT_SHELL="$([ "${CASE_RESOLVE_DSCL:-0}" = 1 ] || printf '%s' "$CASE_LOGIN_SHELL")" \ + SHELL="${CASE_ENV_SHELL-${SHELL-}}" \ FM_REMOTE_JOB_PLATFORM_OVERRIDE="${CASE_PLATFORM_OVERRIDE-}" \ FM_REMOTE_JOB_ACTIVE="${CASE_REMOTE_JOB_ACTIVE-1}" \ "$ROOT/bin/fm-remote-doctor.sh" "$@" 2>&1 @@ -248,23 +339,63 @@ doctor() { set -e } -write_loaded_contract() { # <herdr-path> [properties] - local herdr_bin=$1 properties=${2:-'keepalive | runatload | inferred program'} +write_loaded_contract() { # <herdr-path> [properties] [exec-command] + local herdr_bin=$1 properties=${2:-'runatload | inferred program'} exec_cmd + exec_cmd=${3:-"exec '$GUARD' '$herdr_bin' 'fm-remote'"} cat > "$CASE_STATE/loaded-$LABEL" <<EOF path = $CASE_PLIST -program = $herdr_bin +program = $CASE_LOGIN_SHELL arguments = { - $herdr_bin - server - --session - fm-remote + $CASE_LOGIN_SHELL + -l + -c + $exec_cmd } stdout path = $CASE_HOME/Library/Logs/$LABEL.log stderr path = $CASE_HOME/Library/Logs/$LABEL.log +semaphores = { + successful exit => 0 +} properties = $properties EOF } +# Parse the doctor's owned launch-agent plist and assert the login-shell +# argv contract. The plist is Firstmate's output, so semantic structure is +# in bounds; never match the XML source as a substring. +plist_value() { # <plist> <key> + python3 -c 'import plistlib,sys; value=plistlib.load(open(sys.argv[1], "rb"))[sys.argv[2]]; print(value)' "$1" "$2" +} + +plist_first_value() { # <plist> <array-key> + python3 -c 'import plistlib,sys; print(plistlib.load(open(sys.argv[1], "rb"))[sys.argv[2]][0])' "$1" "$2" +} + +assert_herdr_launch_agent_contract() { # <plist> <herdr-bin> [login-shell] + local plist=$1 herdr_bin=$2 expected_shell=${3:-$CASE_LOGIN_SHELL} json argv0 argv1 argv2 cmd + json=$(python3 -c 'import json,plistlib,sys; print(json.dumps(plistlib.load(open(sys.argv[1], "rb"))))' "$plist") \ + || fail "could not parse $plist as a plist" + argv0=$(printf '%s' "$json" | jq -r '.ProgramArguments[0]') + argv1=$(printf '%s' "$json" | jq -r '.ProgramArguments[1]') + argv2=$(printf '%s' "$json" | jq -r '.ProgramArguments[2]') + cmd=$(printf '%s' "$json" | jq -r '.ProgramArguments[3]') + [ "$argv0" = "$expected_shell" ] || fail "ProgramArguments[0] is $argv0, not the resolved login shell $expected_shell" + [ "$argv1" = -l ] || fail "ProgramArguments[1] is $argv1, not -l" + [ "$argv2" = -c ] || fail "ProgramArguments[2] is $argv2, not -c" + [ "$cmd" = "exec '$GUARD' '$herdr_bin' 'fm-remote'" ] \ + || fail "ProgramArguments[3] is not exec of the guard with $herdr_bin for session fm-remote: $cmd" + [ "$(printf '%s' "$json" | jq -r '.LimitLoadToSessionType')" = Aqua ] \ + || fail "LimitLoadToSessionType is not Aqua" + [ "$(printf '%s' "$json" | jq -r '.RunAtLoad')" = true ] \ + || fail "RunAtLoad is not true" + [ "$(printf '%s' "$json" | jq -r '.KeepAlive.SuccessfulExit')" = false ] \ + || fail "KeepAlive is not {SuccessfulExit=false}: $(printf '%s' "$json" | jq -c '.KeepAlive')" + [ "$(printf '%s' "$json" | jq -r '.ThrottleInterval')" = 10 ] \ + || fail "ThrottleInterval is not 10" + [ "$(printf '%s' "$json" | jq -r '.Label')" = "$LABEL" ] \ + || fail "Label is not $LABEL" +} + assert_no_dangerous_calls() { # <msg> [ ! -s "$CASE_FORBIDDEN_LOG" ] \ || fail "$1"$'\n'"--- attempted ---"$'\n'"$(cat "$CASE_FORBIDDEN_LOG")" @@ -337,14 +468,14 @@ assert_contains "$DOCTOR_OUT" 'check remote-job-worker=ok:' "--fix did not insta assert_contains "$DOCTOR_OUT" 'check remote-job-worker-loaded=ok:' "--fix did not load the remote job worker" assert_present "$CASE_PLIST" "--fix reported success without writing the plist" assert_present "$CASE_JOB_PLIST" "--fix reported success without writing the remote job worker plist" -assert_grep '<string>Aqua</string>' "$CASE_PLIST" "the written plist is not Aqua-scoped" -assert_grep "<string>$LABEL</string>" "$CASE_PLIST" "the written plist does not carry the Firstmate label" -assert_grep '<string>server</string>' "$CASE_PLIST" "the written plist does not run a herdr server" -assert_grep '<string>fm-remote</string>' "$CASE_PLIST" "the written plist does not pin the remote-secondmate session" -assert_no_grep '<string>default</string>' "$CASE_PLIST" "the written plist pins the interactive default session" -assert_grep "<string>$JOB_LABEL</string>" "$CASE_JOB_PLIST" "the worker plist does not carry the Firstmate label" -assert_grep '<string>Aqua</string>' "$CASE_JOB_PLIST" "the worker plist is not Aqua-scoped" -assert_grep "$ROOT/bin/fm-remote-job-worker.sh" "$CASE_JOB_PLIST" "the worker plist does not use the configured code root" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" +[ "$(plist_value "$CASE_JOB_PLIST" Label)" = "$JOB_LABEL" ] \ + || fail "the worker plist does not carry the Firstmate label" +[ "$(plist_value "$CASE_JOB_PLIST" LimitLoadToSessionType)" = Aqua ] \ + || fail "the worker plist is not Aqua-scoped" +[ "$(plist_first_value "$CASE_JOB_PLIST" ProgramArguments)" = "$ROOT/bin/fm-remote-job-worker.sh" ] \ + || fail "the worker plist does not use the configured code root" +assert_absent "$CASE_STATE/dscl-count" "the injected login shell still consulted Directory Services" assert_grep "gui/$(id -u)" "$CASE_LAUNCHCTL_LOG" "the launch agent was not bootstrapped into the GUI domain" cmp -s "$CASE_STATE/interactive-before.plist" "$CASE_INTERACTIVE_PLIST" \ || fail "the fm-remote repair rewrote the interactive default launch agent" @@ -387,7 +518,7 @@ cat > "$CASE_PLIST" <<XML </dict> </plist> XML -write_loaded_contract /obsolete/bin/herdr 'runatload | inferred program' +write_loaded_contract /obsolete/bin/herdr 'keepalive | runatload | inferred program' "exec '/obsolete/bin/herdr' server --session 'fm-remote'" printf 'true\n' > "$CASE_HERDR_RUNNING" doctor expect_code 1 "$DOCTOR_RC" "a stale launch-agent contract was reported ready" @@ -398,10 +529,7 @@ assert_contains "$DOCTOR_OUT" 'check herdr-server=ok:' "the running fixture was doctor --fix expect_code 0 "$DOCTOR_RC" "--fix did not repair launch-agent contract drift" assert_contains "$DOCTOR_OUT" 'check launchagent=ok:' "the repaired launch-agent contract was not confirmed" -assert_grep "<string>$CASE_BIN/herdr</string>" "$CASE_PLIST" "the repaired launch agent does not use the resolved herdr path" -assert_grep '<key>RunAtLoad</key>' "$CASE_PLIST" "the repaired launch agent does not start at login" -assert_grep '<key>KeepAlive</key>' "$CASE_PLIST" "the repaired launch agent is not kept alive" -assert_no_grep '/obsolete/bin/herdr' "$CASE_PLIST" "the obsolete herdr path survived repair" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" pass "a loaded and running launch agent must match the complete owned contract" # --- failed replacement cannot hide a stale loaded launch-agent contract ----- @@ -426,7 +554,7 @@ cat > "$CASE_PLIST" <<XML </dict> </plist> XML -write_loaded_contract /obsolete/bin/herdr 'runatload | inferred program' +write_loaded_contract /obsolete/bin/herdr 'keepalive | runatload | inferred program' "exec '/obsolete/bin/herdr' server --session 'fm-remote'" printf 'true\n' > "$CASE_HERDR_RUNNING" touch "$CASE_STATE/bootout-fail" doctor --fix @@ -466,7 +594,7 @@ doctor --fix expect_code 0 "$DOCTOR_RC" "--fix could not re-scope an existing launch agent" assert_contains "$DOCTOR_OUT" 'check launchagent-scope=ok: LimitLoadToSessionType=Aqua' \ "--fix did not re-scope the launch agent to Aqua" -assert_no_grep 'Background' "$CASE_PLIST" "the Background session scope survived the repair" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" pass "a launch agent outside the Aqua session scope is rewritten in place" # --- launchd start failures are reported and delayed readiness is awaited ---- @@ -492,6 +620,72 @@ assert_contains "$DOCTOR_OUT" 'check herdr-server=ok:' "delayed launchd startup assert_absent "$CASE_STATE/herdr-delay" "the readiness poll stopped before the delayed server became reachable" pass "launchd failures are reported and delayed server readiness is awaited" +# --- a running session served outside the Aqua login session is not ready --- + +new_case Darwin with-herdr gui +doctor --fix +expect_code 0 "$DOCTOR_RC" "the Aqua-owner fixture could not be initialized" +assert_contains "$DOCTOR_OUT" "check herdr-server=ok: session fm-remote is running in the Aqua login session (pid $AQUA_HOLDER_PID, launchd)" \ + "a launchd-born owner was not reported with its pid and birth" + +printf '%s\n' "$BACKGROUND_HOLDER_PID" > "$CASE_STATE/socket-owner" +printf 'background job\n' > "$CASE_STATE/user-loaded-$LABEL" +doctor +expect_code 1 "$DOCTOR_RC" "a label loaded in the user domain was reported Aqua-born" +assert_contains "$DOCTOR_OUT" "check herdr-server=fixable: session fm-remote is served by pid $BACKGROUND_HOLDER_PID born outside the Aqua login session (unknown)" \ + "the user-domain owner was not tagged fixable" +rm -f "$CASE_STATE/user-loaded-$LABEL" + +printf '%s\n' "$XPC_ZERO_HOLDER_PID" > "$CASE_STATE/socket-owner" +doctor +expect_code 1 "$DOCTOR_RC" "an XPC_SERVICE_NAME=0 owner was reported Aqua-born" +assert_contains "$DOCTOR_OUT" "check herdr-server=fixable: session fm-remote is served by pid $XPC_ZERO_HOLDER_PID born outside the Aqua login session (unknown)" \ + "the inherited XPC marker was not tagged fixable" + +printf '%s\n' "$WORKER_HOLDER_PID" > "$CASE_STATE/socket-owner" +doctor +expect_code 0 "$DOCTOR_RC" "the gui-only remote-job worker owner was not reported ready" +assert_contains "$DOCTOR_OUT" "check herdr-server=ok: session fm-remote is running in the Aqua login session (pid $WORKER_HOLDER_PID, worker)" \ + "the gui-only worker was not recognized" + +printf '%s\n' "$SSH_HOLDER_PID" > "$CASE_STATE/socket-owner" +: > "$CASE_LAUNCHCTL_LOG" +doctor +expect_code 1 "$DOCTOR_RC" "a session served by an SSH-born server was reported ready" +assert_contains "$DOCTOR_OUT" "check herdr-server=fixable: session fm-remote is served by pid $SSH_HOLDER_PID born outside the Aqua login session (ssh)" \ + "an SSH-born owner was not tagged fixable with its pid and birth" +assert_contains "$DOCTOR_OUT" 'cannot reach the login keychain' "the consequence of the foreign birth was not named" +assert_contains "$DOCTOR_OUT" 'action: herdr-server:' "the foreign-birth gap came with no operator action" +assert_contains "$DOCTOR_OUT" 'check launchagent-loaded=ok:' "a healthy loaded contract was blamed for the foreign server" +[ ! -s "$CASE_LAUNCHCTL_LOG" ] || assert_not_contains "$(cat "$CASE_LAUNCHCTL_LOG")" kickstart \ + "a read-only doctor run restarted the launch agent" +doctor --fix +expect_code 0 "$DOCTOR_RC" "--fix did not hand the session back to the launch agent" +assert_contains "$DOCTOR_OUT" 'fix herdr-server=applied:' "--fix did not report the takeover through the launch agent" +assert_grep "kickstart -k gui/$(id -u)/$LABEL" "$CASE_LAUNCHCTL_LOG" "the takeover did not go through launchd" +assert_contains "$DOCTOR_OUT" "check herdr-server=ok: session fm-remote is running in the Aqua login session (pid $AQUA_HOLDER_PID, launchd)" \ + "the launch-agent-owned server was not confirmed after the takeover" +assert_no_dangerous_calls "the takeover reached for auto-login, FileVault, or the keychain" + +# A RunAtLoad job also runs at bootstrap; hold that back so the takeover +# depends on the kickstart that is about to fail. +printf '%s\n' "$SSH_HOLDER_PID" > "$CASE_STATE/socket-owner" +touch "$CASE_STATE/kickstart-fail" "$CASE_STATE/bootstrap-does-not-start" +doctor --fix +expect_code 1 "$DOCTOR_RC" "a failed takeover was reported ready" +assert_contains "$DOCTOR_OUT" 'fix herdr-server=failed: launchctl kickstart' "the failed takeover was not reported" +assert_contains "$DOCTOR_OUT" "check herdr-server=fixable: session fm-remote is served by pid $SSH_HOLDER_PID" \ + "a still-foreign server was reported ready after a failed takeover" +rm -f "$CASE_STATE/kickstart-fail" "$CASE_STATE/bootstrap-does-not-start" + +rm -f "$CASE_STATE/socket-owner" +: > "$CASE_STATE/socket-owner" +doctor +expect_code 1 "$DOCTOR_RC" "a session with no provable owner was reported ready" +assert_contains "$DOCTOR_OUT" 'check herdr-server=fixable: session fm-remote is running but no herdr process can be shown to own its socket' \ + "an unprovable owner was not tagged fixable" +pass "a session served outside the Aqua login session is fixable and --fix retakes it through launchd" + # --- no GUI login session: every dependent gap stays human ------------------- new_case Darwin with-herdr no-gui @@ -511,6 +705,75 @@ assert_contains "$DOCTOR_OUT" 'error: this host is not ready for a remote second assert_no_dangerous_calls "the doctor tried to create a login session by force" pass "human gaps are reported with their operator step and never claimed as fixed" +# --- a non-zsh login shell is rendered with separate -l and -c -------------- + +new_case Darwin with-herdr gui /bin/bash +CASE_RESOLVE_DSCL=1 +doctor --fix +expect_code 0 "$DOCTOR_RC" "--fix left a bash-login-shell host unready" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" /bin/bash +pass "a bash Directory Services login shell is rendered with -l -c" + +new_case Darwin with-herdr gui +CASE_LOGIN_SHELL="$CASE_DIR/My Shell/fish&dev" +mkdir -p "$(dirname "$CASE_LOGIN_SHELL")" +printf '#!/bin/sh\nexit 0\n' > "$CASE_LOGIN_SHELL" +chmod +x "$CASE_LOGIN_SHELL" +CASE_RESOLVE_DSCL=1 +doctor --fix +expect_code 0 "$DOCTOR_RC" "--fix rejected a valid custom Directory Services shell" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" "$CASE_LOGIN_SHELL" +pass "custom Directory Services shell paths remain valid plist arguments" + +# --- shell resolution falls back to an executable environment shell, then sh - + +new_case Darwin with-herdr gui /bin/bash +CASE_RESOLVE_DSCL=1 +CASE_DSCL_FAIL=1 +CASE_ENV_SHELL=/bin/bash +doctor --fix +expect_code 0 "$DOCTOR_RC" "--fix rejected an executable SHELL fallback" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" /bin/bash + +new_case Darwin with-herdr gui /bin/sh +CASE_RESOLVE_DSCL=1 +CASE_DSCL_FAIL=1 +CASE_ENV_SHELL="$CASE_DIR/not-a-shell" +doctor --fix +expect_code 0 "$DOCTOR_RC" "--fix rejected the POSIX shell fallback" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" /bin/sh + +new_case Darwin with-herdr gui /bin/sh +CASE_RESOLVE_DSCL=1 +CASE_DSCL_HANG=1 +CASE_ENV_SHELL=/bin/sh +SECONDS=0 +doctor --fix +elapsed=$SECONDS +expect_code 0 "$DOCTOR_RC" "--fix did not fall back after a stalled Directory Services lookup" +[ "$elapsed" -lt 10 ] || fail "a stalled Directory Services lookup blocked doctor for ${elapsed}s" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" /bin/sh +pass "shell resolution bounds Directory Services and uses executable SHELL and POSIX fallbacks" + +# --- one shell resolution is shared by render, validation, and reporting ---- + +new_case Darwin with-herdr gui /bin/sh +CASE_RESOLVE_DSCL=1 +doctor --fix +expect_code 0 "$DOCTOR_RC" "initial repair did not install a healthy login-shell agent" +assert_herdr_launch_agent_contract "$CASE_PLIST" "$CASE_BIN/herdr" /bin/sh +: > "$CASE_LAUNCHCTL_LOG" +rm -f "$CASE_STATE/dscl-count" +CASE_SECOND_LOGIN_SHELL=/bin/bash +doctor --fix +expect_code 0 "$DOCTOR_RC" "repeated repair drifted when a second shell lookup would differ" +assert_contains "$DOCTOR_OUT" 'check launchagent=ok:' "the installed login-shell plist was reported as drifted" +assert_contains "$DOCTOR_OUT" 'check launchagent-loaded=ok:' "the loaded login-shell agent was reported as drifted" +[ "$(cat "$CASE_STATE/dscl-count")" = 1 ] || fail "doctor resolved the account login shell more than once" +assert_no_grep '^bootout\|^bootstrap\|^kickstart' "$CASE_LAUNCHCTL_LOG" \ + "repeated repair reloaded an already healthy login-shell agent" +pass "repeated repair reuses one resolved login shell and remains a no-op" + # --- linux has no launch agent, and --fix starts the server directly --------- new_case Linux with-herdr no-gui diff --git a/tests/fm-remote-herdr-guard.test.sh b/tests/fm-remote-herdr-guard.test.sh new file mode 100755 index 00000000000..721e1fdfc6a --- /dev/null +++ b/tests/fm-remote-herdr-guard.test.sh @@ -0,0 +1,343 @@ +#!/usr/bin/env bash +# tests/fm-remote-herdr-guard.test.sh - the fm-remote launch agent's guard. +# +# Drives the real bin/fm-remote-herdr-guard.sh (and the owner library it +# sources) against a fake herdr CLI, a fake lsof that names a real holder +# process as the session-socket owner, and real holder processes whose +# environment and ancestry carry the birth markers the guard reads. It pins +# the decision table: no server -> start; an Aqua-born owner -> leave it; an +# SSH-born or unprovable owner -> stop it, wait for the socket, start. Nothing +# here touches the runner's own herdr servers, launch agents, or login +# session, and no live harness guard applies: the verdict comes from process +# environment and ancestry, which are kernel facts rather than vendor output. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +command -v jq >/dev/null 2>&1 || { echo "skip: jq not found (the guard parses herdr's JSON, and jq is the holder process)"; exit 0; } +command -v mkfifo >/dev/null 2>&1 || { echo "skip: mkfifo not found (holder processes block on a fifo)"; exit 0; } + +TMP_ROOT=$(fm_test_tmproot fm-remote-herdr-guard) +mkdir -p "$TMP_ROOT" +TMP_ROOT=$(cd "$TMP_ROOT" && pwd -P) +HOLDER_PIDS=() +HOLDER_FD=5 +trap 'if [ "${#HOLDER_PIDS[@]}" -gt 0 ]; then kill "${HOLDER_PIDS[@]}" 2>/dev/null || true; fi; fm_test_cleanup || true' EXIT + +GUARD="$ROOT/bin/fm-remote-herdr-guard.sh" +JQ=$(command -v jq) +SESSION=fm-remote + +# The guard must see only the fixture and the system tools it really needs, +# so a case can also present a host with NO lsof. +TOOLS="$TMP_ROOT/tools" +mkdir -p "$TOOLS" +for tool in ps awk sed grep tr dirname basename sleep cat cp rm env bash sh id head; do + real=$(command -v "$tool") || fail "test host lacks $tool" + ln -sf "$real" "$TOOLS/$tool" +done +ln -sf "$JQ" "$TOOLS/jq" +FAKE="$TMP_ROOT/fake" +mkdir -p "$FAKE" +cat > "$FAKE/lsof" <<'SH' +#!/usr/bin/env bash +# Prints the -F pn shape for every pid listed in the owner file. +[ -f "$FM_FAKE_SOCKET_OWNER" ] || exit 0 +while IFS= read -r pid; do + [ -n "$pid" ] || continue + printf 'p%s\n' "$pid" + printf 'n%s\n' "$FM_FAKE_HERDR_SOCKET" +done < "$FM_FAKE_SOCKET_OWNER" +SH +cat > "$FAKE/launchctl" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" = print ] || exit 0 +domain=${2:-} +scope=${domain%%/*} +label=${domain#*/} +label=${label#*/} +state="$FM_FAKE_STATE/launchctl-$scope-$label" +[ -f "$state" ] || exit 113 +cat "$state" +SH +cat > "$FAKE/herdr" <<'SH' +#!/usr/bin/env bash +set -u +printf '%s\n' "$*" >> "$FM_FAKE_HERDR_LOG" +running=$(cat "$FM_FAKE_HERDR_RUNNING" 2>/dev/null || printf 'false') +case "$*" in + "status --json --session "*) + if [ -f "$FM_FAKE_STATE/release-after" ]; then + left=$(cat "$FM_FAKE_STATE/release-after") + if [ "$left" -gt 0 ]; then + printf '%s\n' "$((left - 1))" > "$FM_FAKE_STATE/release-after" + else + rm -f "$FM_FAKE_STATE/release-after" + printf 'false\n' > "$FM_FAKE_HERDR_RUNNING" + running=false + fi + fi + printf '{"server":{"running":%s,"socket":"%s","version":"0.9.0"},"client":{"version":"0.9.0"}}\n' \ + "$running" "$FM_FAKE_HERDR_SOCKET" + ;; + "server stop --session "*) + if [ -f "$FM_FAKE_STATE/stop-ignored" ]; then + exit 0 + elif [ -f "$FM_FAKE_STATE/stop-releases-after" ]; then + cp "$FM_FAKE_STATE/stop-releases-after" "$FM_FAKE_STATE/release-after" + else + printf 'false\n' > "$FM_FAKE_HERDR_RUNNING" + fi + ;; + "server --session "*) + printf 'pid=%s session=%s\n' "$$" "${3:-}" > "$FM_FAKE_STATE/started" + ;; +esac +exit 0 +SH +chmod +x "$FAKE/lsof" "$FAKE/launchctl" "$FAKE/herdr" +cp "$FAKE/lsof" "$TMP_ROOT/lsof.fake" + +# hold <marker-env...> -> HOLDER_PID: a real non-platform process (jq blocked +# on a fifo this test keeps open) whose environment is exactly the markers. +hold() { + local fifo="$TMP_ROOT/holder-$HOLDER_FD.fifo" + rm -f "$fifo" + mkfifo "$fifo" + # Open read-write so this never blocks on the reader; the holder sees EOF + # only when the descriptor closes at exit. + eval "exec ${HOLDER_FD}<>\"\$fifo\"" + env -i "$@" "$JQ" . "$fifo" & + HOLDER_PID=$! + HOLDER_PIDS+=("$HOLDER_PID") + HOLDER_FD=$((HOLDER_FD + 1)) +} + +# hold_under <argv0> <arg...> -- : a marker-free holder whose PARENT process +# carries the given argv[0] and arguments (the ancestry the guard inspects). +hold_under() { + local argv0=$1 fifo="$TMP_ROOT/holder-$HOLDER_FD.fifo" pidfile="$TMP_ROOT/holder-$HOLDER_FD.pid" + shift + rm -f "$fifo" "$pidfile" + mkfifo "$fifo" + eval "exec ${HOLDER_FD}<>\"\$fifo\"" + ( export FM_HOLDER_JQ="$JQ" FM_HOLDER_FIFO="$fifo" FM_HOLDER_PIDFILE="$pidfile" + exec -a "$argv0" bash -c 'env -i FM_HOLDER=1 "$FM_HOLDER_JQ" . "$FM_HOLDER_FIFO" & printf "%s\n" "$!" > "$FM_HOLDER_PIDFILE"; wait' "$@" ) & + HOLDER_PIDS+=("$!") + HOLDER_FD=$((HOLDER_FD + 1)) + local i=0 + while [ ! -s "$pidfile" ] && [ "$i" -lt 100 ]; do sleep 0.05; i=$((i + 1)); done + [ -s "$pidfile" ] || fail "holder under $argv0 did not report its pid" + HOLDER_PID=$(cat "$pidfile") + HOLDER_PIDS+=("$HOLDER_PID") +} + +CASE_N=0 +new_case() { # [running|stopped] + CASE_N=$((CASE_N + 1)) + CASE_STATE="$TMP_ROOT/case$CASE_N" + mkdir -p "$CASE_STATE" + CASE_LOG="$CASE_STATE/herdr.log" + : > "$CASE_LOG" + CASE_RUNNING="$CASE_STATE/running" + printf '%s\n' "$([ "${1:-running}" = running ] && printf true || printf false)" > "$CASE_RUNNING" + CASE_OWNER="$CASE_STATE/socket-owner" + CASE_SOCKET="$CASE_STATE/herdr.sock" + CASE_PATH="$FAKE:$TOOLS" +} + +load_job() { # <gui|user> <label> [pid] + if [ -n "${3:-}" ]; then + printf 'pid = %s\n' "$3" > "$CASE_STATE/launchctl-$1-$2" + else + printf 'state = running\n' > "$CASE_STATE/launchctl-$1-$2" + fi +} + +guard() { # [extra env assignments...] + set +e + GUARD_OUT=$( + env -i PATH="$CASE_PATH" HOME="$TMP_ROOT" \ + FM_FAKE_STATE="$CASE_STATE" FM_FAKE_HERDR_LOG="$CASE_LOG" FM_FAKE_HERDR_RUNNING="$CASE_RUNNING" \ + FM_FAKE_SOCKET_OWNER="$CASE_OWNER" FM_FAKE_HERDR_SOCKET="$CASE_SOCKET" \ + FM_HOLDER_JQ="$JQ" \ + FM_REMOTE_HERDR_GUARD_STOP_WAIT_TENTHS=8 \ + "$@" "$GUARD" "$FAKE/herdr" "$SESSION" 2>&1 + ) + GUARD_RC=$? + set -e +} + +herdr_calls() { cat "$CASE_LOG"; } +assert_started() { # <msg> + [ -f "$CASE_STATE/started" ] || fail "$1" + assert_grep "session=$SESSION" "$CASE_STATE/started" "the server was started for the wrong session" +} +assert_not_started() { assert_absent "$CASE_STATE/started" "$1"; } +assert_stop_before_start() { + local calls stop_line start_line + calls=$(herdr_calls) + stop_line=$(printf '%s\n' "$calls" | grep -n "^server stop --session $SESSION$" | head -1 | cut -d: -f1) + start_line=$(printf '%s\n' "$calls" | grep -n "^server --session $SESSION$" | head -1 | cut -d: -f1) + [ -n "$stop_line" ] || fail "the guard never asked the foreign server to stop" + [ -n "$start_line" ] || fail "the guard never started its own server" + [ "$stop_line" -lt "$start_line" ] || fail "the guard started its server before stopping the foreign one" +} + +# Prove the holder construction on this host: the environment of a jq holder +# must be readable, or every marker case would be vacuous. +hold FM_PROBE_MARKER=1 +PROBE_PID=$HOLDER_PID +sleep 0.2 +# shellcheck source=bin/fm-remote-herdr-owner-lib.sh +. "$ROOT/bin/fm-remote-herdr-owner-lib.sh" +probe_env=$(fm_remote_herdr_process_env "$PROBE_PID") +case "$probe_env" in + *FM_PROBE_MARKER=1*) ;; + *) fail "this host does not expose a holder's environment (macOS hides platform-binary environments; jq at $JQ must be a non-platform binary): $probe_env" ;; +esac +pass "holder processes expose their environment to the owner library" + +# --- no server: the guard becomes the server --------------------------------- + +new_case stopped +guard +expect_code 0 "$GUARD_RC" "the guard failed when no server owned the session" +assert_started "the guard did not start the server when none owned the session" +assert_not_contains "$(herdr_calls)" 'server stop' "the guard stopped something when no server owned the session" +assert_contains "$GUARD_OUT" "no server owns session $SESSION" "the guard did not report the empty session" +pass "an empty session is started inside the launch agent" + +# --- an Aqua-born owner is left alone ---------------------------------------- + +hold XPC_SERVICE_NAME=dev.firstmate.herdr.fm-remote +LAUNCHD_PID=$HOLDER_PID +hold XPC_SERVICE_NAME=dev.firstmate.herdr.fm-remote +BACKGROUND_PID=$HOLDER_PID +hold XPC_SERVICE_NAME=0 +XPC_ZERO_PID=$HOLDER_PID +hold FM_REMOTE_JOB_ACTIVE=1 +WORKER_PID=$HOLDER_PID +hold SSH_CONNECTION='100.102.217.78 51234 100.100.1.2 22' SSH_CLIENT='100.102.217.78 51234 22' +SSH_PID=$HOLDER_PID +hold FM_NOTHING_TO_SEE=1 +UNMARKED_PID=$HOLDER_PID +hold_under herdr --session "$SESSION" remote-client-bridge +BRIDGE_CHILD_PID=$HOLDER_PID +hold_under 'sshd-session:' kunchen@notty +SSHD_CHILD_PID=$HOLDER_PID +sleep 0.3 + +new_case running +printf '%s\n' "$LAUNCHD_PID" > "$CASE_OWNER" +load_job gui dev.firstmate.herdr.fm-remote "$LAUNCHD_PID" +guard +expect_code 0 "$GUARD_RC" "the guard did not exit 0 for a gui-domain launchd owner" +assert_not_started "the guard started a second server over a gui-domain launchd owner" +assert_not_contains "$(herdr_calls)" 'server stop' "the guard stopped a gui-domain launchd owner" +assert_contains "$GUARD_OUT" "pid $LAUNCHD_PID born in the Aqua login session (launchd)" \ + "the guard did not name the launchd owner" + +new_case running +printf '%s\n' "$WORKER_PID" > "$CASE_OWNER" +load_job gui dev.firstmate.remote-job +guard +expect_code 0 "$GUARD_RC" "the guard did not exit 0 for the gui-domain worker owner" +assert_not_started "the guard started a second server over a gui-domain worker owner" +assert_not_contains "$(herdr_calls)" 'server stop' "the guard stopped a gui-domain worker owner" +assert_contains "$GUARD_OUT" "pid $WORKER_PID born in the Aqua login session (worker)" \ + "the guard did not name the worker owner" +pass "launchd and worker markers require gui-domain launchctl proof" + +# --- a foreign owner is stopped, then the guard becomes the server ----------- + +new_case running +printf '%s\n' "$BACKGROUND_PID" > "$CASE_OWNER" +load_job gui dev.firstmate.herdr.fm-remote +load_job user dev.firstmate.herdr.fm-remote +guard +expect_code 0 "$GUARD_RC" "the guard failed to take over a label also loaded in the user domain" +assert_stop_before_start +assert_contains "$GUARD_OUT" "pid $BACKGROUND_PID born outside the Aqua login session (unknown)" \ + "a user-domain label was trusted as Aqua" + +new_case running +printf '%s\n' "$XPC_ZERO_PID" > "$CASE_OWNER" +guard +expect_code 0 "$GUARD_RC" "the guard failed to take over an XPC_SERVICE_NAME=0 owner" +assert_stop_before_start +assert_contains "$GUARD_OUT" "pid $XPC_ZERO_PID born outside the Aqua login session (unknown)" \ + "XPC_SERVICE_NAME=0 was trusted as Aqua" + +for foreign in "ssh $SSH_PID" "ssh $BRIDGE_CHILD_PID" "ssh $SSHD_CHILD_PID" "unknown $UNMARKED_PID"; do + new_case running + printf '%s\n' "${foreign#* }" > "$CASE_OWNER" + guard + expect_code 0 "$GUARD_RC" "the guard failed to take over from a ${foreign%% *} owner (pid ${foreign#* })" + assert_stop_before_start + assert_started "the guard did not start its own server after the ${foreign%% *} owner released the socket" + assert_contains "$GUARD_OUT" "pid ${foreign#* } born outside the Aqua login session (${foreign%% *})" \ + "the guard did not name the foreign owner and its birth" +done +pass "background, inherited-XPC, SSH-born, SSH-descended, and unprovable owners are taken over" + +# --- an owner nobody can prove is treated as foreign ------------------------- + +new_case running +guard +expect_code 0 "$GUARD_RC" "the guard failed when lsof listed no owner" +assert_contains "$GUARD_OUT" 'no herdr process could be proven to own' "the guard did not report the unprovable owner" +assert_stop_before_start +pass "a running session with no provable owner is taken over rather than trusted" + +new_case running +printf '%s\n' "$SSH_PID" > "$CASE_OWNER" +rm -f "$FAKE/lsof" +guard +cp "$TMP_ROOT/lsof.fake" "$FAKE/lsof" +chmod +x "$FAKE/lsof" +expect_code 0 "$GUARD_RC" "the guard failed when lsof was absent" +assert_contains "$GUARD_OUT" 'lsof does not resolve' "the guard did not report the missing lsof" +assert_stop_before_start +pass "a host without lsof cannot prove an Aqua birth, so the session is taken over" + +# --- a foreign owner that keeps the socket makes the guard fail for a retry -- + +new_case running +printf '%s\n' "$SSH_PID" > "$CASE_OWNER" +touch "$CASE_STATE/stop-ignored" +guard +expect_code 1 "$GUARD_RC" "the guard did not exit 1 when the foreign server kept its socket" +assert_not_started "the guard started a server while the foreign one still held the socket" +assert_contains "$(herdr_calls)" "server stop --session $SESSION" "the guard never asked the foreign server to stop" +assert_contains "$GUARD_OUT" 'did not release its socket within 8 tenths' "the guard did not report the bounded wait" +pass "a foreign server that never releases the socket yields exit 1 so launchd retries" + +# --- the release wait is polled, not assumed --------------------------------- + +new_case running +printf '%s\n' "$SSH_PID" > "$CASE_OWNER" +printf '3\n' > "$CASE_STATE/stop-releases-after" +guard +expect_code 0 "$GUARD_RC" "the guard gave up on a server that released its socket after a few polls" +assert_started "the guard did not start after the delayed release" +assert_contains "$GUARD_OUT" 'released its socket after' "the guard did not report the observed release" +[ "$(grep -c "^status --json --session $SESSION$" "$CASE_LOG")" -ge 4 ] \ + || fail "the guard did not keep polling the session status until the socket was released" +pass "the guard starts as soon as the foreign server releases the socket" + +# --- usage errors never touch a server --------------------------------------- + +new_case running +set +e +env -i PATH="$CASE_PATH" "$GUARD" "$FAKE/herdr" >/dev/null 2>&1 +rc=$? +set -e +expect_code 2 "$rc" "a missing session argument was not a usage error" +set +e +env -i PATH="$CASE_PATH" "$GUARD" "$TMP_ROOT/no-such-herdr" "$SESSION" >/dev/null 2>&1 +rc=$? +set -e +expect_code 1 "$rc" "a non-executable herdr path was not refused" +[ ! -s "$CASE_LOG" ] || fail "a refused invocation still called herdr" +pass "argument errors are refused before any herdr call" diff --git a/tests/fm-remote-secondmate-lifecycle-e2e.test.sh b/tests/fm-remote-secondmate-lifecycle-e2e.test.sh index 1c848ab54a0..c669485f3db 100755 --- a/tests/fm-remote-secondmate-lifecycle-e2e.test.sh +++ b/tests/fm-remote-secondmate-lifecycle-e2e.test.sh @@ -6,6 +6,8 @@ set -u . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" # shellcheck source=tests/remote-herdr-fixture.sh . "$(dirname "${BASH_SOURCE[0]}")/remote-herdr-fixture.sh" +# shellcheck source=tests/herdr-client-pair-fixture.sh +. "$(dirname "${BASH_SOURCE[0]}")/herdr-client-pair-fixture.sh" command -v jq >/dev/null 2>&1 || { echo "skip: jq not found"; exit 0; } ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P) @@ -489,7 +491,10 @@ projects_snapshot() { # <dir> } mkdir -p "$TMP_ROOT/seed-parent/projects" fm_git_init_commit "$TMP_ROOT/seed-parent/projects/resident" -git init -q --bare "$TMP_ROOT/beta.git" +# Pin the bare origin's initial branch to the one fm_git_init_commit creates. +# Left to init.defaultBranch, its HEAD names a branch the push never creates on +# a host that still defaults to master, and cloning it checks out nothing. +git init -q --bare -b main "$TMP_ROOT/beta.git" fm_git_init_commit "$TMP_ROOT/beta-src" git -C "$TMP_ROOT/beta-src" remote add origin "file://$TMP_ROOT/beta.git" git -C "$TMP_ROOT/beta-src" push -q -u origin HEAD @@ -1127,6 +1132,22 @@ launches_after_repair=$(grep -c '^tab create' "$HERDR_LOG" || true) || fail "the endpoint was not probed successfully after readiness repair" pass "startup repairs remote readiness before probing without relaunching" +# --- a stale herdr client shadowing the one the server accepts -------------- +# The remote host's job PATH can resolve an older self-updated herdr ahead of +# the one its running server accepts; the server then refuses every command +# from it with protocol_mismatch. The host-local state read must still reach +# the live endpoint through the accepted client. +make_herdr_client_pair "$TMP_ROOT/client-pair" 0.7.1 14 0.7.5 16 +export FM_HERDR_PAIR_DIR="$TMP_ROOT/client-pair" +SHADOWED_STATE=$(FM_HOME="$REMOTE_HOME" FM_ROOT_OVERRIDE="$REMOTE_ROOT" \ + PATH="$TMP_ROOT/client-pair/stale:$REMOTE_ROOT/bin:$TMP_ROOT/client-pair/tools:/usr/bin:/bin" \ + "$REMOTE_ROOT/bin/fm-remote-secondmate-control.sh" state ios 2>"$TMP_ROOT/shadowed-state.err") +[ "$SHADOWED_STATE" = alive ] \ + || fail "a live endpoint behind a stale shadowing client must still read alive, got: $SHADOWED_STATE ($(cat "$TMP_ROOT/shadowed-state.err"))" +assert_contains "$(cat "$TMP_ROOT/client-pair/stale.log")" 'pane get' "the stale client was not the one the job PATH resolved first" +unset FM_HERDR_PAIR_DIR +pass "the host-local state read steps around a stale shadowing herdr client" + remote_route_meta="$REMOTE_HOME/state/parent-route/ios.meta" cp "$remote_route_meta" "$TMP_ROOT/remote-ios-before-liveness-legacy.meta" cp "$PARENT/state/ios.meta" "$TMP_ROOT/parent-ios-before-liveness-legacy.meta" diff --git a/tests/fm-secondmate-restart.test.sh b/tests/fm-secondmate-restart.test.sh index 8f764de2505..fde665c1020 100755 --- a/tests/fm-secondmate-restart.test.sh +++ b/tests/fm-secondmate-restart.test.sh @@ -550,6 +550,34 @@ test_local_restart_uses_the_home_pin_and_reports_what_ran() { pass "T8 a local restart re-resolves this home's pin and reports the runtime that came up" } +test_native_ultra_restart_keeps_local_and_remote_profiles() { + local dir out rc relaunch_line + dir=$(new_case native-local) + add_local_mate "$dir" sm1 + arm_answer "$dir" sm1 + printf 'pi codex-native/gpt-6-astra ultra\n' > "$dir/home/config/secondmate-harness" + printf 'pi' > "$dir/fake/becomes" + printf '#!/usr/bin/env bash\nprintf "Options: --tui-mode\\n"\n' > "$dir/fakebin/pi" + chmod +x "$dir/fakebin/pi" + out=$(run_restart "$dir" sm1); rc=$? + expect_code 0 "$rc" "native local restart failed: $out" + assert_contains "$out" "restarted: sm1 (pi)" "native local restart did not complete" + assert_contains "$(cat "$dir/home/state/sm1.meta")" "effort=ultra" "local restart dropped native effort" + assert_contains "$(cat "$dir/fake/literal")" "--codex-effort 'ultra'" "local restart dropped native launch flag" + + dir=$(new_case native-remote) + setup_remote_case "$dir" sm2 ok + export FM_FAKE_ANSWER_STATUS="$dir/home/state/sm2.status" + printf 'pi-signed codex-native/gpt-6-astra ultra\n' > "$dir/home/config/secondmate-harness" + out=$(run_restart "$dir" sm2); rc=$? + unset FM_FAKE_ANSWER_STATUS + expect_code 0 "$rc" "native remote restart failed: $out" + relaunch_line=$(grep '^fm-remote-secondmate-control.sh relaunch' "$dir/ssh.log" | head -1) + [ "$relaunch_line" = "fm-remote-secondmate-control.sh relaunch sm2 pi-signed codex-native/gpt-6-astra ultra" ] \ + || fail "remote restart dropped native profile: $relaunch_line" + pass "native Ultra survives local restart and the remote restart transport" +} + # --- T9: an unrelated concurrent reply cannot release the persist gate ------- test_concurrent_reply_cannot_release_persist_gate() { local dir out rc state corr rec @@ -812,6 +840,7 @@ test_unprovable_runtime_falls_back test_unknown_mate_is_accounted_for test_refused_restart_falls_back_without_claiming_a_reload test_local_restart_uses_the_home_pin_and_reports_what_ran +test_native_ultra_restart_keeps_local_and_remote_profiles test_remote_mate_restarts_over_the_transport_hop test_unreachable_host_is_reported_unknown test_concurrent_reply_cannot_release_persist_gate diff --git a/tests/fm-send-remote-delivery.test.sh b/tests/fm-send-remote-delivery.test.sh index ee3736b014c..fb52e7b21e8 100755 --- a/tests/fm-send-remote-delivery.test.sh +++ b/tests/fm-send-remote-delivery.test.sh @@ -299,8 +299,12 @@ test_remote_rerun_is_idempotent() { assert_contains "$err" "$expected_cmd" \ "double transport loss must print the exact correlation-reusing resend command" + # A delivered record escalated for a missed report is genuinely stale: the + # request reached the mate, so the same correlation must never be resent. + fm_pending_reply_set "$pend" delivered_epoch 4242 \ + || fail "could not mark the fixture expectation delivered" fm_pending_reply_set "$pend" phase escalated \ - || fail "could not advance the ambiguous expectation to the escalated fixture phase" + || fail "could not advance the delivered expectation to the escalated fixture phase" ssh_before=$(cat "$ssh_log.count") rc=0 send_env "$fb" "$home" "$ssh_log" FM_PENDING_REPLY_EXISTING_CORR="$corr" \ @@ -314,8 +318,13 @@ test_remote_rerun_is_idempotent() { || fail "a stale explicit correlation minted a replacement expectation" [ "$(find "$rhome/state/parent-route/rsm.inbox" -name '*.msg' | wc -l | tr -d ' ')" = 1 ] \ || fail "a stale explicit correlation created another remote record" - fm_pending_reply_set "$pend" phase delivery_unknown \ - || fail "could not restore the ambiguous expectation for the supported resend" + # The watcher escalates an unknown delivery as soon as it sees it, so the + # printed resend routinely meets an escalated but still undelivered record; + # that record stays the owner's to resend under the same correlation. + fm_pending_reply_set "$pend" delivered_epoch "" \ + || fail "could not restore the undelivered fixture expectation" + [ "$(fm_pending_reply_get "$pend" phase)" = escalated ] \ + || fail "the undelivered fixture expectation must remain escalated before the resend" resend_cmd=$(tail -1 "$dir/err") rc=0 diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index aecb83eae2b..0c86e17a378 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -237,6 +237,8 @@ install_autoarm_scripts() { cat > "$dir/bin/fm-watch-arm.sh" <<'SH' #!/usr/bin/env bash echo "$$" >> "$FM_HOME/state/arm-ran" +printf 'pending:downtime:fixture-generation\n' > "$FM_HOME/state/.watcher-down" +touch "$FM_HOME/state/.last-watcher-beat" printf 'watcher: started pid=%s (beacon fresh)\n' "$$" printf 'stale: fixture-win actionable\n' exit 0 diff --git a/tests/fm-spawn-dispatch-profile.test.sh b/tests/fm-spawn-dispatch-profile.test.sh index 9126ca0dc49..29541f6c513 100755 --- a/tests/fm-spawn-dispatch-profile.test.sh +++ b/tests/fm-spawn-dispatch-profile.test.sh @@ -686,6 +686,76 @@ test_opencode_threads_model_and_ignores_effort_axis() { pass "opencode receives --model and omits the unsupported effort axis" } +test_native_effort_validator_keeps_axes_separate() { + local harness + for harness in pi pi-signed; do + "$ROOT/bin/fm-harness.sh" validate-native-effort "$harness" codex-native/gpt-6-astra ultra \ + || fail "native validator refused supported harness $harness" + done + if "$ROOT/bin/fm-harness.sh" validate-native-effort 'pi:codex-native/forged' '' ultra 2>/dev/null; then + fail "native validator accepted a model prefix embedded in the harness axis" + fi + pass "native effort validator checks harness and model as separate axes" +} + +test_native_pi_ultra_is_explicit_and_model_scoped() { + local rec id out launch harness mode native_profile model + for harness in pi pi-signed; do + for mode in no-mistakes direct-PR; do + id="ultra-$harness-$mode" + rec=$(make_spawn_case "$id" "$harness" "$id") + read_case_record "$rec" + fm_test_spawn_brief "$HOME_DIR" "$id" \ + "$(printf 'brief for %s\n<!-- firstmate-task-branch=fm/%s -->\nDelivery contract: mode=%s' "$id" "$id" "$mode")" + out=$(run_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR" \ + --harness "$harness" --model codex-native/gpt-6-astra --effort ultra --mode "$mode" --yolo off) + expect_code 0 "$?" "native Ultra spawn failed: $out" + assert_meta_profile "$HOME_DIR/state/$id.meta" "$harness" codex-native/gpt-6-astra ultra + launch=$(cat "$LAUNCH_LOG") + assert_contains "$launch" "--model 'codex-native/gpt-6-astra' --codex-effort 'ultra'" "native Ultra flag missing" + assert_not_contains "$launch" "--thinking" "native Ultra was converted into Pi thinking" + assert_not_contains "$launch" "'max'" "native Ultra was aliased to max" + done + done + for native_profile in 'claude:codex-native/gpt-6-astra' 'codex:codex-native/gpt-6-astra' 'pi:openai-codex/gpt-6-astra' 'pi:default' 'pi:codex-native/'; do + harness=${native_profile%%:*}; model=${native_profile#*:}; id="ultra-refused-$RANDOM" + rec=$(make_spawn_case "$id" "$harness" "$id") + read_case_record "$rec" + out=$(run_ship_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR" \ + --harness "$harness" --model "$model" --effort ultra 2>&1) + expect_code 1 "$?" "unsupported Ultra profile should refuse: $native_profile" + assert_contains "$out" "ultra effort requires pi or pi-signed" "native-only refusal missing" + [ ! -e "$HOME_DIR/state/$id.meta" ] || fail "unsupported Ultra published metadata" + [ ! -e "$HOME_DIR/state/$id.busy-gen" ] || fail "unsupported Ultra provisioned lifecycle wiring" + [ ! -s "$LAUNCH_LOG" ] || fail "unsupported Ultra launched an agent" + done + id=ultra-raw-refused + rec=$(make_spawn_case "$id" pi "$id") + read_case_record "$rec" + out=$(run_ship_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR" \ + 'pi --offline' --model codex-native/gpt-6-astra --effort ultra 2>&1) + expect_code 1 "$?" "raw launch silently omitted the native Ultra flag" + [ ! -e "$HOME_DIR/state/$id.meta" ] || fail "raw Ultra launch published metadata" + assert_contains "$out" "canonical --harness pi or pi-signed" "raw launch refusal was not actionable" + pass "Ultra is explicit for native Pi and Pi-signed, including direct-PR, and refuses unsupported profiles before provisioning" +} + +test_batch_preserves_native_ultra() { + local rec id1=ultra-batch-a id2=ultra-batch-b out launch + rec=$(make_spawn_case ultra-batch pi "$id1" "$id2") + read_case_record "$rec" + enable_dispatch_profile "$HOME_DIR" + out=$(run_ship_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" \ + "$id1=$PROJ_DIR" "$id2=$PROJ_DIR" --harness pi --model codex-native/gpt-6-astra --effort ultra) + expect_code 0 "$?" "native Ultra batch failed: $out" + assert_meta_profile "$HOME_DIR/state/$id1.meta" pi codex-native/gpt-6-astra ultra + assert_meta_profile "$HOME_DIR/state/$id2.meta" pi codex-native/gpt-6-astra ultra + launch=$(cat "$LAUNCH_LOG") + assert_contains "$launch" "--codex-effort 'ultra'" "batch dropped native effort" + assert_not_contains "$launch" "--thinking 'ultra'" "batch passed an invalid Pi level" + pass "batch dispatch preserves native Ultra in metadata and launch flags" +} + test_pi_threads_model_and_max_effort() { local rec id out status launch id='profile-pi-z8' @@ -1479,6 +1549,8 @@ SH # An authored role heading must neither suppress nor duplicate the current # worker contract; the launch section is its single, superseding owner. assert_grep 'follow this brief instead of that supervisor contract' "$prompt" "$kind command did not deliver the role correction" + assert_grep 'report to firstmate; do not adopt the supervisor identity' "$prompt" "$kind command lost the worker reporting boundary" + assert_grep 'or address the captain' "$prompt" "$kind command lost the worker address exception" assert_grep 'brief for' "$prompt" "$kind command lost the task" [ "$(grep -c '^# Current worker role contract$' "$prompt")" -eq 1 ] || fail "$brief_kind $kind duplicated the delivered worker contract" @@ -1522,6 +1594,9 @@ test_cursor_threads_model_workspace_and_omits_effort_axis test_cursor_refuses_model_absent_from_live_catalog test_cursor_failed_catalog_probe_does_not_block_spawn test_opencode_threads_model_and_ignores_effort_axis +test_native_effort_validator_keeps_axes_separate +test_native_pi_ultra_is_explicit_and_model_scoped +test_batch_preserves_native_ultra test_pi_threads_model_and_max_effort test_pi_tui_mode_probe_is_safe_for_old_and_new_pi test_pi_signed_threads_shared_pi_profile_and_preserves_identity diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index a9177e1620c..05abc0e51e0 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -2489,14 +2489,25 @@ test_content_in_default_fallback_allows() { # the same net change has independently landed on origin/main via a squash commit. wt_commit_file "$case_dir" feature.txt hello "add feature" land_on_origin_main "$case_dir" feature.txt hello + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" > "${FM_TEST_TREEHOUSE_LOG:?}" +exit 0 +SH + chmod +x "$case_dir/fakebin/treehouse" set +e - run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + FM_TEST_TREEHOUSE_LOG="$case_dir/treehouse.log" \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" rc=$? set -e expect_code 0 "$rc" "content-landed: teardown should succeed when content is already in the default branch" ! grep -q REFUSED "$case_dir/stderr" || fail "content-landed: teardown printed a REFUSED line" + assert_present "$case_dir/treehouse.log" \ + "content-landed: teardown never reached destructive worktree cleanup" + assert_absent "$case_dir/state/task-x1.meta" \ + "content-landed: teardown left task metadata after destructive cleanup" pass "worktree whose content already landed in the default branch is torn down (content fallback)" } diff --git a/tests/fm-test-fixtures.test.sh b/tests/fm-test-fixtures.test.sh index 1ee5baa3a77..9176771dd5d 100755 --- a/tests/fm-test-fixtures.test.sh +++ b/tests/fm-test-fixtures.test.sh @@ -1,11 +1,11 @@ #!/usr/bin/env bash -# Behavior tests for tests/fixtures.sh fake-toolchain and spawn-world builders. +# Behavior tests for tests/lib.sh primitives and tests/fixtures.sh builders. # -# These cases drive the builders as a test would: they write stubs into a -# fakebin and exec those stubs. Assertions are on the binaries' observable -# output, exit status, and files they create - never on fixtures.sh source -# text. Migrated spawn suites cover fm_test_run_spawn through the real -# fm-spawn.sh; this file pins the stubs those suites now share. +# Cases call shared primitives directly or write stubs into a fakebin and exec +# them as a test would. Assertions are on observable output, exit status, and +# filesystem effects - never on helper source text. Migrated spawn suites cover +# fm_test_run_spawn through the real fm-spawn.sh; this file pins the shared +# primitives and stubs those suites use. set -u # shellcheck source=tests/fixtures.sh @@ -13,6 +13,21 @@ set -u TMP_ROOT=$(fm_test_tmproot fm-test-fixtures) +test_touch_epoch_preserves_repeated_dst_hour() { + local TZ=Europe/Paris epoch path actual + export TZ + for epoch in 1761438600 1761442200; do + fm_touch_epoch "$epoch" "$TMP_ROOT/epoch-one" "$TMP_ROOT/epoch two" + for path in "$TMP_ROOT/epoch-one" "$TMP_ROOT/epoch two"; do + actual=$(stat -c %Y "$path" 2>/dev/null || stat -f %m "$path" 2>/dev/null) \ + || fail "could not read fixture mtime for $path" + [ "$actual" = "$epoch" ] \ + || fail "fm_touch_epoch should preserve epoch $epoch, got $actual" + done + done + pass "fm_touch_epoch preserves both epochs in the repeated DST hour" +} + test_no_mistakes_version_constant() { local fakebin out fakebin=$(fm_fakebin "$TMP_ROOT/nm") @@ -124,6 +139,7 @@ test_spawn_home_layout() { pass "spawn-home layout writes harness pin, beat, and brief" } +test_touch_epoch_preserves_repeated_dst_hour test_no_mistakes_version_constant test_no_mistakes_init_doctor_markers test_fake_gh_and_gh_axi diff --git a/tests/fm-test-isolation-proof.test.sh b/tests/fm-test-isolation-proof.test.sh index 08982149ff7..dc39667906b 100755 --- a/tests/fm-test-isolation-proof.test.sh +++ b/tests/fm-test-isolation-proof.test.sh @@ -270,6 +270,24 @@ test_parallel_shards_consume_the_proven_set() { pass "parallel shards consume the proven-isolated set only" } +# A fixture repository names its branch, so a suite that resolves `main` gets +# the same answer wherever it runs. Without the pinned initial branch this +# fails on any host whose init.defaultBranch is still master. +test_fixture_repo_branch_is_pinned() { + local root dir config branch + root=$(fm_test_tmproot fm-fixture-branch-pin) || fail "could not create a fixture root" + dir="$root/repo" + config="$root/gitconfig" + printf '[init]\n\tdefaultBranch = master\n' > "$config" + GIT_CONFIG_GLOBAL="$config" fm_git_init_commit "$dir" + branch=$(git -C "$dir" rev-parse --abbrev-ref HEAD) + [ "$branch" = main ] \ + || fail "fixture repo branch follows init.defaultBranch instead of main: $branch" + git -C "$dir" rev-parse main >/dev/null 2>&1 \ + || fail "fixture repo cannot resolve main" + pass "fixture repositories pin their initial branch to main" +} + test_unknown_pool_is_refused test_family_pool_json_identifies_admission test_list_candidates_nonempty_and_stable diff --git a/tests/fm-turnend-guard.test.sh b/tests/fm-turnend-guard.test.sh index ae2970f0ff0..c577bb6dc3f 100755 --- a/tests/fm-turnend-guard.test.sh +++ b/tests/fm-turnend-guard.test.sh @@ -2070,7 +2070,7 @@ test_hook_away_daemon_allows_beacon_within_poll_derived_grace() { # grace (max(300, FM_POLL + 60) = 660 at FM_POLL=600) - a live daemon that # simply has not finished restarting its watcher yet. beat=$(( $(date +%s) - 400 )) - touch -d "@$beat" "$dir/state/.last-watcher-beat" + fm_touch_epoch "$beat" "$dir/state/.last-watcher-beat" out=$(FM_GUARD_GRACE='' FM_POLL=600 run_hook "$dir" false); status=$? kill "$pid" 2>/dev/null || true wait "$pid" 2>/dev/null || true @@ -2103,7 +2103,7 @@ test_hook_away_daemon_blocks_beacon_older_than_poll_derived_grace() { # 700s exceeds even the wider poll-derived grace (660 at FM_POLL=600), so a # live daemon that has genuinely stopped restarting its watcher still blocks. beat=$(( $(date +%s) - 700 )) - touch -d "@$beat" "$dir/state/.last-watcher-beat" + fm_touch_epoch "$beat" "$dir/state/.last-watcher-beat" out=$(FM_GUARD_GRACE='' FM_POLL=600 run_hook "$dir" false); status=$? kill "$pid" 2>/dev/null || true wait "$pid" 2>/dev/null || true @@ -2127,7 +2127,7 @@ test_hook_no_afk_ignores_poll_derived_grace() { # accept, but away mode is off here, so the strict watcher predicate and its # flat default govern instead - old behavior, unaffected by FM_POLL. beat=$(( $(date +%s) - 400 )) - touch -d "@$beat" "$dir/state/.last-watcher-beat" + fm_touch_epoch "$beat" "$dir/state/.last-watcher-beat" out=$(FM_GUARD_GRACE='' FM_POLL=600 run_hook "$dir" false); status=$? kill "$pid" 2>/dev/null || true wait "$pid" 2>/dev/null || true diff --git a/tests/fm-watch-recovery-loop.test.sh b/tests/fm-watch-recovery-loop.test.sh index 7ce229f5a43..95f3ce38664 100755 --- a/tests/fm-watch-recovery-loop.test.sh +++ b/tests/fm-watch-recovery-loop.test.sh @@ -20,6 +20,7 @@ install_pi_watch_extension_fixture() { "$repo/bin" cp "$ROOT/.pi/extensions/fm-primary-pi-watch.ts" "$repo/.pi/extensions/fm-primary-pi-watch.ts" cp "$ROOT/.pi/extensions/lib/fm-branch-dispatch.ts" "$repo/.pi/extensions/lib/fm-branch-dispatch.ts" + cp "$ROOT/.pi/extensions/lib/fm-native-contract.ts" "$repo/.pi/extensions/lib/fm-native-contract.ts" cp "$ROOT/.pi/extensions/lib/fm-async-exec.ts" "$repo/.pi/extensions/lib/fm-async-exec.ts" cp "$ROOT/.pi/extensions/lib/fm-calm-visibility.ts" "$repo/.pi/extensions/lib/fm-calm-visibility.ts" cp "$ROOT/.pi/extensions/lib/fm-operational-input.ts" "$repo/.pi/extensions/lib/fm-operational-input.ts" diff --git a/tests/fm-watch-triage-waits.test.sh b/tests/fm-watch-triage-waits.test.sh index b777267028e..4cd935bf7eb 100755 --- a/tests/fm-watch-triage-waits.test.sh +++ b/tests/fm-watch-triage-waits.test.sh @@ -43,4 +43,12 @@ test_nonterminal_stale_pause_transitions_reclassify_unchanged_hash test_nonterminal_paused_rechecks_authoritative_state test_paused_authoritative_working_preserves_wedge_timer test_afk_paused_changed_pane_hands_off_plain_stale +test_live_paused_until_controls_recheck_time + +test_paused_until_near_future_is_quiet_before_the_cadence + +test_paused_until_wrong_year_is_bounded_by_the_cadence + +test_paused_until_that_passed_is_rechecked_before_the_cadence + printf '\nall fm-watch-triage-waits tests passed\n' diff --git a/tests/fm-watch-triage.test.sh b/tests/fm-watch-triage.test.sh index 654a8afd71a..8994c60503d 100755 --- a/tests/fm-watch-triage.test.sh +++ b/tests/fm-watch-triage.test.sh @@ -7,6 +7,233 @@ set -u # shellcheck source=tests/watch-triage-helpers.sh . "$(dirname "${BASH_SOURCE[0]}")/watch-triage-helpers.sh" +# --- the away-posture record: captain-held items are never rechecked ---------- +# While state/.afk-contract exists (bin/fm-afk-contract.sh) nobody is there to +# answer a captain-held item and the return brief lists it, so every stale path +# absorbs such a pane silently: the declared-wait cadence, the live-agent first +# sight, the backlog-hold bound, and the daemon-owned one-shot handoff. Archiving +# the record restores the ordinary bounded recheck, so the rule is the record's, +# not a lost alarm. + +# A UTC ISO 8601 stamp for an epoch, on either date flavor. +iso_utc_at() { # <epoch> + date -u -r "$1" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d "@$1" +%Y-%m-%dT%H:%M:%SZ +} + +write_away_record() { # <state> + if ! FM_HOME="$(dirname "$1")" FM_STATE_OVERRIDE="$1" "$ROOT/bin/fm-afk-contract.sh" propose >/dev/null 2>&1 \ + || ! FM_HOME="$(dirname "$1")" FM_STATE_OVERRIDE="$1" "$ROOT/bin/fm-afk-contract.sh" confirm >/dev/null 2>&1; then + fail "could not write the away-posture record in $1" + fi +} + +archive_away_record() { # <state> + FM_HOME="$(dirname "$1")" FM_STATE_OVERRIDE="$1" "$ROOT/bin/fm-afk-contract.sh" archive >/dev/null 2>&1 \ + || fail "could not archive the away-posture record in $1" +} + +test_captain_held_never_rechecked_while_away_record_exists() { + local dir state fakebin out capture_file statusf window key pane_hash sig pid back + dir=$(make_case away-record-held-secondmate); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/secondmate-hold.status" + window="test:fm-secondmate-hold" + printf 'idle awaiting the captain\n' > "$capture_file" + printf 'window=%s\nkind=secondmate\n' "$window" > "$state/secondmate-hold.meta" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$statusf" + back=$(( $(date +%s) - 500 )) + if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$back" '+%Y%m%d%H%M.%S')" "$statusf" + else touch -m -d "@$back" "$statusf"; fi + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-secondmate-hold_status" + key=$(printf '%s' "$window" | tr '.:/' '___') + pane_hash=$(hash_text "idle awaiting the captain") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + write_away_record "$state" + export FM_FAKE_CREW_STATE='state: unknown · source: none · no current-state source available' + # Phase A: the record exists, the hold is well past the cadence, and the + # watcher still absorbs it across whole poll cycles: no wake, no throttle. + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid" || ! wait_poll_cycle "$state" "$pid"; then + reap "$pid"; fail "watcher rechecked a captain-held item while the away-posture record exists: $(cat "$out")" + fi + [ ! -s "$out" ] || fail "a captain-held recheck was printed while the away-posture record exists" + [ ! -s "$state/.wake-queue" ] || fail "a captain-held recheck was queued while the away-posture record exists" + [ ! -e "$state/.paused-resurfaced-$key" ] || fail "the recheck throttle was armed for an item that must never be rechecked" + grep -F 'never rechecked while the away-posture record exists' "$state/.watch-triage.log" >/dev/null \ + || fail "the silent absorb did not name the away-posture rule in the triage log" + reap "$pid" + ack_stopped_cycle "$state" || fail "could not acknowledge the intentional phase-A stop" + # Phase B: archiving the record (the return) restores the bounded recheck. + archive_away_record "$state" + : > "$out" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 100 || { reap "$pid"; fail "archiving the away-posture record did not restore the captain-held recheck"; } + grep -F "awaiting the captain" "$out" >/dev/null || fail "the restored recheck did not name the captain: $(cat "$out")" + unset FM_FAKE_CREW_STATE + pass "a captain-held item is never rechecked while the away-posture record exists, and the recheck returns once the record is archived" +} + +test_live_captain_held_first_sight_silenced_by_away_record() { + local dir state fakebin out capture_file statusf window key sig pid + dir=$(make_case away-record-held-live); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/held-live.status" + window="test:fm-held-live" + printf 'parked at the decision gate\n' > "$capture_file" + printf 'window=%s\nkind=ship\nharness=grok\nbackend=tmux\n' "$window" > "$state/held-live.meta" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-held-live_status" + key=$(printf '%s' "$window" | tr '.:/' '___') + write_away_record "$state" + # A LIVE agent at the gate: without the record pause_state_class answers none + # and the first sight surfaces (test_exited_declared_pause_is_bounded_but_live_gate_surfaces). + export FM_FAKE_CREW_STATE='state: unknown · source: none · no current-state source available' + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_FAKE_TMUX_CURRENT_COMMAND=grok \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=999 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid" || ! wait_poll_cycle "$state" "$pid" || ! wait_poll_cycle "$state" "$pid"; then + reap "$pid"; fail "a live captain-held pane surfaced on first sight while the away-posture record exists: $(cat "$out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "a live captain-held pane was queued while the away-posture record exists" + [ -e "$state/.stale-$key" ] || fail "the silenced first sight did not advance the stale suppressor" + reap "$pid" + unset FM_FAKE_CREW_STATE + pass "a live captain-held pane is absorbed on first sight while the away-posture record exists" +} + +test_backlog_hold_never_rechecked_while_away_record_exists() { + local dir out capture wakes + dir=$(make_hold_home away-record-backlog-hold 'done: PR https://example.test/pr/9 checks green' hold) \ + || fail "could not build the backlog-hold fixture" + out="$dir/watch.out"; capture="$dir/pane.txt" + write_away_record "$dir/state" + # Without the record the FIRST sight of a held delivery alarms + # (test_stale_churn_without_a_captain_call_still_alarms and its siblings). With + # it, even the first sight and every later hash are absorbed. + hold_watch_churn "$dir" "$out" "$capture" 'held delivery, pane tick' 3 \ + || fail "watcher exited while churning a backlog-held delivery under the away-posture record: $(cat "$out")" + wakes=$(hold_stale_wakes "$dir/state") + [ "$wakes" -eq 0 ] || fail "a backlog-held delivery was rechecked $wakes time(s) while the away-posture record exists" + pass "a delivery the captain already holds is never rechecked while the away-posture record exists" +} + +test_afk_one_shot_never_hands_off_captain_held_under_away_record() { + local dir state fakebin out capture_file statusf window key sig pid + dir=$(make_case away-record-held-afk-oneshot); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/held-afk.status" + window="test:fm-held-afk" + printf 'idle awaiting the captain\n' > "$capture_file" + printf 'window=%s\nkind=ship\nharness=grok\nbackend=tmux\n' "$window" > "$state/held-afk.meta" + printf 'captain-held [key=route]: tracked by task-decision-route\n' > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-held-afk_status" + key=$(printf '%s' "$window" | tr '.:/' '___') + date '+%s' > "$state/.afk" + write_away_record "$state" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_FAKE_TMUX_CURRENT_COMMAND=zsh \ + FM_STATE_OVERRIDE="$state" FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid" || ! wait_poll_cycle "$state" "$pid" || ! wait_poll_cycle "$state" "$pid"; then + reap "$pid"; fail "the daemon-owned one-shot handed off a captain-held pane while the away-posture record exists: $(cat "$out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "the daemon-owned one-shot queued a captain-held pane while the away-posture record exists" + [ "$(cat "$state/.stale-$key" 2>/dev/null || true)" = "$(hash_text 'idle awaiting the captain')" ] \ + || fail "the silenced one-shot did not advance the stale suppressor to the pane hash" + reap "$pid" + pass "the daemon-owned one-shot never hands off a captain-held pane while the away-posture record exists" +} + +# --- declared waits are condition-aware: `until <UTC ISO 8601>` -------------- +# A paused: line naming when the wait clears is rechecked at that time when it +# falls within the flat cadence, but a distant or mistyped time cannot extend +# the cadence, and a time that has passed is rechecked at once. +paused_until_fixture() { # <name> <until-epoch> <status-age-secs> + local name=$1 until=$2 age=$3 dir state statusf window key back + dir=$(make_case "$name"); state="$dir/state" + window="test:fm-until" + statusf="$state/until.status" + printf 'idle, waiting for the reset\n' > "$dir/pane.txt" + printf 'window=%s\nkind=secondmate\n' "$window" > "$state/until.meta" + printf 'paused: rate limit resets, until %s, then resuming\n' "$(iso_utc_at "$until")" > "$statusf" + back=$(( $(date +%s) - age )) + if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$back" '+%Y%m%d%H%M.%S')" "$statusf" + else touch -m -d "@$back" "$statusf"; fi + printf '%s' "$(seen_sig "$statusf")" > "$state/.seen-until_status" + key=$(printf '%s' "$window" | tr '.:/' '___') + printf '%s' "$(hash_text 'idle, waiting for the reset')" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + printf '%s\n' "$dir" +} + +until_watch() { # <dir> <cadence> -> pid in UNTIL_PID + local dir=$1 + PATH="$dir/fakebin:$PATH" FM_FAKE_TMUX_WINDOW=test:fm-until FM_FAKE_TMUX_CAPTURE="$dir/pane.txt" \ + FM_FAKE_CREW_STATE='state: unknown · source: none · no current-state source available' \ + FM_STATE_OVERRIDE="$dir/state" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" \ + FM_PAUSE_RESURFACE_SECS="$2" FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$dir/watch.out" 2>&1 & + UNTIL_PID=$! +} + +test_paused_until_near_future_is_quiet_before_the_cadence() { + local dir state + dir=$(paused_until_fixture until-near-future "$(( $(date +%s) + 120 ))" 60); state="$dir/state" + until_watch "$dir" 240 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "a declared wait with a near-future until time was rechecked before that time: $(cat "$dir/watch.out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "a declared wait with a near-future until time was queued for a recheck" + grep -F 'declared time not reached' "$state/.watch-triage.log" >/dev/null \ + || fail "the absorb did not cite the declared time in the triage log" + reap "$UNTIL_PID" + pass "a declared wait naming a near-future until time stays quiet until that time" +} + +test_paused_until_wrong_year_is_bounded_by_the_cadence() { + local dir state + dir=$(paused_until_fixture until-wrong-year "$(( $(date +%s) + 31536000 ))" 300); state="$dir/state" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 \ + || { reap "$UNTIL_PID"; fail "a wrong-year declared time silenced the wait beyond the recheck cadence"; } + grep -F 'stale: test:fm-until' "$dir/watch.out" >/dev/null \ + || fail "the bounded wrong-year recheck did not print a stale wake: $(cat "$dir/watch.out")" + grep -F 'declared time is beyond the recheck cadence' "$dir/watch.out" >/dev/null \ + || fail "the bounded recheck gave the wrong reason: $(cat "$dir/watch.out")" + grep -F 'declared clearing time has passed' "$dir/watch.out" >/dev/null \ + && fail "the bounded recheck falsely claimed the future declared time passed" + pass "a wrong-year declared time cannot silence the watcher beyond the recheck cadence" +} + +test_paused_until_that_passed_is_rechecked_before_the_cadence() { + local dir state + dir=$(paused_until_fixture until-passed "$(( $(date +%s) - 30 ))" 60); state="$dir/state" + until_watch "$dir" 999 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "a declared wait whose until time passed was not rechecked ahead of the cadence"; } + grep -F 'stale: test:fm-until' "$dir/watch.out" >/dev/null || fail "the due recheck did not print a stale wake: $(cat "$dir/watch.out")" + grep -F 'declared clearing time has passed' "$dir/watch.out" >/dev/null \ + || fail "the due recheck did not say the declared time passed: $(cat "$dir/watch.out")" + grep -F 'possible wedge' "$dir/watch.out" >/dev/null && fail "a due declared wait was mislabeled a possible wedge" + # The due recheck fires once per declaration: a second watcher on the same + # unchanged declaration absorbs it again. + ack_stopped_cycle "$state" || fail "could not acknowledge the due recheck" + : > "$dir/watch.out" + until_watch "$dir" 999 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "the due recheck repeated on every poll instead of once per declaration: $(cat "$dir/watch.out")" + fi + reap "$UNTIL_PID" + pass "a declared wait whose until time has passed is rechecked at once, then held to the cadence" +} + + test_status_span_actionable_classifier test_status_span_survives_a_later_routine_append test_status_span_respects_decision_closure @@ -96,4 +323,6 @@ test_heartbeat_backstop_surfaces_a_masked_status test_beacon_stays_fresh_while_absorbing test_afk_signal_records_heartbeat_endpoint test_afk_present_reverts_watcher_to_one_shot +test_busy_pane_native_progress_resets_age + printf '\nall fm-watch-triage tests passed\n' diff --git a/tests/herdr-client-pair-fixture.sh b/tests/herdr-client-pair-fixture.sh new file mode 100644 index 00000000000..52a7fde89c1 --- /dev/null +++ b/tests/herdr-client-pair-fixture.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# tests/herdr-client-pair-fixture.sh - two herdr CLI fakes modeling one host +# that carries a stale client next to a compatible one. +# +# A self-updated ~/.local/bin copy next to a package-managed herdr is a real +# host shape, and Firstmate's fixed remote-job PATH resolves ~/.local/bin first +# (bin/fm-remote-job-lib.sh). A client older than the running server answers +# every command except `status` with error code protocol_mismatch on stderr +# and exit 1 (verified: herdr 0.8.2, protocol 20, against a 0.9.0 server, +# protocol 22). bin/backends/herdr.sh "client selection" owns how the adapter +# steps around that; these fakes exist so its suites can reproduce the shape. +# +# Usage: +# . "$(dirname "${BASH_SOURCE[0]}")/herdr-client-pair-fixture.sh" +# make_herdr_client_pair <dir> [<stale-version> <stale-protocol> <current-version> <current-protocol>] +# +# Installs <dir>/stale/herdr (default 0.8.2, protocol 20, refused by the +# protocol-22 server) and <dir>/current/herdr (default 0.9.0, protocol 22) +# whose pane and agent reads model one live claude agent at fm-remote:wCY:p2, +# plus <dir>/tools/jq so a PATH made of only these directories still parses +# JSON. Each fake appends its argv to <dir>/stale.log or <dir>/current.log; +# callers export FM_HERDR_PAIR_DIR=<dir>. + +make_herdr_client_pair() { # <dir> [stale-version stale-protocol current-version current-protocol] + local dir=$1 stale_version=${2:-0.8.2} stale_protocol=${3:-20} current_version=${4:-0.9.0} current_protocol=${5:-22} + mkdir -p "$dir/stale" "$dir/current" "$dir/tools" + ln -sf "$(command -v jq)" "$dir/tools/jq" + cat > "$dir/stale/herdr" <<SH +#!/usr/bin/env bash +printf '%s\\n' "\$*" >> "\${FM_HERDR_PAIR_DIR:?}/stale.log" +case "\${1:-} \${2:-}" in + "status --json") + printf '{"client":{"version":"$stale_version","channel":"stable","protocol":$stale_protocol},"server":{"status":"running","running":true,"version":"$current_version","protocol":$current_protocol,"compatible":false,"session":"fm-remote","restart_needed":true}}\\n' + exit 0 ;; +esac +printf '{"id":"cli:%s:%s","error":{"code":"protocol_mismatch","message":"client protocol $stale_protocol is older than server protocol $current_protocol; upgrade the Herdr client before using this command"}}\\n' "\${1:-}" "\${2:-}" >&2 +exit 1 +SH + cat > "$dir/current/herdr" <<SH +#!/usr/bin/env bash +printf '%s\\n' "\$*" >> "\${FM_HERDR_PAIR_DIR:?}/current.log" +case "\${1:-} \${2:-}" in + "status --json") + printf '{"client":{"version":"$current_version","channel":"stable","protocol":$current_protocol},"server":{"status":"running","running":true,"version":"$current_version","protocol":$current_protocol,"compatible":true,"session":"fm-remote","restart_needed":false}}\\n' ;; +SH + cat >> "$dir/current/herdr" <<'SH' + "pane get") + if [ "${3:-}" = wCY:p2 ]; then + printf '{"id":"cli:pane:get","result":{"pane":{"agent":"claude","agent_status":"idle","pane_id":"wCY:p2","tab_id":"wCY:t2","workspace_id":"wCY"},"type":"pane_info"}}\n' + else + printf '{"id":"cli:pane:get","error":{"code":"pane_not_found","message":"no such pane"}}\n' >&2; exit 1 + fi ;; + "agent get") + printf '{"id":"cli:agent:get","result":{"agent":{"agent":"claude","agent_status":"idle","pane_id":"wCY:p2"},"type":"agent_info"}}\n' ;; + *) : ;; +esac +exit 0 +SH + chmod +x "$dir/stale/herdr" "$dir/current/herdr" +} diff --git a/tests/herdr-presentation-fixture.sh b/tests/herdr-presentation-fixture.sh index a7bc821aac9..6e895c23817 100644 --- a/tests/herdr-presentation-fixture.sh +++ b/tests/herdr-presentation-fixture.sh @@ -3,6 +3,7 @@ # Each entrypoint calls setup and seed functions in its own process, creating # fresh source/home/lab/evidence state and installing cleanup before provision. # No fixture state or pool slot is transferred between entrypoints. +# FM_HERDR_LAB_LABEL selects the task label used to generate a fresh lab name. fail() { FIXTURE_FAILED=1; printf 'not ok - %s\n' "$1" >&2; cleanup_all; exit 1; } pass() { printf 'ok - %s\n' "$1"; } @@ -22,7 +23,6 @@ presentation_fixture_setup() { command -v treehouse >/dev/null 2>&1 || { echo "skip: treehouse not found"; exit 0; } [ -x "$HERDR_LAB_HELPER" ] || { echo "skip: Herdr lab helper not executable at $HERDR_LAB_HELPER"; exit 0; } - REAL_HERDR=$(command -v herdr) REAL_TREEHOUSE=$(command -v treehouse) HERDR_ORIGINAL_PATH=$PATH TMP_ROOT=$(mktemp -d "$(cd "${TMPDIR:-/tmp}" && pwd -P)/fm-herdr-presentation.XXXXXX") @@ -41,15 +41,15 @@ presentation_fixture_setup() { : > "$MOVE_CALL_LOG" : > "$FOCUS_AUDIT_LOG" REAL_MOVER="$ROOT/bin/backends/herdr-workspace-move.py" - export REAL_HERDR REAL_TREEHOUSE REAL_MOVER HERDR_CALL_LOG TREEHOUSE_CALL_LOG MOVE_CALL_LOG FOCUS_AUDIT_LOG HERDR_ORIGINAL_PATH HERDR_LAB_HELPER + export REAL_TREEHOUSE REAL_MOVER HERDR_CALL_LOG TREEHOUSE_CALL_LOG MOVE_CALL_LOG FOCUS_AUDIT_LOG HERDR_ORIGINAL_PATH HERDR_LAB_HELPER export ACTIVE_SEEDED_CONTROL POST_CREATE_ABORT_CONTROL TMP_ROOT # Log every production-adapter call, remove its already-validated trailing # session flag, and send the operation through the lab helper so that helper # remains the sole process which appends the real trailing session flag. # The adapter's deliberately session-independent version read cannot pass the - # helper's leading-option guard, so the wrapper sends only that read straight - # to the absolute real binary with the same explicit trailing lab session. + # helper's leading-option guard, so derive that read from its guarded status + # response instead of bypassing the helper. cat > "$FAKEBIN/herdr" <<'SH' #!/usr/bin/env bash set -u @@ -80,7 +80,10 @@ for arg in "$@"; do esac done if [ "${1:-}" = --version ]; then - exec env PATH="$HERDR_ORIGINAL_PATH" "$REAL_HERDR" "$@" --session "$HERDR_LAB_SESSION" + [ "$#" -eq 1 ] || exit 1 + status=$(env PATH="$HERDR_ORIGINAL_PATH" "$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" status --json) || exit 1 + printf '%s' "$status" | jq -er '.client.version | select(type == "string" and length > 0) | "herdr \(.)"' + exit "$?" fi focus_snapshot() { local list row workspace tab tabs @@ -271,7 +274,7 @@ SH herdr_forget_inherited_pane HERDR_LAB_SESSION=$(PATH="$HERDR_ORIGINAL_PATH" \ - "$HERDR_LAB_HELPER" name fm-herdr-presentation-projection) + "$HERDR_LAB_HELPER" name "${FM_HERDR_LAB_LABEL:-fm-herdr-presentation-projection}") export HERDR_SESSION="$HERDR_LAB_SESSION" HERDR_LAB_SESSION LAB_READY=0 RECORDED_WORKTREES="$EVIDENCE_ROOT/worktrees" diff --git a/tests/lib.sh b/tests/lib.sh index e62c0369d3e..9db0195c8e4 100644 --- a/tests/lib.sh +++ b/tests/lib.sh @@ -601,6 +601,31 @@ SH chmod +x "$fakebin/$tool" } +# --- portable file timestamps ----------------------------------------------- + +# fm_touch_epoch <epoch> <path> [path...]: set each path's modification time to +# an absolute epoch second on every supported host. +# +# There is no portable touch(1) flag that takes an epoch: `touch -d @<epoch>` is +# a GNU extension and BSD touch rejects it outright ("out of range or illegal +# time specification"), leaving the file at its current mtime. A test that wants +# a beacon aged 700 seconds then silently measures a brand-new one. +# `touch -t [[CC]YY]MMDDhhmm[.SS]` is POSIX and both accept it, so the only +# host-specific step left is turning the epoch into that stamp, and date(1) +# spells that two incompatible ways. Probe them in this order: GNU date rejects +# `-r <seconds>` (its -r takes a file), while BSD date rejects `-d` as an +# illegal option, so whichever runs is the one that understood the request. +# TZ is pinned to UTC for date and touch so repeated DST hours stay unambiguous. +fm_touch_epoch() { + local epoch=$1 stamp + shift + stamp=$(TZ=UTC0 date -d "@$epoch" +%Y%m%d%H%M.%S 2>/dev/null) \ + || stamp=$(TZ=UTC0 date -r "$epoch" +%Y%m%d%H%M.%S 2>/dev/null) \ + || fail "fm_touch_epoch: date(1) accepted neither -d @<epoch> nor -r <epoch>" + TZ=UTC0 touch -t "$stamp" "$@" \ + || fail "fm_touch_epoch: touch -t $stamp failed for $*" +} + # --- deterministic git identity and fixtures -------------------------------- # fm_git_identity [name] [email]: export a fixed author/committer identity so @@ -612,11 +637,13 @@ fm_git_identity() { # fm_git_init_commit <dir>: create a git repo at <dir> with a README and one # commit. Uses an inline identity so it works whether or not fm_git_identity was -# called. +# called. The initial branch is pinned rather than inherited from +# init.defaultBranch, so a fixture that names main resolves the same on a +# developer machine and on a runner that still defaults to master. fm_git_init_commit() { local dir=$1 mkdir -p "$dir" - git -C "$dir" init -q + git -C "$dir" init -q -b main printf '# %s\n' "$(basename "$dir")" > "$dir/README.md" git -C "$dir" add README.md git -C "$dir" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' commit -qm initial @@ -676,6 +703,16 @@ fm_write_secondmate_meta() { # --- common assertions ------------------------------------------------------ +# assert_equals <expected> <actual> <msg> +assert_equals() { + [ "$1" = "$2" ] || fail "$3 (expected '$1', got '$2')" +} + +# assert_not_equals <unexpected> <actual> <msg> +assert_not_equals() { + [ "$1" != "$2" ] || fail "$3 (unexpectedly got '$1')" +} + # assert_contains <haystack> <needle> <msg> assert_contains() { case "$1" in diff --git a/tests/remote-herdr-fixture.sh b/tests/remote-herdr-fixture.sh index b01419066cc..e849e8d6e61 100644 --- a/tests/remote-herdr-fixture.sh +++ b/tests/remote-herdr-fixture.sh @@ -52,7 +52,7 @@ for ((i=0; i<${#args[@]}; i++)); do done case "${1:-} ${2:-}" in "status --json") - printf '{"client":{"version":"0.7.5","protocol":16},"server":{"running":true}}\n' ;; + printf '{"client":{"version":"0.7.5","protocol":16},"server":{"running":true,"protocol":16,"compatible":true}}\n' ;; "server "*|"server") : ;; "workspace list") jq_state '{result:{workspaces:.workspaces}}' ;; "workspace create") diff --git a/tests/watch-triage-helpers.sh b/tests/watch-triage-helpers.sh index 74f42743e87..86e6cff1eec 100644 --- a/tests/watch-triage-helpers.sh +++ b/tests/watch-triage-helpers.sh @@ -5588,3 +5588,162 @@ test_afk_paused_changed_pane_hands_off_plain_stale() { || fail "AFK paused stale was not queued with the plain window identity" pass "AFK changed paused panes hand off plain stale identities for daemon-owned pause triage" } + +iso_utc_at() { # <epoch> + date -u -r "$1" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d "@$1" +%Y-%m-%dT%H:%M:%SZ +} + +paused_until_fixture() { # <name> <until-epoch> <status-age-secs> + local name=$1 until=$2 age=$3 dir state statusf window key back + dir=$(make_case "$name"); state="$dir/state" + window="test:fm-until" + statusf="$state/until.status" + printf 'idle, waiting for the reset\n' > "$dir/pane.txt" + printf 'window=%s\nkind=secondmate\n' "$window" > "$state/until.meta" + printf 'paused: rate limit resets, until %s, then resuming\n' "$(iso_utc_at "$until")" > "$statusf" + back=$(( $(date +%s) - age )) + if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$back" '+%Y%m%d%H%M.%S')" "$statusf" + else touch -m -d "@$back" "$statusf"; fi + printf '%s' "$(seen_sig "$statusf")" > "$state/.seen-until_status" + key=$(printf '%s' "$window" | tr '.:/' '___') + printf '%s' "$(hash_text 'idle, waiting for the reset')" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + printf '%s\n' "$dir" +} + +until_watch() { # <dir> <cadence> -> pid in UNTIL_PID + local dir=$1 + PATH="$dir/fakebin:$PATH" FM_FAKE_TMUX_WINDOW=test:fm-until FM_FAKE_TMUX_CAPTURE="$dir/pane.txt" \ + FM_FAKE_CREW_STATE='state: unknown · source: none · no current-state source available' \ + FM_STATE_OVERRIDE="$dir/state" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" \ + FM_PAUSE_RESURFACE_SECS="$2" FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$dir/watch.out" 2>&1 & + UNTIL_PID=$! +} + +test_live_paused_until_controls_recheck_time() { + local dir state fakebin out capture_file statusf window key sig wakes future past + dir=$(make_case live-paused-until); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/parked.status" + window="test:fm-parked" + printf 'window=%s\nkind=ship\nharness=grok\nbackend=tmux\n' "$window" > "$state/parked.meta" + future=$(iso_utc_at "$(( $(date +%s) + 7200 ))") + printf 'paused: rate limit until %s\n' "$future" > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-parked_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + printf 'parked, elapsed 1s' > "$capture_file" + printf '%s' "$(hash_text 'parked, elapsed 1s')" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + parked_watch_round "$state" "$fakebin" "$out" "$capture_file" "$window" absorb \ + || fail "a live worker woke before its declared future time" + printf 'parked, elapsed 2s' > "$capture_file" + parked_watch_round "$state" "$fakebin" "$out" "$capture_file" "$window" absorb \ + || fail "pane churn bypassed a live worker's declared future time" + wakes=$(awk -F '\t' -v w="$window" '$3 == "stale" && $4 == w { n++ } END { print n + 0 }' \ + "$state/.wake-queue" 2>/dev/null || echo 0) + [ "$wakes" -eq 0 ] || fail "a live worker produced $wakes wakes before its declared time" + + past=$(iso_utc_at "$(( $(date +%s) - 120 ))") + printf 'paused: rate limit until %s\n' "$past" >> "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-parked_status" + printf 'parked, elapsed 3s' > "$capture_file" + parked_watch_round "$state" "$fakebin" "$out" "$capture_file" "$window" exit \ + || fail "a live worker did not wake when its declared time passed" + wakes=$(awk -F '\t' -v w="$window" '$3 == "stale" && $4 == w { n++ } END { print n + 0 }' \ + "$state/.wake-queue" 2>/dev/null || echo 0) + [ "$wakes" -eq 1 ] || fail "a passed declared time produced $wakes wakes instead of one" + ack_stopped_cycle "$state" || fail "could not acknowledge the due declared-time recheck" + printf 'parked, elapsed 4s' > "$capture_file" + parked_watch_round "$state" "$fakebin" "$out" "$capture_file" "$window" absorb \ + || fail "a due declared time bypassed the reset long cadence" + wakes=$(awk -F '\t' -v w="$window" '$3 == "stale" && $4 == w { n++ } END { print n + 0 }' \ + "$state/.wake-queue" 2>/dev/null || echo 0) + [ "$wakes" -eq 0 ] || fail "a due declared time rechecked again inside the long cadence" + pass "a live paused worker stays absorbed until its declared time, then rechecks" +} + +test_busy_pane_native_progress_resets_age() { + local dir state fakebin out capture_file window key pane_hash sig pid + dir=$(make_case busy-native-progress-resets-age); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; window="test:fm-busy-reset" + printf 'Working...' > "$capture_file" + printf 'window=%s\nkind=ship\nharness=pi\n' "$window" > "$state/busy-reset.meta" + record_pi_busy "$state" busy-reset + printf 'working: setup complete\n' > "$state/busy-reset.status" + sig=$(seen_sig "$state/busy-reset.status"); printf '%s' "$sig" > "$state/.seen-busy-reset_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "Working...") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + # A wedge is already mid-escalation, as if several over-age polls already ran. + echo $(( $(date +%s) - 500 )) > "$state/.stale-since-$key" + printf '1\n' > "$state/.wedge-escalations-$key" + # The worker has progressed without completing its long native turn. + touch "$state/busy-reset.progress" + touch -t 200001010000 "$state/busy-reset.meta" + touch -t 200001010000 "$state/busy-reset.turn-ended" + prime_turnend_seen "$state/busy-reset.turn-ended" + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_BUSY_TURN_MAX_SECS=3600 FM_STALE_ESCALATE_SECS=240 FM_POLL=1 FM_SIGNAL_GRACE=1 \ + FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if ! wait_poll_cycle "$state" "$pid"; then + reap "$pid"; fail "a fresh native activity on a busy pane was still escalated: $(cat "$out")" + fi + [ ! -s "$out" ] || fail "a fresh native activity on a busy pane printed a wake reason" + [ ! -e "$state/.stale-since-$key" ] || fail "a fresh native activity did not clear the wedge timer" + [ ! -e "$state/.wedge-escalations-$key" ] || fail "a fresh native activity did not clear the escalation counter" + reap "$pid" + pass "native progress resets busy age without a completed turn or notification" +} + +test_paused_until_near_future_is_quiet_before_the_cadence() { + local dir state + dir=$(paused_until_fixture until-near-future "$(( $(date +%s) + 120 ))" 60); state="$dir/state" + until_watch "$dir" 240 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "a declared wait with a near-future until time was rechecked before that time: $(cat "$dir/watch.out")" + fi + [ ! -s "$state/.wake-queue" ] || fail "a declared wait with a near-future until time was queued for a recheck" + grep -F 'declared time not reached' "$state/.watch-triage.log" >/dev/null \ + || fail "the absorb did not cite the declared time in the triage log" + reap "$UNTIL_PID" + pass "a declared wait naming a near-future until time stays quiet until that time" +} + +test_paused_until_wrong_year_is_bounded_by_the_cadence() { + local dir state + dir=$(paused_until_fixture until-wrong-year "$(( $(date +%s) + 31536000 ))" 300); state="$dir/state" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 \ + || { reap "$UNTIL_PID"; fail "a wrong-year declared time silenced the wait beyond the recheck cadence"; } + grep -F 'stale: test:fm-until' "$dir/watch.out" >/dev/null \ + || fail "the bounded wrong-year recheck did not print a stale wake: $(cat "$dir/watch.out")" + grep -F 'declared time is beyond the recheck cadence' "$dir/watch.out" >/dev/null \ + || fail "the bounded recheck gave the wrong reason: $(cat "$dir/watch.out")" + grep -F 'declared clearing time has passed' "$dir/watch.out" >/dev/null \ + && fail "the bounded recheck falsely claimed the future declared time passed" + pass "a wrong-year declared time cannot silence the watcher beyond the recheck cadence" +} + +test_paused_until_that_passed_is_rechecked_before_the_cadence() { + local dir state + dir=$(paused_until_fixture until-passed "$(( $(date +%s) - 30 ))" 60); state="$dir/state" + until_watch "$dir" 999 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "a declared wait whose until time passed was not rechecked ahead of the cadence"; } + grep -F 'stale: test:fm-until' "$dir/watch.out" >/dev/null || fail "the due recheck did not print a stale wake: $(cat "$dir/watch.out")" + grep -F 'declared clearing time has passed' "$dir/watch.out" >/dev/null \ + || fail "the due recheck did not say the declared time passed: $(cat "$dir/watch.out")" + grep -F 'possible wedge' "$dir/watch.out" >/dev/null && fail "a due declared wait was mislabeled a possible wedge" + # The due recheck fires once per declaration: a second watcher on the same + # unchanged declaration absorbs it again. + ack_stopped_cycle "$state" || fail "could not acknowledge the due recheck" + : > "$dir/watch.out" + until_watch "$dir" 999 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "the due recheck repeated on every poll instead of once per declaration: $(cat "$dir/watch.out")" + fi + reap "$UNTIL_PID" + pass "a declared wait whose until time has passed is rechecked at once, then held to the cadence" +}