fix: improve fleet supervision reliability by cisrd · Pull Request #13 · cisrd/firstmate

cisrd · 2026-09-08T17:07:04Z

Intent

Le capitaine demande d enchaîner les tâches restantes en les regroupant en lots cohérents plutôt que de produire des dizaines de petites PR. Il autorise maintenant deux ou trois ouvriers supplémentaires, avec un maximum de deux sessions Grok, tout en surveillant la charge de la machine.

Ce lot regroupe cinq défauts déjà mesurés de la supervision Firstmate :

le verdict de shell sans agent doit primer quel que soit l ordre des classifications propres à Cursor, Grok, Muse, Codex ou Kimi ;
une boucle de surveillance qui se termine proprement sans événement ne doit pas être déclarée en échec ni laisser la flotte sans relève ;
un résultat terminal déjà présenté ne doit pas être annoncé à nouveau à chaque balayage ;
les variables temporaires d un instantané ne doivent jamais contaminer les lectures suivantes ni faire passer tous les ouvriers pour perdus ;
un agent volontairement arrêté pendant que sa PR attend un événement externe ne doit pas déclencher indéfiniment de fausses alarmes.

Ce brief est en français, ta livraison ne l est pas.

Ajout du capitaine pendant l exécution : « il reste 97% sur grok si ca coupe tu changes switch ». Cette autorisation vise les ouvriers Grok actuellement lancés : si Grok interrompt réellement cette tâche, Firstmate la relancera sur GPT-5.6 Sol en effort élevé sans redemander, dans la même copie et sans perdre le travail.

What Changed

Prioritize shell-without-agent detection across harnesses, preserve settled task states, and isolate and validate snapshot overrides.
Recover clean, eventless watcher exits with bounded successor retries and backoff; deduplicate previously presented terminal outcomes across status-log rewrites.
Record voluntary agent exits during armed PR waits, throttle stale alarms without dropping PR polling, and clear wait records on relaunch or teardown.

Risk Assessment

⚠️ Medium: The change spans several supervision lifecycle transitions, but the approved validator centralization is implemented and no additional actionable issues were substantiated beyond previously declined findings.

Testing

Six focused test scripts and additional executable evidence scenarios passed without skips, demonstrating all five supervision fixes using real CLI subprocesses with isolated homes and mocked backends. Captured CLI transcripts, snapshot JSON, and durable state; no live fleet, full-suite, lint, or delivery phases were run.

Evidence: Shell-without-agent reconciliation preserves reasons but rejects stale working

Source: Shell-without-agent reconciliation preserves reasons but rejects stale working

state: done · source: status-log · PR https://example.invalid/pr/1 opened · agent gone, pane shell remains

=== live pane with no agent still reports its terminal status-log state ===
state: done · source: status-log · PR https://example.invalid/pr/1 opened · agent gone, pane shell remains
state: failed · source: status-log · the release job could not be retried · agent gone, pane shell remains
state: blocked · source: status-log · which provider? · agent gone, pane shell remains
state: parked · source: status-log · choose REST or RPC · agent gone, pane shell remains
state: paused · source: status-log · holding for the vendor maintenance window · agent gone, pane shell remains

=== live pane with no agent keeps failed, blocked, needs-decision, and paused with their reasons ===
state: paused · source: status-log · holding for the vendor maintenance window · agent gone, pane shell remains
state: unknown · source: pane · harness state unavailable (dead shell-no-agent)

=== live pane with no agent reports unknown for a stale working status log ===
state: unknown · source: pane · harness state unavailable (dead shell-no-agent)
state: blocked · source: status-log · waiting on the captain to pick a provider · agent gone, pane shell remains

=== the shell-without-agent verdict outranks an un-retired busy lifecycle record ===
state: blocked · source: status-log · waiting on the captain to pick a provider · agent gone, pane shell remains

Evidence: Clean watcher exit receives a healthy successor

Source: Clean watcher exit receives a healthy successor


=== watch-arm: a clean empty close starts one owned successor instead of failing ===
watcher: attached pid=1805571 (beacon 0s)
watcher: started pid=1811814 (beacon fresh)

Evidence: Terminal deduplication and recurring blocker presentation

Source: Terminal deduplication and recurring blocker presentation


=== a rewritten log does not re-announce the same terminal result and does announce a new identity ===
STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
ident done: PR https://example.test/identity/pull/1 checks green

=== an identical nonterminal captain event appended later surfaces again, then falls silent ===
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/recurring-blocker/fourth.out
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/recurring-blocker/third.out
STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
recur blocked [key=bad/value]: waiting on the captain to pick a provider
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/recurring-blocker/second.out
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/recurring-blocker/first.out
STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
recur blocked [key=bad/value]: waiting on the captain to pick a provider
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/identity-stable/new.out
STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
ident failed: the follow-up PR could not be opened
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/identity-stable/retry.out
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/identity-stable/first.out
STATUS OUTCOME BACKSTOP (newest captain-facing task event has no covering branch outcome):
ident done: PR https://example.test/identity/pull/1 checks green
/tmp/fm-wake-drain-outcome-backstop-tests.jG8hJ3/identity-stable/state/.status-outcome-identity
ident	e1:1033566637-44

Evidence: Fleet snapshot under a foreign ambient override

Source: Fleet snapshot under a foreign ambient override


=== fleet snapshot confines override variables to each child and refuses a foreign captured path ===
{
  "schema": "fm-fleet-snapshot.v1",
  "generated": "2026-09-08T17:02:31Z",
  "fm_home": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak",
  "roots": {
    "fm_root": "~/.no-mistakes/worktrees/a03e7f5d4084/01M20Y4BVJS67Y30T1QYD7T8AW",
    "state": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/state",
    "data": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/data",
    "config": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/config",
    "projects": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/projects"
  },
  "backlog": {
    "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/data/backlog.md",
    "present": false,
    "records": []
  },
  "tasks": [
    {
      "id": "alpha",
      "kind": "ship",
      "harness": "claude",
      "mode": "no-mistakes",
      "yolo": "",
      "project": "alpha",
      "spawn_gen": null,
      "backend": "tmux",
      "remote": null,
      "paths": {
        "meta": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/state/alpha.meta",
          "present": true
        },
        "status_log": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/state/alpha.status",
          "present": true,
          "kind": "event_history",
          "last_event": {
            "state": "working",
            "note": "alpha live",
            "raw": "working: alpha live"
          }
        },
        "worktree": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/projects/alpha-worktree",
          "present": true
        },
        "home": {
          "path": null,
          "present": false
        },
        "report": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/data/alpha/report.md",
          "present": false
        }
      },
      "secondmate_projects": [],
      "current_state": {
        "state": "working",
        "source": "status-log",
        "detail": "alpha live",
        "raw": "state: working · source: status-log · alpha live",
        "observed_at": "2026-09-08T17:02:31Z",
        "freshness": "fresh"
      },
      "endpoint": {
        "target": "firstmate:fm-alpha",
        "exists": true,
        "agent_alive": "not_checked",
        "status": "unknown",
        "observed_at": "2026-09-08T17:02:31Z",
        "freshness": "fresh"
      },
      "pr": {
        "url": null,
        "source": "absent"
      },
      "hints": {
        "pending_decision": false,
        "blocked_event": false,
        "open_decisions": [],
        "scout_report_present": false,
        "last_event_text": "working: alpha live"
      },
      "actions": {
        "watch": "bin/fm-peek.sh fm-alpha",
        "steer": "bin/fm-send.sh fm-alpha '<instruction>'",
        "return_channel_note": null
      },
      "backlog": null
    },
    {
      "id": "beta",
      "kind": "ship",
      "harness": "claude",
      "mode": "no-mistakes",
      "yolo": "",
      "project": "alpha",
      "spawn_gen": null,
      "backend": "tmux",
      "remote": null,
      "paths": {
        "meta": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/state/beta.meta",
          "present": true
        },
        "status_log": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/state/beta.status",
          "present": true,
          "kind": "event_history",
          "last_event": {
            "state": "working",
            "note": "beta live",
            "raw": "working: beta live"
          }
        },
        "worktree": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/projects/alpha-worktree",
          "present": true
        },
        "home": {
          "path": null,
          "present": false
        },
        "report": {
          "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/data/beta/report.md",
          "present": false
        }
      },
      "secondmate_projects": [],
      "current_state": {
        "state": "working",
        "source": "status-log",
        "detail": "beta live",
        "raw": "state: working · source: status-log · beta live",
        "observed_at": "2026-09-08T17:02:31Z",
        "freshness": "fresh"
      },
      "endpoint": {
        "target": "firstmate:fm-beta",
        "exists": true,
        "agent_alive": "not_checked",
        "status": "unknown",
        "observed_at": "2026-09-08T17:02:31Z",
        "freshness": "fresh"
      },
      "pr": {
        "url": null,
        "source": "absent"
      },
      "hints": {
        "pending_decision": false,
        "blocked_event": false,
        "open_decisions": [],
        "scout_report_present": false,
        "last_event_text": "working: beta live"
      },
      "actions": {
        "watch": "bin/fm-peek.sh fm-beta",
        "steer": "bin/fm-send.sh fm-beta '<instruction>'",
        "return_channel_note": null
      },
      "backlog": null
    }
  ],
  "main_inventory": {
    "valid": true,
    "reason": null,
    "orphan_in_flight": [],
    "unstructured_current_count": 0
  },
  "scout_reports": [],
  "secondmate_current": {
    "registry": {
      "present": false,
      "available": true,
      "complete": true,
      "reason": null,
      "provenance": "registered-table",
      "path": "/tmp/fm-fleet-snapshot.yqkRiZ/override-leak/data/secondmates.md",
      "freshness": {
        "status": "fresh",
        "observed_at": "2026-09-08T17:02:31Z"
      },
      "records": [],
      "input_truncated": false,
      "records_truncated": false,
      "reasons": [],
      "lines_in_window": 0,
      "records_in_window": 0
    },
    "records": [],
    "total_registered": 0,
    "total": 0,
    "shown": 0,
    "truncated": 0
  },
  "secondmate_landed": {
    "records": [],
    "truncated": [],
    "unreadable": [],
    "partial": []
  },
  "secondmate_guidance": {
    "note": "For kind=secondmate, bearings selects validated structured state from that registered home; parent events and bounded terminal evidence are fallback-only supplements and never current-state authority."
  }
}

Evidence: Voluntary-wait suppression, invalid-record alarms, and heartbeat recovery

Source: Voluntary-wait suppression, invalid-record alarms, and heartbeat recovery

ok - watcher validates the shared voluntary-exit corpus
ok - voluntary exit with an armed PR poll surfaces once, absorbs churn, keeps the poll, and still alarms a true death
ok - terminal identity dedupe never hides a genuinely new result earlier in the scanned span

=== Actual watcher outputs and persisted triage/queue records (mock backend, real watcher) ===
/tmp/fm-watch-triage-tests.uOrBOa/heartbeat-identity-buried/watch.out
heartbeat
/tmp/fm-watch-triage-tests.uOrBOa/heartbeat-identity-buried/state/.wake-queue
1788886884	1	heartbeat	heartbeat	heartbeat
/tmp/fm-watch-triage-tests.uOrBOa/heartbeat-identity-buried/state/.watch-triage.log
[2026-09-08T19:01:19+0200] absorbed heartbeat (no captain-relevant change)
/tmp/fm-watch-triage-tests.uOrBOa/voluntary-pr/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/voluntary-pr/state/.wake-queue
1788886875	3	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/voluntary-pr/state/.watch-triage.log
[2026-09-08T19:00:49+0200] absorbed stale (voluntary exit waiting on an armed PR poll): test:fm-held-merge
[2026-09-08T19:00:58+0200] absorbed stale (voluntary exit waiting on an armed PR poll): test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-symlink/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-symlink/state/.wake-queue
1788886836	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-missing/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-missing/state/.wake-queue
1788886826	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-extra/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-extra/state/.wake-queue
1788886816	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-duplicate/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-duplicate/state/.wake-queue
1788886804	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-epoch/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-epoch/state/.wake-queue
1788886792	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-wait/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-wait/state/.wake-queue
1788886781	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-reason/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-reason/state/.wake-queue
1788886771	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-schema/watch.out
stale: test:fm-held-merge
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-schema/state/.wake-queue
1788886760	2	stale	test:fm-held-merge	stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-reordered/watch.out
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-reordered/state/.watch-triage.log
[2026-09-08T18:59:05+0200] absorbed stale (voluntary exit waiting on an armed PR poll): test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-reordered/state/.wake-queue
/tmp/fm-watch-triage-tests.uOrBOa/record-valid/watch.out
stale: test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-valid/state/.watch-triage.log
[2026-09-08T18:58:49+0200] absorbed stale (voluntary exit waiting on an armed PR poll): test:fm-held-merge
/tmp/fm-watch-triage-tests.uOrBOa/record-valid/state/.wake-queue

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅

⚠️ bin/fm-crew-state.sh:164 - A newly spawned worker can have metadata and a busy agent before writing its first status event. snapshot_capture_optional intentionally creates no capture when that optional log is absent, but prefetch_task_observations still passes the capture pathname. This new validation immediately emits unknown, bypassing both authoritative run lookup and live-agent classification. Direct crew-state reads still work, so the fleet snapshot silently mislabels that worker. Preserve the captured absence as an empty status observation rather than treating it as an invalid capture, while retaining rejection of genuinely invalid overrides.
⚠️ bin/fm-control.sh:479 - The approved decision requires: "Centralize the voluntary-exit record grammar in one shared fm_voluntary_exit_record_valid helper used by control and watcher." The target still introduces independent validators here and in fm-watch.sh:1176; fm-pr-lib.sh has no shared helper, and the requested shared valid/invalid corpus is absent. Remove the duplicate grammar and implement the already-approved shared validator and consumer regressions.

🔧 Fix: Centralize voluntary-exit validation with shared consumer regressions
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

bash bin/fm-test-run.sh --jobs 1 --per-script-timeout-secs 300 tests/fm-busy-state.test.sh tests/fm-crew-state.test.sh tests/fm-fleet-snapshot-view.test.sh tests/fm-wake-drain-outcome-backstop.test.sh tests/fm-watch-arm.test.sh tests/fm-control.test.sh
bash ~/.no-mistakes/evidence/01M20Y4BVJS67Y30T1QYD7T8AW/targeted-watcher.sh: shared valid/invalid record corpus, voluntary-exit alarm throttling, and heartbeat deduplication with a buried new result.
Executed evidence drivers crew-evidence.sh, snapshot-evidence.sh, drain-evidence.sh, and continuity-evidence.sh from the evidence directory; captured actual CLI responses, snapshot JSON, and persisted watcher records.
git status --short: confirmed no worktree changes or leftover generated artifacts.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…atively

…-state

… consumer regressions

…ecovery behavior

Alex William added 8 commits September 8, 2026 15:22

Harden supervision reliability transitions

14afd7e

no-mistakes(review): bound clean-empty watcher successor retries iter…

da5ea75

…atively

no-mistakes(review): preserve terminal status log for agent-gone panes

fd6da85

no-mistakes(review): scope content-identity dedupe to terminal outcomes

fa29e0d

no-mistakes(review): delegate snapshot override validation to fm-crew…

5fd701e

…-state

no-mistakes(review): keep settled status states through a gone agent

1eeafd1

no-mistakes(review): Centralize voluntary-exit validation with shared…

2b3b592

… consumer regressions

no-mistakes(document): Align supervision documentation with current r…

b86b9d2

…ecovery behavior

Navigation Menu

fix: improve fleet supervision reliability - #13

fix: improve fleet supervision reliability#13
cisrd wants to merge 8 commits into
mainfrom
fm/fm-lot-supervision-fiabilite

cisrd commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cisrd commented Sep 8, 2026

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant