fix: simplify evidence receipts and No-Mistakes handoff - #211
Merged
Merged
Conversation
…nting Receipts had grown a second validation lifecycle beside No-Mistakes: a risk classifier whose low path never fired in production, plan generations, run binding by ancestry, restamp and content-tree proofs, branch-custody evidence, terminal sealing, and a validation_* metadata family that eleven follow-up PRs kept patching. fm-receipt-check.sh now answers one question - did the worker account for every declared acceptance criterion - through the default check, --criterion, and --parse-criteria, emitting fm-evidence-check.v2. The latest receipt per criterion decides it, so a finding is recorded as a later failure and satisfied again only by a fresher success; no generations. Run identity moves to the PR-ready owner. fm-pr-check.sh requires complete evidence for every ship mode and, for no-mistakes tasks, proves the run from No-Mistakes' own axi status (task branch, PR URL, full head_sha against the forge's PR head, passed or CI-green), records nm_run_id=, and runs the ask-user decision audit as its single PR-ready owner. fm-crew-state.sh accepts a ship done only with a clean worktree, complete evidence, pr= for the PR modes or a clean fm/<id> branch for local-only, and the same ask-user audit against the recorded or attributed run. fm-spawn.sh --relaunch carries pr=, pr_head=, and nm_run_id= forward so a restart mid-handoff can still satisfy those gates. The PR-publication race protection that rode on the validation-plan lock survives as state/.<id>.pr-publication.lock. Briefs, validation-supervision, ship-landing, AGENTS.md section 7, and the verification record state the one scope: receipts certify nothing about review, CI, No-Mistakes completion, or merge readiness. Binding, sealing, restamp, generation, and classifier tests are deleted; the behavioral set (complete, missing, failed, expected-negative, accepted-blocked, unknown AC, invalidation, one handoff per mode, wrong or foreign run refused, restart preservation) is added across the receipt, pr-check handoff, crew-state, and relaunch suites.
…ir owners The per-task publication lock gets its operational-home-layout entry, the no-mistakes timeout fm-pr-check.sh reads gets its configuration.md line, and the GitLab merge-watch record states that only direct-PR registers a merge request because no-mistakes PR-ready compares the forge head with the run's head_sha.
fm-pr-merge.sh registers the PR once more before every merge, and reconciliation re-arms a skipped poll; a PR this task already records as pr= passed the evidence, run-identity, and ask-user gates at its registration, so the re-registration refreshes pr_head= and re-arms without consulting No-Mistakes again, while a different URL is gated in full. The merge, CI-watch, and teardown fixtures that register a PR now carry a complete direct-PR evidence contract instead of a no-mistakes mode they never exercised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Simplify Firstmate's evidence-receipt system so receipts prove only acceptance-criterion accounting and No-Mistakes exclusively owns its validation lifecycle, per the captain-approved consolidated design (data/fm-receipt-simplification-review/consolidated.md, merging the Astra and Fable reviews).
Accepted requirements:
axi statusfor that run matches the task branch, the PR URL and the PR's full head SHA (forge head via gh headRefOid against the run's head_sha) and the run is passed or CI-green; it then runs the existing ask-user decision audit (fm_nm_ask_user_decisions, the sqlite read-only adapter) as that guarantee's single PR-ready owner, and the evidence completeness check. The ask-user audit stays in firstmate because No-Mistakes exposes no CLI for per-finding gate decision provenance. A PR already recorded as pr= for the task passed those gates at registration, so re-registering the same URL (fm-pr-merge.sh does this before every merge; reconciliation re-arms skipped polls) refreshes pr_head= and re-arms without re-running the gates; a different URL is gated in full.Decisions and constraints: delivery mode and yolo stay fixed at intake (AGENTS.md section 7), never chosen after implementation. Mechanical merge guards in fm-pr-merge.sh / fm-merge-local.sh (accepted-blocked and red-check refusals) are explicitly OUT of scope for a separate follow-up PR. The receipt writer no longer stamps a commit head; the schema still tolerates a legacy head field so old ledgers stay readable. The legacy done-artifact check remains owned by bin/fm-classify-lib.sh and bin/fm-teardown.sh. fm_nm_ci_checks_state keeps parsing the CI log because axi status exposes no live checks-ready field. A no-mistakes PR-ready fails closed when the forge head cannot be observed (GitLab), consistent with No-Mistakes publishing GitHub PRs only. Tests use existing fakes only; no live Herdr, No-Mistakes runs, Boat or RunPod. Test fixtures in fm-pr-merge, fm-main-ci-watch and fm-teardown that register PRs carry a complete direct-PR evidence contract because registration now gates on evidence. Tests that fail identically on untouched origin/main (kimi tomllib, OMP/Pi TS versions, check-unregister, prepush-guard, send-turn-start, treehouse-orphan-recovery, secondmate-safety) are pre-existing environment failures and not in scope.
Firstmate-Validation-Generation: 599d0d30f079b483b226eb76b405aebb
What Changed
Risk Assessment
🚨 High: Captain, the done gate can bypass the required decision audit for existing tasks, so this change needs correction before merging.
Testing
Receipt suites, manual CLI checks, and focused handoff, completion, relaunch, and publication-race checks passed using real Firstmate scripts with required external-service fixtures. A tracing-induced stderr failure passed after tracing was removed. CLI and metadata evidence was captured; no rendered UI changed. No lint, full suite, or other pipeline phase ran.
Evidence: Receipt CLI transcript
Evidence: Handoff and done CLI output
Evidence: Publication race and watcher output
Evidence: Persisted metadata after relaunch
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-crew-state.sh:133- AC4 requires the "same ask-user decision audit at done against the recorded nm_run_id or the attributed run." The newif [ -n "$audit_run" ]; thenguard skips that audit entirely when neither identity is available. A pre-upgrade task with pr=, complete receipts, a clean worktree, and an idle worker reporting done is therefore accepted when No-Mistakes is unavailable. The changed fixture at tests/fm-crew-state.test.sh:1557 supplies precisely this metadata, and tests/fm-crew-state.test.sh:1573 expects done. Fail closed in emit() when no audit run can be established, covering both status-log and coarse-run completion, and correct that fixture's expectation.docs/verification/evidence-receipts.md:96- AC7 requires that "the report states production and test lines removed versus added." The rewritten verification report lists guarantees and commands but contains no such accounting. Add separate production and test additions/deletions for the reviewed commit range.🔧 Fix applied.
2 issues (1 error, 1 warning) still open:
bin/fm-crew-state.sh:133- AC4 requires the "same ask-user decision audit at done against the recorded nm_run_id or the attributed run." The newif [ -n "$audit_run" ]; thenguard skips that audit entirely when neither identity is available. A pre-upgrade task with pr=, complete receipts, a clean worktree, and an idle worker reporting done is therefore accepted when No-Mistakes is unavailable. The changed fixture at tests/fm-crew-state.test.sh:1557 supplies precisely this metadata, and tests/fm-crew-state.test.sh:1573 expects done. Fail closed in emit() when no audit run can be established, covering both status-log and coarse-run completion, and correct that fixture's expectation.bin/fm-pr-check.sh:131- AC3 requires that “re-registering the same URL ... re-arms without re-running the gates,” but[ -z "$NM_RUN_ID" ] || ALREADY_REGISTERED=1adds an unrequired prerequisite. Pre-upgrade no-mistakes tasks have registered pr= records without nm_run_id, which the old registration never wrote. When fm-pr-merge.sh:349 re-registers that URL with No-Mistakes unavailable, registration now refuses and blocks merging. Remove the extra prerequisite and let exact recorded-URL matching control re-registration. Sibling sites: bin/fm-pr-check.sh:136 repeats evidence checks; bin/fm-pr-check.sh:152 repeats run-identity and decision checks. This is separate from declined R1’s done-time audit.✅ **Test** - passed
✅ No issues found.
TMPDIR="$PWD/.phase-test-tmp" bash tests/fm-receipt-check.test.shTMPDIR="$PWD/.phase-test-tmp" bash tests/fm-receipt.test.shTMPDIR="$PWD/.phase-test-tmp" bash -x tests/fm-pr-check-handoff.test.shManualfm-receipt.shandfm-receipt-check.shcalls with isolated task records and adversarial ledger inputsSelected crew-state tests: evidence completeness, malformed brief, registered PR, clean local branch, recorded-run decision audit, and fast-mode isolationSelected relaunch tests: preserve delivery identity mid-handoff and avoid inventing delivery recordsSelected publication tests: lock serialization, fresh-lock deferral, stale-lock refusal, and concurrent watcher publicationCaptured CLI transcripts, watcher output, and persisted relaunch metadata; removed disposable fixtures and confirmed clean worktree✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.