fix: preserve run authority across capped reruns - #70
Conversation
The capped overview fallback kept run identities but discarded their live state before selecting from SQLite. If a rerun cancelled the predecessor between those reads, its later cancelled status falsely appeared consistent even after a replacement had started. Preserve the readable live identities, refresh the read-only same-branch inventory once when the selected predecessor becomes terminal, and report unknown when no replacement can be established. Retain every observed ID on an unreadable refresh and preserve genuine newer failures, final status and head/custody checks, and the unchanged normal-path lookup count. Add deterministic inventory/status interleavings and bounded-refresh regressions, register all existing crew-state cases in their original order, and document the controlled-fixture versus live-evidence boundary.
Final verification: 4cc19b0At 2026-09-18T06:09:46.440510+00:00, 20/20 automatic checks are successful for
Retry history retainedThe first CI attempt passed 19 checks, including the full current-state suite, but portable serial 4 reported two failures in unchanged PR-poll/startup fixtures. The eight skips are existing optional gates, not new live passes. Bug-fix proofThe deterministic race test failed twice before the fix with The four local gates, source-aware lint, named-case/order audit and focused regressions passed. |
Preserve PR #71's generation-bound replacement reports alongside PR #70's bounded capped-rerun selection. Resolve the shared current-state registry as the ordered union of both parents: 142 cases, with named selection and listing retained and the full-suite success banner emitted only for a full execution. Keep incoming production behavior and PR #70's selector unchanged.
Merge conflict resolved: c32582aMerged fork main All four local gates and ten focused cases passed on tree Fresh CI and Fork CI are running for this exact head. |
Conflict-resolution head fully verified: c32582aAt 2026-09-18T12:46:47.452426+00:00, all 20 automatic checks passed for
Both sides of the conflict retained and exercisedThe case registry is the ordered union of the two parents: 128 shared + 9 capped-rerun + 5 revived-worker cases = 142, with no duplicates or missing definitions. Locally, all four repository gates and ten focused cases passed, alongside source-aware lint and fresh scope/order/routing audits. The PR remains open, non-draft, mergeable/clean and unmerged, with auto-merge off. |
1 participant
Summary
Fix the capped no-mistakes rerun-selection race documented as a remaining limitation in canonical PR #4476, after synchronizing with merged PR #68.
unknownwith every observed candidate ID if no successor is established, the refresh fails, identities disappear or conflict, or the replacement changes again.The production change is confined to
bin/fm-nm-run-lib.sh's current-state selector.The coarse runs-ledger/teardown attribution functions and
bin/fm-crew-state.sh's final identity/liveness/head/custody checks are unchanged.No database write, runtime deadline change, new dependency, workflow change or concurrency admission is introduced.
The separate normal-path three-CLI-call cost noted by kunchenguid#4476 is not changed by this fix.
Root cause and deterministic reproduction
The capped overview fallback retained candidate IDs but discarded their earlier live state before recursively selecting from the complete inventory.
Consequently, an overview showing A live, followed by an inventory showing A cancelled and a replacement B starting before the final status read, could produce a falsely consistent terminal verdict for A.
The regression invokes the real state reader over disposable Git and SQLite fixtures.
A shim lets the actual read-only query finish, then publishes the replacement into that fixture database before the next observation, without sleeps or production-source mutation.
Before the fix, two independent runs reproduced:
The same scenario now preserves
01RERUNand its review gate.An additional red/green case ensures partial displayed-head semantics cannot erase a readable live identity before the complete lookup; the fetched successor still needs full validation.
Regression coverage
Nine new named cases cover:
All 133 cases from current main retain their order, including the original 128 and the five added by PR #71.
The nine rerun cases make 142 registered cases, preserving both parents' order through the shared
fm_test_run_casesinterface.The documentation records these as controlled interleavings, not newly captured live no-mistakes runs.
No live pipeline was initialized or controlled and no vendor session was started.
Conflict resolution with PR #71
Normally merged PR #71's fork-main update into this branch; no squash, rebase or force-push.
The sole conflict was the case registry at the end of
tests/fm-crew-state.test.sh.The combined registry keeps all 142 cases exactly once and preserves the relative order of each parent's cases.
It retains main's array-style registration and PR #70's full-suite-only success banner, so listing or one selected case is not mislabeled as a complete suite pass.
PR #70's selector and its two documentation changes are byte-identical to
4cc19b0.The 13 other incoming paths are byte-identical to new main, preserving generation-bound reporting, active-run precedence, later busy-turn authority, recovery receipts, and the default bounded snapshot.
The PR still changes exactly four paths against its actual base: +300/-18, with no existing file-mode changes.
Workflow definitions, runner algorithms, proof admissions, package pins, live gates and production deadlines are unchanged.
Current local verification
10 focused behavior cases passed, alongside all four repository gates, explicit source-aware lint, the executable case-list/order audit, and refreshed lane/changed-path inventories.
Case listing and inventory checks are inspection evidence, not suite executions.
All checks used explicit Git-for-Windows Bash, serial selected-case execution, cache off, and
FM_LIVE=0.shell-syntaxrepository-lintsource-aware-lintdocumentation-audienceslane-coveragecase-registration-and-scopeci-lane-inventorylane-coverage-retrycapped-rerun-between-inventory-and-status-keeps-replacement-gatecapped-rerun-keeps-liveness-before-complete-head-validationcapped-stable-inventory-uses-one-readercapped-refresh-preserves-a-genuinely-newer-failureactive-run-is-authoritativecurrent-launch-report-beats-only-launch-seedcurrent-launch-without-report-requires-live-agentcurrent-launch-report-reaches-bounded-snapshotcurrent-launch-report-survives-dead-endpointlaunch-report-does-not-retire-secondmateactual-base-changed-routingThe first coverage attempt timed out after 184,734 ms at a 180-second external cancellation bound, without an assertion diagnostic.
That bound was shorter than the earlier same-host successful coverage run (198,240 ms at a 300-second bound).
One serial retry used the previous 300-second bound and passed unchanged in 151,827 ms.
The timeout remains a timeout; neither production/CI deadlines nor test assertions changed, and no performance-regression repair is claimed.
The behavior rows above use the existing public runner, with the row name prefixed by
test_and hyphens replaced by underscores forFM_TEST_ONLY:Other command owners:
The case audit proves 128 common + 9 PR #70 + 5 PR #71 = 142, no duplicate or missing registration, both parent orders retained.
The 64 selected scripts all map to existing CI producers; no broad local suite was run.
The fresh coverage partition is 240 = 24 parallel + 200 serial + 16 Herdr, five serial shards, 21 unhinted serial subjects and zero unhinted parallel subjects.
Documentation validation reports 108 surfaces / 479 local links.
The full current-state suite remains owned by Behavior portable parallel 2.
Native management, stock Bash, package and real-Herdr guarantees remain with their existing workflow jobs; optional skips never become live passes.
Fresh CI
All 20 automatic producers passed on the current head, verified at 2026-09-18T12:46:47Z.
Both workflows completed successfully on attempt 1, with one verified workflow/job producer per expected check and no missing, pending, failed or mismatched required producer.
The complete 142-case current-state suite passed in portable parallel 2:
exit=0 duration_ms=38463 gate_skip=false, including the capped-rerun and default-bound replacement-snapshot assertions.No current-head CI rerun was needed; the older retry below remains historical evidence only.
The manual-only Windows Herdr experiment was not dispatched.
Historical evidence before this base update (4cc19b0)
These results apply only to the original published head, not to the merge commit above.
The original 137-case suite and all 20 checks passed, with one targeted retry of portable serial 4 and its dependent summary.
That initial shard reported failures in unchanged PR-poll/startup fixtures; both assertions passed on its same-head retry, without code/deadline/assertion changes, and the initial causes remain unestablished.
The first failed log and both attempts remain retained; a narrow underlying
gh apifallback retrieved that complete read-only log aftergh-axitruncated it.The original local runs also retained three pre-existing Windows fixture-setup failures: one quoted-path Git/SQLite fixture and two no-Python restricted-PATH fixtures (
/usr/bin/env: bash: No such file or directory).These were not counted as passes or skipped coverage; the old full-suite CI passed independently.
This update does not claim those local fixture setups were repaired.
Original workflows: CI, Fork CI.
The original PR body, case logs, deterministic red/green evidence and audit are archived outside the repository.
Provenance and delivery
4cc19b0a989c592bf29324ff607f77210cf48760.765b4f91b4e086a0483a4cb8c9d8d6d494ca65a7.c32582a64c4e8d9bcda17092f4e5a2dfe008ddf9.51d69810c3740ba156803b17ce4854c539089a51.C:/src/firstmate2.1c11c416fe4d70b475332795e7c698f3afb03dd6; this request did not authorize updating it.%TEMP%/firstmate2-rerun-race-20260918/base-update-765b4f9/, including every command plan/log/result, scope/order audit and fresh CI observations.Leave this ordinary, non-draft PR unmerged, with auto-merge off pending separate approval.