Skip to content

test: wait on each stall checkpoint's own effect in wake-queue tests - #34

Merged
talktejas merged 2 commits into
mainfrom
fm/repo-dev-branch-ask-store-r2
Sep 30, 2026
Merged

talktejas merged 2 commits into
mainfrom
fm/repo-dev-branch-ask-store-r2

Conversation

@talktejas

@talktejas talktejas commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Intent

The captain's ruling, 2026-09-10, in his words: "for every repo we are working on you should ask captain what is the main development branch and store that value in the repo and every time merge or get worktree against that branch."

This is round two of that task. Round one's agent was lost when its terminal disappeared, but all of its work survives.

You are changing firstmate's own shared tracked material, so load the firstmate-coding-guidelines skill before editing anything and follow it: one sentence per line, plain dashes, shellcheck-clean scripts, colocated tests, and never an agent as a commit co-author.

Write tests and RUN them. This is firstmate's own tooling and its tests are fast.

Never cd as a separate step before a read command - pass the path or use git -C <dir>. Compact your context before 40%.

What Changed

  • Replaced wait_for_checkpoint_alert and the fixed --seconds 1 / FM_TEST_ALERT_CEILING checkpoint runs in tests/fm-wake-queue.test.sh with a new checkpoint_until helper. It reruns fm-watch-checkpoint.sh with a bound that grows by one second each round, up to FM_TEST_CHECKPOINT_ROUNDS rounds (12 by default), and stops as soon as the step's effect shows up.
  • Added the predicates observed and published. With observed, each quiet step now waits until the watcher has written its .secondmate-wake-progress-<mate> record for that fake instant and oldest row. Before this change, a watcher killed early on a loaded runner skipped that record without any error.
  • With published, alert steps wait for their alert line to appear. Quiet steps now fail loudly if the watcher never records its observation. This covers both the foreign-queue stall case and the reprovisioned-generation case.

Pi package pin in CI (a hold, not a fix)

  • .github/workflows/ci.yml now installs @earendil-works/pi-coding-agent@0.87.1 in both Pi install steps instead of the unpinned latest.
  • This pin is a hold against an upstream break published the same day, not a fix for it.
  • Pi 0.99.2 was published on npm at 2026-09-30T19:30Z, about ten minutes before this pull request's first CI run, and it broke tests/fm-calm-pi-extension.test.sh and tests/fm-pi-branch-extension.test.sh, which this pull request does not touch.
  • 0.87.1 is the last version CI was green on (main, 2026-09-28).
  • The extension code and those tests are unchanged, and nothing else is pinned or unpinned; lift the pin once the extension is made compatible with Pi 0.99.x.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change only touches tests: it replaces fixed-duration checkpoint steps with retries that stop once each step's own recorded effect appears. That is sound because the stall tick runs before the rearm-resurface wake in the watcher loop, and a retried alert step is deduplicated by the queued-key and receipt checks.

Testing

I ran the changed wake-queue test file in full (all cases pass). I then ran its two foreign-queue stall tests three times under heavy CPU load: all passed at the target commit, and the base commit failed all three runs. That reproduces the flake the change fixes: 'no actionable wake within 60s' and 'check: rearm-resurface'. The round-one test for the recorded base branch also passes. Logs are in the evidence directory. Temporary scripts and load processes were removed, and the worktree is clean.

  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
The foreign secondmate queue stall test alerts exactly once per no-progress episode, and stays quiet on the first sighting and on progress ✅ pass live fm-wake-queue.test.log: 'ok - foreign secondmate queue alerts once per no-progress episode...'
A reprovisioned queue generation starts a fresh no-progress interval and still alerts when frozen ✅ pass live fm-wake-queue.test.log: 'ok - a reprovisioned queue generation starts a fresh no-progress interval'
Adversarial: both stall tests stay green with the CPU saturated (3x nproc busy loops), 3 rounds ✅ pass live stall-under-load.log (3/3 rounds ok)
Regression: the base commit's version of the same tests fails under the same load, proving the fix matters ✅ pass live base-stall-under-load.log: 3/3 'not ok - a foreign queue with no progress did not alert' (no wake within 60s / rearm-resurface)
Round-one intent still holds: the worker and scout start from the project's recorded development (base) branch ✅ pass live fm-spawn-base-branch.test.log
Evidence: Target commit: stall tests under CPU load (3/3 pass)

Source: Target commit: stall tests under CPU load (3/3 pass)

ok - foreign secondmate queue alerts once per no-progress episode without age-only or cascade noise
ok - a reprovisioned queue generation starts a fresh no-progress interval
round 1 189.36s
ok - foreign secondmate queue alerts once per no-progress episode without age-only or cascade noise
ok - a reprovisioned queue generation starts a fresh no-progress interval
round 2 153.30s
ok - foreign secondmate queue alerts once per no-progress episode without age-only or cascade noise
ok - a reprovisioned queue generation starts a fresh no-progress interval
round 3 162.85s
Evidence: Base commit: stall tests under CPU load (3/3 fail, reproduces flake)

Source: Base commit: stall tests under CPU load (3/3 fail, reproduces flake)

not ok - a foreign queue with no progress did not alert: checkpoint: no actionable wake within 60s
base round 1 rc=1
not ok - a foreign queue with no progress did not alert: check: rearm-resurface
base round 2 rc=1
not ok - a foreign queue with no progress did not alert: checkpoint: no actionable wake within 60s
base round 3 rc=1
Evidence: Full fm-wake-queue test run

Source: Full fm-wake-queue test run

ok - an abandoned same-process lock hold is reclaimed; a parent's live hold is not
ok - without BASHPID a subshell cannot release or reclaim its parent lock and owns its own hold
ok - bounded acquire hands ownership to the waiting caller after contention
ok - presentation lock waits are bounded and retriable without weakening acknowledgement atomicity
ok - malformed presentation locks report acquire failure instead of contention
ok - foreign secondmate queue alerts once per no-progress episode without age-only or cascade noise
ok - declared external-wait pause rows do not feed secondmate wake-loop escalation
ok - a reprovisioned queue generation starts a fresh no-progress interval
ok - an active turn defers the secondmate stall escalation without cancelling it
ok - a long-lived mate mid-turn is not a stall, but a queue frozen past the busy bound still alarms
ok - secondmate stall markers reject symlinks without touching their targets
ok - stall publication acknowledgement closes the pre-marker crash window
ok - empty prefix mate cleanup preserves another mate's stall receipt
ok - self-announced appends suppress only their own bytes and fail toward waking
ok - historical annotations replay nothing already announced and keep everything new
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 25 --recovery-generation 29675.1790797426.GrU9iX
ok - concurrent append plus drain preserves durable records through acknowledgement
ok - signal written while no watcher runs is caught on next run
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation 55724.1790797441.rRvWke
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - stale wake is queued before suppressor state is advanced
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation 59392.1790797444.sE52W1
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - a not-provably-working stale wake is queued before its suppressor is advanced
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation 61609.1790797447.YCRT1s
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - registered custom check output is queued before cadence suppression
ok - concurrent drains replay until one post-handling acknowledgement consumes records
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 4 --recovery-generation 65754.1790797451.V3EswL
ok - drain collapses obvious duplicate heartbeat and signal records
ok - drain asserts watcher liveness: warns on a lapse, stays silent for a live watcher with a fresh beacon
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 8 --recovery-generation 68578.1790797455.jf6DOx
ok - structural signal enrichment is separate, deduped, home-local, and tier-zero for other wakes
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 13 --recovery-generation 70551.1790797456.h8c5nX
ok - every readable unread status line is annotated in full while invalid status files preserve their raw wakes
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation 73136.1790797460.xzRyv0
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 2 --recovery-generation 73136.1790797460.xzRyv0
ok - slow annotation releases the append lock and a deleted status file fails open
ok - a branch-actor scoped ack never swallows an unacked main-owned row, and main's later drain sees exactly what remains
ok - main drain and acknowledgement exclude an active branch grant
ok - a branch-held row raises no queued-wake warning for main, and the same row is presented and acknowledged once the grant clears
ok - a queue that cannot be counted keeps the queued-wake alarm up
ok - structurally unusable rows are retired by main alone, leaving every remaining row presentable and acknowledgeable
ok - branch grant cannot take a row already claimed by main
ok - actor ownership filtering precedes same-key deduplication
ok - main reclaims rows granted to an exited branch owner
ok - a branch-actor drain with no eligible-row snapshot refuses loudly instead of draining nothing
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation existing
ok - wake append publishes atomic recovery evidence before durable rows
ok - wake drain: generation-less legacy wakes are adopted and acknowledged
ok - wake drain: a stale acknowledgement cannot retire or consume a newer recovery episode
ok - wake drain: an acknowledgement that consumes nothing says so and names the exact command for the current wake
ok - wake drain: a branch acknowledgement that consumes nothing names the exact command for its granted wake
ok - wake drain: recovery acknowledgement failures are explicit and retryable
ok - interruptions preserve durable rows until post-handling acknowledgement
Evidence: Base-branch spawn test run

Source: Base-branch spawn test run

ok - base= parses alongside the delivery mode without changing any existing line's mode
ok - a cold slot for a develop-based project starts the worker on develop
ok - a project with no base record still starts from the repo default branch
ok - a project whose default branch is the right base passes the assertion
ok - an unregistered project falls back to the default branch and spawns as today
ok - a base= record naming a branch the repo does not have is refused
ok - a local-only project starts the worker from the local default branch
ok - a scout also starts from the project's recorded base branch
ok - an explicit --base wins over the project's registry base record
ok - an explicit --base is never written back to the shared registry
ok - the registry base still applies when --base is absent
ok - an explicit base the project cannot resolve refuses instead of falling back
ok - a scout also starts from an explicit --base
ok - a scout on a local-only project starts from the local base branch
ok - --relaunch refuses --base and keeps the task's own recorded base
ok - --secondmate refuses --base
ok - a local-only task lands on the base it was created against, not the standing one
ok - a task dispatched without --base lands on the project's current declaration
ok - a landing whose recorded base has vanished falls back to the project's standing base
ok - a declaration landed only on the development branch is still read by a clone on the default branch
ok - the repository's own declaration wins over a home-private registry record
ok - the private registry remains the fallback for a project that has not adopted the file
ok - a malformed declaration is rejected rather than handed to git
ok - the resolver reads a declaration from an unchecked-out branch under stock /bin/bash
# all fm-spawn-base-branch tests passed

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
The foreign secondmate queue stall test alerts exactly once per no-progress episode, and stays quiet on the first sighting and on progress ✅ pass live fm-wake-queue.test.log: 'ok - foreign secondmate queue alerts once per no-progress episode...'
A reprovisioned queue generation starts a fresh no-progress interval and still alerts when frozen ✅ pass live fm-wake-queue.test.log: 'ok - a reprovisioned queue generation starts a fresh no-progress interval'
Adversarial: both stall tests stay green with the CPU saturated (3x nproc busy loops), 3 rounds ✅ pass live stall-under-load.log (3/3 rounds ok)
Regression: the base commit's version of the same tests fails under the same load, proving the fix matters ✅ pass live base-stall-under-load.log: 3/3 'not ok - a foreign queue with no progress did not alert' (no wake within 60s / rearm-resurface)
Round-one intent still holds: the worker and scout start from the project's recorded development (base) branch ✅ pass live fm-spawn-base-branch.test.log
  • bash tests/fm-wake-queue.test.sh (full file, idle machine): all 42 cases ok, including the two stall cases
  • Only the two stall tests (test_secondmate_foreign_queue_stall_tracks_progress_and_alerts_once, test_secondmate_reprovisioned_queue_starts_a_fresh_interval) at the target commit, 3 rounds with 3x nproc busy loops saturating the CPU
  • The same two tests at base commit f84f2f35, 3 rounds under the same load (regression reproduction)
  • bash tests/fm-spawn-base-branch.test.sh (round-one behaviour: the worker starts from the project's recorded base branch)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

… wake

Round one of the development-branch task (ask for the branch, store it in the
repo as .firstmate-base, keep fleet sync on it) already landed on main as #1;
this carries its one open thread.

The foreign-queue stall cases in tests/fm-wake-queue.test.sh failed under load.
Their quiet steps ran fm-watch-checkpoint.sh for one second, and on a loaded
runner the watcher is killed before it reaches its stall observation, so the
step silently records nothing. The alert step then starts the no-progress
interval at its own frozen fake instant, so it can never alert; retrying only
wakes on check: rearm-resurface for the previous checkpoint's exit, which is
why a longer ceiling could not help.

checkpoint_until now runs each step until its observable effect lands: a quiet
step until the watcher has recorded that step's observation, an alert step
until the alert is published. Each round's bound grows by a second, and the
checkpoint exits on the first wake, so a healthy run pays nothing extra.
@talktejas talktejas changed the title fix(tests): wait on each stall checkpoint's own effect, not the first wake test: wait on each stall checkpoint's own effect in wake-queue tests Sep 30, 2026
…figuration only. In `.github/workflows/ci.yml`, both Pi install steps (the portable parallel job and the portable serial job) now run `npm install -g @earendil-works/pi-coding-agent@0.87.1`. That is the last version main's CI was green on. Next to each install is a comment saying the pin is a hold against an upstream Pi 0.99.x break published 2026-09-30, not a fix for it, and should be lifted once the extension works with Pi 0.99.x. I did not touch the extension code or the failing tests (`fm-calm-pi-extension`, `fm-pi-branch-extension`), and nothing else is unpinned. Checks: the workflow file loads as valid YAML in Python, and `npm view` confirms version 0.87.1 exists on npm. I could not run `tests/fm-ci-workflow.test.sh` locally because it needs Ruby, which is not installed here. Before this, the existing Pi extension tests were not run against 0.87.1. For the PR body, whoever owns that phase must state plainly that the Pi pin is a hold against an upstream break published the same day, not a fix for it. On ci-3: Behavior portable serial 1 was cancelled, which is a missing result, not a pass. The new push re-runs it, and its real verdict must be the one taken
@talktejas
talktejas merged commit 9d1590f into main Sep 30, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant