[Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout-tests.yml #945

Activity

1. don-petry

   added

   health-check

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   fleet-tracker

   on Aug 7, 2026
2. don-petry commented on Aug 7, 2026

   

   don-petry

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Dev-Lead Implementation Plan

   Issue: #945 — [Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout-tests.yml

   Scope

   Fix the flakiness (41.2% failure rate, 102s→180s duration spread) of canary-rollout-tests.yml by making the single non-hermetic test in tests/canary_rollout.bats stop hitting the real GitHub API.

   Root Cause

   Diagnosed by running the full suite under a PATH trap that records any gh/git/sleep invocation not covered by a test's own stub. Exactly one of the 250+ tests leaks real calls:

   orchestrator: evaluate warning for a fail-closed frontier does NOT claim PRE_EXISTING

   It overrides _frontier_state but then calls the full cmd_evaluate ci-failure-analyst with no gh/git stub in PATH. Before reaching the overridden _frontier_state, cmd_evaluate resolves the candidate/channel tags (_resolved_channel) and the cut date (candidate_cut_date) via gh api on the agent's cross-repo host petry-projects/.github-private. On CI those become real network round-trips (11 gh api calls per run) whose auth/rate-limit/latency variance is nondeterministic — the source of both the failure rate and the duration spread. Every other test in the suite stubs gh/git; this one is the sole exception, contradicting the suite's own contract ("stubs gh/git via PATH … without any real API access").

   Implementation Checklist

   • Add a hermetic gh + git PATH stub to the leaking test (mirroring the established stub pattern in this file) so cmd_evaluate's channel/cut-date resolution stays local. The branch under test remains driven by the existing _frontier_state override; assertions are unchanged.

   Tests to Write / Verify

   • Red: trap harness records 1 leaking test / 11 real gh calls before the fix.
   • Green: trap harness records 0 real gh/git calls after the fix.
   • Full bats tests/canary_rollout.bats suite stays green.
   • Assertions of the fixed test (BLOCKED, not PRE_EXISTING, cut date) still hold.

   Note: this is a test-hermeticity fix — the shipped scripts (scripts/canary-rollout.sh, scripts/lib/canary-rollout.sh) are unchanged (behavior-preserving; Phase 2 exception noted).

   Test Command

   GH_TOKEN=dummy-token-for-stubbed-tests bats --print-output-on-failure tests/canary_rollout.bats
3. don-petry commented on Aug 7, 2026

   

   don-petry

   on Aug 7, 2026

   ContributorAuthor

   More actions

   ✅ Completion report — #945 (canary-rollout-tests.yml DEGRADED, 41.2% failure rate)

   Root cause: a single non-hermetic test. Of the 252 tests in tests/canary_rollout.bats, exactly one — orchestrator: evaluate warning for a fail-closed frontier does NOT claim PRE_EXISTING — invoked the full cmd_evaluate ci-failure-analyst without a gh/git PATH stub. Because ci-failure-analyst is cross-repo hosted (petry-projects/.github-private), cmd_evaluate resolves the candidate/channel tags (_resolved_channel) and cut date (candidate_cut_date) via gh api before reaching the test's overridden _frontier_state. That produced 11 real network round-trips per run, whose auth/rate-limit/latency variance drove both the ~41% failure rate and the 102s→180s duration spread. This was the only test violating the suite's documented hermeticity contract ("stubs gh/git via PATH; a dummy token satisfies the token precondition without any real API access").

   Fix (behavior-preserving test hermeticity — Phase 2 exception). Added a benign gh+git PATH stub to that one test so candidate/channel/cut-date resolution stays local. The BLOCKED / "cut date" / not-PRE_EXISTING assertions are driven entirely by the existing _frontier_state override (triage="-" → indeterminate branch), so the stub changes nothing observable. Assertions unchanged. No shipped script touched (scripts/canary-rollout.sh and scripts/lib/canary-rollout.sh are byte-for-byte identical).

   Verification

   Check	Result
   Network-leak trap (whole suite, trap gh/git on PATH)	0 leaks (was 11 from this test)
   Full bats suite	252/252 pass, 0 failures
   Target test #198	✅ pass
   .dev-lead/scripts/dev-lead-lint.sh	✅ pass (shellcheck 29 files + CODEOWNERS)
   shellcheck --severity=warning -x scripts/canary-rollout.sh scripts/lib/canary-rollout.sh	✅ exit 0

   Diff: tests/canary_rollout.bats, +22 lines (one test). No commit/push performed — the CI workflow handles git operations.
4. don-petry

   added

   dev-lead:needs-humandev-lead could not complete this issue; needs human attentiondev-lead could not complete this issue; needs human attention

   on Aug 7, 2026
5. don-petry commented on Aug 7, 2026

   

   don-petry

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Dev-Lead: cannot implement issue #945 — needs human attention

   A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

   • Cause: timeout
   • Tier: action
   • Elapsed: 2100s
   • Budget: 2100s
   • Run: https://github.com/petry-projects/.github/actions/runs/31159605135

   This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

   Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-945-20260807-0756 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.
6. don-petry commented on Aug 11, 2026

   

   don-petry

   on Aug 11, 2026

   ContributorAuthor

   More actions

   ✅ Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-08-08 (3 days) — the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.
7. don-petry

   closed this as completedon Aug 11, 2026