Repository navigation
cmd_rollback moves the bare <agent>/<ring> tag, not <agent>/v<M>-<ring> — the emergency lever is a silent no-op for every v-scoped agent #1103
Description
cmd_rollback moves the bare <agent>/<ring> tag, not the v-scoped <agent>/v<M>-<ring> tag that callers actually pin. For any agent on a v-scoped channel family, a rollback therefore silently moves a tag nobody reads — and reports success.
This is the rollback-side counterpart of #1046 (which fixed v-scope handling in _gh_candidate_cut_date). cmd_promote is major-scope-aware; cmd_rollback is not.
Observed
dev-lead is on major 139, and all seven fleet stubs pin the v139-* family. Attempting to put stable on a specific soaked release:
$ canary-rollout rollback dev-lead stable --to v139.8.0 --dry-run
rolling back dev-lead/stable -> v139.8.0 (90dcc01ca854) on petry-projects/.github-private
[DRY-RUN] would: gh api PATCH repos/petry-projects/.github-private/git/refs/tags/dev-lead/stable \
sha=90dcc01ca854d7acd65044bb8defcc32789a8335 (force)
It targets dev-lead/stable. The tag every stub reads is dev-lead/v139-stable. Nothing in the fleet would have changed, and the command exits 0.
Why this is dangerous rather than merely broken
Rollback is the emergency lever. It is what you reach for when a bad candidate has reached the fleet and you need it off, quickly, under pressure. In that moment this command reports a successful rollback, moves a tag with no consumers, and leaves every repo still running the bad code.
Silent no-op plus success exit is the worst possible failure mode for that operation.
It also removed the safe option in a real decision on 2026-09-08: with 152 clean runs on 90dcc01c and a need to advance stable, rollback --to v139.8.0 was exactly the right tool — one step, landing the soaked commit. Because it is unusable, the only path left was promote --override walked three times, which shipped 9e709367 (zero canary coverage) instead. See #1102.
Cause
cmd_rollback resolves its target ring via the bare channel_tag "$agent" "$ring" form, without passing the agent's current major. cmd_promote and _resolved_channel both go through the v-scope-preferring path (_agent_current_major → channel_tag "$agent" "$tier" "$major", falling back to bare), added in epic #657 F4. cmd_rollback was not updated with them.
Acceptance criteria
-
cmd_rollbackresolves the target ring tag the same waycmd_promoteand_resolved_channeldo: prefer<agent>/v<M>-<tier>when the agent has a current major, fall back to the bare tier only when no v-scoped tag exists. - The dry-run output names the exact ref it will PATCH, so an operator can see
v139-stablevsstablebefore committing. The current output showing onlydev-lead/stableis what made this invisible. - Fail loudly rather than no-op: if the resolved tag does not exist, or resolves to a tag no caller pins, say so instead of exiting 0.
- Audit the other subcommands for the same gap — anything calling
channel_tagwithout a major on an agent that has one.promote,evaluateanddriftappear correct;rollbackwas not. - Regression test with the real shape: agent on major 139,
v139-*family present,rollback <agent> stable --to vX.Y.Zmust targetv139-stable.
Related
- canary gate: a v-scoped channel tag shadows the release tag in _gh_candidate_cut_date, so any cross-repo agent adopting v<M>-<tier> channels is permanently BLOCKED (indeterminate) — blocks #1592 #1046 — the same v-scope blindness in
_gh_candidate_cut_date(fixed) - Story (F): major-version boundary in the channel/ring model [#604] #657 F4 — where major-scoped channels were introduced
- Frontier resets to ring0 whenever autocut advances 'next', so outer rings can be unreachable indefinitely — stable sat 3 weeks while next advanced 8 releases (follow-up to #1066) #1102 — the frontier-reset issue that made rollback the preferred tool here
- Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member) #1086 — vacuous soak measurement
CI on PR #1117 is red. Maintainer note 2026-09-13, in the body because dev-lead cannot read comments (#1566)
Lint and bats → Run bats suite fails on PR head dbac61d1 (run 34737293858). main is green at 782e005f.
not ok 113 orchestrator: rollback fails loudly when the resolved ring tag does not exist (no silent no-op) (#1103)
not ok 114 orchestrator: rollback falls back to the bare ring tag when no v-scoped channel tag exists (#1103)
# ::error::rollback target ring tag dev-lead/v139-stable does not exist on petry-projects/.github-private — refusing to create-and-move a channel tag nobody pins
Likely cause, to verify rather than assume: PR #1097 (#1065) merged to main at 04:03Z, just before this branch picked up main. It changed channel-tag resolution to key on the channel major via _agent_current_channel_major, which is the highest M for which <agent>/v<M>-next exists. It no longer uses the release major. Both failing tests resolve dev-lead/v139-stable, which suggests the rollback fixtures seed a v139-next anchor, or none at all, and still assume the old release-major resolution.
- Make
cmd_rollbackresolve its target through the same resolver_resolved_channeluses now. Do not add a second resolution path. - Update the two fixtures so their channel tags match the scenario each test names. Test 114 ("no v-scoped channel tag exists") must seed no
v<M>-nextanchor, so resolution takes the legacy bare path. - Do not weaken test 113's fail-loud assertion.
Activity
don-petry commentedon Sep 13, 2026
Dev-Lead Implementation Plan
Issue: #1103 — cmd_rollback moves the bare
<agent>/<ring>tag, not<agent>/v<M>-<ring>— the emergency lever is a silent no-op for every v-scoped agentScope
Make
cmd_rollbackresolve the target ring tag through the same v-scope-preferring path ascmd_promote/_resolved_channel(prefer<agent>/v<M>-<ring>, fall back to bare only when no v-scoped tag exists), name the exact ref it will PATCH in dry-run output, and fail loudly instead of no-op'ing when the resolved ring tag does not exist.Implementation Checklist
scripts/canary-rollout.sh— incmd_rollback, replace the bare$agent/$ringtarget with_resolved_channel "$agent" "$ring"(tag + commit), mirroringcmd_promote's_resolved_channel_tag.v139-stablevsstable.::error::) when the resolved ring tag does not exist on the host, instead of creating-and-moving a tag nobody pins.Audit (AC #4)
_gh_move_tagwrite paths arecmd_promote(uses_resolved_channel_tag✓),cmd_autocut(passes an explicit major tochannel_tag✓), andcmd_rollback(the bug).cmd_evaluate/cmd_driftare read-only and go through_resolved_channel. The$agent/$tolookup in rollback is the immutable release tag (correctly bare) — only the ring tag was wrong.Tests to Write (
tests/canary_rollout.bats)v139-*family present —rollback dev-lead stable --to v139.8.0 --dry-runmust targetdev-lead/v139-stable, never baredev-lead/stable.<agent>/<ring>tag.Test Command
./node_modules/.bin/bats --print-output-on-failure tests/canary_rollout.batsdon-petry commentedon Sep 13, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
cmd_rollbackresolve the target ring tag through_resolved_channel— the same v-scope-preferring pathcmd_promote/_resolved_channeluse (prefer<agent>/v<M>-<ring>, fall back to bare<agent>/<ring>only when no v-scoped tag exists).v139-stablevsstable._gh_move_tagwrite paths arecmd_promote(already_resolved_channel_tag),cmd_autocut(passes an explicit major tochannel_tag), andcmd_rollback(this fix).evaluate/driftare read-only via_resolved_channel. The--torelease lookup is the immutable bare<agent>/vX.Y.Zand is correctly left bare.Local check output (provisional — not the pushed run)
Files touched (this workspace, not yet pushed)
scripts/canary-rollout.sh:cmd_rollbackresolves the ring tag v-scope-aware + fails loudly on a non-existent resolved tag.tests/canary_rollout.bats: three regression tests — v-scoped target (real major-139 shape), fail-loud on absent tag, bare-tier fallback preserved.Notes
The fail-loud error names the resolved tag; on a current-major agent with no v-scoped tag,
_resolved_channelfalls back to the bare tier, so the error names the bare tag it would otherwise have no-op'd on — matchingcmd_promote's resolution exactly.don-petry commentedon Sep 13, 2026
Dev-Lead: Implementation Complete — PR #1117
The implementation for issue #1103 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
1d3987d009ca177fd781db3a64bed7909f30e28cReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
fix: cmd_rollback falls back to bare ring tag when v-scoped tag doesn…