Skip to content

cmd_rollback moves the bare <agent>/<ring> tag, not <agent>/v<M>-<ring> — the emergency lever is a silent no-op for every v-scoped agent #1103

Open

Description

@don-petry

cmd_rollback moves the bare <agent>/<ring> tag, not the v-scoped <agent>/v<M>-<ring> tag that callers actually pin. For any agent on a v-scoped channel family, a rollback therefore silently moves a tag nobody reads — and reports success.

This is the rollback-side counterpart of #1046 (which fixed v-scope handling in _gh_candidate_cut_date). cmd_promote is major-scope-aware; cmd_rollback is not.

Observed

dev-lead is on major 139, and all seven fleet stubs pin the v139-* family. Attempting to put stable on a specific soaked release:

$ canary-rollout rollback dev-lead stable --to v139.8.0 --dry-run

rolling back dev-lead/stable -> v139.8.0 (90dcc01ca854) on petry-projects/.github-private
[DRY-RUN] would: gh api PATCH repos/petry-projects/.github-private/git/refs/tags/dev-lead/stable \
                 sha=90dcc01ca854d7acd65044bb8defcc32789a8335 (force)

It targets dev-lead/stable. The tag every stub reads is dev-lead/v139-stable. Nothing in the fleet would have changed, and the command exits 0.

Why this is dangerous rather than merely broken

Rollback is the emergency lever. It is what you reach for when a bad candidate has reached the fleet and you need it off, quickly, under pressure. In that moment this command reports a successful rollback, moves a tag with no consumers, and leaves every repo still running the bad code.

Silent no-op plus success exit is the worst possible failure mode for that operation.

It also removed the safe option in a real decision on 2026-09-08: with 152 clean runs on 90dcc01c and a need to advance stable, rollback --to v139.8.0 was exactly the right tool — one step, landing the soaked commit. Because it is unusable, the only path left was promote --override walked three times, which shipped 9e709367 (zero canary coverage) instead. See #1102.

Cause

cmd_rollback resolves its target ring via the bare channel_tag "$agent" "$ring" form, without passing the agent's current major. cmd_promote and _resolved_channel both go through the v-scope-preferring path (_agent_current_major → channel_tag "$agent" "$tier" "$major", falling back to bare), added in epic #657 F4. cmd_rollback was not updated with them.

Acceptance criteria

  • cmd_rollback resolves the target ring tag the same way cmd_promote and _resolved_channel do: prefer <agent>/v<M>-<tier> when the agent has a current major, fall back to the bare tier only when no v-scoped tag exists.
  • The dry-run output names the exact ref it will PATCH, so an operator can see v139-stable vs stable before committing. The current output showing only dev-lead/stable is what made this invisible.
  • Fail loudly rather than no-op: if the resolved tag does not exist, or resolves to a tag no caller pins, say so instead of exiting 0.
  • Audit the other subcommands for the same gap — anything calling channel_tag without a major on an agent that has one. promote, evaluate and drift appear correct; rollback was not.
  • Regression test with the real shape: agent on major 139, v139-* family present, rollback <agent> stable --to vX.Y.Z must target v139-stable.

Related


CI on PR #1117 is red. Maintainer note 2026-09-13, in the body because dev-lead cannot read comments (#1566)

Lint and bats → Run bats suite fails on PR head dbac61d1 (run 34737293858). main is green at 782e005f.

not ok 113 orchestrator: rollback fails loudly when the resolved ring tag does not exist (no silent no-op) (#1103)
not ok 114 orchestrator: rollback falls back to the bare ring tag when no v-scoped channel tag exists (#1103)
# ::error::rollback target ring tag dev-lead/v139-stable does not exist on petry-projects/.github-private — refusing to create-and-move a channel tag nobody pins

Likely cause, to verify rather than assume: PR #1097 (#1065) merged to main at 04:03Z, just before this branch picked up main. It changed channel-tag resolution to key on the channel major via _agent_current_channel_major, which is the highest M for which <agent>/v<M>-next exists. It no longer uses the release major. Both failing tests resolve dev-lead/v139-stable, which suggests the rollback fixtures seed a v139-next anchor, or none at all, and still assume the old release-major resolution.

  • Make cmd_rollback resolve its target through the same resolver _resolved_channel uses now. Do not add a second resolution path.
  • Update the two fixtures so their channel tags match the scenario each test names. Test 114 ("no v-scoped channel tag exists") must seed no v<M>-next anchor, so resolution takes the legacy bare path.
  • Do not weaken test 113's fail-loud assertion.

Activity

  1. don-petry commented on Sep 13, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #1103 — cmd_rollback moves the bare <agent>/<ring> tag, not <agent>/v<M>-<ring> — the emergency lever is a silent no-op for every v-scoped agent

    Scope

    Make cmd_rollback resolve the target ring tag through the same v-scope-preferring path as cmd_promote/_resolved_channel (prefer <agent>/v<M>-<ring>, fall back to bare only when no v-scoped tag exists), name the exact ref it will PATCH in dry-run output, and fail loudly instead of no-op'ing when the resolved ring tag does not exist.

    Implementation Checklist

    • scripts/canary-rollout.sh — in cmd_rollback, replace the bare $agent/$ring target with _resolved_channel "$agent" "$ring" (tag + commit), mirroring cmd_promote's _resolved_channel_tag.
    • Emit the resolved ring tag in the "rolling back …", dry-run PATCH, real-move, and success/error lines so an operator sees v139-stable vs stable.
    • Fail loudly (non-zero, ::error::) when the resolved ring tag does not exist on the host, instead of creating-and-moving a tag nobody pins.

    Audit (AC #4)

    • Confirmed the only _gh_move_tag write paths are cmd_promote (uses _resolved_channel_tag ✓), cmd_autocut (passes an explicit major to channel_tag ✓), and cmd_rollback (the bug). cmd_evaluate/cmd_drift are read-only and go through _resolved_channel. The $agent/$to lookup in rollback is the immutable release tag (correctly bare) — only the ring tag was wrong.

    Tests to Write (tests/canary_rollout.bats)

    • Regression (the real shape): dev-lead on major 139 with a v139-* family present — rollback dev-lead stable --to v139.8.0 --dry-run must target dev-lead/v139-stable, never bare dev-lead/stable.
    • Fail-loud: a rollback whose resolved ring tag does not exist on the host exits non-zero with an error, rather than exiting 0.
    • Bare-tier fallback preserved: an agent with no v-scoped family still rolls back the bare <agent>/<ring> tag.

    Test Command

    ./node_modules/.bin/bats --print-output-on-failure tests/canary_rollout.bats

  2. don-petry commented on Sep 13, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Made cmd_rollback resolve the target ring tag through _resolved_channel — the same v-scope-preferring path cmd_promote/_resolved_channel use (prefer <agent>/v<M>-<ring>, fall back to bare <agent>/<ring> only when no v-scoped tag exists).
    • Dry-run / move / success / error lines now name the resolved ring tag, so an operator sees v139-stable vs stable.
    • Added a fail-loud guard: when the resolved ring tag has no commit on the host, it errors and returns non-zero instead of creating-and-moving a tag nobody pins.
    • Audit note (AC Add workflow, environment, and orchestration guidance #4): the only _gh_move_tag write paths are cmd_promote (already _resolved_channel_tag), cmd_autocut (passes an explicit major to channel_tag), and cmd_rollback (this fix). evaluate/drift are read-only via _resolved_channel. The --to release lookup is the immutable bare <agent>/vX.Y.Z and is correctly left bare.

    Local check output (provisional — not the pushed run)

    # scripts/canary-rollout.sh cmd_rollback change + 3 new bats tests (#1103)
    ./node_modules/.bin/bats tests/canary_rollout.bats   → all local cases pass locally (incl. 3 new #1103 tests)
    bash .dev-lead/scripts/dev-lead-lint.sh              → all checks passed locally
    shellcheck -S warning scripts/canary-rollout.sh      → clean locally
    

    Files touched (this workspace, not yet pushed)

    • scripts/canary-rollout.sh: cmd_rollback resolves the ring tag v-scope-aware + fails loudly on a non-existent resolved tag.
    • tests/canary_rollout.bats: three regression tests — v-scoped target (real major-139 shape), fail-loud on absent tag, bare-tier fallback preserved.

    Notes

    The fail-loud error names the resolved tag; on a current-major agent with no v-scoped tag, _resolved_channel falls back to the bare tier, so the error names the bare tag it would otherwise have no-op'd on — matching cmd_promote's resolution exactly.

  3. don-petry commented on Sep 13, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete — PR #1117

    The implementation for issue #1103 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions