Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.

fix(deploy): skip the staging acceptance verdict on cancelled jobs - #3507

Closed
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/uadd7xtykrlo
Closed

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/uadd7xtykrlo

Conversation

@polylane

@polylane polylane Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

The staging acceptance check reported a failed verdict on release pull-request runs that a concurrency cancellation had already stopped, so each stopped run carried a red check although no staging stack was deployed. The check now reports a verdict only for a run that no cancellation touched, so a stopped run stays unreported.

flowchart LR
  A["new PR event for the same head"] --> B["validate concurrency group, cancel-in-progress"]
  B --> C["validate result = cancelled"]
  C --> D["deploy job skipped"]
  D --> E["acceptance if: always() and not cancelled()"]
  E -->|"observed: cancelled() false for a job-level cancellation"| F["verdict step: DEPLOY_RESULT=skipped, exit 1"]
  F --> G["red staging / live Review on a cancelled run"]
  E -->|"fix: read the dependency results"| H["job skipped, no verdict"]
Loading
Detection evidence

Affected: int_14cf586860018m832vjyg5di

Why this fix

The merged guard from #3442 requires that acceptance is reported only for a run that could have deployed and accepted, and nine of 28 failed verdict jobs in the October window sit on runs GitHub records as cancelled. In those runs the validate job was cancelled by its own concurrency group one second after its step finished, the deploy job resolved skipped, and cancelled() still returned false because it reports only a run-level cancellation. Reading the dependency results in the job condition leaves every ineligible refusal and genuine failure red, and the required release status still refuses while no successful acceptance exists.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:28). Recorded for commit 3959c65e8c8bfd3ce0bc5727ba77d129838c0faa.

Out of scope
  • The eleven validate refusals and two authorize failures keep their red verdict: the merged guard declares an ineligible refusal fail-closed, so I left that path alone.

Production impact: CI-only verdict gate change: the staging Workers keep their traffic

The changed path is a workflow condition that gates only the staging / live Review verdict job; it changes no Worker code, no deployment input and no deploy job condition.

I checked the six staging Workers that this workflow deploys in coreplane-infra Production over the 48 hours to 2026-11-02T15:00Z: 1,361,136 logged events with 405 at error level (switchboard 253,634 events, switchboard-memory 934,674, switchboard-sandbox 133,042, switchboard-resident 39,786). None of them reads this job, and the deploy job's condition is untouched, so no production path changes on merge.

The required staging / Review E2E status is published from the deploy job's steps by the status workflow, which refuses it when the deploy job is skipped, so the merge gate keeps its current behaviour.

Checked and not affected (6)
  • switchboard
  • switchboard-memory
  • switchboard-sandbox
  • switchboard-resident
  • switchboard-docs
  • switchboard-controlled-acceptance-source

Observability

  • The staging / live Review job conclusion shows whether the verdict is reported; it is the signal that changes.
  • The step summary line from the verdict step (Staging E2E is automatic for release PRs only…, Full-stack staging deployment and live Review passed.) and its ::error:: annotation show why a verdict was or was not reported.
  • The published staging / Review E2E status description stays the release gate signal; no new log or metric is added.

Polylane assessed this pull request at 3959c65.

Causal chain
  • Signal (alert): Staging acceptance check goes red when the deploy job is skipped or cancelled (switchboard)
  • Surfacing site: GitHub Actions job staging / live Review of the deploy staging workflow, coreplanelabs/switchboard, on Depot runners at coreplanelabs/switchboard:.github/workflows/deploy-staging.yml#acceptance
  • Mechanism: A newer event for the same pull request enters the validate job's concurrency group with cancel-in-progress, so the validate job result becomes cancelled while the run continues. The deploy job then resolves skipped, and cancelled() in the acceptance condition reports only a run-level cancellation, so the job runs and its verdict step exits 1 on DEPLOY_RESULT=skipped.
  • Producer: deploy staging workflow acceptance verdict job, instance coreplanelabs/switchboard runs 40412752761, 40412833867, 40412957273, 40413412912, 40415405192, 40418129035, 40420095850, 40421056862, 40439537761, at coreplanelabs/switchboard:.github/workflows/deploy-staging.yml#acceptance
  • Trigger: A newer pull request event for the release head entering the validate job's concurrency group, which has cancel-in-progress: true
  • What happens to the failed unit today: The acceptance job is the verdict of a requested run. The release gate is the staging / Review E2E status that staging-status.yml publishes, and it refuses that status when the deploy job is skipped.
  • Cadence check: 9 of 28 failed acceptance jobs between 2026-10-26T21:22Z and 2026-10-27T03:07Z; each ran 2 to 3 seconds
  • Blast radius: 0 other resource(s), 0 other tenant(s); data at risk: none named
  • Producer evidence:
    • The job condition used always() && !cancelled() plus an event filter, with no read of needs.authorize.result, needs.validate.result or needs.deploy.result.
    • Run 40439537761: staging:validate succeeded at 02:39:42Z, validate cancelled at 02:39:43Z, deploy skipped, acceptance failed at 02:39:45Z, run conclusion cancelled.
    • Nine of 28 failed acceptance jobs in the window have a cancelled run conclusion, a cancelled validate job and a skipped deploy job.
    • Each verdict job ran 2 to 3 seconds, the length of the gate step that exits 1 on DEPLOY_RESULT=skipped.
3 files changed (+26/-12)
  • .github/workflows/deploy-staging.yml: modified, +3/-1
  • docs/reference/specs/release-and-deploy.md: modified, +1/-1
  • src/deploy/staging.test.ts: modified, +22/-10

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment.

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added polylane severity:low Polylane autofix severity: low labels Nov 2, 2026
@boristane

Copy link
Copy Markdown
Contributor

Closing: Switchboard is being decommissioned and the repository archived (coreplanelabs/infrastructure#268).

@boristane boristane closed this Nov 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

polylane severity:low Polylane autofix severity: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant