🐛🧑‍💻 - No supported recovery for an unpublished candidate when promotion PR CI requires a source fix #732

Is there an existing issue for this?

• Searched repo:vypdev/copilot is:issue "reprepare", "source branch changed", and "candidate" "restart" on 2026-10-26; no matching report was found.

Which actions are affected?

Issue, Pull Request; release/hotfix deployment preparation and promotion recovery.

Which platforms are affected?

Linux (observed PR contract failure); orchestration policy is platform independent.

Description

A prepared release candidate passes every platform gate, then a promotion-PR-only CI contract discovers a correctable test-harness problem. The promotion operation has already frozen its source SHA before that PR's CI exists. Correcting the existing managed branch and rerunning preparation would be refused as a nonretryable source-identity change. We could not find a supported way to supersede this unmerged and unpublished candidate while preserving its version and ordinary Copilot branch/deployment ownership.

Please own the upstream investigation and define/implement the supported recovery. Keeping the accepted/tagged SHA immutable is correct; this report requests a safe pre-acceptance replacement path, not accepting changed content under old evidence or weakening publication checks.

Consumer: vypdev/zstandard#170 .
Promotion: vypdev/zstandard#173 .
Operation: 594f6029-5636-4733-ab71-85313f456d5f, version 1.5.2, phase promotion_pr_pending, stateVersion1/revision3.
Managed source: release/1.5.2@f55bc5bc2d913aa66f188d847b6d726ed29e6060.
Tested tree: c47cb276c658514cee33ad5a506fb3f94c649a65.
Production base: master@aa8bdc020b37db8d91c89762f4512d396e556ab5.
Origin: develop@22eb4de4ce8f5d959b735cb4c7a91c3ca35cf73e.

Observed facts:

1. Candidate preparation https://github.com/vypdev/zstandard/actions/runs/40389276368 completed successfully. It built distributables in deployment and passed Android AGP9+legacy, Linux, Web, Windows, static/package checks, and iOS/macOS with SwiftPM and CocoaPods.
2. Its real artifact release-candidate-1.5.2 ID 12686865234 attests the exact source/tree above, run40389276368/attempt1/issue170. Downloaded ZIP SHA256: bff7f4f080d949e23c3d2c56e28ae1743c51c4563ea8c9948f497c6b69963f54.
3. Prepare-promotion job https://github.com/vypdev/zstandard/actions/runs/40389276368/job/121825813166 created PR173 and enabled native auto-merge. The maintainer disabled only this PR's auto-merge to review all technical CI.
4. Promotion contract https://github.com/vypdev/zstandard/actions/runs/40396033211/job/121829201153 failed before any upload. Its four publisher fixture tests replace the environment without forwarding LD_LIBRARY_PATH; the Python3.11 interpreter installed by setup-python on ubuntu-runner-4 then cannot find libpython3.11.so.1.0.
5. A consumer correction preserves only PATH/LD_LIBRARY_PATH/SYSTEMROOT (matching the neighboring fixtures), while retaining private HOME/BASH_ENV and the mock registry. All four local fixture tests pass with the correction. This is a consumer test problem, not a Copilot-caused Python loader error.
6. We have not pushed the correction, changed the pinned branch, edited orchestration state, merged PR173, or deliberately invoked the failing resume. The predicted refusal below is established by the exact executed Core source, not claimed as an observed new failed run.
7. The live operation has no productionSha or publication receipt, publicationVerified=false, and no reconciliation targets. GitHub matching refs returns no v1.5.2 tag. The old partial1.5.1 tag/packages are immutable and unrelated to replacement of this still-unpublished1.5.2 candidate.

Reproducing the issue

1. Have a consumer perform full candidate validation before prepare_deployment_action.
2. Let that action open a production promotion PR and persist its exact source.
3. Have a PR-triggered contract uncover a legitimate small fix.
4. Prepare the correction on the same Action-owned source branch.
5. Inspect or invoke preparation again: PreparePromotionHandler.resume compares the current source tip against the stored sourceSha and calls runtime.block(..., false) on any difference.
6. In that nonretryable state, subsequent preparation returns the blocked result through isBlockedOutsidePreparation. The documented resume mode is reconciliation-only after verified publication.

For this live consumer, step4 is only a local unpushed patch and step5 is source inspection. Please reproduce using isolated repositories/fakes rather than mutating the live candidate.

copilot Version

3.10.5. The workflow uses vypdev/copilot@v3; its job log resolves that tag to 39679e26a22b40a5a4a40e8f167f46273157182e. This SHA identifies inspected source and is not an Action pin.

Relevant Log Output

Actual consumer failure:

test_all_packages_publish_in_order_across_expiring_credentials ... python3: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory
subprocess.CalledProcessError: Command '['setup-sdk']' returned non-zero exit status 127.
Ran 4 tests in 0.171s
FAILED (errors=4)

Exact Core source:
https://github.com/vypdev/copilot/blob/39679e26a22b40a5a4a40e8f167f46273157182e/src/application/usecases/actions/deployment_handlers/prepare_promotion_handler.ts

if (operation.phase === "preparing" || operation.phase === "promotion_pr_pending") {
  const currentSourceSha = await this.runtime.dependencies.git.getBranchSha(operation.sourceBranch);
  if (currentSourceSha !== operation.sourceSha) {
    return await this.runtime.block(
      context, operation, "promotion",
      "The prepared source branch changed after its immutable SHA was stored.",
      false,
    );
  }
}

Documentation inspected:
https://github.com/vypdev/copilot/blob/39679e26a22b40a5a4a40e8f167f46273157182e/docs/issues/deployment-orchestration.mdx#recovery-by-visible-phase

Additional context and comments

The consumer's Copilot engine, generated workflows, profile, approval policy and credentials are unchanged. No branch lifecycle bypass or manual package publication was performed.

Please document whether there is an already-supported maintainer procedure. If a replacement capability is needed, suggested acceptance cases:

• Explicitly authorized premerge replacement requires fresh complete validation/evidence for the new source/tree and preserves the version/origin/managed branch.
• Old operation and PR/evidence remain auditable, and all superseded acceptance/publication callbacks are rejected.
• Concurrent attempts, moved heads, changed base, and stale events fail closed.
• No replacement is allowed after production acceptance, tag creation, publication or a conflicting registry receipt.
• No force-push, tag movement, weakened checks, fabricated candidate artifact, manual publisher, or silent change of sourceSha is required.
• A normal PR CI correction has an actionable supported route; users are not required to edit hidden durable JSON to keep the same unpublished patch version.