Avoid redeploying production after an evicted candidate was refused before deployment #4937

🤖 Generated by the Agentic Engineer

Description

A candidate which the queue preflight correctly refuses can still start a full restore-main deployment. The recovery condition treats every failed deploy job alike, even when the production composite was skipped. This consumes hosted runners and the shared deployment lock without recovering a candidate deployment.

Observed today: run 38781024186, for the closed #4926, failed its queue preflight, skipped Deploy to Production, and then queued Heal Prod. The same preflight-only failure occurred in run 38776984758. No incident or monetary saving is inferred from these records.

Steps to Reproduce

1. Admit a merge-group run, then remove its candidate before its deploy job starts.
2. Let the existing queue preflight refuse that evicted or replaced entry.
3. Observe that the production composite is skipped but the restore-main job is still eligible.

Expected Behavior

Skip recovery only when an explicit, trustworthy outcome proves this attempt stopped before any production operation. Preserve recovery for every actual deployment failure or cancellation and every missing, malformed, partial or unknown outcome. Keep all required checks, queue/source binding, deployment serialization, signatures, authorization and recovery fences intact.

Acceptance Criteria

• A regression executes the real preflight/deploy/recovery wiring and proves a verified pre-production refusal cannot invoke either production action.
• Actual deployment failures and cancellations still recover main.
• Unknown, missing, malformed and interrupted outcomes retain the conservative recovery path.
• Current queue membership, exact source and all existing recovery controls remain enforced.
• Review, hosted CI and merge pass; record a subsequent evicted-run outcome without interrupting an active recovery.

Follow-up to #4891; separate from the larger candidate-deletion design in #4443. The currently queued recoveries are left intact: cancelling an already-starting recovery is not an acceptable workaround. Priority Medium; Effort Medium.

Refinement (2026-10-17)

Where it is decided today (.github/workflows/ci.yaml):

• heal-prod-on-failure runs when deploy-prod.result is failure or cancelled, or success with merge-group-queue-membership.outputs.evicted == 'true'. It reads nothing else from deploy-prod.
• The queue preflight is the queue-preflight step inside deploy-prod (go run ./scripts/verify-merge-group-queued check). In check mode it writes no output; a refusal is exit 1. deploy-prod has no outputs: block, so a preflight refusal reaches the heal as an ordinary failure.
• scripts/validate-merge-group-heal/ pins the exact heal condition (main.go) and the preflight step (queue_preflight_test.go); the changes job runs it as 🩹 Validate merge-group heal contract.

Proposed change (one PR):

1. Give deploy-prod one output, written by a final if: always() step, that is true only when steps.queue-preflight.outcome == 'failure' and every step that can touch production (the deploy composite and anything after the preflight) has outcome skipped. Any other combination, including a missing step, writes nothing.
2. Add && needs.deploy-prod.outputs.stopped_before_production != 'true' to the heal condition. An absent or empty output keeps today's behaviour, so every unknown case still heals.
3. Update the pinned condition in scripts/validate-merge-group-heal/ and add cases: preflight refusal with the composite skipped does not heal; a composite failure or cancellation heals; a refusal plus a run composite heals; a missing or malformed output heals.

Acceptance criteria: unchanged from the list above; the heal-contract tests above are the regression the first criterion asks for.

Priority Medium, Effort Medium, Service class Standard (as already stated in the body). Each needless heal holds the production deploy lock and a hosted runner for a full restore deploy while the queue waits behind it; two such runs on 2026-10-15. No prerequisites. Part of #4979.