A deploy passes its Flux step when Flux could not delete a removed resource #4734

🤖 Generated by the Agentic Engineer

Evidence

In merge-queue run 38113692499 (2026-10-11), Flux recorded these events on the apps layer:

The deploy's 🔁 Trigger Flux reconciliation step printed ✔ apps reconciled, and the wait for the published revision passed. The run failed only because a later, unrelated check (the gateway route check) happened to notice the leftover route. A leftover object of any other kind would have gone through as a green deploy.

Who is affected

Anyone removing a resource from the platform. The change merges and deploys green while the resource keeps running, unmanaged, and nothing reports it. For prune-protected kinds the orphan alert covers this; for every other kind nothing does.

Expected behaviour

A deploy in which Flux failed to delete a resource it tried to prune does not pass. The failure names the resource and the reason Flux gave.

Acceptance criteria

• After reconciliation, the deploy fails when any Flux layer reported a prune failure for the revision it just applied.
• Resources that are prune-protected on purpose do not trip it (Flux skips those; it does not fail on them).
• A test shows a failed delete fails the step and a clean prune passes it.

Notes

Flux retries the prune on the next reconcile, so one option is to read each layer's events or status for the applied revision rather than to change Flux settings. Which signal is reliable needs checking first.

Size: S–M.

Refinement (2026-10-17)

Where the gap is. The deploy's 🔁 Trigger Flux reconciliation step (.github/actions/deploy-prod/action.yml) runs scripts/reconcile-flux-workloads.sh, which runs ksail workload reconcile; KSail decides "reconciled" inside its own binary. The next step, scripts/wait-for-platform-flux-revision.sh, waits only for infrastructure-controllers to report the published revision and Ready. Nothing in the repository reads a Kustomization's events or a prune result: PruneFailed appears nowhere. Flux records a failed prune with the reason PruneFailed (fluxcd/pkg/apis/meta, PruneFailedReason).

Proposed change (one PR): a new bash check, run in deploy-prod straight after the exact-revision wait, that reads the Kubernetes events for the platform's Flux Kustomizations (bootstrap, infrastructure-controllers, infrastructure, apps, infrastructure-overprovisioning) since the reconcile step started, and fails when any carries reason PruneFailed. The failure prints the Kustomization and the kind, namespace and name of the object Flux could not delete, from the event, and nothing else from the message. Tenant Flux layers are out of scope for this slice; file them as a follow-up if the check proves reliable.

Decided here so the implementer does not re-open it: read events rather than change Flux settings; never retry or force the prune from the deploy; a failed or empty event read is a failure of the check, not a pass, so the check also proves it can see events (for example the ReconciliationSucceeded event the same reconcile wrote).

Acceptance criteria (in addition to the list above):

• A self-test under scripts/tests/ with a fake kubectl covers: a PruneFailed event in the window fails and names the object; one from before the window does not; a clean reconcile passes; an unreadable or empty event list fails.
• The test runs in CI's validate job, with the k8s path filter listing the script and the test.
• The first merge-group deploys after merge pass the new step, read from their logs.

Priority Medium, Effort Medium, Service class Risk reduction. It closes the one case under #4979 where a broken deploy is judged healthy: a removed resource keeps running unmanaged with no signal. Effort includes a new deploy step, its test, CI wiring and two production deploys to verify. No prerequisites. Part of #4979.