A failed merge-group deploy leaves prod on never-merged code, because the heal shares the deploy path and fails with it

@devantler

🔴 Production delivery is still halted — one cluster write unblocks it

Status at 2026-08-07 05:2xZ: ~14 hours halted. Every GitOps delivery to prod is refused. This
needs one manual cluster write from @devantler; nothing in CI or Git can reach it, for the reason
in Why nothing automatic can fix this below.

Verified live, this tick:

	Value
`ocirepository/flux-system` (`flux-system`)	`Ready=False` — `failed to verify the signature using provider 'cosign keyless': no matching attestations: unsupported: multiple identities are not supported at this time`
Live `matchOIDCIdentity` entries	3 (separate list entries)
`main` declares	1 entry, with the three signers OR'd inside the subject regex
Source of the 3 entries	`fluxinstance/flux` → `spec.kustomize.patches[6]`, applied by a never-merged candidate

The fix restores the control, it does not remove it. main already carries the correct
single-entry form (k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml,
patch index 6, same as live). The write below simply makes the live object match main.

kubectl --context admin@prod edit fluxinstance flux -n flux-system

In spec.kustomize.patches[6].patch, replace the three matchOIDCIdentity entries with this
single entry — copied verbatim from main:

matchOIDCIdentity:
  - issuer: '^https://token\.actions\.githubusercontent\.com$'
    subject: '^https://github\.com/devantler-tech/platform/\.github/workflows/(ci\.yaml@refs/heads/gh-readonly-queue/main/.+|(cd|dr-rebuild)\.yaml@refs/heads/main)$'

Then it self-heals, with no drift. flux-operator re-renders the OCIRepository with one matcher →
cosign verifies → the root source goes Ready → infrastructure-controllers reconciles and applies
main's FluxInstance, which carries that identical value.

Why nothing automatic can fix this

The trust configuration is delivered through the channel it gates. The OCIRepository's matchers
are rendered by fluxinstance/flux, which is owned by the infrastructure-controllers Kustomization
(kustomize.toolkit.fluxcd.io/name on the live object) — and that Kustomization's source is the very
OCIRepository that is refusing every artifact. So the corrected value on main cannot be delivered.

Two escape routes were checked and both are void:

gh workflow run cd.yaml — cd.yaml:127 uses the same ./.github/actions/deploy-prod
composite as ci.yaml, so it reaches the same reconcile step and fails identically. (Confirmed by
the heal at 03:21:41Z, job 92754831196, and again by attempt 2's merge-group CI 31140471256:
publish/sign/attest/promote all succeed, then it waits exactly 30 min in
🔁 Trigger Flux reconciliation and is refused with the same VerificationError.)
ksail cluster update — it syncs Talos machine config and does not own the FluxInstance; its
if: also requires cilium_rollout_gate.active != true (active 12 days) and
wait_flux_revision.outcome == success (skipped), so either condition alone skips it.

The agent runtime denies cluster mutation at the command class, so this write is the maintainer's.

Evidence — the originating incident

A merge-group candidate deploy applied to real production, then failed; the heal that exists to
undo that also failed, for the same underlying reason. Production has been running never-merged
code for over four hours with no automatic recovery.

Measured on the merge group for #3001
(run 31110692734, branch
gh-readonly-queue/main/pr-3001-…):

Step	Window	Outcome
`🚀 Deploy to Prod`	14:26:06 → 14:57:00Z	failure — but it had already applied to the cluster
`CI - Required Checks`	14:57:03 → 14:57:06Z	failure → PR evicted from the queue
`🩹 Heal Prod (restore main…)`	14:57:06 → 15:02:13Z	failure, at step 3 `🩹 Re-deploy main to Production`

The apply is visible on the live object. fluxinstance/flux in flux-system carries
kustomize-controller Apply at 14:28:51Z, and its spec.kustomize.patches contains an
OCIRepository/flux-system → add /spec/verify patch:

target: {"kind":"OCIRepository","name":"flux-system","namespace":"flux-system"}
patch:  - op: add / path: /spec/verify / value: provider: cosign …

At the time of the incident that patch existed only on #3001's branch — main's
flux-instance.yaml then contained zero occurrences of spec/verify. So the running cluster's Flux
configuration was a candidate that had never merged.

#3001 has since merged (2026-08-06 21:50:44Z) carrying the corrected single-entry form, which
is why main and the live object now disagree in the direction described at the top of this issue:
main declares 1 matcher, the cluster still runs the candidate's 3. The divergence outlasted the
branch that caused it, because the delivery channel needed to close it is the one that is broken.

Nothing has restored it. Checked against the unfiltered run list (not a --branch main filter, which
is unreliable on a merge-queue repo): no deploy has succeeded since the 14:24:50Z merge group.

Why this is a distinct defect

#2828 and #2838 both describe the heal running successfully at the wrong moment — ordering races
that leave prod behind main, i.e. running code that has at least landed. This is the opposite and
more severe case:

the deploy and the heal share the same deploy path, so they fail together rather than
independently;
the outcome is prod running code that never merged — and which outlived the branch it came
from, since fix(security): enforce cosign verify on the live root OCIRepository #3001 has since merged a different, corrected value that cannot be delivered; and
there is no second line of defence — a heal is the only automatic recovery, and it is
structurally correlated with the thing it recovers from.

Any blocker that breaks deploys therefore breaks recovery at the same time. The live instance of that
blocker is the gated Cilium rollout in #2963: the DaemonSet health check cannot converge, so the
handoff quiesce times out for the deploy and the heal identically.

Impact

Production's most privileged configuration is set by whichever partially-successful candidate last
touched it, not by main. Today's divergence happens to be benign — the candidate adds signature
verification — but the mechanism is direction-agnostic. The same path could leave prod running a
candidate that removes a control, and the recovery job would fail to undo it in exactly the same way.

It also silently invalidates reasoning about live state: a reader who measures the cluster and finds a
control present will conclude it is configured in Git, when it is not. (That already occurred — see
the correction on #3001.)

Expected behaviour

A failed merge-group deploy leaves production on a state that is reachable from main, or it raises
an alert that it could not.

Acceptance criteria

When the heal cannot restore main, that is surfaced as an explicit, visible failure signal —
not just a red job on an evicted PR that nobody owns.
The heal's success is asserted against the cluster, not the job's exit status: after a
failed candidate deploy, the live revision is reachable from main, or the discrepancy is
reported.
A regression test covers "deploy fails and heal fails" — the correlated case — rather than
only the ordering cases in heal-prod-on-failure can overwrite a newer deploy that has succeeded but not yet merged #2828/A merge-group heal can leave prod behind a merge that lands just after it #2838.

Size

Small-to-medium. The detection half (assert the live revision against main, and alert when the heal
cannot converge) is independently shippable and is the useful first child; making the heal robust to a
broken deploy path is the larger follow-up and may depend on #2963.

Related: #2828, #2838 (heal ordering races), #2963 (the live blocker that breaks both halves), #3001
(the candidate currently on prod).

🔗 Related PRs

#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2622 - fix(ci): heal cancelled merge-group deploys [merged]
#2793 - feat(cilium): activate homogeneous device selection [merged]
#2962 - fix(deploy): make rollout-gate suppression of machine-config sync visible and bounded [merged]
#3001 - fix(security): enforce cosign verify on the live root OCIRepository [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

Escalation — the live instance is worse than "prod runs never-merged code": it halted delivery

I filed this an hour ago describing the mechanism and called today's divergence "benign, because the
candidate adds signature verification". That was wrong, and I am correcting it. Reading the root
source's conditions rather than its spec:

ocirepository/flux-system   Ready False  VerificationError   since 2026-08-06T14:29:01Z
                            SourceVerified False             since 2026-08-06T14:29:01Z
  failed to verify the signature using provider 'cosign keyless': no matching attestations:
  unsupported: multiple identities are not supported at this time

The never-merged candidate installed a matchOIDCIdentity list Flux's verifier rejects outright, so
production's root source has been failing verification for 4h 45m and flux-system/apps is still
pinned to the previous revision. Details on
#3001.

Two things this sharpens about the defect filed here

1. The blast radius is not bounded by "the candidate was an improvement". I argued the mechanism
is direction-agnostic in principle; it turns out the very first observed instance was harmful in
practice. A speculative deploy does not need to contain a malicious or regressive change to break
production — an ordinary, well-reviewed, CI-green change that is merely wrong about an external
system's capabilities is enough. Static validation cannot catch that class, which is exactly why the
candidate must not be the thing production runs.

2. The failure is self-locking, which removes the last informal recovery route. I noted here that
the heal is correlated with the deploy. This instance shows a third correlation: the broken config
travels through the delivery channel it breaks. flux-operator writes the offending field from the
live FluxInstance; the FluxInstance arrives via kustomize-controller from the artifact; the
artifact is what verification now refuses. So the cluster cannot reconcile its way back even once the
deploy path is repaired — recovery requires an out-of-band change to the live FluxInstance.

Any fix for this issue that assumes "a later successful deploy converges prod back onto main" is
therefore insufficient on its own. A candidate deploy can leave the cluster in a state where no
subsequent deploy can reach it.

Consequence for the acceptance criteria above

The third criterion — a regression test for the correlated "deploy fails and heal fails" case —
should be widened to cover the self-locking case: after a failed candidate deploy, assert that the
cluster is still able to receive a deploy at all, not merely that it is behind main.

Live recovery is a maintainer action; I hold read-only cluster access by design and have changed
nothing.

Independent confirmation, and the exact minimal remedy

Confirming this from a separate session at ~19:2xZ — still live, now ~5h. I am not re-filing the
diagnosis above; this adds only the precise action and one blocker on the automated path.

Live state. ocirepository/flux-system in flux-system:

Ready          False  VerificationError
SourceVerified False  VerificationError
  failed to verify the signature using provider 'cosign keyless': no matching attestations:
  unsupported: multiple identities are not supported at this time

So the root source is not reconciling and all GitOps delivery to prod is halted, not merely
divergent.

The remedy is one patch, not a config rewrite. fluxinstance/flux carries 7 entries under
spec.kustomize.patches. Exactly one of them adds /spec/verify to the root OCIRepository, with
the three-entry matchOIDCIdentity cosign rejects. Removing that single entry (7 → 6) restores the
configuration main actually declares.

Verified against main rather than assumed — k8s/providers/hetzner/.../flux-instance/flux-instance.yaml
at origin/main is 149 lines and contains zero occurrences of verify (control: confirmed the
file exists and is non-empty first, so the zero is a real count and not a failed read). Its patches
target the kustomize-controller Deployment only.

Why not "fix the matcher to one identity" instead: that keeps verification on, but requires
picking which of the three subjects signed the artifact currently on latest. Guessing wrong leaves
prod exactly as stuck. Removing the patch returns prod to its declared state and is trivially
reversible; a corrected single-identity verify can then land through normal review.

⚠️ The automated recovery path is unavailable right now, independent of this defect. GitHub
Actions is in an active major outage (githubstatus.com, opened 15:22Z; at 18:46Z "workflow runs
are still failing, and jobs may remain queued for an extended period or may time out"). So re-running
🩹 Heal Prod cannot succeed until Actions recovers — the two failures compound, and waiting for the
heal is not currently a recovery strategy.

That leaves a direct cluster edit as the only path that works now, which is outside my runtime
authority (mutation is denied to me, deliberately). I have not attempted it.

Disclosure: the candidate that applied this came from my own lane's PR #3001. Flagging that
plainly rather than reporting it as an anonymous defect.

The exact one-command remedy, and the measured blast radius

Third session on this at ~20:0xZ. The diagnosis above is right and I am not restating it. Two things
it stops short of: the command for the direct cluster edit it correctly identifies as the only
working path, and how far the stall has actually spread.

1. The self-lock is confirmed at the field-ownership level, not inferred

kustomize-controller owns spec.kustomize.patches on fluxinstance/flux (server-side apply,
last at 14:28:51Z), and flux-operator owns f:spec.f:verify on the root OCIRepository
(14:28:53Z, 2s later). That is the loop stated precisely: the field that has to change is written
only by the GitOps channel that the field itself breaks. It is not merely that the heal is
inconvenient right now — no reconcile can correct it from inside.

2. Blast radius: 3 of 11 Kustomizations, ~5.5h

Kustomization	Ready	Applied revision	Since
`apps`	False `DependencyNotReady`	`latest@sha256:82e0d0bf…`	14:28:42Z
`infrastructure`	False `DependencyNotReady`	`latest@sha256:82e0d0bf…`	14:29:02Z
`infrastructure-controllers`	Unknown `Progressing`	churning on `…e6039d0a`	19:54:06Z

The other 8 are Ready. Workloads are still serving — this is a stalled delivery pipeline, not a
production outage — but the whole apps + infrastructure tree has been frozen on a pre-incident
revision for about five and a half hours, and nothing will move it without the edit below.

3. The command

patches currently has 7 entries; index 6 is the offender, verified live
(target={"kind":"OCIRepository","name":"flux-system","namespace":"flux-system"}, patch body
contains /spec/verify). The test op makes it fail closed rather than remove the wrong entry
if the array has shifted in the meantime:

kubectl --context admin@prod -n flux-system patch fluxinstance flux --type=json -p '[
  {"op":"test","path":"/spec/kustomize/patches/6/target/kind","value":"OCIRepository"},
  {"op":"remove","path":"/spec/kustomize/patches/6"}
]'

This converges rather than fights: with the patch gone, flux-operator drops spec.verify from the
root OCIRepository, verification stops failing, delivery resumes, and kustomize-controller then
reconciles the 6 patches that main actually declares. The manual edit and the declared state agree,
so there is nothing to undo afterwards.

Expected within a minute or two:

kubectl --context admin@prod -n flux-system get ocirepository flux-system \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
# want: Ready=True Succeeded / SourceVerified absent or True

Still not attempted by me — cluster mutation is denied to my runtime by design, and I am not
routing around that. This is a maintainer action. Actions remains in a critical outage as of
19:43Z, so 🩹 Heal Prod is still not an alternative.

4. Why the whole tree is stuck, not just the source — the loop is circular

Added after posting, from the Kustomization events. infrastructure-controllers is not merely
"churning": its health check is waiting on FluxInstance/flux-system/flux status: InProgress
and timing out at the full 25m, repeatedly (19:24:56Z, 19:52:01Z — so it re-enters the same
wait about every 27 minutes and can never exit it).

That closes the causality:

patches[6] puts an unsatisfiable verify on the root OCIRepository;
the OCIRepository fails verification, so it is not Ready;
the FluxInstance that manages it therefore reports InProgress;
infrastructure-controllers health-checks that same FluxInstance → 25m timeout, forever;
infrastructure → DependencyNotReady; 6. apps → DependencyNotReady.

Note lastAppliedRev is still …82e0d0bf while lastAttemptedRev is …e6039d0a — the new artifact
is attempted every cycle and never applied. So all three not-Ready Kustomizations have a single cause,
and the one patch removal above should release the entire chain in dependency order without any
further intervention.

It also means the tree is architecturally sensitive here: infrastructure-controllers health-checks
the FluxInstance that manages the source infrastructure-controllers is itself delivered from, so
any FluxInstance-level fault freezes the whole delivery tree in 25-minute retry cycles rather than
failing fast. That is a separate resilience concern from this incident and I am filing it separately
rather than widening this issue.

Status update — the remedy has INVERTED. Do not run the one recorded earlier.

main now carries the fix, so the previously-recorded remedy would delete a security control
rather than restore one. Re-measured live at 21:57Z.

What changed

#3001 merged at 21:50:44Z, carrying the single-entry, regex-alternation matcher.

	`matchOIDCIdentity` entries	State
`main` (after #3001)	1 (alternation inside one `subject`)	correct
live cluster	3 (one per workflow)	rejected by cosign

The earlier note on this issue said "main declares 6 patches, patches[6] is the sole offender, a
JSON-patch test+remove breaks the lock and converges — nothing to undo." That was true when it
was written and is false now. main declares 7 patches, and patches[6] is the correct verify
patch. Removing it today would strip cosign verification off the root source and diverge from main.

Correct remedy now: apply main. Either let a deploy run, or — if patching by hand — replace
patches[6]'s value with main's single-entry form. Never remove it.

Why the merge did not heal prod

The fix is on main and still cannot reach the cluster, because the only two paths to prod are:

ci.yaml → deploys on merge_group only
cd.yaml → workflow_dispatch only (manual, by design)

#3001's merge produced no merge_group deploy — GitHub Actions has been in major_outage since
16:33Z (still major at 21:30Z). The only post-merge run on main is a GitHub-managed CodeQL run.
So the self-lock in this issue still holds: the artifact in storage is frozen at the 14:28:42Z digest,
and the new one carrying the fix cannot be verified by the stale 3-entry config that is rejecting it.

Live blast radius (21:57Z) — narrower than at 19:04Z

Kustomization	State
`apps`, `bootstrap`, `flux-system`	Ready=True, converged on `…e6039d0a`
`infrastructure-controllers`	`Progressing` (health-checks the FluxInstance, which cannot converge)
`infrastructure`	`DependencyNotReady`

Workloads are serving. This is a stalled delivery pipeline, not an outage: apps reconciled as
recently as 21:44Z. What is actually blocked is new commits reaching prod.

Expected resolution — no manual action required

This clears on its own: once Actions recovers, the next PR through the merge queue runs a
merge_group deploy, which applies the FluxInstance directly (outside the broken OCI channel) and
restores the correct single-entry matcher. A manual cd.yaml dispatch does the same thing sooner, if
someone wants it back before then.

Not escalated further, on that basis — apps are up and the heal is automatic.

(Cluster mutation is denied to this agent by design; every figure above is from read-only reads and
I did not route around that.)

Status at 2026-08-06T23:3xZ — still locked, and the auto-heal did not arrive

The prediction recorded when this was filed — the next queued PR heals prod by itself once Actions returns — has not held. Actions recovered in practice around 22:1xZ (verified by green pull_request runs across three repos while the status page still read major_outage), and prod is unchanged nine hours in.

The reason is simply that no platform PR has merged since, so no merge_group event has occurred. There is no eligible one waiting either: the portfolio currently has no trusted non-draft PR anywhere, and the two open platform drafts in this lane both have CI wedged at head.

Verified current state

Declared on main (k8s/providers/hetzner/.../flux-instance.yaml, at 0a00b2cd — this issue's own fix, merged 21:50:43Z) — one matcher, alternation inside the subject regex:

matchOIDCIdentity:
  - issuer: '^https://token\.actions\.githubusercontent\.com$'
    subject: '^…/(ci\.yaml@refs/heads/gh-readonly-queue/main/.+|(cd|dr-rebuild)\.yaml@refs/heads/main)$'

Live on the cluster — three separate entries, which Flux rejects as a set:

Ready=False  VerificationError: no matching attestations:
             unsupported: multiple identities are not supported at this time

Consequence, unchanged since 14:28Z: infrastructure-controllers Progressing (health-checking a FluxInstance that can never go Ready), with infrastructure and apps DependencyNotReady behind it. Workloads are serving normally — every tenant Kustomization is Ready=True. This is a halted delivery pipeline, not an outage.

0a00b2cd is the only commit on main not yet delivered, so the corrective deploy has a one-commit blast radius.

The remedy, and why it is not mine to run

cd.yaml is the documented path for exactly this state — its own header describes it as covering "the rare direct-push-to-main case, which has no merge_group event to deploy". It carries the same validate-eks-authorization gate as the merge-queue deploy and shares the prod-deploy concurrency group, so it cannot race one.

gh workflow run cd.yaml --repo devantler-tech/platform --ref main

I attempted this and the agent runtime's permission layer denied the dispatch. That is a guard doing its job on a production deploy, so I have not tried to work around it — flagging it instead. This is the one step that needs a hand other than mine; everything up to it is done and verified.

Self-locking remains the underlying defect (a bad FluxInstance change can only be corrected through the channel it breaks), and the structural fix for that is #3006.

The heal is running — state verified live at 01:2xZ

GitHub Actions is operational again (githubstatus: all systems operational), which is what unblocked this. The sequence t1006 laid out has executed as designed:

fix(ci): accept the shared cosign matcher in the DR publication contract #3008 merged, so main passes its own publication contract again and stops reddening every pull request.
fix(rbac): enumerate the github-config tenant Role's managed-resource kinds #3004 is queued and its merge_group run is live — 🚀 Deploy to Prod has been in its Deploy to Production step since 00:50Z. fix(rbac): enumerate the github-config tenant Role's managed-resource kinds #3004 changes k8s/, so it satisfies the k8s: ['k8s/**'] path filter that gates the deploy job; fix(ci): accept the shared cosign matcher in the DR publication contract #3008 alone could not, touching only scripts/.

Not yet healed, as of this comment. The live root OCIRepository still carries 3 matchOIDCIdentity entries and reports Ready=False VerificationError: … multiple identities are not supported. main declares the corrected single entry with the three signers as a subject alternation. The live FluxInstance still shows the 3-entry patch, so ksail cluster update has not reached the apply yet.

infrastructure-controllers is Progressing; infrastructure and apps are DependencyNotReady. Every other Kustomization is ReconciliationSucceeded and workloads have served throughout — a halted pipeline, not an outage.

What settles it: the live FluxInstance patch dropping from 3 matcher entries to 1, then the root OCIRepository reporting Ready=True. Both are one read each:

kubectl --context admin@prod get fluxinstance -n flux-system flux -o json \
  | jq '[.spec.kustomize.patches[] | select(.patch | test("matchOIDCIdentity"))] | length'
kubectl --context admin@prod get ocirepository -n flux-system flux-system -o json \
  | jq -r '.status.conditions[] | select(.type=="Ready") | "\(.status) \(.reason)"'

No action is needed from anyone while the deploy runs. If it completes without the live matcher count dropping to 1, that is a new fact and this issue should carry it.

The merge-queue deploy cannot heal this, and now there is a measured run proving it

main is no longer the blocker: #3008 merged and both cosign validators now pass on df72a9ee
(verified by running them, not by reading CI). #3004 was then driven green and enqueued, on the
expectation — recorded in an earlier note on this issue — that its merge-queue deploy would apply
the corrected matcher directly and recover delivery.

That expectation was wrong, and the deploy disproved it. Run 31135949968, 🚀 Deploy to Prod,
failed at:

✗ reconcile oci source: flux-system: flux OCIRepository is still reconciling the triggered
  request - the pushed artifact was not served before the timeout

#3004 was evicted from the queue and production is unchanged: the live FluxInstance still declares
3 matchOIDCIdentity entries, and the root OCIRepository is still Ready=False
VerificationError.

Why no CI path can repair it

The deploy action delivers Kubernetes manifests by exactly one mechanism — publish to GHCR, then ask
Flux to pull:

Order	Step	Delivers
5	`📦 Publish evidenced manifests to GHCR`	the artifact
7	`🔁 Trigger Flux reconciliation`	fails here — waits on the broken `OCIRepository`
8	`⏳ Wait for the newly published Flux revision`	never reached
9	`🔄 Update cluster`	never reached — and it is Talos machine config, not manifests

So the only step that touches the cluster without Flux is Talos-only and sits downstream of the
failing reconcile. The FluxInstance itself is delivered through Flux (per #3006,
infrastructure-controllers applies the object that owns its own source), so a wrong FluxInstance
cannot be corrected by the channel it breaks. cd.yaml shares this same composite action, so the
manual path fails identically — it is not an escape hatch for this failure mode.

This is a genuine deadlock, not a slow convergence, and re-queuing #3004 will reproduce it exactly.

What actually breaks it open

One of:

A direct cluster write — patch the live FluxInstance so matchOIDCIdentity carries the
single alternation entry main already declares. Agents are denied cluster mutation by design,
so this is a maintainer action.
A deploy-path change — apply the FluxInstance (or the root OCIRepository's verify block)
directly, before the reconcile gate, so a bad matcher is self-correcting rather than
self-locking. This is the durable fix and belongs on this issue.

Option 1 restores delivery immediately; option 2 stops it recurring. They are complementary.

Impact remains a halted pipeline, not an outage — apps, infrastructure and
infrastructure-controllers are not reconciling, while every deployment stays at its desired
replica count and workloads serve normally.

Update — the deploy failed, and the heal is tracking the same profile

#3004's merge_group 🚀 Deploy to Prod ran 55 minutes in its Deploy to Production step and then failed; #3004 was evicted from the queue and is back to OPEN and CLEAN. 🩹 Heal Prod picked up automatically at 01:26:12Z and re-deploys main, which carries the corrected single-matcher config — so that job alone can heal production without #3004 merging.

It has now been in its own deploy step for ~35 minutes with no change to the cluster: the live FluxInstance patch still declares 3 issuer entries, the root OCIRepository still reports matchers=3 / VerificationError, and apps, infrastructure and infrastructure-controllers are still not Ready.

Two readings, and they are worth separating rather than assuming:

The apply happens late. If ksail cluster update applies and then waits on health, an unchanged cluster at 35 minutes is expected and the fix lands near the end.
The deploy waits on something that cannot converge while the OCI channel is failing — the self-locking shape already described here. Then the heal times out exactly as the deploy did, and the automatic recovery path does not exist.

The 55-minute failure is what makes (2) worth taking seriously: that is a wait, not a crash. The deploy job's log settles it, and it cannot be read until the run completes (--log-failed refuses on an in-progress run) — run 31135949968, job 92735412751. That is the first thing to read next, before any remedy is chosen; the remedy for (1) is patience and the remedy for (2) is not.

Workloads have served throughout. Nothing here needs a human yet — the log is available to the next run.

Live recurrence, measured this tick (2026-08-07)

This scenario is active right now, and the loop ran twice in one night on the same PR (#3004):

attempt	`merge_group` run	window (UTC)	outcome
1	`31135949968`	00:49 → ~01:26	`🚀 Deploy to Prod` / step `🚀 Deploy to Production` failed → PR evicted → `🩹 Heal Prod` also failed
2	`31140471256`	02:13 → 02:49	same job, same step, failed again → evicted again → heal running at time of writing

So the heal-shares-the-failing-path thesis in this issue is confirmed on attempt 1: the deploy failed,
and the heal that exists to restore prod failed with it.

Why this is worse than a stuck PR

Prod is currently pinned on undeliverable state. The root flux-system OCIRepository is failing
verification:

False VerificationError: failed to verify the signature using provider cosign keyless:
no matching attestations: unsupported: multiple identities are not supported at this time

The live object carries 3 matchOIDCIdentity entries; cosign accepts exactly one and fails
closed for all. main already carries the corrected 1-entry form. The only path that applies it is
the merge-queue deploy — which is the thing that will not complete. That is the self-lock this issue
describes, closed into a loop.

infrastructure-controllers → infrastructure → apps are consequently stalled. Workloads are
serving (all six tenant Kustomizations Ready=True, no unhealthy pods) — this is a halted delivery
pipeline, not an outage.

What is NOT yet established

Attempt 1 surfaced Error: create website devantler.tech failed: 500 during umami website
provisioning, while umami CNPG was failing over. I am not claiming that is attempt 2 as cause.
Umami is healthy again (app pods Running, DB endpoints populated, provisioning CronJob Completed
seconds before this comment), so "transient failover" does not explain a second failure 80 minutes
later. Attempt 2 logs are withheld while the run is in progress
(run 31140471256 is still in progress; logs will be available when it is complete), so its cause is
undiagnosed — deliberately left open rather than guessed.

Next concrete step: once run 31140471256 completes, pull the 🚀 Deploy to Prod job log
(job id 92749257163) and root-cause the step before any further queue attempt. #3004 has been
marked DO-NOT-RE-QUEUE in its body so a pentad-clean sweep does not spend another ~36 minutes on it.

Attempt 2's cause is already decided by live state — and no deploy path can fix this

Building on the 03:03Z comment, which left attempt 2 undiagnosed pending its log. The log is not
needed, and waiting for it will cost another run: the cluster carries a decisive control taken at
02:5xZ, while attempt 2's own reconcile was in flight.

The control

lastHandledReconcileAt: 2026-08-07T02:51:41Z   <- attempt 2's reconcile request WAS handled
ArtifactOutdated=True  NewRevision  @02:51:48Z <- Flux SAW attempt 2's freshly published artifact
Ready=False  VerificationError      @02:51:48Z <- ...and refused it anyway
artifact.lastUpdateTime: 2026-08-06T14:28:42Z  <- stored artifact still the one from 14:28Z

A newly published, correctly signed, fully attested artifact from the current pipeline was seen and
rejected. So the cause is not the signature, not the umami 500, and not transient — those
can be closed out. It is the matcher count, and it is deterministic:

unsupported: multiple identities are not supported at this time

repeated once per entry. The live object carries 3 matchOIDCIdentity entries; cosign accepts
exactly one and fails closed. No signature can satisfy a 3-entry matcher, so every attempt fails
identically at the same step. Attempt 3 would too.

Why this is closed, not merely stuck

The live FluxInstance that renders those matchers is labelled

kustomize.toolkit.fluxcd.io/name: infrastructure-controllers

— it is owned by a Kustomization delivered through the very OCIRepository it configures. So the
corrected 1-entry form on main reaches the cluster only via a channel that cannot open until the
corrected form is already applied. That is a closed loop with no internal exit, which is why both the
deploy and the heal fail: neither writes cluster-side verification config. infrastructure-controllers
is Progressing; infrastructure and apps are DependencyNotReady. Workloads keep serving.

The remedy recorded in earlier ticks is void

Earlier ticks recorded gh workflow run cd.yaml as the remedy. It cannot work: cd.yaml line 127
uses the same ./.github/actions/deploy-prod composite as ci.yaml, so a manual dispatch runs the same
publish -> reconcile -> timeout and fails identically after ~35 minutes. Nothing in
.github/workflows/ breaks this loop. Please do not spend a run on it.

What actually clears it

One out-of-band cluster write, reducing the live matcher list to the single alternation entry main
already declares. It is convergent, not divergent: once the OCIRepository verifies,
infrastructure-controllers reconciles the FluxInstance from Git to that same value.

This is outside the agent permission boundary (cluster mutation is hard-denied, verified this tick on a
server-side dry-run no-op), so it needs the maintainer. Raised on the private channel with the exact
command; deliberately not reproduced here.

The durable fix

A delivery channel's own trust configuration must not be delivered solely through that channel. Filing
that separately as the design defect — this issue stays on the heal-shares-the-failing-path thesis,
which attempt 1 confirmed.

Correction / narrowing of my previous comment — prod is NOT on never-merged code

My comment above described the loop as active and cited the heal failing with the deploy. That part
stands for attempt 1. But I should be precise about the part that gives this issue its title, because
I verified it directly and the worst case has not materialised:

# live prod, 03:07Z — the github-config tenant Role
groups=repo.github.m.upbound.io,team.github.m.upbound.io,actions.github.m.upbound.io,
       enterprise.github.m.upbound.io,github.m.upbound.io
  resources=*                       <-- WILDCARD = main's state

flux-system lastAppliedRevision = latest@sha256:e6039d0a…   (the old pinned artifact)

The wildcard is main. #3004's enumerated-kinds Role is not live. So the two failed candidate
deploys did not leave prod carrying never-merged code on this occasion — either the deploy failed
before applying that manifest, or the heal reverted it.

Why this matters for triage: it separates two failure modes this issue can produce, which have
very different urgency:

Prod left on never-merged code — the severe case in the title. Not currently happening.
Delivery halted while prod sits on the last-good revision — what is happening. Workloads
serve; new changes cannot land.

We are in (2). That is a stuck pipeline, not a correctness or security exposure, and it should not be
escalated as though the tenant Role had been silently narrowed or widened in production.

Attempt 2 remains undiagnosed — run 31140471256 is still in progress at 03:07Z (the 🩹 Heal Prod
job has been running 18 min), so its logs are still withheld. Next step is unchanged: pull job
92749257163 once the run completes.

Attempt 2 is now fully diagnosed — the logs became readable, and they close the last open question

My earlier comments had to reason around a log GitHub withholds while a run is in progress. Run
31140471256 (merge-group attempt 2 for #3004) has now completed, so I read both failing jobs
directly. Nothing here changes the remedy; it removes the remaining guesswork about why.

The deploy did not crash — it waited, then was refused

Job 92749257163 (🚀 Deploy to Prod, 02:13:00Z → 02:49:02Z). The publish half succeeded
completely: stage → cosign sign → SBOM → SLSA provenance attest → verify evidence → promote to
latest all passed by 02:15:58Z. It then entered 🔁 Trigger Flux reconciliation at 02:16:00Z and
sat there for exactly 30 minutes before failing at 02:46:03Z with:

✗ flux-system   VerificationError · failed to verify the signature using provider 'cosign keyless':
                no matching attestations: unsupported: multiple identities are not supported at this time
✗ reconcile oci source: flux-system: flux OCIRepository is still reconciling the triggered request

So a freshly published, signed, attested artifact was served and rejected. That rules out the
transient explanations I had been carrying: it is not a signing failure, not umami, not a flake.
Attempt 1's 55-minute failure has the same shape.

`ksail cluster update` was never going to help — it is doubly gated and aimed at the wrong object

I had been treating the suppressed 🔄 Update cluster step as a possible route to correcting the
FluxInstance. It is not, for two independent reasons:

Both of its conditions were false. Its if: requires
cilium_rollout_gate.outputs.active != 'true' (the gate has been active 12 days, threshold 7)
and wait_flux_revision.outcome == 'success' (that step was skipped, because the reconcile
above it had already failed). Either alone skips it.
It does not own the FluxInstance anyway. The live object carries
kustomize.toolkit.fluxcd.io/name: infrastructure-controllers, so it is applied by that
Kustomization — not by ksail cluster update, which syncs Talos machine config, as its name says.

The heal cannot recover this class, and now there is direct evidence

Job 92754831196 (🩹 Heal Prod, 03:24:34Z) failed at 03:21:41Z with the byte-identical error.
That is the point this issue's title makes, and it is now demonstrated rather than inferred: the heal
re-deploys main through the same OCIRepository that is refusing every artifact. The safety net
shares the failure mode of the thing it is meant to catch, so it cannot fire when it is most needed.

Current live state (read directly, 03:5xZ)

ocirepository/flux-system      matchers=3   Ready=False/VerificationError
fluxinstance/flux              generation=26  Ready=False/ReconciliationFailed
                               patch still carries 3 matchOIDCIdentity entries
kustomization/infrastructure-controllers   Unknown/Progressing
kustomization/infrastructure               False/DependencyNotReady
kustomization/apps                         False/DependencyNotReady

main already declares the corrected single alternation matcher. Workloads continue to serve
throughout — this is a delivery freeze, not an outage of the running apps.

Why this is closed, and what breaks it

The FluxInstance renders the OCIRepository's matchOIDCIdentity list. The FluxInstance is applied by
the infrastructure-controllers Kustomization. That Kustomization is delivered through the
OCIRepository. A config error in the trust policy therefore disables the only channel that could
deliver its own correction, and every CI path — cd.yaml, the merge-group deploy, and the heal —
enters through that same channel.

The break-glass is unchanged and is a single cluster write that replaces the three matcher
entries with the one-entry alternation main already declares. Once the OCIRepository verifies, Flux
pulls main, and the FluxInstance reconciles itself back into a consistent state — no further
manual step. I do not hold cluster-mutation permission (verified again this run: denied at the
command class, including a server-side dry-run no-op), so this remains the maintainer's one action.
It was sent privately with the exact command.

I will keep re-reading the matchers at the top of each run and will report the moment they change.

@devantler

Resolved — the in-place fix is applied and verified live

@devantler applied the single cluster write. Verified against the live cluster, not inferred:

Check	Value
`ocirepository/flux-system` `matchOIDCIdentity` entries	1 (the alternation form `main` declares)
`SourceVerified`	True — `Succeeded`
`Ready`	True — `Succeeded`

And it self-healed exactly as predicted — all five Kustomizations are now Ready=True at the same
new revision latest@sha256:e996cd9b…, up from the stale e6039d0a… they had been pinned to:

apps                        ready=True   e996cd9b
bootstrap                   ready=True   e996cd9b
flux-system                 ready=True   e996cd9b
infrastructure              ready=True   e996cd9b
infrastructure-controllers  ready=True   e996cd9b

So the divergence is closed in the convergent direction: flux-operator re-rendered the source from
main, and cluster and Git now agree. Delivery is restored, prod is no longer running never-merged
code, and the merge queue is unblocked (#3004 re-queued; its stale DO NOT RE-QUEUE body note has
been cleared).

Closing the incident, not the lesson. The structural loop this exposed is real and unfixed: the
root source's trust config is still delivered through the source it gates, and the heal still shares
the deploy path, so the same deadlock recurs on any future bad value. That recovery gap is now
tracked as a focused, actionable issue — #3014 — with a decisive acceptance test. Prevention
(catching a non-verifying matcher before it ships) remains #3007.

A failed merge-group deploy leaves prod on never-merged code, because the heal shares the deploy path and fails with it #3005

Description

🔴 Production delivery is still halted — one cluster write unblocks it

Why nothing automatic can fix this

Evidence — the originating incident

Why this is a distinct defect

Impact

Expected behaviour

Acceptance criteria

Size

Activity

coderabbitai commented on Aug 6, 2026

devantler commented on Aug 6, 2026

Escalation — the live instance is worse than "prod runs never-merged code": it halted delivery

Two things this sharpens about the defect filed here

Consequence for the acceptance criteria above

devantler commented on Aug 6, 2026

Independent confirmation, and the exact minimal remedy

devantler commented on Aug 6, 2026

The exact one-command remedy, and the measured blast radius

1. The self-lock is confirmed at the field-ownership level, not inferred

2. Blast radius: 3 of 11 Kustomizations, ~5.5h

3. The command

4. Why the whole tree is stuck, not just the source — the loop is circular

devantler commented on Aug 6, 2026

Status update — the remedy has INVERTED. Do not run the one recorded earlier.

What changed

Why the merge did not heal prod

Live blast radius (21:57Z) — narrower than at 19:04Z

Expected resolution — no manual action required

devantler commented on Aug 6, 2026

Status at 2026-08-06T23:3xZ — still locked, and the auto-heal did not arrive

Verified current state

The remedy, and why it is not mine to run

devantler commented on Aug 7, 2026

The heal is running — state verified live at 01:2xZ

devantler commented on Aug 7, 2026

The merge-queue deploy cannot heal this, and now there is a measured run proving it

Why no CI path can repair it

What actually breaks it open

devantler commented on Aug 7, 2026

Update — the deploy failed, and the heal is tracking the same profile

devantler commented on Aug 7, 2026

Live recurrence, measured this tick (2026-08-07)

Why this is worse than a stuck PR

What is NOT yet established

devantler commented on Aug 7, 2026

Attempt 2's cause is already decided by live state — and no deploy path can fix this

The control

Why this is closed, not merely stuck

The remedy recorded in earlier ticks is void

What actually clears it

The durable fix

devantler commented on Aug 7, 2026

Correction / narrowing of my previous comment — prod is NOT on never-merged code

devantler commented on Aug 7, 2026

Attempt 2 is now fully diagnosed — the logs became readable, and they close the last open question

The deploy did not crash — it waited, then was refused

ksail cluster update was never going to help — it is doubly gated and aimed at the wrong object

The heal cannot recover this class, and now there is direct evidence

Current live state (read directly, 03:5xZ)

Why this is closed, and what breaks it

devantler commented on Aug 8, 2026

Resolved — the in-place fix is applied and verified live

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

`ksail cluster update` was never going to help — it is doubly gated and aimed at the wrong object