The deploy bridge loses its own sync lease mid-run, and each failure fences a node that fails the next deploy

Evidence

Three consecutive prod deploys failed the same way on 2026-08-10, each leaving prod-control-plane-2 cordoned with a drain fence that failed the next one closed:

Run	Job window (UTC)	Failure
`CD` 31416462549	18:06:34 → 18:16:15	`The GHCR synchronization lease was lost; refusing further cluster mutation.`
`pr-3066` deploy 31417048057	18:16:41 → 18:18:05	`Could not select Talos nodes requiring GHCR synchronization.` (fails closed on the fence above)
`pr-3066` heal 31417048057	18:18:51 → 18:31:15	`The GHCR synchronization lease was lost; refusing further cluster mutation.`

Each lease-loss run had already cordoned prod-control-plane-2 and written
platform.devantler.tech/ghcr-auth-drain-owner, then refused further mutation — correctly, since it can no longer prove it holds the lease. The fence survives, and the next deploy fails closed on it. That is a self-sustaining loop: every failure manufactures the precondition for the next failure.

What it is not

Ruled out by direct measurement rather than reasoning:

Not concurrent deploys. Job windows do not overlap, and the prod-deploy concurrency group is correctly applied to both cd.yaml (workflow level) and ci.yaml's deploy-prod job. The heal ran alone from 18:18:51 to 18:31:15 and still lost the lease at 18:31:07.
Not Flux overwriting the Lease. ghcr-auth-refresh appears in no manifest under k8s/, carries no Flux labels or annotations, and its only managedFields manager is kubectl-patch — the bridge itself.
Not a stale-holder takeover. Takeover of a non-empty holder is refused by design, so nothing could have seized it from a live holder.

Why it matters

The fail-closed behaviour is right: a process that cannot prove it holds the lease must not keep mutating nodes. The defect is that losing the lease is apparently reachable while running alone, and that the resulting fence has no automatic release — so a single occurrence wedges the lane until a human clears it, and the heal that is supposed to restore main re-creates the condition instead.

Where to look

renew_sync_lease CASes on resourceVersion, and assert_sync_lease_held fails the run when the holder no longer matches. A renewal that loses its CAS race (heartbeat versus a foreground renewal, or a renewal that overruns the 120s leaseDurationSeconds under a long step) would produce exactly this signature with no second actor involved. The heal ran for 12+ minutes, and the CD run for ~10, both well beyond a single lease duration.

Worth capturing the holder identity and leaseTransitions at each refusal so the next occurrence distinguishes "someone else took it" from "we failed to renew it".

Related: #3034 adds --fences, which reports these fences with liveness evidence and the exact release commands, and records the run reference in the holder identity so a holder can be resolved against the API instead of by timestamp correlation.

Blocker: the next production occurrence of the lease-loss failure, or 2026-10-24 if there is none | outcome | last-verified 2026-10-17: no recurrence; 0 of 226 failed CD, CI, heal and restore runs since 2026-10-07 carry the lease-loss message (the same search finds it in the 2026-08-10 run 31416462549). Close as not reproduced if that still holds on 2026-10-24

🔗 Related PRs

#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2793 - feat(cilium): activate homogeneous device selection [merged]
#2962 - fix(deploy): make rollout-gate suppression of machine-config sync visible and bounded [merged]
#2977 - fix(ci): retry the cordon claim when an unrelated Node write breaks its CAS test [merged]
#2983 - fix(ci): keep the claim conflict when the retry re-reads the node [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

The lease is not lost to a CAS race — the API endpoint disappears underneath the run

This answers the question left open above ("distinguishes 'someone else took it' from 'we failed to renew it'"). It is neither: the renewal never reaches an API server, because the staging step reboots the one control-plane node the CD kubeconfig points at.

Measured, CD run 31416462549

🔑 Stage Flux and consumer GHCR pull credential ran 18:06:50 → 18:16:13 (duration_ms=563738). Inside that single step:

Time (UTC)	Observation	Source
18:07:32	`secret/variables-base patched (no change)` — last successful mutation	job log
18:15:25.86 / .97 / 18:15:26.07	three × `couldn't get current server API group list … connect: connection refused`	job log
18:15:26.18	`The connection to the server … was refused - did you specify the right host or port?`	job log
18:15:32	`Warning Rebooted node/prod-control-plane-2`	cluster events
18:16:10	`The GHCR synchronization lease was lost; refusing further cluster mutation.`	job log

The client dialled exactly one address and never failed over — that address is prod-control-plane-2's. So the eight minutes of silence between 18:07:32 and 18:15:25 is the step doing its node work, and it ends by rebooting the node serving its own API connection.

It replicates on the heal run

The heal in the table above lost its lease at 18:31:07. The cluster recorded Rebooted node/prod-control-plane-2 at 18:30:28 — 39 seconds earlier. Two independent runs, same ordering: node reboot, then lease loss.

Two things this rules out

Not ksail cluster update. In this run cluster_update is outcome=skipped — the job had already failed. The fleet reboot came from the bridge's own reassert path, so this is a different mechanism from Every prod deploy rolling-reboots the whole fleet: ksail cluster update invalidates the GHCR node proof #3042, which attributes fleet reboots to cluster update. Both exist.
Not a lease-duration overrun or CAS race. No renewal contended; the API was refusing connections outright.

Why it is always `prod-control-plane-2`

Because it is the node the kubeconfig targets. It is not that this node is special to the drain — it is that rebooting it specifically is the one reboot that severs the run's own control connection, so that is the node the run dies on and therefore the node left cordoned with ghcr-auth-drain-owner. The self-sustaining loop described above is real; this is the step that starts each iteration.

How strong the claim is

Strong replicated correlation (2 of 2 lease losses immediately preceded by a reboot of that node), not a proven invariant — prod-control-plane-2 also rebooted at 18:47:57, followed by prod-control-plane-4 (18:51:40) and prod-control-plane-5 (18:54:47), and the deploy in flight across that window has so far survived it. So timing within the reboot matters; the mechanism does not require every such reboot to be fatal.

What this changes about the fix

Hardening lease renewal would not have helped — there was nothing to renew against. The candidates worth weighing are, in rough order of how much they address the cause rather than the symptom:

Give the API a stable endpoint (Give the prod k8s API a stable endpoint (VIP/LB) so CP-node recreates don't break kubeconfig #2120 / chore(prod): verify the stable API endpoint during control-plane replacement #2764). A VIP or LB in front of the three control-plane members means rebooting one member does not sever the connection at all. This also fixes the class, not just this instance.
Order node selection so the kubeconfig's own endpoint node is drained last, and re-verify reachability before continuing — turns a fatal mid-run severance into a final, expected one.
Do not reboot when the credential is unchanged (Every prod deploy rolling-reboots the whole fleet: ksail cluster update invalidates the GHCR node proof #3042's direction 3) — removes most of these reboots entirely, and with them most of the exposure.

(1) and (3) are independently valuable and neither blocks the other. Releasing the fence on abort (#3034) remains correct regardless — it bounds the damage rather than preventing it.

#3072 adds the missing datum: the Lease's holder, same_holder, transition count, renew time, duration and resourceVersion, printed at the moment a renewal is refused. same_holder=true means renewal failed; a changed holder with a bumped transition count means it was taken. Nothing today separates those, which is why this issue has no cause.

One hypothesis is now excluded by test, not by reasoning. I proposed that the heartbeat latches on its first renewal failure — : >"${sync_lease_lost_file}" with no retry, and assert_sync_lease_held never clears it — and that the existing recovery (recover_sync_lease_heartbeat_after_transport_interruption) is wired into only two call sites, both in the drain path, while the prod failures occurred during credential staging.

Injecting transient lease-read failures through the fake harness refuted it: the run succeeded with the one-strike code in place, and the ablation of a candidate fix passed too, which made the would-be regression test worthless. So a transient read failure is survivable on main, and the latch is not the cause on its own.

The still-unexplained observation is that both failing runs lost the Lease while running alone — no overlapping deploy job, no Flux ownership of the object (kubectl-patch is its only managedFields manager), and takeover of a non-empty holder is refused by design.

New occurrence (2026-08-16), reached by a different trigger

The fence-residue half of this issue recurred tonight, but not via lease loss. Recording it because
it widens the trigger set and narrows one claim in the original write-up.

Run 31976321946 — merge_group CI for pr-3171, 22:25:59Z. 🚀 Deploy to Prod died inside
stage_flux_ghcr_auth at 22:39:44Z:

🩹 Heal Prod then failed at 22:42:02Z on the residue that refusal left behind:

…ending in Could not select Talos nodes requiring GHCR synchronization. and
Could not safely release the GHCR synchronization lease. — the exact fail-closed pair this issue
describes, so the consequence is unchanged even though the cause is new.

What is new: the guard fires on ordinary autoscaler churn

The fenced node is autoscale-cx43-…, i.e. cluster-autoscaler-managed. Its identity changing while
the transaction waited for post-reboot lifecycle taints to clear is the autoscaler doing its job,
not a safety violation — but the guard cannot tell those apart, so it kills the transaction and leaves
the fence. Every prod deploy that overlaps a scale-down of a node it has claimed is exposed to this.

What is narrower than stated: residue on an ephemeral node self-releases

The original write-up says the fence "has no automatic release — so a single occurrence wedges the
lane until a human clears it". Measured tonight, that holds for a permanent node (the original
evidence fenced prod-control-plane-2) but not for an autoscaler node: the autoscaler deleted
autoscale-cx43-6c6b81dd28b85e09, and the annotation went with the node object.

Verified read-only at ~23:00Z, ~18 minutes after the heal failed:

7 nodes; the fenced node is gone, replaced by autoscale-cx43-7646e1143157fa02.
No node carries platform.devantler.tech/ghcr-auth-drain-owner.
Every node reports unschedulable=false — no residual cordon.
All 11 Flux Kustomizations Ready=True; root OCIRepository Ready=True.

Nothing was half-applied: the deploy died before publish_platform_manifest, which is recorded
skipped, so prod stayed on the last good manifest throughout.

So the blast radius is a function of which node gets claimed: a control-plane or static worker
wedges the lane until a human clears it, an autoscaler node clears itself on the next scale-down. That
makes the severity of an individual occurrence effectively random, which is worth designing against
rather than relying on.

Lane state

Not a standing wedge — merge_group is healthy at 10 of the 12 most recent runs, and the other
failure (pr-3090, 16:17Z) carries none of these markers, so it is an unrelated cause. pr-3171 has
been re-enqueued at the same evaluated head and is at queue position 1.

Related: this is the same fence-residue family as #3046 and #3065; #3034's --fences reporting would
have made tonight's residue self-describing.

Correction to the comment above — it did NOT stay self-released, and the lane is now wedged

My earlier note reported that the residue self-cleared when the autoscaler reaped the fenced node.
That was true of the node annotation, and only of it. A killed transaction leaves residue in two
places, and the Lease outlives the node.

Re-enqueuing pr-3171 on the strength of the clean node check produced a second eviction (run
31978017291, 23:02Z), this time on the other half:

Lease flux-system/ghcr-auth-refresh — holderIdentity
4384f07cc5a864b0-gh31976321946.1-2690-25297, renewTime 2026-08-16T22:42:05Z,
leaseDurationSeconds 120, leaseTransitions 240. The holder embeds run 31976321946, which is
completed/failure at 22:42:11Z, and no platform run is in progress — so the holder is provably
dead and the lease has been held ~27 minutes past expiry.

🩹 Heal Prod failed the same way in the same run, so main cannot deploy either. This is the
self-sustaining loop this issue describes, reached from the autoscaler-churn trigger rather than from
lease loss.

So the "no automatic release" claim in the original write-up is right after all — my correction
above narrowed it too far. The right statement is: the node fence can self-release when it lands
on an ephemeral node, and the Lease never does. Since the Lease alone is sufficient to wedge the
lane, an occurrence is wedging regardless of which node was claimed. The apparent recovery was the
less important half clearing.

Practical note for whoever automates the check: verifying node annotations and cordons is not
sufficient evidence that a fence has cleared. Lease flux-system/ghcr-auth-refresh has to be read
too, and --fences (#3034) is the thing that would have reported both together.

Escalated to the maintainer for the release, since it is a cluster mutation. pr-3171 is parked
rather than re-queued.

New occurrence 2026-08-23 — and it is the discriminating measurement this issue asked for

This issue closes with "worth capturing the holder identity and leaseTransitions at each refusal so
the next occurrence distinguishes 'someone else took it' from 'we failed to renew it'." That occurrence
happened today, --fences (from #3034) supplied the holder identity, and the answer is neither —
it is a third mode this issue does not yet describe.

What happened

merge_group run 32646748943 (PR #3320), job 🚀 Deploy to Prod:

Time (UTC)	Event	Source
14:54:24.00	lease acquired	`.spec.acquireTime`
14:56:33.34	`✅ Synchronised every existing consumer and refreshed root Flux GHCR auth from Git/SOPS.`	job log
14:56:33.00	lease renewed (written 14:56:34Z)	`.spec.renewTime`, `managedFields`
14:56:35.19	`##[error]Could not safely release the GHCR synchronization lease.`	job log
14:57:07.15	`🩹 Heal Prod` refused: "Another GHCR synchronization transaction holds the synchronization lease"	job log

Why this is a different mode from the 2026-08-10 evidence

The three runs above reported The GHCR synchronization lease was lost — the holder had moved away.
Today the opposite is true, and it is stronger evidence:

The deploy's actual work SUCCEEDED. The success line precedes the failure by 1.85 s. Nothing was
left half-applied; this is purely an exit-path defect.
The holder never changed. The lease still reads
holderIdentity: 4384f07cc5a864b0-gh32646748943.1-2831-12212 — that run, hours after it ended.
There was provably no second actor. leaseTransitions: 534, and the only managedFields manager
is kubectl-patch — so the "not Flux, not concurrent deploys" findings above still hold, and here
they are not needed as an argument: nobody else touched it.

So the failure is release losing its CAS race against its own heartbeat, not a renewal losing one.
The timing fits precisely: the heartbeat wrote at 14:56:34Z, and the release failed ~2 s later, which is
exactly what a test op on a resourceVersion captured before that write would do.

One more datum for the "renewal overruns the duration" hypothesis in Where to look: acquire → last
renew is 129 s against leaseDurationSeconds: 120. The lease had already expired by the time it was
renewed, so the renewal path is running late even on a ~2-minute step.

Why this raises the severity

This issue's Why it matters says a single occurrence wedges the lane until a human clears it. Today
shows the precondition is weaker than "a deploy fails": a completely successful deploy can wedge the
whole queue on its way out. Every platform PR entering the merge queue is evicted at the deploy step,
and 🩹 Heal Prod cannot restore main because it fails closed on the same fence — so the heal path is
unavailable in exactly the situation it exists for.

The fail-closed behaviour remains right and I am not proposing to weaken it. But the release path
failing to release is not a safety property — it is the bug, and it is now the more likely trigger of the
two.

Where I would look next

release_sync_lease almost certainly re-uses a resourceVersion captured at acquire time (or before the
last heartbeat) for its CAS test. The release only needs to prove we still hold it, which
holderIdentity alone establishes — testing resourceVersion as well makes the release fail whenever
the heartbeat has legitimately advanced it, which is precisely when the release is safest. Re-reading the
lease immediately before the release, or dropping the resourceVersion conjunct from the release CAS
while keeping the holderIdentity one, would both close it without loosening the exclusion guarantee.

Current state

The lease is still held and stale (last renewed 14:56:33Z, duration 120 s). Recovery is a deliberate
human step per docs/dr/runbook.md Scenario 11 and needs a cluster write I deliberately do not hold, so
I verified the holder is terminal (status=completed, conclusion=failure), confirmed --fences reports
exactly one fence held with nothing to release before it, and handed the maintainer the exact CAS-guarded
command it printed. PR #3320 was evicted on this, not on its own content.

Correction to my Where I would look next — and the actual mechanism

I then read the code, and the guess in my previous comment is wrong. Correcting it because it would
send someone to a fix that changes nothing.

release_sync_lease does not re-use a resourceVersion from acquire time. It kills the heartbeat,
re-reads the lease, and builds its CAS from that fresh read. So "re-read before releasing" is already
what it does.

The real asymmetry

Both paths face the same race. Only one tolerates it.

renew_sync_lease, on a failed patch, re-reads and forgives the conflict — with a comment saying so:

It returns success when the holder is still this transaction and the lease is still live.

release_sync_lease has no equivalent. One get, one patch, and any failure is return 1 —
fatal. The race the renew path is documented to expect is unhandled two functions away, even though
the release's safety requirement is weaker: it only needs to prove we still hold the lease, which
holderIdentity alone establishes. resourceVersion adds nothing there except a way to lose.

How the heartbeat still writes after being killed

kill + wait targets the heartbeat shell, not the kubectl child it may be blocked in. That
child survives and its write lands afterwards — inside the release's get → patch window.

Today's timings fit that exactly, at a 30 s heartbeat interval:

14:56:33.34 work completes; EXIT trap runs; heartbeat killed — but a renewal was already in flight
14:56:34 a kubectl-patch write lands (managedFields time; renewTime 14:56:33) — the orphaned
heartbeat, since the release's own patch never applied
14:56:35.19 release's CAS tests the pre-14:56:34 resourceVersion → conflict → fatal

Why a successful deploy reports failure

cleanup_refresh_work ends with if ((original_status == 0 && cleanup_status != 0)); then exit 1.
So a deploy that did all its work correctly still exits 1 on a release conflict — and leaves the lease
held, because the release it just failed was the thing that would have cleared it. That is both halves
of today's incident from one defect.

Fix direction, revised

Give release_sync_lease the same conflict tolerance renew_sync_lease already has: on patch
failure, re-read; if holderIdentity is still ours, retry the release against the fresh
resourceVersion, bounded. Only a holder that is no longer ours is a genuine refusal — and that
case needs no release anyway.
Drop the resourceVersion conjunct from the release CAS. The holderIdentity test already carries
the safety property, and it is immune to a benign heartbeat write.
Make the heartbeat's own in-flight write impossible to orphan, so ordering stops mattering.

(1) alone closes today's occurrence. (2) makes the class unreachable rather than merely retried. Neither
weakens the exclusion guarantee: a lease held by someone else still refuses, exactly as now.

I have not shipped this — platform's queue is currently wedged on the very lease this describes, and the
recovery is the maintainer's step. Flagging it as well-specified and ready for whoever picks it up.

chore: agent-claim #3071 nonce=1702a309266e3317c28ae46463676ae7

Shipping the release-path half — and what stays open

I have a reproduction and a fix for the mode this incident exhibited, in #3340. It is
deliberately Part of rather than Fixes, because it closes one of the two modes on this issue and
I do not want the other to be closed silently with it.

Reproduced, in the harness rather than by argument

The fixture models the event — a same-holder renewal landing between the release's read and its
patch — and lets the ordinary CAS comparison decide the outcome, so the failure is produced the way
an apiserver would produce it rather than injected. Against the current script it reproduces this
incident's exact signature:

✅ Synchronised every existing consumer and refreshed root Flux GHCR auth from Git/SOPS.
::error::Could not safely release the GHCR synchronization lease.

A transaction that did all of its work, reporting failure and leaving the Lease held.

What the fix does

Per the revised direction above: the release CAS now tests holderIdentity only, and a release that
fails for a passing reason is retried against a fresh read rather than abandoned.

A negative control pins the half that must not move: when the Lease genuinely belongs to another
transaction, the release still refuses and still leaves that holder untouched.

One deliberate asymmetry, so it does not read as an oversight: the operator-facing recovery command
--fences prints keeps its resourceVersion conjunct. There the holder is terminal, nothing is
racing the write, and pinning the version is what guarantees the Lease has not changed between the
report a human read and the command they paste.

What stays open

The 2026-08-10 mode — The GHCR synchronization lease was lost, holder moved away — is not
addressed by this and is a different defect.

One datum toward it, from this occurrence: acquire → last renew was 129 s against a 120 s
leaseDurationSeconds. The heartbeat sleeps its interval and then renews, so each renewal's own
API latency is added to the period rather than absorbed by it, and the interval drifts longer the
slower the API is. At 30 s against 120 s there is enough headroom that this alone should not lose the
lease — so I am recording it as a lead with a measurement behind it, not as the cause.

fix(deploy): treat an already-cleared lease as released, and bound th…

fix(deploy): sanitize the release diagnostic, and stop a deleted leas…

fix(deploy): keep the release backoff working for a decimal sync inte…

Measurement: the open mode (The GHCR synchronization lease was lost) has not recurred since the release-path fix.

I scanned every failed platform push, merge_group and workflow_dispatch run created on or after 2026-08-24: 81 runs, 0 with a lease-loss annotation on any failed job. A control confirms the method works. The same annotation query run against the original 2026-08-10 CD run 31416462549 finds the message.

The lease code is still live on main (scripts/refresh-flux-ghcr-auth.sh). #4080, merged today, fences nodes only when a real pull needs the new credential, which should also shrink how often the lease path runs at all.

So this stays open as a latent defect, with the heartbeat-drift lead above still the only datum. It is not an active incident. It will be re-checked at the next lease-loss occurrence, or on 2026-10-24, whichever comes first. If there is still no recurrence by then, closing it as not reproduced is the honest outcome.

The deploy bridge loses its own sync lease mid-run, and each failure fences a node that fails the next deploy #3071

Description

Evidence

What it is not

Why it matters

Where to look

Activity

coderabbitai commented on Aug 10, 2026

devantler commented on Aug 10, 2026

The lease is not lost to a CAS race — the API endpoint disappears underneath the run

Measured, CD run 31416462549

It replicates on the heal run

Two things this rules out

Why it is always prod-control-plane-2

How strong the claim is

What this changes about the fix

devantler commented on Aug 10, 2026

devantler commented on Aug 16, 2026

New occurrence (2026-08-16), reached by a different trigger

What is new: the guard fires on ordinary autoscaler churn

What is narrower than stated: residue on an ephemeral node self-releases

Lane state

devantler commented on Aug 16, 2026

Correction to the comment above — it did NOT stay self-released, and the lane is now wedged

devantler commented on Aug 23, 2026

New occurrence 2026-08-23 — and it is the discriminating measurement this issue asked for

What happened

Why this is a different mode from the 2026-08-10 evidence

Why this raises the severity

Where I would look next

Current state

devantler commented on Aug 23, 2026

Correction to my Where I would look next — and the actual mechanism

The real asymmetry

How the heartbeat still writes after being killed

Why a successful deploy reports failure

Fix direction, revised

devantler commented on Aug 23, 2026

Shipping the release-path half — and what stays open

Reproduced, in the harness rather than by argument

What the fix does

What stays open

devantler commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Why it is always `prod-control-plane-2`