Deploy GHCR step can time out on the Flux policy pause without saying which condition never held #4873

🤖 Generated by the Agentic Engineer

Description

A merge-group production deploy failed in the GHCR credential step with
Flux did not acknowledge a stable pause of the image-verification policy owner.
The pull request (#4865, a one-line action bump) was removed from the merge queue at 04:55 UTC on
2026-10-14. The failure prints nothing else, so the run cannot show which condition never held.

Evidence

• Merge-group CI run 38510782818, job 🚀 Deploy to Prod, step 🔑 Reassert Flux and consumer GHCR pull credential (./scripts/refresh-flux-ghcr-auth.sh --reuse-runtime-proof …): failed after 322 s. Every earlier step passed (cluster update, API stability, Talos and kubelet read-backs).
• The message comes from the post-pause loop in pause_flux_policy_handoff. It needs, on two consecutive reads, the child fence owned and flux_policy_parent_is_stable (parent owned and no Reconciling=True condition) with an unchanged resourceVersion. On timeout it returns without printing the parent's or the child's conditions.
• The earlier merge-group deploy failures between 11 and 13 October had other causes (deadline exceeded, image-factory schematic registration, a stalled infrastructure-controllers). This is the first time this message appears in that window.
• The next queued deploy (chore(deps): bump step-security/harden-runner from 2.22.0 to 2.22.1 #4866, merge-group CI run 38513911689) passed the same step at 05:30 UTC, and chore(deps): bump devantler-tech/.github/actions/validate-naming from 7.10.3 to 7.10.4 #4865 was re-queued once. So the failure is intermittent; it is still a lost merge-queue slot (about 36 minutes) with no recorded cause.

Suspected cause (not proven)

The parent is checked for quiescence before it is suspended. The same function's comment says a suspended reconciliation "can leave its pre-suspension Reconciling=True condition behind", and the post-pause loop therefore does not use status as a clock for the child. The parent still has to be quiescent after suspension. If the parent started a reconciliation between the quiescence check and the suspend patch, its Reconciling=True would stay frozen, and the loop could never succeed. The log cannot confirm or rule this out.

Proposed repair and acceptance criteria

• On this timeout, print the parent's and the child's conditions, spec.suspend and resourceVersion (the same flux_policy_report_conditions used before acquisition), so the next occurrence names its cause.
• With that evidence, or a reproduction in the script's self-tests, decide whether a frozen pre-suspension Reconciling=True on the suspended parent should block the stable-pause proof. Add a self-test case for that state either way.
• The fix does not weaken the fence: a parent or child changed by another writer still resets the proof.

Refinement (2026-10-17)

What has changed since this was filed. #4877 (merged 2026-10-14) made the timeout print fixed counters: observations= unreadable= child_not_owned= parent_unstable= version_changes= (scripts/refresh-flux-ghcr-auth.sh, end of pause_flux_policy_handoff). It deliberately prints no conditions, versions or owner identities, because those can carry private data. So the first criterion is met in its safe form, and printing raw conditions, spec.suspend or resourceVersion is not wanted.

Why the suspected cause is likely, not just possible. The post-pause loop requires flux_policy_parent_is_stable, which includes flux_policy_parent_is_quiescent (no Reconciling=True). Before that loop the function suspends the parent and then calls restart_flux_kustomize_controller_for_handoff. A suspended object is not reconciled again, so a Reconciling=True condition written before the suspend can never be cleared by the new controller. If the parent began a reconcile between the pre-claim quiescence wait and the suspend patch, every post-pause observation counts as parent_unstable until the loop gives up.

Proposed change (one PR):

1. Add a self-test in scripts/tests/refresh-flux-ghcr-auth/ (the table in quiet_observation_test.go has a parent reconciling case that only checks the counter): a parent that is owned and suspended but carries a frozen Reconciling=True from before the suspend. Today it times out — the failing baseline.
2. After suspension, prove the parent the same way the child is proved: owned and suspended, with an unchanged resourceVersion across two consecutive reads. A Reconciling=True on the suspended parent no longer blocks on its own. A pre-suspension reconcile is already retired by the controller restart, which is why that restart exists.
3. Split parent_unstable in the timeout line into parent_not_owned and parent_changed, fixed names and counts only, so a recurrence names which predicate failed.

Acceptance criteria (replacing the list above):

• The frozen-Reconciling=True case fails before the change and passes after it.
• A parent changed by another writer (owner annotation, spec.suspend or resourceVersion moves) still resets the proof and still times out, covered by tests.
• The timeout line carries only fixed predicate names and counts; no condition message, version or identity is printed.
• go test ./scripts/tests/refresh-flux-ghcr-auth and the 🔐 Validate Flux GHCR credential bridge CI step pass.
• After merge, the next merge-group deploys pass the step; any recurrence names its predicate on this issue.

Priority Medium, Effort Low, Service class Risk reduction. One recorded eviction (about 36 minutes of queue time) and a mechanism that, once hit, can only time out. Small, test-led change inside one function; no prerequisites. Part of #4979.