Repository navigation
Production deploys fail intermittently at the data product rollout check and evict unrelated pull requests #4545
Description
🤖 Generated by the Agentic Engineer
What happens
The production deploy sometimes fails at its last step, the check that the data product pages are fully rolled out, with the single line {"complete":false,"failure":"rollout_changed"}. The pull request being deployed is then removed from the merge queue, and production is restored to main, although the pull request had nothing to do with the data product pages.
Seen twice on 2026-10-05, on two unrelated changes:
| Merge group | Change | Failed at |
|---|---|---|
| run 37323560971 (#4513) | a dependency bump | 15:51Z |
| run 37380959392 (#4531) | a policy-fixture check script | 22:55Z |
The other deploys in the same window (#4509, #4521, #4522, #4528) passed the same step. Each failure costs about an hour of queue time plus a restore deploy, and the queue was nine deep at the time.
Why it is hard to act on
scripts/verify-data-product-ui-rollout.sh takes a snapshot of the rollout, runs the public page checks, takes a second snapshot, and fails with rollout_changed when the second snapshot no longer passes or differs from the first in any field. It does not say which object or field differed, and it does not look again, so a change that settles a moment later is reported the same way as a real regression. In both runs above the step failed about 40 seconds in, well inside its 300 second limit.
The cause of the difference is not established. The snapshot includes the condition lists of the Flux objects it reads, so one candidate is an ordinary periodic reconcile starting between the two snapshots; that is a guess until a failing run names the field.
Acceptance criteria
- When the two snapshots differ, the step's output names which object changed (no secret values, cluster addresses or pod IPs in the public log).
- A difference that settles within the step's existing time limit does not fail the deploy: the step takes a fresh before/after pair and passes only on a pair that is identical and valid.
- A rollout that keeps changing until the time limit still fails, and a rollout that is wrong but stable still fails, both covered by the script's own tests.
- The cause of the two failures above is identified from a run that names the changed object, and recorded here.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
devantler commentedon Oct 6, 2026
Status 2026-10-06 19:55Z: the first three acceptance criteria shipped in #4546 (merged 05:30Z today); the fourth is waiting for a recurrence.
Nothing is left to build. The next step is to read the named change from the first run that prints one, or to close this as fixed-by-retry if none appears by 2026-10-20 (two weeks of deploys).
chore: agent-claim #4545 nonce=f8a3f6432cadd824fa109ace11efaa0f
devantler commentedon Oct 7, 2026
Why the fourth criterion could not be met by waiting, and what #4635 changes
The status above waits for "the first run that prints a named change". Reading the check as merged in #4546 shows that no passing run ever can: it names what moved only when it refuses the deploy after the third snapshot pair. A pair that differs and then settles on the retake leaves nothing in the log. So since the fix, the one event that would identify the cause is a failure the fix exists to prevent, and "no eviction since" says the retake works without saying what it is absorbing.
#4635 (in the merge queue) makes a passing run say so. When the check had to look again, its result line gains two fields: how many pairs were retaken, and the role and field names that moved across them. A deploy where nothing moved prints exactly what it prints today.
After it merges, the cause can be read from ordinary deploys. In each production deploy log, the line from the data product rollout check is one of:
"retaken":N,"settled":[…]: something moved and then held, and the names say what.The first deploy that carries the fields answers this issue's last criterion. If two weeks of deploys carry none, the retake is absorbing nothing, and the two failures on 5 October stay unexplained but no longer recur; closing on that basis is then reasonable. Nothing else is left to build.
devantler commentedon Oct 7, 2026
#4635 is merged (
e3911bf8, 2026-10-07 23:00Z) and the first two production deploys with it are read.No extra fields means nothing moved under the check in either deploy, so neither one names a cause yet. That is the expected common case, not a failure of the change.
What closes this issue, unchanged from the note above: the first deploy whose result line carries
retakenandsettlednames what moved, and answers the last criterion. If deploys through 2026-10-21 carry none, close it on that basis: the retake is then absorbing nothing and the two failures of 5 October did not recur. Nothing is left to build.devantler commentedon Oct 20, 2026
New occurrence, different signature — 2026-10-20 23:39Z, evicted #5129 (a Kyverno policy migration, unrelated to the data product pages).
{"complete":false,"failure":"public_contract_incomplete","phase":"public_contract","role":"descriptor_library","reason":"transport_failed"}What this tells us:
descriptor.js). The earlier fetches to the same host in the same pass succeeded.transport_failedis the catch-all:fetch()inscripts/verify-data-product-ui-rollout.shmaps only curl exits 6, 28, 35 and 60, and it drops curl's exit code and stderr (2>/dev/null). So the log cannot say whether this was a refused connection (7), an empty reply (52), a reset (56) or the 256 KiB size cap (63).descriptor.jsis 13,301 bytes, and three probes shortly after the failure (~23:41Z) returned 200 in under 0.3 s.Suggested follow-up for this issue (not done here): put curl's numeric exit code into the refusal reason, so the next occurrence names its cause, and retry a public fetch once when the connection itself fails (exits 7, 52, 56), staying within the existing deadline.
#5129 is being put back in the queue at the same head; its own checks and review are unchanged.
devantler commentedon Oct 21, 2026
Verification on the agreed date (2026-10-21): the fourth criterion is answered, so this issue is complete.
I read the rollout check's result line from every production deploy since #4635 merged (2026-10-07 23:00Z to 2026-10-21 02:00Z):
rollout_changedWhat moved, named by the one retake (run 38559308446, #4875, 2026-10-14 11:49Z,
"retaken":1):deployment-probe.status.availableReplicas,deployment-probe.status.readyReplicas,pods.items.status,replicasets.items.statusSo the object that changes under the check is the contract probe deployment (
data-product-controller-contract-probe): its pod's readiness flipped between the two snapshots and settled before the retake. It was not the Flux condition lists this issue guessed at. The retake absorbs it as designed, and no deploy has been evicted byrollout_changedsince #4546.Against the acceptance criteria:
Not this issue: the 2026-10-20 eviction of #5129 (
public_contract_incomplete,descriptor_library,transport_failed) happened in the public page checks, not the rollout comparison, and has its own cause. It is filed separately as #5170 so this record can close. The other refusals in the window (deadline_exceeded×3,invalid_expected_state×1) carry norollout_changedand belong to the changes those deploys carried.Closing as completed.