Repository navigation
Make offline UI rejection controls reach their intended predicate #5079
Description
🤖 Generated by the Agentic Engineer
Description
The complete post-merge cluster-policy replay at signed main 119968c6b55a7aad8d65f024136353fa02d45656 stopped in the offline UI-rollout regression because an unsafe fixture did not record an actual snapshot-predicate rejection. A second bounded run of the unchanged script stopped at a different fixture with the same missing proof. This blocks a clean local replay; it is not evidence of a production outage or a current hosted CI failure.
Steps to reproduce and evidence
Run the actual scripts/tests/test-data-product-ui-rollout.sh with its unchanged commands and case deadlines. Both observations were on a shared macOS host, using only the script's synthetic resources and fake commands:
- The complete policy-family replay stopped after 189.62 seconds at
ui-pod-unplanned-publisher:rejection did not reach snapshot validation. - A separate unchanged-script diagnostic run stopped after 291.87 seconds at
active-false-connector: the same missing rejection record.
Those controls have a three-second case budget. The missing journal proves that the intended predicate rejection was not observed; the actual failure phase still needs to be captured before assigning a cause. Resource contention/startup cost is a hypothesis, not a confirmed root cause. The earlier source PR's complete hosted policy job passed, so these samples must not be represented as a reproduced hosted regression.
Expected behavior
Every unsafe-fixture control exercises the real snapshot predicate and proves its intended rejection and cleanup. A setup/read/deadline failure must remain a failed or unknown observation, never substitute for that proof.
Acceptance criteria
- Capture the actual helper failure classification and elapsed phase before fixture cleanup, without changing the test's behavior or deadline.
- Separate shared-host capacity effects from a deterministic fixture/helper defect using a clean, bounded observation on the existing standard hosted runner.
- Repair the measured bottleneck or fixture scheduling at its root. Keep all cases, refusal assertions, cleanup checks and production behavior; do not skip controls, accept timeout as predicate rejection, or simply raise the three-second budgets.
- Record complete before/after fixture-family timings and output identities; prove independent failures still propagate and successful controls all execute.
- Pass the complete policy-family replay and actual hosted checks at the accepted head before claiming delivery.
Additional context
The split in #5069 changes job scheduling, not this helper or its fixtures. Keep that delivery and this newly observed verification limitation distinct. The broader CI lead-time theme is #4569; this reliability defect belongs to the production-gate reliability Epic #4979 as a real sub-issue. Size: Medium, including diagnosis, controlled measurement, repair and hosted verification.
Activity
devantler commentedon Oct 19, 2026
The bounded diagnostic now captured the actual failure classification before normal fixture cleanup. At the same unchanged signed main, a third run stopped after 219.85 seconds at
ksail-failed-product, again with no predicate-rejection journal:{"complete":false,"failure":"deadline_exceeded","phase":"initial_readiness"}The diagnostic only observed the last synthetic fixture's report during the existing EXIT cleanup. It did not change the helper, fake responses, test selection, assertions or three-second case deadline. The first two failures remain recorded, not erased by retries.
This confirms the third control exhausted its readiness budget before proving its intended predicate rejection. It does not establish whether the dominant cost is process startup, shared-host contention or another collection-phase expense, and it is still not a reproduced hosted failure. The next useful measurement is that phase's command/startup breakdown on the standard hosted runner, followed by a separately reviewed performance repair. No timeout increase or weaker negative-control acceptance is proposed.
devantler commentedon Oct 20, 2026
First hosted CI occurrence. The same missing proof now also fails a GitHub-hosted runner, so this is no longer only a shared-host replay problem.
⚖️ Validate Cluster Policieson platform#5142, head9fe8248157dc6f30d9bfd7ea0007435f8be4b41d, 2026-10-20 21:56Z.FAIL: active-foreign-probe-image: rejection did not reach snapshot validation, after the fixture's ownPASSline. The script started at 21:48:55Z and stopped at 21:56:04Z, about 7 minutes in.scripts/reconcile-talos-kernel-args/). It touches neither the UI rollout verifier nor its test, so the PR did not cause this failure.A fourth fixture now fails the same way (after
ui-pod-unplanned-publisher,active-false-connectorandksail-failed-product). That fits the three-second case budget running out under load rather than any single fixture's predicate. I am re-running the failed job once to unblock the dependency update; this observation stands either way.devantler commentedon Oct 20, 2026
Cause found by reading the code at
main3c931f4a. The test is starved of time; the verifier is not broken.What a mutation control has to do inside its budget. Every
mutate_caseandksail_mutationcontrol (124 of them) runs the verifier with--timeout 3. Before the verifier can judge anything, its firstcollecthas to finish about 26 bounded reads. Each read starts a watchdog subshell, the fakekubectland thejqwrapper, so the first judgement needs well over a hundred short-lived processes. Only then doescheck.jqrun, and only that run writes therejectionsrecord the test asserts.Why 3 seconds is really 2 to 3. The deadline is counted in whole seconds:
started_at=$SECONDSanddeadline=$((started_at + timeout_seconds)), andremainingrefuses as soon asleftreaches 0. If the script starts late in a second, it gets just over 2 seconds of wall-clock time.How that matches all four failures. On a busy host, the first
collectsometimes does not finish in that window. The verifier then correctly reportsdeadline_exceededin phaseinitial_readiness, which is exactly what the 2026-10-19 diagnostic captured. The test's first assertion accepts that, because the rollout was refused. Its second assertion then fails because the predicate never ran. Which fixture is hit is random (four different ones so far), which fits load, not any one predicate. The same design explains the ~7-minute job: each of the 124 controls keeps polling until its own deadline, so at least 124 × 2–3 s are spent waiting.Proposed fix (test-only, no change to the verifier). When a control fails with
deadline_exceededininitial_readinessand has written norejectionsrecord, the predicate was never reached. That run proves nothing either way, so retry that one control once with a larger budget, say 10 s. The strict assertion stays: the retry must still record a real predicate rejection. A control that reaches the predicate and is refused for any other reason still fails immediately. Raising every control's budget to 10 s instead would add about 15 minutes to every run.Acceptance criteria
deadline_exceeded,initial_readiness) is retried once with a larger budget, and the retry must record a rejection.