Coroot warns after successful health-reconciliation job retention cleanup #4537

🤖 Generated by the Agentic Engineer

Blocker: kubernetes/kubernetes#141319 | upstream | last-verified 2026-10-14: still open and unmerged against master, so no supported Kubernetes release carries the CronJob deletion fix yet; it must land in a supported release before deployment and live retention readback.

Why

Coroot raised a cleanup warning for a health-reconciliation job that had already completed successfully. Native production events show completion at 18:46:39 UTC on 2026-10-05, followed by one controller FailedDelete / NotFound event at 18:47:40. The exact referenced job is absent; 24 later runs completed successfully by the 19:12 readback.

Impact

The warning requires investigation even though that run completed its work. This is separate from the earlier deadline failure and does not establish that every run is healthy.

Root cause and delivery dependency

Fresh read-only evidence on 2026-10-06 reproduces the same sequence: a health-reconciliation Job completed at 05:20:41 UTC, then its CronJob reported FailedDelete / NotFound at 05:21:41. There is no Job TTL; concurrency remains Forbid, with one successful and three failed Jobs retained. Subsequent reconciliation continues successfully.

The controller's deletion helper emits the warning for every deletion error, including NotFound. The existing upstream issue describes the same stale-informer retention race, and upstream PR #141319 is still open. Its maintainer review requests treating NotFound as completed deletion, including removal from the active list, while preserving warnings for actual failures. The existing patch has not yet incorporated that requested behavior.

The observed lifecycle and code path match that upstream defect. The first deletion request's exact actor has not been established by these reads. Do not claim that historical events establish deployed clearance, create a duplicate contribution, change retention to hide the symptom, or suppress general cleanup errors. Delivery requires an accepted supported upstream release, reviewed platform integration, and fresh retention and alert readback.

Acceptance criteria

• Correlate the successful job identity, the deletion path, and the controller's subsequent NotFound response.
• Deliver a supported root-cause fix with a regression demonstrating repeated cleanup succeeds while real deletion failures remain visible.
• Verify normal production retention and fresh successful reconciliation after deployment.
• Verify a separate execution failure still raises its normal alert.