Nothing alerts on an unhealthy Crossplane provider fleet, so a total outage read green for 18 hours #3457

🤖 Generated by the Agentic Engineer

Evidence

A total Crossplane outage ran for more than 18 hours while every surface a run inspects read green.

#3454 went live at the 2.3.4 → 2.4.0 core bump: a duplicate Provider for one package source poisoned the Lock's dependency graph, and every provider in the cluster went Healthy=False. Measured live during the incident:

• All 5 Provider objects Healthy=False, each with the same message — cannot initialize dependency graph from the packages in the lock: node xpkg.upbound.io/upbound/provider-family-aws already exists.
• Zero provider pods in crossplane-system — only crossplane and crossplane-rbac-manager were running.
• Live AWS managed resources depending on that fleet: role/eks-ci, policy/eks-ci-smoke-boundary, and openidconnectprovider/github-actions — the last of which is what lets GitHub Actions authenticate to AWS at all.

And simultaneously, throughout:

• flux-system/infrastructure — the Kustomization that owns those Providers — reported Ready=True, "Applied revision: …".
• Every GitHub surface was green. Two consecutive scheduled runs surveyed the portfolio and recorded nothing_on_fire.

The outage was found only because a run happened to query the cluster directly.

Why the existing coverage does not reach this

docs/dr/alerting.md states the design: the four top-level Kustomizations wait: true on their children, "so a failed controller / app / HelmRelease surfaces here as its parent going NotReady". That is exactly what did not happen. wait: true health-gates on kstatus, and kstatus has no readiness rule for a Crossplane Provider — the same gap infrastructure's own comment already documents for KRO's ResourceGraphDefinition and works around with an explicit healthCheckExprs entry. With no rule, a Provider is not treated as unready; it is simply not gated at all.

So the parent never goes NotReady, and the documented escalation path never fires. This is the same class as #3099 (nothing alerts on a stuck Longhorn volume) and the CNPG gap that cron-job-cnpg-degraded-alert was built for — a resource whose degradation no existing signal covers.

Coroot's auto-inspections do not cover it either: there is no crashloop and no restart. There are simply no pods, and "a Deployment that should exist but does not" is not a thing an inspection on a running workload can see.

Hypothesis

An independent check that reads Provider health directly, on the same CronJob pattern as cnpg-degraded-alert, turns an 18-hour silent outage into a page within one tick.

Success signal

• Baseline: time-to-detection for All Crossplane providers unhealthy: Crossplane 2.4.0 auto-created a duplicate provider-family-aws, poisoning the lock dependency graph #3454 was ~15 hours to first human-visible signal, and it was incidental rather than produced by any monitor.
• Target: a provider that has been Healthy=False past the grace period pages within one CronJob interval.
• Guardrail: no alert during a normal provider upgrade, when a provider is briefly unhealthy while its new revision installs.

Smallest useful change

A crossplane-provider-unhealthy-alert CronJob in observability, modelled directly on k8s/bases/infrastructure/controllers/coroot/cron-job-cnpg-degraded-alert.yaml:

• lists providers.pkg.crossplane.io through the API server with a narrowly-scoped SA;
• alerts on any Provider whose Healthy condition has been False longer than a grace period;
• reuses the allow-coroot CiliumNetworkPolicy and the same SOPS-substituted webhook, and skips the reserved example.invalid host so local and CI stay quiet;
• carries kustomize.toolkit.fluxcd.io/substitute: disabled, for the same reason that CronJob does — an unescaped ${GRACE_SECONDS} is why the CNPG check had never once succeeded.

🔴 The grace period needs measuring first, and that CronJob's header says so

Its own comment warns explicitly:

Crossplane's Synced condition behaves the OPPOSITE way — it is rewritten every reconcile, which would make a grace period silently unreachable and the check a permanent no-op. Do not copy this pattern to a Crossplane resource without re-measuring.

That warning is about Synced on a managed resource. This check keys on Healthy on a Provider, which is a different condition on a different kind, so the warning neither applies nor is safely assumed away — it has to be measured.

First measurement, taken live during the incident (2026-08-29T15:09:53Z): all five Providers carried Healthy=False with lastTransitionTime stable at 14:51:40–14:51:42Z, i.e. unchanged for ~18 minutes across an active reconcile loop. That is consistent with a real transition timestamp rather than a per-reconcile rewrite, but it is one sample. Confirm with the sampling method that CronJob's header records — two samples ~45s apart, checking that metadata.resourceVersion advances while lastTransitionTime does not — and take it against a healthy fleet as well, since a stuck provider may simply not be reconciling.

If the timestamp turns out to be rewritten, key the grace period on observed state across ticks instead, and say so in the manifest.

Acceptance criteria

• The Healthy-condition lastTransitionTime behaviour on a Provider is measured (both a healthy and an unhealthy fleet), and the result recorded in the manifest's header.
• A CronJob alerts on any Provider unhealthy past the grace period, independent of every Flux health gate — so relaxing a gate cannot silently remove this coverage.
• Proven in both directions against the real API shape: an unhealthy-past-grace fleet alerts, and a provider mid-upgrade inside the grace does not.
• The delivery path is proven — an undeliverable webhook fails the Job rather than dropping the alert silently, matching cnpg-degraded-alert.
• example.invalid is skipped, so local and CI stay quiet by design.

Explicitly out of scope

Adding a Provider entry to flux-system/infrastructure's healthCheckExprs. It looks like the obvious fix and it is the wrong one here: #3100 documents, with two incidents, that making that Kustomization strict wedges the merge queue for the whole platform. Detection belongs on an independent alert, exactly as cnpg-degraded-alert's header argues.

Rough size: small — one manifest plus its kustomization entry and tests, closely following an existing file.

Refinement (2026-10-17)

Moved from Icebox to Ready: #4981 names this as its smallest useful change, and it has the longest measured silence of any class there.

State on main today. No manifest reads providers.pkg.crossplane.io health. The pattern to copy is still k8s/bases/infrastructure/controllers/coroot/cron-job-cnpg-degraded-alert.yaml with its own ClusterRole and ClusterRoleBinding; two later alerts (cron-job-posture-staleness-alert.yaml, cron-job-stranded-volume-attach-alert.yaml) follow the same shape and can be read for the current conventions.

The first acceptance criterion is answered, with one check left. The 2026-08-29 comment shows from crossplane-runtime source that Condition.Equal ignores LastTransitionTime and SetConditions skips an equal write, so Healthy's transition time is real and a grace period keyed on it is reachable. That was read for the version running then. Before building, re-read the same two functions in the crossplane-runtime release used by the Crossplane version production runs now (from the released module source), and record the version in the manifest header. If they changed, key the grace period on state observed across ticks instead, as the body says.

Decisions kept: independent CronJob, not a healthCheckExprs entry on flux-system/infrastructure (see Explicitly out of scope); the RBAC grants get and list on providers.pkg.crossplane.io only; the alert names the provider and its condition reason, never package credentials or account identifiers.

Priority High, Effort Low, Service class Risk reduction. An unnoticed provider outage stops the platform's AWS resources from reconciling, including what lets GitHub Actions authenticate to AWS, and the measured silence was 18 hours. One manifest set closely following three existing ones, plus its tests and a controlled proof in both directions. No prerequisites. Part of #4981.