A platform fault never reads green: every silent-failure class gets an alert #4981

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

Several production faults stayed invisible for hours because the component reported itself healthy, or because nothing watched the signal that failed:

• Nothing alerts on an unhealthy Crossplane provider fleet, so a total outage read green for 18 hours #3457 — a total Crossplane provider outage read green for 18 hours; nothing alerts on an unhealthy provider fleet.
• Cilium's Gateway API controller never started after an operator restart, so route changes stopped reaching the gateway #4198 (closed) — Cilium's Gateway API controller never started after an operator restart, so route changes silently stopped reaching the gateway.
• Nothing alerts on a Longhorn volume stuck unattachable, so a storage fault surfaced as a merge-queue outage #3099 — a Longhorn volume stuck unattachable surfaced only as a merge-queue failure, not as an alert.
• A HelmRelease whose post-renderer fails at reconcile still reports Ready #4376 — a HelmRelease whose post-renderer fails at reconcile still reports Ready.
• A scan result refused for being too large is only a warning in a log #4637 — a scan result refused for being too large is only a warning in a log.
• Kubescape runtime detection loads no rules: the node agent checks nothing #4968 — Kubescape runtime detection loads no rules, so the node agent checks nothing (tracked under roadmap: Kubescape security stack → 100% and hold (posture · CVE · runtime) #2447).

The opposite problem costs attention too: #4537 (Coroot warns after a successful cleanup) and #4329 (a remaining Coroot latency incident) are warnings a reader has to investigate before learning whether anything is wrong. In the 30 days to 2026-10-17, main took 20 fix(coroot) and 16 fix(observability) commits.

Audience and problem

The maintainer and the agent lanes who operate the platform, and every tenant whose service depends on it. When a component fails silently, the first signal is a stalled deploy or a user report hours later; when a warning fires for a healthy state, real warnings get read later or not at all.

Hypothesis

If each silent-failure class above gets an alert on the signal that actually failed, and each known false warning is fixed at its source, then the time from a platform fault to an alert falls from hours to minutes without raising the number of warnings that need no action.

Success signal

• Signal: detection delay, from the start of a production fault to its first alert, for faults in the classes above.
• Baseline: 18 hours (Nothing alerts on an unhealthy Crossplane provider fleet, so a total outage read green for 18 hours #3457) and about 15 hours (Cilium's Gateway API controller never started after an operator restart, so route changes stopped reaching the gateway #4198) for the two cases where the delay was measured.
• Target: every class above produces an alert within 30 minutes of the fault, demonstrated by a controlled trigger or a recorded real occurrence.
• Guardrail: warnings that resolve without action do not increase week on week; no existing alert is silenced or raised to a less sensitive threshold to meet this.

Smallest useful change

#3457: alert when the Crossplane provider fleet is unhealthy. It has the longest measured silence and a clear health signal.

Acceptance criteria

• Each child is fixed with a demonstrated alert, or closed with a reason.
• Every class above has a recorded detection delay under 30 minutes.
• The no-action warning count over the last four weeks is recorded and has not grown.

Size: medium to large — six children, each small to medium.