Report event source failures and node coverage in policy status

[JimBugwadia]

What this changes and why

An event source that fails to load or stops reading now appears in per-source availability/failure metrics and in affected monitor policies' EventSourcesAvailable condition. Previously, these failures only produced logs, making missing observations indistinguishable from quiet workloads.

Each daemon writes policy-relevant source status into its node shard. Cluster aggregation preserves failures across healthy-node updates and reports Unknown when an expected daemon has not reported. Expected nodes come from DaemonSet pod placement and desired count, including pending pods; node deletion and placement changes trigger recomputation. Monitor Applied reflects incomplete observation coverage without changing enforcement semantics.

Dependencies cover every producer: open/exec use openexec-observe, network/protocol use egress-observe, DNS uses dnsquery, and exec additionally uses exec-trace. Empty behaviors and rules add no dependency. Nonempty expressions conservatively retain dependencies across empty evaluations because later reevaluation can produce targets.

Reader readiness and retry transitions share one lifecycle callback; poll sources announce readiness only after a successful poll, and exec tracing reopens its reader on retry and synchronizes reader startup with concurrent closure. Initialization failures require a daemon restart. Status is last-reported availability, without a stale-daemon heartbeat detector; membership changes converge through Kubernetes controller/cache updates and the status flush interval.

Fixes #92.

How it was validated

• make build and make test pass, including race detection.
• make kind-install and make smoke-quickstart pass.
• make test-e2e-lsm passed both enforcement and syscall-storm suites on kind with BPF-LSM active (capability,bpf,landlock); daemon restart count remained zero.
• make lint-docs passes.
• Live kind checks: adding a temporary expected node produces Unknown; deleting it restores True after controller convergence. Removing BPF privileges produces source gauges of 0, initialization-failure counters, and EventSourcesAvailable=False / Applied=False. Restoring privileges returns source gauges to 1 and source availability to True. Temporary resources were removed and original privileges restored.
• Regression tests cover lifecycle readiness/failure/retry, healthy-node flush ordering, missing reports, policy behavior/mode changes, source recovery, and membership changes.
• Reader startup/close concurrency regression reproduced the race and passes with the fix. Fresh Linux exec-trace kernel tests for argv reporting and unadmitted-cgroup exclusion passed inside kind without skips; build, race-enabled unit tests, install, and quickstart passed again after the fix.
• Review-fix validation: all-producer dependency and poll-readiness race regressions; full build/test, kind install, quickstart, and BPF-LSM suites passed again. Live API checks verified omitted-mode defaulting, open and egress source projection, empty-rule exclusion, and conservative dependencies for a valid expression returning an empty list.

Generated artifacts

• make verify-crds and regeneration of deepcopy artifacts produce no drift.
• No BPF C or generated BPF artifacts changed.
• No new BPF program added.

Documentation

• docs/dev/DESIGN.md and development guidance updated.
• RuntimePolicy and metrics references updated, including operational limits.

Commits

• Every commit is signed off with git commit -s.

[Copilot]

🟡 Changes recommended

Unresolved source-dependency and monitor-policy classification issues can produce inaccurate coverage and Applied status.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR adds event-source lifecycle observability, per-node coverage aggregation, and monitor-policy status gating.

Changes:

• Adds readiness, failure, retry, and recovery metrics.
• Tracks expected DaemonSet nodes and aggregates source status.
• Updates APIs, CRDs, Helm resources, documentation, and tests.

Review findings:

• Critical (1 vote): Include open, network, and protocol poll sources in dependency coverage.
• Moderate (1 vote): Treat omitted modes and empty behaviors correctly when deriving dependencies.
• Nit (1 vote): Clarify the openexecmgr.New failure message regarding filename coverage.
• Nit (1 vote): Use compiler.IsObserveMode for mode classification.

File summaries

File	Summary
pkg/runtimeevent/iface.go	Defines source lifecycle states and callbacks.
pkg/runtimeevent/iface_test.go	Tests readiness callback behavior.
pkg/metrics/metrics.go	Adds source availability and failure metrics.
pkg/metrics/metrics_test.go	Tests source metrics.
pkg/controller/statuswriter.go	Aggregates source status and policy coverage conditions.
pkg/controller/statuswriter_test.go	Tests aggregation, recovery, and membership changes.
pkg/controller/daemonplacement.go	Discovers expected DaemonSet nodes.
pkg/controller/daemonplacement_test.go	Tests placement detection.
pkg/collector/pollsource.go	Announces poll-source readiness.
pkg/collector/collector.go	Tracks lifecycle transitions and retries.
pkg/collector/collector_test.go	Tests lifecycle and retry behavior.
pkg/bpf/exectrace/source.go	Reopens readers and reports readiness.
pkg/bpf/exectrace/source_test.go	Tests reader recovery.
pkg/bpf/dnsquery/source.go	Reports DNS reader readiness.
docs/users/reference/runtimepolicy.md	Documents source coverage and conditions.
docs/users/reference/metrics.md	Documents source metrics.
docs/dev/DEVELOPMENT.md	Adds validation guidance.
docs/dev/DESIGN.md	Documents lifecycle and aggregation architecture.
cmd/kyverno-runtime/daemon.go	Wires lifecycle reporting and placement tracking.
charts/kyverno-runtime/templates/daemonset.yaml	Supplies placement identity.
charts/kyverno-runtime/templates/clusterrole.yaml	Grants DaemonSet read access.
charts/kyverno-runtime/crds/runtime.nirmata.io_runtimepolicies.yaml	Updates the source-status schema.
api/v1alpha1/zz_generated.deepcopy.go	Updates generated deepcopy support.
api/v1alpha1/runtimepolicy_types.go	Adds source status fields and conditions.
Agents.md	Updates package and filtering guidance.

Review details

Files not reviewed (1)

• api/v1alpha1/zz_generated.deepcopy.go: Generated file

Suppressed comments (4)

pkg/controller/statuswriter.go:690

• When openexecmgr.New fails, the daemon records exec-trace with DependencyUnavailable and does not register the open/exec manager or either related source, so filename observations cannot remain available in that failure path. This generic message is therefore misleading; generate a dependency-specific message (or otherwise include the reason) that reports filename coverage as unavailable too.

		if state == runtimeevent.SourceStateUnavailable {
			return "exec trace source is unavailable; argv observations are unavailable, but exec filename observations may remain available"
		}

pkg/controller/statuswriter.go:662

• The dependency is inferred from a non-nil behavior pointer, but an empty Behavior{} (or a rule with no values or expression) compiles to an empty AllowDenyPair and creates no observation attachment. A no-op monitor policy will therefore be marked EventSourcesAvailable=Unknown/False and its Applied condition will be gated even though it cannot produce events; derive this from the evaluated pair's HasEntries() (or otherwise distinguish behaviors that can produce targets).

		if behavior.Exec != nil && !slices.Contains(dependencies, execTraceSource) {
			dependencies = append(dependencies, execTraceSource)
		}
		if behavior.DNS != nil && !slices.Contains(dependencies, dnsQuerySource) {
			dependencies = append(dependencies, dnsQuerySource)

pkg/controller/statuswriter.go:653

• When spec.mode is omitted, the status path deliberately treats the policy as NoMode (baseAppliedCondition), but this default makes the same policy look like monitor mode here. A policy with an exec or dns behavior can therefore publish EventSourcesAvailable=Unknown/False and source entries even though it is not attached or observed; treat a nil mode as non-observe before deriving dependencies.

	mode := compiler.ModeMonitor
	if spec.Mode != nil {
		mode = string(*spec.Mode)
	}
	if !compiler.IsObserveMode(mode) {

pkg/controller/statuswriter.go:293

• This new monitor-only gate should use the repository's compiler.IsObserveMode helper rather than comparing the mode string directly. Otherwise this path can diverge from the mode classification used by sourceDependencies if another observe mode is introduced.

	if mode == compiler.ModeMonitor {

• Files reviewed: 24/25 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.