feat(observability): discover tenant metrics and alert on missing game signals #880

Production readiness audit of Gameplane v0.4.0, commit 07bee47b. Tracking issue and PR merge order: #867.

Finding

The agent exports CPU, RSS, disk and available player counts. Numeric tick/frame measurements depend on module declarations. The five sampled official modules supply none. Extra namespaces and scrape access need deliberate configuration.

Work

Make safe discovery and scrape access available in every managed namespace. Document per-engine metric support, add validated declarations where engines expose them, preserve unknown player counts, and integrate working set, throttling, OOM/scheduling and engine-specific profiling with explicit stale/missing signal alerts.

Acceptance criteria

Demonstrate discovery in two tenant namespaces, an expected tenant target that disappears from discovery, a lost scrape, unknown player count, tick slowdown, memory growth and relay saturation. Distinguish process RSS, container pressure and engine churn. Verify tenant isolation and bounded label cardinality.

PR ownership and prior work

Own collectors, dashboards, alerts and compatible official-module declarations. Reuse H02’s network boundary and M02’s relay counters. Keep observability changes independent where those interfaces are already stable. Extend the declaration/unknown-value conventions, ServiceMonitor and Cilium coverage in #725/#838, including Prometheus rules that currently select one namespace.

Agent collector · Game performance collector · Monitor scope · Monitoring setup

Qualification evidence

The authenticated Palworld dedicated server endpoint returned 59 instantaneous frames/s on Steam build 25247047. The module consumes that field and keeps unknown observations distinct from a measured zero. Module validation and both amd64 and arm64 devcontainer checks passed. Captured engine response · Module PR #60

The panel alert and dashboard consume the relay counters qualified against client and server observations in #879. The local Tailscale forwarding profile and the measured FRP trial limits are recorded with their applicable topologies. Relay qualification · Panel PR #908

On the panel commit b8f443f, the enforcing Cilium and Prometheus drill passed in 382.65 seconds. Two identically named running games appeared in separate tenant namespaces. The drill invalidated one player-count response, observed a real game-process RSS increase, removed and restored one discovery target, revoked and restored scrape access, denied unauthorized clients and forwarded player traffic through FRP. The chart's promtool tests also passed for target loss, engine slowdown, memory pressure and relay saturation. Live monitoring test · Evaluated alert rules · Cilium legacy policy upgrade