Skip to content

feat(suse-observability): Add monitors for uncovered metrics-path failures - #25

Merged
VioletCranberry merged 1 commit into
mainfrom
230-metrics-path-monitors
Sep 23, 2026
Merged

VioletCranberry merged 1 commit into
mainfrom
230-metrics-path-monitors

Conversation

@VioletCranberry

Copy link
Copy Markdown
Contributor

Why

The stackpack ships ~34 metrics-path monitors, and six failure modes could occur while every one of them stayed green. Each is a case the sizing work (#178, #206) needs visibility on:

Monitor Gap it closes
VictoriaMetrics - Storage is read only The two existing disk monitors predict exhaustion; nothing reported that VictoriaMetrics had already stopped accepting writes. #178 AC1.iv asks for this.
Insert / Select concurrency saturated The internal stackstate-monitoring PrometheusRules have ConcurrentFlushesHitTheLimit; the customer-facing pack had no equivalent, and the query path had none at all.
vmagent - Samples dropped on remote write persistentqueue_bytes_dropped and remotewrite_packets_dropped were covered; silently dropped samples were not.
VictoriaMetrics - Metrics are not fresh No staleness signal existed for the metrics path.
Monitor execution time The existing checks monitor covers failure percentage, not runs that overrun their interval - which is #206's monitor pass condition.

Thresholds are set from observed baselines, not guessed: execution time triggers above 60s because the slowest monitor currently averages 25s; concurrency at 90% of capacity; freshness at 30s to match the topology SLO.

What the freshness monitor is not

It measures the age of the newest arriving sample, not ingest-to-queryable latency. That satisfies #178 and needs no new deployable, so it works on customer installs - but it does not satisfy #206's freshness pass condition, which requires an independent probe writing timestamped samples through the ingress and querying them back. That probe is still outstanding; recorded in #230 so the two don't get conflated.

Deliberately not added

A pending-rows growth monitor. stackstate_vm_pending_rows has a large environment-dependent baseline (~22k storage, ~600 indexdb on one live namespace), so any absolute threshold either pages on normal behaviour or never fires. The dashboard panel from #225 covers it, and vmagent queue saturation catches the downstream effect.

I also checked settings/services/ before scoping: disk headroom, rows rejected, persistent-queue drops and monitor failure rate already have monitors, so none of these duplicate one.

Testing

Validated and test-deployed to nightly-champagne as 0.0.9-cli-test.10000. All six register ENABLED; 65 monitors across the pack with no duplicate identifiers and every remediation-hint include resolving.

Ran each via sts monitor run against a live namespace:

  • Four gauge-based monitors return a CLEAR check state, correctly templated per pod.
  • Two counter-based monitors return no check states - and so do the equivalent already-merged monitors (vmagent-remotewrite-packets-dropped, monitors-failure-percentage-pod). Counter monitors here only materialise a state once the counter moves, so these match the established pattern. Worth being explicit though: that means they are not distinguishable from broken until a counter increments.

Refs #230

…lures

Six failure modes could occur while every existing monitor stayed green:
VictoriaMetrics switching to read-only, insert and select concurrency
reaching the CPU-derived limit, vmagent discarding samples on remote write,
metrics going stale, and monitors overrunning their interval.

Thresholds come from observed baselines rather than guesses. Monitor
execution time triggers above 60s because the slowest monitor currently
averages 25s. Concurrency triggers at 90% of capacity. Freshness uses 30s
to match the topology SLO.

Skip a pending-rows growth monitor: the baseline is environment-dependent
(~22k storage, ~600 indexdb on one live namespace), so any absolute
threshold either pages on normal behaviour or never fires. The dashboard
panel covers it.

Refs #230
@VioletCranberry
VioletCranberry merged commit 934c930 into main Sep 23, 2026
12 checks passed
@VioletCranberry
VioletCranberry deleted the 230-metrics-path-monitors branch September 23, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants