feat(suse-observability): Add monitors for uncovered metrics-path failures - #25
Merged
Merged
Conversation
…lures Six failure modes could occur while every existing monitor stayed green: VictoriaMetrics switching to read-only, insert and select concurrency reaching the CPU-derived limit, vmagent discarding samples on remote write, metrics going stale, and monitors overrunning their interval. Thresholds come from observed baselines rather than guesses. Monitor execution time triggers above 60s because the slowest monitor currently averages 25s. Concurrency triggers at 90% of capacity. Freshness uses 30s to match the topology SLO. Skip a pending-rows growth monitor: the baseline is environment-dependent (~22k storage, ~600 indexdb on one live namespace), so any absolute threshold either pages on normal behaviour or never fires. The dashboard panel covers it. Refs #230
rb3ckers
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The stackpack ships ~34 metrics-path monitors, and six failure modes could occur while every one of them stayed green. Each is a case the sizing work (#178, #206) needs visibility on:
stackstate-monitoringPrometheusRules haveConcurrentFlushesHitTheLimit; the customer-facing pack had no equivalent, and the query path had none at all.persistentqueue_bytes_droppedandremotewrite_packets_droppedwere covered; silently dropped samples were not.Thresholds are set from observed baselines, not guessed: execution time triggers above 60s because the slowest monitor currently averages 25s; concurrency at 90% of capacity; freshness at 30s to match the topology SLO.
What the freshness monitor is not
It measures the age of the newest arriving sample, not ingest-to-queryable latency. That satisfies #178 and needs no new deployable, so it works on customer installs - but it does not satisfy #206's freshness pass condition, which requires an independent probe writing timestamped samples through the ingress and querying them back. That probe is still outstanding; recorded in #230 so the two don't get conflated.
Deliberately not added
A pending-rows growth monitor.
stackstate_vm_pending_rowshas a large environment-dependent baseline (~22k storage, ~600 indexdb on one live namespace), so any absolute threshold either pages on normal behaviour or never fires. The dashboard panel from #225 covers it, and vmagent queue saturation catches the downstream effect.I also checked
settings/services/before scoping: disk headroom, rows rejected, persistent-queue drops and monitor failure rate already have monitors, so none of these duplicate one.Testing
Validated and test-deployed to nightly-champagne as
0.0.9-cli-test.10000. All six register ENABLED; 65 monitors across the pack with no duplicate identifiers and every remediation-hint include resolving.Ran each via
sts monitor runagainst a live namespace:vmagent-remotewrite-packets-dropped,monitors-failure-percentage-pod). Counter monitors here only materialise a state once the counter moves, so these match the established pattern. Worth being explicit though: that means they are not distinguishable from broken until a counter increments.Refs #230