Add OpenTelemetry metrics for retry and archive operations - #5836
Merged
Merged
Conversation
rbev
approved these changes
Sep 1, 2026
Comment on lines
+44
to
+51
| static string ResultTag(ScopeOutcome outcome) => outcome switch | ||
| { | ||
| ScopeOutcome.Success => "success", | ||
| ScopeOutcome.Empty => "empty", | ||
| ScopeOutcome.Cancelled => "cancelled", | ||
| ScopeOutcome.Failed => "failed", | ||
| _ => "failed" | ||
| }; |
Contributor
There was a problem hiding this comment.
Not necessarily better but you could avoid all the string constants using enum.ToString()
Suggested change
| static string ResultTag(ScopeOutcome outcome) => outcome switch | |
| { | |
| ScopeOutcome.Success => "success", | |
| ScopeOutcome.Empty => "empty", | |
| ScopeOutcome.Cancelled => "cancelled", | |
| ScopeOutcome.Failed => "failed", | |
| _ => "failed" | |
| }; | |
| static string ResultTag(ScopeOutcome outcome) => (Enum.IsDefined(outcome) ? outcome : ScopeOutcome.Failed).ToString().ToLower(); |
RetryingManager now depends on RetryMetrics, so tests that register it must also provide a test double to satisfy the dependency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds telemetry for error management, the follow-up to the ingestion (#5816) and retention (#5831) metrics. Retry and archive were the last long-running operations in the error instance with no signal at all: a retry of a large group can run for hours across four stages and two polling loops, and until now the only way to see it was the log.
What is instrumented
sc.retry.*covers the whole retry pipeline: an end-to-end operation duration from request to completion, a duration histogram per stage (prepare, stage, forward), a message counter by outcome, an in-progress gauge, and the depth of the bulk request queue. Theabandonedmessage outcome is the one worth alerting on, since it counts messages that hit the staging retry limit and were dropped from their batch, which previously existed only as a log line. The in-progress gauge is what surfaces a stuck operation, because an operation that never completes never records a duration.sc.archive.*covers group archive and unarchive: whole-operation duration, per-batch duration, a message counter, and an in-progress gauge. The instrumentation lives once in the sharedInMemoryArchive/InMemoryUnarchivestate machines, but only the EF persistence registersArchiveMetrics, so on RavenDB the family does not exist rather than being half-populated. No RavenDB file is touched.The host also gains the standard ASP.NET Core, HTTP client and runtime instrumentation packages, behind the existing
OTEL_EXPORTER_OTLP_ENDPOINTgate. That covers the read APIs per route, the scatter-gather calls to remote instances, and GC/thread pool context.AddIngestionMetricsis renamedAddTelemetrynow that it wires more than ingestion.All tag values come from enums, so series counts are bounded; group ids, endpoint names and queue addresses stay out and are recovered from the audit log when a specific operation needs investigating. The full instrument contract is documented in
docs/telemetry.md.