feat(metrics): export latency as bucketed histograms, not summaries - #179
Merged
Merged
Conversation
With no bucket configuration, metrics-exporter-prometheus 0.16 renders
every histogram!() as a rolling summary (quantile series + monotonic
_sum/_count) rather than a bucketed Prometheus histogram. That makes
latency hard to reason about operationally: _sum/_count give a
since-process-start average that never decays, and the exporter's summary
quantiles use an opaque slow-decaying window, so a p99 stays pinned at an
old spike long after latency recovers and can't be recomputed over an
arbitrary window at query time.
Configure explicit buckets so all *_duration_seconds metrics export as
true histograms (_bucket{le="..."}), letting downstream compute
histogram_quantile() over any window:
- Suffix("_duration_seconds") -> request-scale buckets (5ms..60s), covering
http_request_duration_seconds and master_scan_duration_seconds.
- Full-name overrides for the job-scale metrics master_task_duration_seconds
and rollout_compaction_duration_seconds -> coarser buckets extending to
1800s. A Full matcher outranks the Suffix matcher (exporter precedence is
Full > Prefix > Suffix), so minute-plus jobs don't collapse into +Inf and
distort high percentiles.
Counters are unaffected. Extends the metrics test to assert histogram TYPE
+ _bucket series and the absence of summary quantiles.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the metrics issue:
lance-context-metricsinstalls the Prometheus recorder with no bucket config, so metrics-exporter-prometheus 0.16 renders everyhistogram!()as a rolling summary (quantile series + monotonic_sum/_count) instead of a bucketed histogram (_bucket{le="..."}).Verified against the exporter source (
distribution.rs::get_distribution): with no buckets/overrides it falls through toDistribution::new_summary(...), confirming the issue's core claim. Summary quantiles use an opaque, slow-decaying window — a p99 stays pinned at an old spike long after latency recovers, and consumers can't recompute a quantile over an arbitrary window.Change
Configure explicit buckets so all
*_duration_secondsmetrics export as true histograms, enabling query-timehistogram_quantile(0.99, rate(..._bucket[5m])):Suffix("_duration_seconds")→ request-scale buckets (5ms…60s): covershttp_request_duration_seconds,master_scan_duration_seconds.Full(...)overrides → job-scale buckets (…1800s) formaster_task_duration_secondsandrollout_compaction_duration_seconds. These jobs (compaction / WAL merge / index) routinely exceed 60s; the issue's proposed single 60s-max bucket set would dump every long job into+Infand pin high percentiles. AFullmatcher outranks theSuffixmatcher (exporter precedence Full > Prefix > Suffix), so the override wins for these two names while everything else uses the request set.Counters (
http_requests_total,master_task_enqueued_total, …) are unaffected — this only concernshistogram!-emitted metrics.Test plan
cargo test -p lance-context-metrics— extended the render test to assert# TYPE ... histogram,_bucket{le="..."}series (both the requestle="1"and the job-scalele="300"), and the absence of{quantile="..."}summary series.cargo fmt --check+cargo clippy --all-targetsclean.🤖 Generated with Claude Code