Skip to content

feat(metrics): let native histograms recover lost resolution - #255

Merged
TheJokr merged 2 commits into
cloudflare:mainfrom
ethanolchik:min-reset-duration-v2
Aug 18, 2026
Merged

feat(metrics): let native histograms recover lost resolution#255
TheJokr merged 2 commits into
cloudflare:mainfrom
ethanolchik:min-reset-duration-v2

Conversation

@ethanolchik

@ethanolchik ethanolchik commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Problem

NativeHistogramBuilder never passed a minimum reset duration to prometheus-client, so exceeding max_buckets reduced the schema by one step with nothing to restore it. Resolution loss was permanent for the life of the process, per label set, and ratcheted further on each subsequent breach. That forces callers to pick a coarse bucket_factor purely to stay clear of the cap.

Change

Forward min_reset_duration to NativeHistogramConfig. Once it has elapsed, a limit breach resets the histogram and restores its configured schema instead of degrading it. Default is one hour; Duration::ZERO disables resets and preserves the current behaviour. No timer needed because the wrapped crate resets lazily from reset_if_scheduled, which runs on both the observe and snapshot paths.

A reset also clears counts, sum, and classic buckets, so it surfaces as a counter reset in _bucket/_sum/_count. rate() handles that by design, but worth knowing if you are dual-emitting during a migration.

Breaking change

min_reset_duration is a new public field, so full struct-literal initialisers need updating; ..NativeHistogramBuilder::new(..) and the with_* setters are unaffected. Happy to default it inside config() without a public field instead if you would rather avoid the break.

`NativeHistogramBuilder` never passed a minimum reset duration to
`prometheus-client`, so when a histogram exceeded `max_buckets` the only
available mitigation was reducing its schema by one step. Nothing restored the
original schema afterwards, making the resolution loss permanent for the
lifetime of the process, per label set, and ratcheting further on every
subsequent breach.

Expose `min_reset_duration` on the builder and forward it to the wrapped
configuration. Once the duration has elapsed, a bucket limit breach resets the
histogram and restores its configured schema instead of degrading it. The
default of one hour bounds any resolution loss; `Duration::ZERO` disables
resets and preserves the previous behaviour.

Bounding degradation is what makes finer bucket factors practical: callers no
longer have to choose a coarse factor purely to stay clear of the bucket cap.

Note that a reset also clears the counts, sum, and classic buckets, which is
visible as a counter reset in `_bucket`, `_sum`, and `_count` series.
@TheJokr
TheJokr self-requested a review August 18, 2026 11:00
@TheJokr
TheJokr merged commit 173658d into cloudflare:main Aug 18, 2026
18 checks passed
@ethanolchik
ethanolchik deleted the min-reset-duration-v2 branch August 18, 2026 12:16
@TheJokr TheJokr mentioned this pull request Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants