Repository navigation
Conversation
bwplotka
force-pushed
the
bwplotka/staleness-detection
branch
from
November 12, 2026 07:30
099e550 to
2b0161b
Compare
bwplotka
commented
Nov 12, 2026
bwplotka
marked this pull request as ready for review
November 12, 2026 09:03
bwplotka
force-pushed
the
bwplotka/staleness-detection
branch
from
November 13, 2026 04:32
0d63e9c to
4df05f2
Compare
bwplotka
added a commit
that referenced
this pull request
Nov 13, 2026
…bench-v4 Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
saswatamcode
left a comment
Member
There was a problem hiding this comment.
Couple of AI review comments.
Add STALE_NAN_BITS, STALE_NAN, DATA_POINT_FLAG_NO_RECORDED_VALUE, is_stale_nan, lazy flags column materialization on NumberDataPoints, SummaryDataPoints, and HistogramDataPoints, and zero-steady-state-allocation per-target StalenessTracker support in ArrowDecoder. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Explain that OTAP treats data-point flags as default_values_optional so omitting all-zero flags columns preserves schemas and avoids UInt32 allocations on live-only batches. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…or StalenessTracker Fix the module rustdoc describing StaleNaN as quiet, note bitwise-only handling for signaling NaN bits on STALE_NAN and is_stale_nan, and implement Debug on StalenessTracker. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Use NONE_IDX sentinels in batch_key_to_tracker and batch_val_to_tracker instead of sequential catch-up loops so ephemeral exemplar label keys and values in batch interners never leak into StalenessTracker. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Normalize counter family names by stripping trailing _total suffixes and include MetricType and saw_total_suffix in TrackedFamily keys so # TYPE changes never attach data points to mismatched UnivariateMetrics rows and # TYPE req counter vs # TYPE req_total counter share one family. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
… scrape completion Stage per-scrape family metadata updates and preserve committed active flags until commit_scrape runs after final RecordBatch construction so mid-scrape errors and early DecodeStream drops roll back cleanly. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…d lazy allocation Make StalenessTracker::new() zero-allocating until first use, shrink TrackedSeries layout, clear arenas when active_count drops to zero, compact dead series, families, and interned strings under label churn, and document the opt-in memory model. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
… OTAP batches Emit NULL for NumberDataPoints.double_value and HistogramDataPoints.sum (and 0.0 for SummaryDataPoints.sum with empty buckets, quantiles, and exemplars) when FLAG_NO_RECORDED_VALUE is set, matching OTLP and OTAP semantics without poisoning Arrow float aggregations. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Add DecodeOptions::track_timestamps_staleness (default false, matching Go Prometheus scrape/scrape.go) so client-timestamped samples are excluded from StalenessTracker unless explicitly enabled. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…ith_staleness Enforce entry and duration DecodeBudget limits inside StalenessTracker::finish_scrape_step so cooperative streams yield during large target drops, and add step_decode_with_staleness and finish_decode_with_staleness for lifetime-free DecodeCursor callers. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…ess in SPEC.md Drop ArrowDecoder::encode_stale_target in favor of flush_target_stale to keep the public API surface minimal, and document StalenessTracker, FLAG_NO_RECORDED_VALUE, and track_timestamps_staleness in specs/SPEC.md. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…talenessTracker loop Keep the stateless ArrowDecoder::decode OpenMetrics bounded-memory check intact and add separate opt-in StalenessTracker checks covering steady-state scrapes and alternating full/subset disappearing-series cycles. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…and budgets Add tests covering flushed metadata rollback, zero-capacity rehashing, key-order resorting, exemplar interner isolation, churn compaction, wire NaN vs StaleNaN, # TYPE transitions, UntypedPolicy::Drop, track_timestamps_staleness, and budgeted cooperative stale draining. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Monomorphize ingest_entries and step_cursor over const TRACKED: bool so untracked decode, decode_stream, and step_decode paths compile out StalenessTracker branches and register state. Split OptionalValueCol::push_value and SparseFlagsCol::push_zero from #[cold] null/flag slow paths so live scalar samples avoid Option<f64> construction and inlined Vec growth paths. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…samples When track_timestamps_staleness is false and an already-tracked family emits an explicit-timestamp sample, sync the family's current_batch_id and pending metadata in StalenessTracker so any stale markers emitted for disappeared untimestamped series in that family share the same UnivariateMetrics row. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Track active_scrape_id on StalenessTracker and validate in step_cursor that callers do not swap StalenessTracker instances or mix tracked and untracked step_decode calls on the same DecodeCursor. Also roll back tracker state in finish_decode_with_staleness whenever tracker.active_scrape_id matches cursor.scrape_id. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…l_suffix Pass CurrentFamily::canonical_key() to emit_family_row so stale counter emission does not strip a second trailing _total on counter families ending in _total_total. Also avoid mutating saw_total_suffix after the first sample of a family has been emitted, and skip staging histogram exemplars when group.is_stale is already set. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
… or abort Release large tracker arena allocations when active_count drops to zero or when abort_scrape rolls back uncommitted entries, and bound next_keys pre-allocation by active_count in compact(). Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…nd edge cases Add unit test coverage for SeenState::SeenStale, OptionalValueCol null-to-valid transitions, compact() with # UNIT metadata and inverted key IDs, update_existing_family metadata updates and rollback, _total_total counter stale emission, mixed explicit-timestamp + untimestamped families, and cursor/tracker mismatch validation. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Deduplicate sorted (tk, tv) label slices across metric families within each target via an open-addressed AttrSetEntry table so targets with shared label sets store each unique label set only once in series_attrs. Pack intern::Bucket, FamilyBucket, AttrSetBucket, SeriesBucket, AttrSetEntry, and TrackedSeries from 16 bytes down to 8 bytes each, size hash tables for the 70% load factor threshold, shrink Vec doubling tail capacity after initial scrape population, and compact deactivated series once churn exceeds 12.5% of active series. On a 10,840-series Avalanche target, per-target StalenessTracker retained heap drops 5.2x from 2.19 MiB to 417.8 KiB (34.18 MiB to 6.53 MiB across 16 targets), and post-rotation retention drops 10.1x from 4.24 MiB to 417.8 KiB. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Call tracker.abort_scrape() before the tracker.is_empty() early return in ArrowDecoder::flush_target_stale so dangling uncommitted scrapes do not leave active_scrape_id or staged seen flags behind. Add ArrowDecoder::flush_target_stale_stream so targets with more than 65,536 families or per-batch data-point limits can flush stale series in bounded chunks, and document the single-batch u16 metric limit on flush_target_stale. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
bwplotka
force-pushed
the
bwplotka/staleness-detection
branch
from
November 19, 2026 05:59
8fbbfff to
54f574e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Prometheus series staleness detection and optional cross-scrape disappeared series tracking to
expfmt::arrow::ArrowDecoder.STALE_NAN_BITS,STALE_NAN,DATA_POINT_FLAG_NO_RECORDED_VALUE, andis_stale_nan(f64)inexpfmt::arrow.flagscolumn onNumberDataPoints,SummaryDataPoints, andHistogramDataPointsonly when a batch contains a non-zero flag, and omits stale data point values per OTLPFLAG_NO_RECORDED_VALUEsemantics rather than writingSTALE_NANinto OTAP payloads.StalenessTrackeralong withArrowDecoder::decode_with_staleness,decode_stream_with_staleness,step_decode_with_staleness,finish_decode_with_staleness, andflush_target_stale, withDecodeOptions::track_timestamps_stalenessdefaulting tofalse, transactional rollback on failed or aborted scrapes, and dead-series arena compaction under series churn.prombenchyotap-rustworkload is0 BwhenStalenessTrackeris not used, or17.1 MiBin initial steady state and29.1 MiBat the2xcompaction watermark under 1h full series churn when enabled across all 16 targets.Static live-heap estimate (
prombenchyotap-rust)Workload from
avalanche.exampletarget.yamlandmanifests/scenarios/otap-rust/1_collector.yaml:--num-cores=1andmax_data_points_per_batch: 200scraping 16avalanchepods every 15s.158gauges,280counters,28classic 10-bucket histograms,482-objective summaries),--series-count=10, 12 labels per series (series_id,cycle_id, and 10label_key_kkkkk_*labels), and--series-interval=3600rotatingcycle_idevery hour.StalenessTrackertracks assembledHistogramDataPointsandSummaryDataPointsrows rather than per-bucket or per-quantile lines.Stateless mode without
StalenessTracker(decode,decode_stream,step_decode):0 Badditional heap when no wireStaleNaNis present becauseSparseFlagsColandOptionalValueCol::validstay at capacity0(104 Binline struct fields on the sharedArrowDecoder).Opt-in
StalenessTrackerenabled (1 tracker per target, 64-bitVecdoubling capacities and 70% load-factor hash tables):Veccapacity (0 < t < 1h)2xchurn watermark (1h <= t < 2h)compact()(t >= 2h)series_attrs((u32, u32)@ 8 B, 61,680 active pairs)family_bytes(514 names + ~361 B# HELPtext)series(TrackedSeries@ 16 B, 5,140 active)series_buckets(SeriesBucket@ 16 B)families(TrackedFamily@ 52 B) +family_bucketskey_interner+val_interner+ remap scratch + struct5,140OTAP /9,940Prom series)82,240OTAP /159,040Prom series)Notes on churn and label cardinality:
commit_scrape()runscompact()onceseries.len() > 2 * active_count, so the first 1h churn (5,140active +5,140dead =10,280) retains one dead generation (29.10 MiBacross 16 targets) and the second 1h churn (15,420 > 10,280) compacts back to20.00 MiB, with a ~2.3 MiB single-target transient spike while scraping and compacting one target at a time (~31.4 MiB collector peak).avalanchereuses 21 distinct label values across all 5,140 series per target; if a non-synthetic target instead carried 5,140 distinct 16-byte label values per target,val_internerwould add ~368 KiB per target (~5.8 MiB across 16 targets).Commits in order
2b0161barrow: add Prometheus series staleness detection and StalenessTrackera92e491arrow: document why SparseFlagsCol omits all-zero flags columns9d0f192arrow: clarify signaling StaleNaN bit semantics and implement Debug for StalenessTracker31b5eb4arrow: lazily intern only series attributes in StalenessTrackerbc491a0arrow: key StalenessTracker families by canonical name and MetricType6cff462arrow: defer StalenessTracker active and family metadata commit until scrape completiona076c06arrow: bound StalenessTracker live heap via dead-series compaction and lazy allocation00bde12arrow: omit stale data point values instead of writing STALE_NAN into OTAP batchese65fe06arrow: skip staleness tracking for explicit wire timestamps by default87d4b18arrow: honor DecodeBudget during stale draining and add step_decode_with_staleness03ab0a3arrow: remove duplicate encode_stale_target alias and document staleness in SPEC.md2ee1003tests: preserve stateless OpenMetrics dhat check alongside optional StalenessTracker loop37b3679arrow: add unit and OTAP tests for staleness edge cases, compaction, and budgets0d63e9carrow: specialize untracked decode loop and fast-path live scalar valuesBenchmarks (
benchstat before.txt after.txt)Comparing
fd6c3c1(before.txt) vs0d63e9c(after.txt):