QueueManager Shard-Level Coalescing - #5
Draft
dashpole wants to merge 2 commits into
Draft
Conversation
Implement Prototype Option B for Prometheus Issue prometheus#17857 to correlate, buffer, and coalesce samples and exemplars in QueueManager shard worker queues prior to PRW 2.0 protobuf serialization. - Introduce shardCoalescer with a fixed-size circular ring buffer (2,048 entries), slot generation tracking, and exact scrape timestamp matching (<= 50ms). - Support sample-first and exemplar-first ingestion for float samples, native histograms, and float histograms with zero string allocations on the hot path. - Enforce PRW 2.0 specification invariants (zero empty TimeSeries, zero duplicate samples, drop unmatched exemplars on deadline with metric instrumentation). - Synchronize lifecycle and drainage during dynamic resharding. - Add unit tests, resharding concurrency tests with -race, and 100k churn benchmarks.
…tion, and metric accounting
dashpole
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this change necessary?
Possible fix for prometheus#17857
The Prometheus Remote Write 2.0 (PRW 2.0) specification states:
Currently, Prometheus TSDB writes samples and exemplars into separate WAL records. Remote Write
QueueManageringests them as independent queue items (tSamplevstExemplar). When serializing for PRW 2.0 inpopulateV2TimeSeries,tExemplaremits awritev2.TimeSeriesmessage with attached exemplars but 0 samples and 0 histograms. This violates the PRW 2.0 specification, causing downstream PRW 2.0 receivers (OTel Collector, Mimir, Google Cloud Managed Prometheus) to reject payloads or fail parsing.Previous attempts (such as PR prometheus#18014) attempted to match exemplars across a single popped batch slice, which dropped exemplars whenever samples and exemplars landed in adjacent batches.
What does this PR do?
This PR implements a QueueManager Shard-Level Coalescing Window, resolving the specification violation completely within
storage/remote/with zero TSDB or WAL on-disk modifications:Lock-Free Shard Coalescing Ring (
storage/remote/shard_coalescer.go):shardCoalescerinto each shard queue worker with a pre-allocated circular ring buffer (default 2,048 slots) and 64-bit slot generation counters (uint64) to eliminate dangling index risks on buffer wrap-around.map[chunks.HeadSeriesRef]coalescerIndexEntry) guaranteeing zero heap string allocations on the hot ingestion path.Exact Scrape Timestamp Matching ($\le 50\text{ms}$ ):
Strict PRW 2.0 Invariants & Drop Accounting:
populateV2TimeSeriespopulateswritev2.TimeSeriescontainingprometheus_remote_write_unmatched_exemplars_dropped_total(never emitting an invalid 0-sample series and never retransmitting duplicate samples).Resharding Synchronization:
QueueManager.reshardLoop()flushes active batches and drains shard coalescers during dynamic shard scaling, preventing orphaned buffered exemplars.Verification & Testing
go test -v -race ./storage/remote/...(All PASS)TestQueueManager_PRW2_Coalescing: Verifies all sample arrival orders, native histograms, and float histograms.TestQueueManager_100kSeriesChurn_HeapStability: Ingested 100,000 distinct series under continuous churn; confirmed stable bounded heap memory (~13.6 MB).BenchmarkQueueManager_PRW2: Validated minimal allocation footprint on PRW 2.0 sending path.