feat(ingestion-consumer): count gap walks and clear stale confirmation lag - #95873
Merged
Conversation
|
😎 Merged successfully - details. |
Contributor
🤖 CI report✅ Trunk lane — non-backend laneThis PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes. |
The ledger fills an undelivered offset with pre-completed filler, so a frontier commit steps past it silently. With the frontier owning commits the commit sentinel no longer reports these walks as gaps, so nothing records that a commit skipped an offset Kafka never delivered. The partition ledger now marks filler as undelivered and reports how many filler offsets a take drained. The topic ledger counts them at take as kafka_consumer_ledger_gap_offsets_total by topic and partition. A take without gaps emits nothing.
…voke The gauge is refreshed from OffsetFetch results, which only cover assigned partitions. A partition revoked while a commit was still unconfirmed kept reporting that lag for the life of the process, so a pod that held a partition for one incremental assign showed a constant nonzero lag for days and dominated the per-partition max on the dashboard. forget_partitions now sets the gauge to zero for every revoked partition that had a commit attempted.
pl
force-pushed
the
pl/ingestion/ledger-gap-walks
branch
from
September 7, 2026 07:06
16f27f9 to
9875461
Compare
pl
marked this pull request as ready for review
September 7, 2026 07:08
Contributor
|
Reviews (1): Last reviewed commit: "fix(ingestion-consumer): zero the commit..." | Re-trigger Greptile |
jose-sequeira
approved these changes
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Operators reading the offset ledger dashboard see two things that are not true now that the ledger frontier owns commits (#91766).
Changes
CommitSentinel::forget_partitionszeroes it for every revoked partition that had a commit attempted.kafka_consumer_ledger_gap_offsets_totalby topic and partition, counts undelivered offsets a frontier commit walked over. The topic ledger emits it when a take drains filler; a take without gaps emits nothing.TakenFrontiergainsgap_offset_count.How did you test this code?
gap_filler_carries_no_charge_and_the_frontier_walks_over_itnow asserts the gap count.a_take_counts_each_gap_offset_oncecatches a take that counts drained filler twice, or counts filler the frontier did not reach.Automatic notifications
Docs update
None.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Claude Code. Skills invoked: /writing-pr-descriptions. Both gaps surfaced while reviewing the first days of commit mode in production: a stale per-partition lag gauge on a pod that briefly held partitions during an incremental assign, and the sentinel gap counter reading zero once frontier commits replaced per-batch commits. Searched open ledger PRs; none covers either.