feat: metrics for the WAL backlog blind spots - #285
Merged
Merged
Conversation
Today a store sat at 17k pending generations, answering 503 to every read for five hours, and nothing said so: the only signal was a customer. Four metrics close that gap, one per blind spot: - master_stores_pending_over_read_cap (gauge): stores whose reads the cap is refusing right now; alert when > 0. Also master_stores_pending_over_1k, master_wal_pending_generations_total and _max, refreshed every stats scan. - master_stats_pending_recounted_total (counter): the version-unchanged shortcut found a different pending count than the row it would have reused (the lance-format#281 condition); warns when the growth exceeds 256. - rollout_reads_refused_pending_total now carries a `store` label, so the unreadable store is named, not just counted. - rollout_merge_index_built_on_demand_total (counter): a merge had to build the key BTree itself because no master had (the lance-format#282 condition). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Today
mai3_bigclimb_run6p5_77b_t0r1sat at 17,254 pending WAL generations, every read answered 503 for ~5 hours, and no metric or log said so — a customer noticed first. The stats scanner was reusing a stale row (#281), no master had ever indexed the store (#282), and the read cap fired silently. 40+ more stores were in the same state.Fix
One metric per blind spot:
master_stores_pending_over_read_capmaster_stores_pending_over_1k,master_wal_pending_generations_total,master_wal_pending_generations_maxmaster_stats_pending_recounted_totalwarn!when growth > 256rollout_reads_refused_pending_total{store}rollout_merge_index_built_on_demand_totalGauges come from a small
Backlog::summarizeover the scan snapshot.Verification
backlog_summary_counts_stores_over_the_read_cap(17,254 / 1,200 / 3 → total 18,457, max 17,254, 1 over cap, 2 over 1k). Existing scanner recount tests and core read-cap/merge tests pass; clippy clean.🤖 Generated with Claude Code