fix(master): recount pending WAL generations even when the base version is unchanged - #281
Merged
beinan merged 1 commit intoSep 30, 2026
Conversation
…on is unchanged The scanner reuses the previous stats row when a store's base-table version has not moved. WAL flushes never move it: a store nobody has merged keeps its version forever while its shards fill up. Its stats row kept saying "3 pending" while 20 shards accumulated 17,254 generations, so the merge sweep -- which reads that row -- never scheduled it, and reads on it were refused by the pending-generation cap. The shortcut still skips the base-table row count, but now recounts pending generations (one small manifest read per shard) and writes the new number into the reused row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Sep 30, 2026
beinan
added a commit
that referenced
this pull request
Sep 30, 2026
…tats now (#283) ## Problem The merge sweep picks targets from the stats table, which the scanner refreshes on its interval (300 s in production). When a row is known to be wrong — `mai3_bigclimb_run6p5_77b_t0r1` showed 3 pending generations while its shards held 17,254 (#281) — an operator had no direct way to correct it: wait for the next scan, or call `GET /experiments/{name}?fresh=true`, which reads as a query rather than a scheduler input change. ## Fix `POST /api/v1/experiments/{name}/rescan`: resolves the store through the registry (404 if absent), runs the existing `scanner::refresh_one` (full observation under the stats-writer lock, upserts the row) and returns the refreshed `ExperimentDetail`. ## Verification - `rescan_experiment_refreshes_pending_generations` (etcd-backed): 0 → 2 flushed generations; the response and the persisted row both report 2; unknown name → `NotFound`. - Master suite incl. `--include-ignored` passes; clippy clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Sep 30, 2026
…table (#282) ## Problem The delete-only `merge_insert` from #280 probes the key BTree. When the base table has none, Lance falls back to a hash join over the entire base table. `mai3_bigclimb_run6p5_77b_t0r1` (174 GB, 67k rows, 2.6 MB/row) had never been indexed because the master's stats row for it was stale (#281), so every worker that ran a merge on it — the master's fan-out or a manual `/internal/merge-wal` call — was OOMKilled within ~15 s. Building the BTree first took 19 s and the same merge then peaked at 9 GiB. ## Fix `StorageBase::merge_prepared_batches` builds the key BTree (`create_key_btree_index`, idempotent) when the base table has fragments and `has_key_btree_index()` is false, before the delete-only merge. An empty base table has nothing to join and is left unindexed. The master's pre-fan-out build (#277) stays; this is the worker-side guarantee that does not depend on the master having visited the store. ## Verification - `merge_builds_the_id_btree_when_the_base_has_rows_but_no_index`: first merge into an empty base builds nothing; the second merge, against a populated base, creates the BTree and rows still read back exactly once. - Full core suite (249 tests) passes; clippy clean across the workspace. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Sep 30, 2026
## Problem Today `mai3_bigclimb_run6p5_77b_t0r1` sat at 17,254 pending WAL generations, every read answered 503 for ~5 hours, and no metric or log said so — a customer noticed first. The stats scanner was reusing a stale row (#281), no master had ever indexed the store (#282), and the read cap fired silently. 40+ more stores were in the same state. ## Fix One metric per blind spot: | metric | kind | meaning / alert | |---|---|---| | `master_stores_pending_over_read_cap` | gauge | stores whose reads the cap is refusing right now; **alert when > 0** | | `master_stores_pending_over_1k`, `master_wal_pending_generations_total`, `master_wal_pending_generations_max` | gauge | fleet backlog shape, refreshed every stats scan | | `master_stats_pending_recounted_total` | counter | version-unchanged shortcut found a pending count different from the row it would have reused (the #281 condition); `warn!` when growth > 256 | | `rollout_reads_refused_pending_total{store}` | counter | now labeled by store so the unreadable store is named | | `rollout_merge_index_built_on_demand_total` | counter | a merge built the key BTree because no master had (the #282 condition) | Gauges come from a small `Backlog::summarize` over the scan snapshot. ## Verification `backlog_summary_counts_stores_over_the_read_cap` (17,254 / 1,200 / 3 → total 18,457, max 17,254, 1 over cap, 2 over 1k). Existing scanner recount tests and core read-cap/merge tests pass; clippy clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The scanner's unchanged-version shortcut reuses the previous stats row wholesale. WAL flushes never bump the base-table version, so a store that no worker or master has ever merged keeps its version forever while its shards fill up. In production
mai3_bigclimb_run6p5_77b_t0r1sat at "3 pending" in the stats table while its 20 shards accumulated 17,254 generations; the merge sweep reads that row, so it never scheduled a merge, and after #278/#279 every read on the store was refused (503) by the pending-generation cap.Fix
Both
observe_genericandobserve_onestill skip the base-table row count on an unchanged version, but recountpending_wal_generations(one small shard-manifest read per shard, under the existing observe timeout) and write it into the reused row.Verification
skipped_round_recounts_pending_generations: 1 → 4 flushed generations with the base version unchanged; the skipped row reports 4.generic_store_is_observed_with_pending_walextended the same way (3 → 5).--include-ignoredpasses; clippy clean.🤖 Generated with Claude Code