feat(master): auto-sweep MergeWal from pending WAL generations - #162
Merged
Conversation
Previously MemWAL merge only fired via the manual "Merge WAL"/"Optimize" UI buttons — the master's periodic sweep only enqueued Compact, and the data-plane's local merge triggers default to off. Add a second periodic sweep that enqueues a MergeWal task for every experiment whose scanned pending_wal_generations crosses MERGE_WAL_MIN_GENERATIONS, reusing the existing fan-out to worker endpoints. Depless MergeWal now de-dupes so a slow broadcast is not re-queued. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
4 tasks
beinan
added a commit
that referenced
this pull request
Jul 22, 2026
## Summary - `RolloutStore::spawn_periodic_cleanup` and its `cleanup_interval_secs` option/field had **no production caller**. WAL merge is driven by two live paths instead: the server's process-wide `spawn_global_sweeper` and the master's fan-out `MergeWal` sweep (#162), both calling `cleanup_own_shard` directly. - Removes the per-store timer, its `RolloutStoreOptions.cleanup_interval_secs` option and the store field, the two dead tests, and stale doc/comment references. This eliminates a confusing third WAL-scheduling path. - The **server-level** `rollout_cleanup_interval_secs` (which drives the active global sweeper) is retained unchanged. Net: 5 insertions, 220 deletions. ## Test plan - [x] `cargo build -p lance-context-core -p lance-context-server -p lance-context` - [x] `cargo test -p lance-context-core rollout` (27 passed) - [x] `cargo clippy` clean on all three crates - [x] `cargo fmt --all` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MergeWaltasks, so MemWAL merge no longer depends on someone clicking the UI "Merge WAL"/"Optimize" button.MergeWalfor every experiment whosepending_wal_generations >= MERGE_WAL_MIN_GENERATIONS, reusing the existingrun_merge_walfan-out toWORKER_ENDPOINTS(each worker folds its own shard).MergeWalnow de-dupes (viashould_dedupe) so a slow broadcast is not re-queued by the next sweep; the Optimize chain's dependent MergeWal (hasdepends_on) is unaffected.Why
Before this, of the three merge trigger paths only the manual UI button actually fired
MergeWal: the master auto-sweep only enqueuedCompact, and the data-plane local triggers (ROLLOUT_MERGE_AFTER_GENERATIONS/ROLLOUT_CLEANUP_INTERVAL_SECS) default to off. This gives the master a single, evolvable place to drive WAL merge automatically.Config
MERGE_WAL_INTERVAL_SECS(default 600,0disables)MERGE_WAL_MIN_GENERATIONS(default 8)Correctness
cleanup_own_shard); master is only the trigger, so no live writer is fenced.merge-wal-sweepcoordination lock, mirroring the compaction sweep.STATS_SCAN_INTERVAL_SECS; acceptable for background convergence.Test plan
cargo check/cargo clippy -p lance-context-master --all-targetscleancargo test -p lance-context-master(newsweep_merge_wal_enqueues_over_threshold_and_dedupesis etcd-gated like the other sweep/dedupe tests)pending_wal_generationsdropping.Follow-up (not in this PR)
Smarter per-worker targeting (introspection endpoint + endpoint→shard routing) instead of global-count broadcast.
🤖 Generated with Claude Code