perf(master): speed up rollout record browsing - #169
Merged
Conversation
beinan
added a commit
that referenced
this pull request
Jul 22, 2026
…ll) (#170) ## Problem `GET /experiments/{name}/records` (master data browser) lists via `RolloutStore::list_filtered` → `LsmScanner`, which unions the base table with **every** pending MemWAL generation. For a high-write experiment the pending count runs into the hundreds (700+ seen), and each generation is a separate object-store open+load — so browsing takes tens of seconds and intermittently 500s when any one generation's abfss request times out. List latency is proportional to WAL backlog even though the caller usually just wants already-merged data. PR #169 late-materialized pagination but the scan was still base∪WAL. ## Change Add a `source` selector to the record-list read path, defaulting to base-table-only: - **fragments** (default) — scan only the base table, skipping all WAL generations. Fast and bounded; latency independent of WAL backlog. May lag the most recent un-merged writes. - **wal** — scan only the pending flushed MemWAL generations (the un-merged tail). - **all** — the base ∪ WAL union (previous behavior, fully consistent). ### Surfaces - **core** (`rollout_store.rs`): `ListSource` enum + `list_filtered_source`, reusing lance's `LsmScanner` primitives — `LsmScanner::new(base, [], pk)` for base-only and `LsmScanner::without_base_table(..)` for wal-only — so the existing two-pass pagination and `batch_to_rollout_records` are untouched. `list_filtered` remains a thin `All` wrapper (existing callers/tests unchanged). Fragments skips WAL manifest discovery entirely (zero per-shard reads). `get_by_id`/`get_blob` stay on the full union so point lookups still find un-merged rows. - **master** (`routes.rs`, `error.rs`, api `ExperimentRecordsResponse`): `?source=fragments|wal|all` (absent/empty → fragments; unknown → 400), echoed in the response. - **ui** (`App.tsx`, `api.ts`, `styles.css`): Fragments/WAL/All tabs, default Fragments, with a note that Fragments may lag un-merged writes. ## Non-goals Read latency only. Write latency / WAL-backlog growth are separate. Durability and the union semantics of `all` are unchanged. ## Testing - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -- -D warnings` (clean) - `cargo test -p lance-context-core -p lance-context-master` — 148 lib tests pass, incl. new `list_source_splits_base_and_wal` (base-only omits un-merged rows; wal-only shows exactly them; both flip after a merge) and `parse_list_source_maps_and_defaults`. - `crates/lance-context-master/ui && npm run build` (tsc + vite) clean. Note: the pre-existing integration test `wal_merge_generation_cleanup::serial_merge_deletes_merged_generation_dirs` fails on `origin/main` as well (asserts `row_count == 30` but `observe()` reports 32) — unrelated to this change, which does not touch the merge/observe path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark
Local debug build, 25-row page over 90,000 rows and 52 fragments:
Testing
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p lance-context-core -p lance-context-master --no-runcargo test -p lance-context-core -p lance-context-master --lib(core: 147 passed; master: 7 passed)cargo test -p lance-context-core bench_master_pagination_90k_52_fragments -- --ignored --nocaptureuv run --frozen ruff format --check python/uv run --frozen ruff check python/uv run --frozen pyrightuv run --frozen pytest(184 passed, 12 existing unrelated failures, 2 skipped, 1 xfailed; Python CI is not triggered by this Rust-only change)