feat(master): split rollout record listing by source (fragments/wal/all) - #170
Merged
Conversation
Browsing records via the master data browser scanned base ∪ every pending
MemWAL generation. For high-write experiments the pending count runs into
the hundreds, and each generation is a separate object-store open — so the
list took tens of seconds and intermittently 500'd when one generation's
request timed out. Latency was proportional to WAL backlog even though the
caller usually just wants already-merged data.
Add a `source` selector to the record-list read path, defaulting to
base-table-only:
- fragments (default): scan only the base table, skipping all WAL
generations. Fast and backlog-independent; may lag un-merged writes.
- wal: scan only the pending flushed MemWAL generations (the un-merged tail).
- all: the base ∪ WAL union (the previous behavior).
core: `ListSource` enum + `RolloutStore::list_filtered_source`, reusing
lance's `LsmScanner` primitives (empty snapshots = base-only;
`without_base_table` = wal-only) so the existing two-pass pagination and
batch conversion are unchanged. `list_filtered` stays as the `All` wrapper.
Fragments skips WAL manifest discovery entirely, so its latency is
independent of merge backlog. `get_by_id`/`get_blob` stay on the full union.
master: `?source=fragments|wal|all` on `/experiments/{name}/records`
(unknown value → 400), echoed back in the response.
ui: Fragments/WAL/All tabs in the records browser, defaulting to Fragments
with a note that it may lag the most recent un-merged writes.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
3 tasks
beinan
added a commit
that referenced
this pull request
Jul 22, 2026
…ace (#171) ## Summary Adds a runnable test environment + harness so the master's rollout-record HTTP surface — in particular the `?source=fragments|wal|all` selector shipped in #170 — can be exercised end to end. That path is only reachable with a live etcd + object store, so its integration test is `#[ignore]`d and never runs in CI; this closes that gap. Two interchangeable ways to bring the same stack up (etcd + master + worker(s), shared `DATA_DIR`): - **Docker Compose** (`test/docker-compose.yml` + single multi-stage `test/Dockerfile`): MinIO-backed `s3://lance-context` shared by master + two workers. WAL self-merge disabled so the source split is observable. - **Containerless** (`test/harness/native-*.sh`): same stack as plain host processes (static etcd binary, local-filesystem `DATA_DIR`, `cargo`-built binaries) for sandboxes where the kernel forbids `unshare`/netlink and no container can start. `test/harness/smoke.sh` makes real HTTP calls and asserts: `fragments`=0, `wal`=3, `all`=3 before any merge; response echoes the resolved source; default (no param) = `fragments`; unknown source → HTTP 400. ## Files - `test/Dockerfile` — multi-stage (ui → builder → master/worker) - `test/docker-compose.yml` — MinIO + etcd + master + 2 workers - `test/harness/{up,down,smoke}.sh` — Docker path + shared smoke assertions - `test/harness/native-{up,down}.sh` — containerless path - `test/harness/README.md`, `.dockerignore` ## Test plan / what was actually verified - [x] **Containerless path run locally and PASSES all smoke assertions** (real etcd + master + worker over HTTP): `fragments=0 / wal=3 / all=3`, default→fragments, unknown source→400. - [x] UI builds (`npm run build`); compose YAML parses; all shell scripts `bash -n` clean. - [ ] Docker Compose path: statically validated only — not run end to end here because this sandbox's kernel blocks container startup (`unshare: operation not permitted`). Needs a runner with a working container runtime to exercise `up.sh && smoke.sh`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GET /experiments/{name}/records(master data browser) lists viaRolloutStore::list_filtered→LsmScanner, which unions the base table with everypending MemWAL generation. For a high-write experiment the pending count runs into the
hundreds (700+ seen), and each generation is a separate object-store open+load — so
browsing takes tens of seconds and intermittently 500s when any one generation's abfss
request times out. List latency is proportional to WAL backlog even though the caller
usually just wants already-merged data.
PR #169 late-materialized pagination but the scan was still base∪WAL.
Change
Add a
sourceselector to the record-list read path, defaulting to base-table-only:and bounded; latency independent of WAL backlog. May lag the most recent un-merged writes.
Surfaces
rollout_store.rs):ListSourceenum +list_filtered_source, reusinglance's
LsmScannerprimitives —LsmScanner::new(base, [], pk)for base-only andLsmScanner::without_base_table(..)for wal-only — so the existing two-pass paginationand
batch_to_rollout_recordsare untouched.list_filteredremains a thinAllwrapper (existing callers/tests unchanged). Fragments skips WAL manifest discovery
entirely (zero per-shard reads).
get_by_id/get_blobstay on the full union so pointlookups still find un-merged rows.
routes.rs,error.rs, apiExperimentRecordsResponse):?source=fragments|wal|all(absent/empty → fragments; unknown → 400), echoed in the response.App.tsx,api.ts,styles.css): Fragments/WAL/All tabs, default Fragments,with a note that Fragments may lag un-merged writes.
Non-goals
Read latency only. Write latency / WAL-backlog growth are separate. Durability and the
union semantics of
allare unchanged.Testing
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warnings(clean)cargo test -p lance-context-core -p lance-context-master— 148 lib tests pass, incl.new
list_source_splits_base_and_wal(base-only omits un-merged rows; wal-only showsexactly them; both flip after a merge) and
parse_list_source_maps_and_defaults.crates/lance-context-master/ui && npm run build(tsc + vite) clean.Note: the pre-existing integration test
wal_merge_generation_cleanup::serial_merge_deletes_merged_generation_dirsfails on
origin/mainas well (assertsrow_count == 30butobserve()reports 32) —unrelated to this change, which does not touch the merge/observe path.
🤖 Generated with Claude Code