feat(master-ui): add read-only SQL query tab for experiments - #177
Merged
Conversation
Adds a SQL console to the master control-plane UI so operators can run
ad-hoc read-only SELECT queries against a chosen experiment's rollout
records (exposed as a table named `records`), covering aggregations and
group-bys the fixed-filter record browser cannot express.
Backend:
- RolloutStore::query_sql materializes the merged base+WAL view into an
in-memory DataFusion table and runs the query; binary_payload excluded.
- SELECT-only enforcement via DataFusion's SQL parser; DML/DDL, multiple
statements, and non-queries are rejected as InvalidInput (HTTP 400).
- Bounded by SQL_MAX_SCAN_ROWS (materialization) and SQL_MAX_RESULT_ROWS
(result); the latter sets a `truncated` flag rather than dropping rows
silently.
- POST /api/v1/experiments/{name}/query route; SqlQueryRequest/Response
DTOs; MasterError::from_lance_user maps user SQL errors to 400.
- datafusion 53 + arrow-json 58 pinned to the versions lance 7 already
resolves, so no duplicate arrow/datafusion is introduced.
Frontend:
- New SQL tab/route with an experiment picker, SQL editor (Cmd/Ctrl+Enter
to run), results grid, error box, and row-count/truncation notes.
Tests: core unit tests (count, group-by over merged data, blob column
rejected, DELETE rejected, empty-experiment columns) and an (etcd-gated)
route test for 200/400/404.
Co-Authored-By: Claude <noreply@anthropic.com>
- rustfmt: collapse over-wrapped assert!/let statements in the SQL tests. - clippy (redundant_clone/useless_conversion): drop the identity `ArrowError::from` around arrow-json writer errors, which already return `ArrowError`; map straight through `LanceError::from`. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SELECTqueries against its rollout records (exposed as a table namedrecords). Covers aggregations / group-bys the fixed-filter record browser can't express.SELECT/CTE is accepted; DML/DDL/multi-statement input is rejected as a 400.Backend
RolloutStore::query_sqlmaterializes the merged non-blob rows into an in-memory DataFusionMemTableand runs the validated query (binary_payloadexcluded).ensure_select_only).SQL_MAX_SCAN_ROWS(200k, hard error) andSQL_MAX_RESULT_ROWS(10k, setstruncatedinstead of silently dropping rows).POST /api/v1/experiments/{name}/query;SqlQueryRequest/SqlQueryResponseDTOs;MasterError::from_lance_usermaps user SQL errors to 400, others to 500.datafusion 53+arrow-json 58— the exact versions lance 7 already resolves (both on arrow 58.3.0), so no duplicate arrow/datafusion is pulled in.Cargo.lockdelta is just two lines.Frontend
SqlConsolecomponent +/sqlroute and nav tab: experiment picker, monospace SQL editor (Cmd/Ctrl+Enter to run), results grid (reuses the horizontally-scrollable.table-wrap), error box, and row-count / truncation notes.Test plan
cargo test -p lance-context-core— new unit tests: count, group-by over merged data,binary_payloadrejected,DELETErejected, empty-experiment column resolution. Full lib suite: 158 passed.#[ignore], like the existing record-endpoint tests): 200 valid / 400DROP/ 404 unknown experiment.lance-context-core+lance-context-masterbuild warning-free; UI builds clean (tsc + vite).SELECT problem_id, count(*) FROM records GROUP BY problem_id ORDER BY 2 DESC; confirm a malformed query surfaces the SQL error; confirm a just-written (un-merged/WAL) row appears.Note: the pre-existing
serial_merge_deletes_merged_generation_dirsintegration test fails onmainindependently of this change.🤖 Generated with Claude Code