Add claim-check offloaded message field columns to rollout schema - #172
Merged
Merged
Conversation
Add five nullable LargeUtf8 columns to the rollout schema for oversized rollout-message fields written via the claim-check path: model_input_string, model_output_string, rationale, problem_text, user_metadata. Each is an individually-projectable column so a reader can select one without materializing the rest. Wired through the RolloutRecord struct, rollout_schema(), records_to_batch, batch_to_rollout_records, the AddRolloutRequest/RolloutRecordDto DTOs, and both from_add_request/to_dto converters.
beinan
approved these changes
Jul 22, 2026
beinan
added a commit
that referenced
this pull request
Jul 23, 2026
## Summary - evolve legacy base tables to the latest additive `rollout_schema()` before WAL merge, adding missing nullable columns as all-null arrays - align flushed WAL batches to the evolved base table schema so old generations null-fill missing fields and newer generations preserve populated fields - reject missing required columns, type mismatches, and unknown columns instead of silently dropping data - preserve the existing base-first `scan_one_by_id` and `lsm_scanner_for_source` read path - keep legacy WAL point-lookup projections to columns guaranteed across generations; absent post-#172 fields decode as `None` - cover old base/old WAL merge, old base/new-schema value preservation, and base/WAL point lookups ## Testing - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -- -D warnings` - `cargo test -p lance-context-core -p lance-context-master --lib` - `cargo test -p lance-context-core --lib claim_check -- --nocapture`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds five nullable
LargeUtf8columns to the rollout schema for oversized rollout-message fields written via the claim-check path:model_input_stringmodel_output_stringrationaleproblem_textuser_metadataEach is an individually-projectable column, so a reader can select one without materializing the rest (rather than packing them into
content/binary_payload).Motivation
A downstream consumer persists oversized rollout-message fields as a claim-check row (one Lance row per message, one column per large field). The current published schema only supports
content,input_tokens,output_tokens,output_logprobs; the other five fields were silently dropped onadd()because they weren't inrollout_schema(). This adds them as first-class projectable columns.Changes
Wired the five fields through every symmetric site:
crates/lance-context-core/src/rollout.rs—RolloutRecordstructcrates/lance-context-core/src/rollout_store.rs—rollout_schema(),records_to_batch(builders + appends + array inserts),batch_to_rollout_records(reads + construction)crates/lance-context-api/src/lib.rs—AddRolloutRequest+RolloutRecordDtocrates/lance-context-core/src/api_impl.rs+crates/lance-context-server/src/routes/rollouts.rs— bothfrom_add_request/to_dtoconvertersmaster/routes.rs,master/scheduler.rs,tests/wal_merge_generation_cleanup.rsspecs/rollout-schema-design.md— documented the new column tierAll columns are nullable, so existing datasets and rows remain readable unchanged.
Test plan
add()→get()and each column projects independently