Consolidate the storage layer of the three fixed schemas (rollout as reference) + support arbitrary user-defined schemas

[beinan]

Summary

lance-context-core currently ships three independently-written fixed-schema stores:

Store	File	Key column	Schema
RolloutStore	crates/lance-context-core/src/rollout_store.rs	id	rollout_schema() (36 cols)
ContextStore	crates/lance-context-core/src/store.rs	id	schema_with_options() (26 cols)
DatagenStore	crates/lance-context-core/src/datagen_store.rs	event_id	datagen_log_schema() (33 cols)

They are structurally parallel but share almost no code. Over time they have forked at the storage layer, and their observable behavior now differs in ways that are bugs, not design choices.

This is a parent ticket covering two related goals:

1. Consolidate the low-level storage layer of the three stores onto one shared implementation, taking RolloutStore as the reference/canonical behavior.
2. Support arbitrary user-defined schemas on top of that shared layer, with a small set of enforced constraints (notably a required id field), while keeping add and merge semantics identical to the fixed-schema stores.

---

Part 1 — Consolidate the storage layer (rollout store is the reference)

Divergences we need to fix

Write / add path

• RolloutStore and DatagenStore keep a resident ShardWriter and do put → force_seal_active → wait_for_flush_drain on every add, so a generation is durable and cross-instance-visible when add returns.
• ContextStore opens and closes a new writer per call (write_entries), with no seal and no drain. Its writes are only visible after an internal Lance flush. It also has no fence-retry handling, because it never holds a writer.
• Result: add means three different things depending on which store you call.

WAL -> base merge

• RolloutStore::merge_own_shard is the correct implementation: close writer, scan generations, WriteMode::Append, claim_epoch + commit_update with a surgical generation drain, then remove the shard dir. Triggered by both a count threshold (maybe_merge_own_shard) and a time threshold (cleanup_own_shard).
• DatagenStore::merge_own_shard is a near line-for-line copy (plus a background cleanup task), and buffers every flushed generation fully in memory.
• ContextStore never merges at all. WAL generations accumulate forever and every read pays a full LsmScanner reconstruction over all shard manifests.

Compaction

• RolloutStore: synchronous, single-compactor, defer_index_remap: true to work around a Lance panic on the fieldless MemWAL index.
• ContextStore: has compaction plus a background task guarded by CompactionState, but without defer_index_remap.
• DatagenStore: no compaction at all — base fragments grow unbounded.

Indexing

• Rollout: manual ZoneMap on id.
• Context: automatic ZoneMap-or-BTree on open and post-compaction, but reopens with bare Dataset::open(uri) — dropping storage_options, which breaks on credentialed object stores. No vector index is ever built; search is a brute-force in-memory scan.
• Datagen: none.

Open / create

• The load_with_options / create_with_options / DatasetNotFound-fallback triad is written out four times (three stores + registry.rs), and the WriteParams + ObjectStoreParams + StorageOptionsAccessor idiom appears ~6 times.

Read path

• ContextStore::get scans self.dataset.scan() directly, bypassing the LSM scanner, so it can miss unflushed WAL rows — inconsistent with its own get_by_id.
• get_by_id / get_by_external_id do a full list() + linear find() instead of using the id index.
• ContextStore truncates limit/offset in memory instead of pushing down.
• Only RolloutStore's decode path tolerates missing columns (column_as_optional).

Errors

• Three conventions in-tree: inline LanceError::from(ArrowError::...) (rollout), a local fn invalid_input (datagen), native LanceError::invalid_input (eval), plus Result<_, String> in record.rs and the filter parsers.
• api_impl.rs::to_ctx_err string-matches Display output to recover a typed error, contradicting the stated intent of re-exporting lance::Error from lib.rs. is_fenced_error is likewise string-matching and duplicated in two files.

Misc

• update_visible_record mints Uuid::new_v4() even though id.rs documents UUIDv7 for index-tail clustering.
• RolloutFilters::from_json_value allows 6 keys while its own expression() supports 9 — dataset and content_type are unreachable from JSON.
• Nested structs (relationships, state_metadata) are decoded positionally; field reordering silently corrupts data.

Target

A single shared storage layer (working name lance-context-core::store_base) owning:

• open / create / load_with_options / create_with_options / storage-options plumbing
• resident writer lifecycle, fence detection + retry, Drop / close()
• add: encode -> ensure_mem_wal -> put -> force_seal_active -> wait_for_flush_drain (rollout semantics, for all stores)
• WAL merge: one merge_own_shard, count + time triggers, surgical generation drain, streaming (not fully buffered)
• compaction with defer_index_remap, single-compactor guard, optional background task
• id index management (ensure-on-open, re-ensure post-compaction), always via load_with_options
• LSM scanner construction with a configurable merge key
• one error taxonomy, with typed fence/not-found detection instead of string matching

RolloutStore, ContextStore, DatagenStore become thin schema+encode/decode layers over it. The features unique to ContextStore (lifecycle, supersession, tombstones, id-uniqueness validation) move up into the shared layer as opt-in capabilities rather than being dropped.

Existing seams to build on: crate::store's column_as / column_as_optional / timestamp_from_micros / CompactionConfig / CompactionStats, rollout_store::derive_shard_id, crate::storage (join_uri, validate_store_name), crate::id::generate_id.

Behavior changes to call out explicitly: ContextStore.add becomes synchronously durable and visible; ContextStore starts merging WAL into the base table; DatagenStore gains compaction and an event_id index. These are the point of the ticket, but they are visible changes and need release notes.

---

Part 2 — Arbitrary user-defined schemas

Users should be able to declare their own schema at store-creation time instead of picking one of three baked-in ones — but the storage behavior underneath must stay identical, especially add and merge.

Constraints to enforce at creation

• An id column is mandatory: Utf8, non-nullable, tagged lance-schema:unenforced-primary-key=true. It is always the LSM merge key.
• Reserved / prefixed names rejected: _rowid, _mem_wal, and the tombstone content_type sentinel.
• Only Arrow types supported by the generic encode/decode path.
• Blob columns declared explicitly (today VALID_BLOB_COLUMNS is a hardcoded 2-element list).
• Vector columns declared with dimension + distance metric so an index can actually be built (today this is schema metadata lance-context:distance_metric).
• Schema is immutable after creation in v1; evolution is a follow-up (only migrate_relationships_column exists today).

Why this is currently hard

Each store spells out its field list in five to seven independent places: schema fn, record->Arrow builders, Arrow->record decode, projection list, filter parser, merge key, index name. DatagenStore::events_to_batch is positional and unchecked.

The one genuinely schema-driven piece already in-tree is the final assembly step of both records_to_batch implementations, which iterates the dataset's fields and pulls from a name->array map, erroring on unknown columns. That already handles column reordering and omission — it just doesn't handle addition. That is the seed of the generic path.

ContextStore::schema_with_options is the only schema that is built rather than declared constant, so it is the natural insertion point.

API surface work

There is currently no schema abstraction at the API boundary at all — lance-context-api, -server and -client import zero arrow types, and transport is hand-written REST/JSON over axum with no proto/OpenAPI/codegen.

• CreateContextRequest / CreateRolloutStoreRequest must carry a schema declaration, and registry.rs must persist it (today it stores only name / uri / created_at).
• AddRecordRequest (20 fields, and it has no id — the server generates one), AddRolloutRequest (44 fields, does take a client id, so it is the better template), RecordDto, RolloutRecordDto, RecordPatchDto + its hand-written 16-way is_empty() all become schema-driven.
• Six duplicated conversion fns need deduping first: record_from_add_request and rollout_record_from_add_request each exist in both routes/ and api_impl.rs; server handlers bypass the API traits and call the inherent add directly.
• SearchRequest / RetrieveRequest need explicit column names once more than one vector/text column can exist.
• RolloutStoreApi::get_trajectory's default body hardcodes rollout_id / sequence_order inside the trait — must move out.
• Client add_rollouts selects binary parts by binary_payload.is_some(); must become schema-driven.
• Validation is scattered across four layers with no choke point, and no deny_unknown_fields anywhere (unknown keys are silently dropped). Needs one schema-driven validation entry point.
• DatagenStore has no API/server/client surface at all — decide whether it gets one or stays internal.

specs/rollout-schema-design.md explicitly documents the current one-schema-at-a-time-by-hand approach and should be superseded.

---

Proposed child tickets

1. Extract shared open/create/storage-options module; collapse the 4 duplicated triads.
2. Extract resident writer + fence-retry + close/Drop; migrate ContextStore onto it.
3. Unify the add path on rollout semantics (put -> seal -> drain); migrate ContextStore and DatagenStore.
4. Extract one merge_own_shard + count/time triggers; make it streaming; wire up ContextStore (new behavior).
5. Unify compaction: defer_index_remap everywhere, single-compactor guard, opt-in background task; add compaction to DatagenStore.
6. Unify index management; fix the Dataset::open storage-options drop; add an event_id index to DatagenStore.
7. Unify the error taxonomy; delete to_ctx_err string-matching and the duplicated is_fenced_error.
8. Read-path fixes: LSM-based ContextStore::get, index-accelerated point lookups, pushdown of limit/offset.
9. Name-based (not positional) nested struct encode/decode.
10. Generic schema-driven encode/decode engine (name-keyed builders + decoders).
11. SchemaSpec type + validation rules (required id, reserved names, blob/vector declarations) + registry persistence.
12. API/server/client: schema-carrying create, generic record DTOs, dedupe the 6 conversion fns, explicit column names in search/retrieve.
13. Docs: supersede specs/rollout-schema-design.md, write the arbitrary-schema guide + migration/release notes.

Out of scope

• Schema evolution after creation (follow-up).
• Changing the wire protocol away from REST/JSON.
• Vector index creation for ContextStore search (tracked separately, though the shared index layer should make room for it).