fix(rollout): seal the memtable in cleanup_own_shard before merging

[beinan]

Fixes #184.

Problem

ROLLOUT_FLUSH_INTERVAL_SECS's documentation says 0 disables periodic flush and rows "then only become visible when the cleanup/merge path flushes them". That fallback did not exist.

cleanup_own_shard delegates to merge_own_shard_if_ready(1), which reads the shard manifest and short-circuits:

let pending = manifest.flushed_generations.len();
if pending == 0 || pending < threshold.max(1) {
    return Ok(0);
}

With no periodic flush, nothing seals the active memtable, so flushed_generations is always empty, so this returns 0 and never reaches merge_own_shard — which contains the only other close() (and therefore the only other seal) on that path.

Net effect with ROLLOUT_FLUSH_INTERVAL_SECS=0: add persists the WAL entry, so no data loss, but the rows are invisible to every reader indefinitely — until the process restarts and replays the WAL.

Verification

Confirmed with a throwaway probe before writing the fix:

before_close_visible=0
observe_row_count=0 pending_gens=0
cleanup_merged=0          <-- cleanup could not rescue it
after_cleanup_visible=0   <-- still invisible
after_close_visible=1     <-- only close() sealed it

Change

1. cleanup_own_shard flushes before the threshold check (rollout_store.rs). One line, plus a comment explaining why the ordering matters. Cleanup becomes a genuine standalone fallback, as the config already claimed.

2. Startup warning when both intervals are 0 (main.rs). Cleanup-alone is now sufficient, but flush=0 and cleanup=0 still leaves nothing to seal. That is a legitimate (if unusual) configuration, so it warns rather than refusing to boot.

3. Config doc corrected to describe what each combination actually does.

Testing

New cleanup_own_shard_seals_before_merging asserts a single cleanup pass seals, merges, and exposes the row, with the count trigger explicitly disabled so cleanup is the only path that can help.

I verified it is a real guard by reverting just the self.flush().await? line:

FAILED: assertion `left == right` failed: cleanup must seal then merge the generation

Full runs: lance-context-core --lib 161 passed / lance-context-server 49 passed.

Note on scope

ShardWriter::close() does seal the active memtable (lance-7.0.0 write.rs:1911-1921), and RolloutStore's Drop spawns a detached close — so LRU eviction is not affected by this bug. That was the concern in #185, which I will close separately with these findings.

🤖 Generated with Claude Code