fix: merge the WAL as delete-only merge_insert + append, not an upsert

[beinan]

Problem

The WAL merge uses WhenMatched::UpdateAll. Lance's upsert takes every matched target row with all of its columns to join and rewrite the fragment. Rollout rows carry multi-megabyte inline blobs, so in production:

• a merge matching 17k rows read 7.6 GB (459 KB/row); one matching 1.4k rows read 4.0 GB (3 MB/row)
• two such merges in flight held a 32 GiB worker at its limit — cgroup anon 27–30 GiB, dropping to 8 GiB within two minutes of the merges finishing
• 8 workers OOMKilled in two hours on the fix: stop reads and merges from opening the whole shard at once #278 image even with merges bounded to two per worker

Fix

merge_prepared_batches is now two commits:

1. a delete-only merge_insert on the key column (WhenMatched::Delete, WhenNotMatched::DoNothing) — probes the id BTree, touches only the key column and deletion vectors;
2. a plain Dataset::append of the new rows into fresh fragments.

No target blob is ever read. Last-write-wins is unchanged (read_flushed_generations already reduces to one newest row per key). A crash between the commits leaves the old rows deleted and the new rows still in the WAL; the retry deletes nothing and appends once — same end state.

Debug builds of the two commits overflow the 2 MiB test-thread stack inside DataFusion's optimizer walk (release builds fit). Tests get 8 MiB via .cargo/config.toml [env] RUST_MIN_STACK and the same variable in the CI workflow env.

Verification

• New merge_deletes_old_rows_and_appends_instead_of_rewriting_fragments: after overwriting one of two keys, the original fragment is kept with a deletion vector and the new row lands in a second fragment; the read returns the newer content exactly once.
• Full core suite (247 tests), server, master (incl. etcd --ignored) and lance-context suites pass; clippy clean.

🤖 Generated with Claude Code