Skip to content

Support INSERT OVERWRITE for memory tables - #58

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
snowflake-insert-overwrite-memory
Sep 4, 2026
Merged

Support INSERT OVERWRITE for memory tables#58
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
snowflake-insert-overwrite-memory

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

  • Supports the Rustice Snowflake compatibility case for INSERT OVERWRITE on development memory tables.

Rationale for this change

MemTable currently rejects InsertOp::Overwrite, so valid INSERT OVERWRITE statements fail during physical planning. This prevents DataFusion-backed local environments from matching Snowflake DML behavior even though the memory sink already buffers the complete input before modifying the table.

What changes are included in this PR?

  • Add an opt-in overwrite mode to MemSink while preserving append as the default.
  • Allow MemTable to plan InsertOp::Overwrite; InsertOp::Replace remains unsupported.
  • Replace each target partition only after the input stream completes successfully, so an input error leaves existing batches untouched.
  • Add integration coverage for replacing existing rows and clearing a table with an empty overwrite input.

Are these changes tested?

  • cargo +1.97.0 fmt --all -- --check
  • cargo +1.97.0 test -p datafusion datasource::memory_test --lib (15 passed)
  • cargo +1.97.0 clippy --all-targets --all-features -- -D warnings
  • Extended workspace suite: all memory-table tests passed. One unrelated existing memory-limit runner failed because the process baseline RSS (158.8 MB) already exceeded its fixed 152.6 MB limit.

Are there any user-facing changes?

Yes. DataFusion memory tables now support INSERT OVERWRITE; append behavior and existing MemSink::try_new callers are unchanged.

@osipovartem
osipovartem requested review from Vedin and rampage644 and removed request for rampage644 September 4, 2026 16:23
@osipovartem
osipovartem merged commit e140250 into embucket-sync-df55.0.0 Sep 4, 2026
43 of 66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant