Skip to content

Support INSERT OVERWRITE for MemTable - #24969

Open
osipovartem wants to merge 3 commits into
apache:mainfrom
Embucket:upstream-memtable-insert-overwrite
Open

Support INSERT OVERWRITE for MemTable#24969
osipovartem wants to merge 3 commits into
apache:mainfrom
Embucket:upstream-memtable-insert-overwrite

Conversation

@osipovartem

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

DataFusion parses and plans INSERT OVERWRITE, but MemTable rejects every insert operation except append. Users therefore cannot replace the contents of an in-memory table with the standard overwrite operation.

What changes are included in this PR?

  • allow InsertOp::Overwrite in MemTable::insert_into
  • let MemSink replace each target partition after the input stream completes successfully
  • continue rejecting the distinct InsertOp::Replace operation

What is the testing strategy for this PR?

Two unit tests verify that overwrite replaces existing rows and that an empty overwrite clears the table.

Validated with:

  • cargo test -p datafusion --lib test_insert_overwrite -- --nocapture
  • cargo clippy -p datafusion -p datafusion-datasource -p datafusion-catalog --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check

Are there any user-facing changes?

Yes. INSERT OVERWRITE now replaces all existing data in a MemTable. There are no breaking public API changes.

@github-actions github-actions Bot added core Core DataFusion crate catalog Related to the catalog crate datasource Changes to the datasource crate labels Sep 6, 2026
@codecov-commenter

codecov-commenter commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.12346% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.67%. Comparing base (262936e) to head (6a6f047).

Files with missing lines Patch % Lines
datafusion/core/src/datasource/memory_test.rs 89.85% 4 Missing and 3 partials ⚠️
datafusion/catalog/src/memory/table.rs 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #24969   +/-   ##
=======================================
  Coverage   81.67%   81.67%           
=======================================
  Files        1126     1126           
  Lines      414842   414919   +77     
  Branches   414842   414919   +77     
=======================================
+ Hits       338841   338905   +64     
- Misses      56070    56079    +9     
- Partials    19931    19935    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@osipovartem
osipovartem force-pushed the upstream-memtable-insert-overwrite branch from 2d7b03b to 2e11c24 Compare September 6, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate core Core DataFusion crate datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants