Core: Preserve explicit delete_data_file() in _DeleteFiles - #3858
Open
qzyu999 wants to merge 2 commits into
Open
Core: Preserve explicit delete_data_file() in _DeleteFiles#3858qzyu999 wants to merge 2 commits into
qzyu999 wants to merge 2 commits into
Conversation
qzyu999
force-pushed
the
fix-delete-files-explicit-data-file
branch
2 times, most recently
from
August 26, 2026 04:34
1fc26cd to
b87a47c
Compare
…eletes Fixes apache#3857 _compute_deletes resets self._deleted_data_files before scanning manifests by predicate. Files added via the inherited delete_data_file() method were silently dropped. Preserve them and include them in the should_delete check alongside predicate-matched files.
qzyu999
force-pushed
the
fix-delete-files-explicit-data-file
branch
from
August 26, 2026 05:37
b87a47c to
9813657
Compare
rambleraptor
suggested changes
Aug 26, 2026
| @@ -0,0 +1,257 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
Collaborator
There was a problem hiding this comment.
Can you consolidate these tests into https://github.com/apache/iceberg-python/blob/main/tests/table/test_snapshots.py?
This is where the rest of the snapshot tests live.
| existing_manifests = [] | ||
| total_deleted_entries = [] | ||
| partial_rewrites_needed = False | ||
| # Preserve files explicitly requested via delete_data_file() before resetting. |
Collaborator
There was a problem hiding this comment.
This seems reasonable.
| from pyiceberg.types import LongType, NestedField | ||
|
|
||
|
|
||
| @pytest.fixture() |
Collaborator
There was a problem hiding this comment.
Can we reuse existing fixtures?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3857
Rationale for this change
_DeleteFiles._compute_deletesresetsself._deleted_data_files = set()before scanning manifests by predicate. Files added via the inheriteddelete_data_file()method were silently dropped because the reset discards them before the manifest scan begins.This means calling
delete_data_file()on a_DeleteFilesinstance (viaupdate_snapshot().delete()) produces no error and no effect.Root cause
_compute_deletesrebuilds_deleted_data_filesfrom predicate-matched entries only. Explicit file references added before the computation were lost.Fix
Preserve the explicit set before resetting, and include those files in the
should_deletecheck alongside predicate evaluation. This is a 6-line production change.Are these changes tested?
Two new tests covering:
delete_data_file()on a_DeleteFilesinstance deletes the fileBoth pass across all 3 catalog backends (memory, sql, sql_without_rowcount). Existing snapshot and commit-retry tests (169 total) continue to pass.
Are there any user-facing changes?
delete_data_file()on a_DeleteFilesinstance now correctly deletes the specified file instead of silently doing nothing.