fix: treat empty DataFrame writes as noop on immediate-commit path - #769
Open
yuw1 wants to merge 3 commits into
Open
fix: treat empty DataFrame writes as noop on immediate-commit path#769yuw1 wants to merge 3 commits into
yuw1 wants to merge 3 commits into
Conversation
…path
Spark sink contract: when a DataFrame pipeline produces zero rows
(eg after a filter/aggregate), df.write.format("lance").save(uri)
should succeed silently, not raise.
Lance-Java Append.builder() rejects empty fragments with
"IllegalArgumentException: fragments cannot be null or empty", so
the immediate-commit path of LanceBatchWrite.commit() raised. Short-
circuit before building the operation when both stagedCommit is null
(immediate-commit path) and fragments is empty (no upstream data).
The staged-commit path goes through Overwrite.builder(), which does
not enforce the empty-fragments check and is the canonical way to
create an empty table via Spark v2 DataFrameWriter (CTAS / RTAS), so
we leave it untouched. A dedicated unit test covers the staged path to
guard against future regressions.
Adds:
- LanceBatchWrite: 1-line short-circuit + 4-line comment explaining
why this only applies to the non-staged path.
- LanceBatchWriteTest.testCommitEmptyFragmentsIsNoop: unit test for
the bug repro on the immediate-commit path.
- LanceBatchWriteTest.testCommitEmptyFragmentsStagedIsNoop: unit test
for the staged path (path-based staged create of an empty table).
- integration-tests/test_lance_spark.py.test_write_empty_dataframe_is_noop:
pyspark end-to-end test that mirrors the original bug report.
Address lance-gatekeeper review feedback on the previous commit: - The earlier guard short-circuited every empty immediate-commit write, including `overwrite=true` / `write_mode=OVERWRITE`. Empty overwrites must still go through `Overwrite.builder()` so the table is truncated and the pinned-version transaction is recorded. - Move the `isOverwrite` computation ahead of the guard so the condition can reference it. - Add `testCommitEmptyOverwriteTruncatesTable` covering the populated-table → empty-overwrite transition. - Add a parallel PySpark integration test (local filesystem backend). Tests: `mvnw test` / `spotless:check` / `checkstyle:check` / `git diff --check` clean.
Address lance-gatekeeper review feedback:
The existing `test_write_empty_dataframe_is_noop` exercises the staged
table creation path (CTAS-style `df.writeTo('t').create()`), which
already accepted empty fragments via `Operation::Overwrite` before this
PR. As a result, that test still passes if the new immediate-append
guard in `LanceBatchWrite.commit()` is removed, and does not provide
the integration coverage the behavior change requires.
Add `test_append_empty_dataframe_to_existing_path_is_noop`:
- Seed an existing Lance path with three rows.
- Run `empty_df.write.format('lance').mode('append').save(path)`.
- Assert the seed rows remain unchanged.
This routes through `LanceBatchWrite.commit()` on the
immediate-commit path (`stagedCommit == null && !isOverwrite`) and
hits the new guard end-to-end via PySpark.
Tests: `mvnw test` / `spotless:check` / `checkstyle:check` / `git diff --check` clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a Spark DataFrame pipeline produces zero rows upstream,
df.write.format("lance").save(uri)currently fails withIllegalArgumentException: fragments cannot be null or empty. This PR makes the immediate-commit path ofLanceBatchWrite.commit()a no-op on empty fragments.The staged-commit path (Spark v2 DataFrameWriter's CTAS-style API, i.e.
df.writeTo("t").create()) already accepts empty fragments viaOperation::Overwriteand is the canonical way to create an empty table — that path is intentionally left untouched.Fix
Short-circuit
LanceBatchWrite.commit()on the immediate-commit path:Tests
LanceBatchWriteTest.testCommitEmptyFragmentsIsNoop— unit test reproducing the bug on the immediate-commit path.LanceBatchWriteTest.testCommitEmptyFragmentsStagedIsNoop— unit test guarding the staged-commit path (path-based staged create of an empty table).LanceBatchWriteTest.testCommitEmptyOverwriteTruncatesTable— regression guard for the empty overwrite truncate path.integration-tests/test_lance_spark.py::TestDMLInsert::test_write_empty_dataframe_is_noop— pyspark end-to-end test mirroring the original repro.integration-tests/test_lance_spark.py::TestDMLInsert::test_write_empty_dataframe_overwrite_clears_table— pyspark end-to-end test for the empty overwrite truncate (local filesystem backend).integration-tests/test_lance_spark.py::TestDMLInsert::test_append_empty_dataframe_to_existing_path_is_noop— pyspark end-to-end test exercising the immediate-append guard on an existing Lance path (seed → empty append → seed rows unchanged).Validation:
LanceBatchWriteTest: 586 / 586 tests pass onlance-spark-base_2.12.spotless:check/checkstyle:checkclean.git diff --checkclean.testCommitEmptyFragmentsIsNoopfails with the sameIllegalArgumentException(fragments cannot be null or empty) atLanceBatchWrite.commit→Append.<init>(thePreconditions.checkArgument(!isEmpty())guard), matching the original bug report.Diff
Closes #768