Skip to content

fix: treat empty DataFrame writes as noop on immediate-commit path - #769

Open
yuw1 wants to merge 3 commits into
lance-format:mainfrom
yuw1:fix/spark-empty-write-skip-append
Open

fix: treat empty DataFrame writes as noop on immediate-commit path#769
yuw1 wants to merge 3 commits into
lance-format:mainfrom
yuw1:fix/spark-empty-write-skip-append

Conversation

@yuw1

@yuw1 yuw1 commented Aug 16, 2026

Copy link
Copy Markdown

Summary

When a Spark DataFrame pipeline produces zero rows upstream, df.write.format("lance").save(uri) currently fails with IllegalArgumentException: fragments cannot be null or empty. This PR makes the immediate-commit path of LanceBatchWrite.commit() a no-op on empty fragments.

The staged-commit path (Spark v2 DataFrameWriter's CTAS-style API, i.e. df.writeTo("t").create()) already accepts empty fragments via Operation::Overwrite and is the canonical way to create an empty table — that path is intentionally left untouched.

Fix

Short-circuit LanceBatchWrite.commit() on the immediate-commit path:

// Spark sink contract: empty immediate append writes should be a noop (not raise).
// Lance-Java Append rejects empty fragments, so skip building the operation.
// StagedCommit goes through Overwrite which does not enforce this check, so we
// only short-circuit the non-staged path. Empty overwrites must still go through
// Overwrite.builder() so the table is truncated and OCC is enforced.
if (stagedCommit == null && !isOverwrite && fragments.isEmpty()) {
  return;
}

Tests

  • LanceBatchWriteTest.testCommitEmptyFragmentsIsNoop — unit test reproducing the bug on the immediate-commit path.
  • LanceBatchWriteTest.testCommitEmptyFragmentsStagedIsNoop — unit test guarding the staged-commit path (path-based staged create of an empty table).
  • LanceBatchWriteTest.testCommitEmptyOverwriteTruncatesTable — regression guard for the empty overwrite truncate path.
  • integration-tests/test_lance_spark.py::TestDMLInsert::test_write_empty_dataframe_is_noop — pyspark end-to-end test mirroring the original repro.
  • integration-tests/test_lance_spark.py::TestDMLInsert::test_write_empty_dataframe_overwrite_clears_table — pyspark end-to-end test for the empty overwrite truncate (local filesystem backend).
  • integration-tests/test_lance_spark.py::TestDMLInsert::test_append_empty_dataframe_to_existing_path_is_noop — pyspark end-to-end test exercising the immediate-append guard on an existing Lance path (seed → empty append → seed rows unchanged).

Validation:

  • LanceBatchWriteTest: 586 / 586 tests pass on lance-spark-base_2.12.
  • spotless:check / checkstyle:check clean.
  • git diff --check clean.
  • Manually reverted the fix to confirm testCommitEmptyFragmentsIsNoop fails with the same IllegalArgumentException (fragments cannot be null or empty) at LanceBatchWrite.commitAppend.<init> (the Preconditions.checkArgument(!isEmpty()) guard), matching the original bug report.

Diff

 integration-tests/test_lance_spark.py              |  67 +++++++++++
 .../org/lance/spark/write/LanceBatchWrite.java     |   9 ++
 .../org/lance/spark/write/LanceBatchWriteTest.java | 123 +++++++++++++++++++++
 3 files changed, 80 insertions(+), 7 deletions(-)

Closes #768

…path

Spark sink contract: when a DataFrame pipeline produces zero rows
(eg after a filter/aggregate), df.write.format("lance").save(uri)
should succeed silently, not raise.

Lance-Java Append.builder() rejects empty fragments with
"IllegalArgumentException: fragments cannot be null or empty", so
the immediate-commit path of LanceBatchWrite.commit() raised. Short-
circuit before building the operation when both stagedCommit is null
(immediate-commit path) and fragments is empty (no upstream data).

The staged-commit path goes through Overwrite.builder(), which does
not enforce the empty-fragments check and is the canonical way to
create an empty table via Spark v2 DataFrameWriter (CTAS / RTAS), so
we leave it untouched. A dedicated unit test covers the staged path to
guard against future regressions.

Adds:
- LanceBatchWrite: 1-line short-circuit + 4-line comment explaining
  why this only applies to the non-staged path.
- LanceBatchWriteTest.testCommitEmptyFragmentsIsNoop: unit test for
  the bug repro on the immediate-commit path.
- LanceBatchWriteTest.testCommitEmptyFragmentsStagedIsNoop: unit test
  for the staged path (path-based staged create of an empty table).
- integration-tests/test_lance_spark.py.test_write_empty_dataframe_is_noop:
  pyspark end-to-end test that mirrors the original bug report.
@github-actions github-actions Bot added the bug Something isn't working label Aug 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 16, 2026
Address lance-gatekeeper review feedback on the previous commit:

- The earlier guard short-circuited every empty immediate-commit write,
  including `overwrite=true` / `write_mode=OVERWRITE`. Empty overwrites
  must still go through `Overwrite.builder()` so the table is truncated
  and the pinned-version transaction is recorded.
- Move the `isOverwrite` computation ahead of the guard so the condition
  can reference it.
- Add `testCommitEmptyOverwriteTruncatesTable` covering the
  populated-table → empty-overwrite transition.
- Add a parallel PySpark integration test (local filesystem backend).

Tests: `mvnw test` / `spotless:check` / `checkstyle:check` / `git diff --check` clean.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 16, 2026
Address lance-gatekeeper review feedback:

The existing `test_write_empty_dataframe_is_noop` exercises the staged
table creation path (CTAS-style `df.writeTo('t').create()`), which
already accepted empty fragments via `Operation::Overwrite` before this
PR. As a result, that test still passes if the new immediate-append
guard in `LanceBatchWrite.commit()` is removed, and does not provide
the integration coverage the behavior change requires.

Add `test_append_empty_dataframe_to_existing_path_is_noop`:
  - Seed an existing Lance path with three rows.
  - Run `empty_df.write.format('lance').mode('append').save(path)`.
  - Assert the seed rows remain unchanged.

This routes through `LanceBatchWrite.commit()` on the
immediate-commit path (`stagedCommit == null && !isOverwrite`) and
hits the new guard end-to-end via PySpark.

Tests: `mvnw test` / `spotless:check` / `checkstyle:check` / `git diff --check` clean.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 16, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The mode-specific empty-write behavior is now correct, and the added integration test exercises the immediate empty-append path against an existing dataset.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Empty DataFrame writes fail with "fragments cannot be null or empty"

1 participant