Skip to content

Fix/spark iceberg sort merge - #4234

Closed
fkzhao wants to merge 3 commits into
apache:mainfrom
fkzhao:fix/spark-iceberg-sort-merge
Closed

Fix/spark iceberg sort merge#4234
fkzhao wants to merge 3 commits into
apache:mainfrom
fkzhao:fix/spark-iceberg-sort-merge

Conversation

@fkzhao

@fkzhao fkzhao commented Sep 4, 2026

Copy link
Copy Markdown

Purpose

Fix Spark reads for Fluss lake-enabled primary-key tables.

Brief change log

  • Handle empty log-record batches without throwing NoSuchElementException.
  • Add a sorted Iceberg reader for lake snapshot and log merge reads.
  • Support comparator-only readers when no Iceberg split is available.
  • Correct primary-key projection positions during sort-merge reads.
  • Preserve timestamp-with-time-zone comparisons.
  • Avoid reusing mutable Iceberg row objects during sorting.

Tests

  • IcebergLakeSourceTest: 5 tests passed.
  • Added regression coverage for:
    • Empty log records.
    • Missing Iceberg split.
    • Projected primary-key rows.
    • Timestamp-with-time-zone primary keys.

API and Format

  • No external API changes.
  • No data format or storage format changes.
  • No migration is required.

Documentation

No documentation changes are required.

@fkzhao fkzhao closed this Sep 4, 2026
@fkzhao
fkzhao deleted the fix/spark-iceberg-sort-merge branch September 7, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant