Skip to content

[flink] Avoid creating empty lake writer in TieringSplitReader#3291

Merged
luoyuxia merged 2 commits intoapache:mainfrom
luoyuxia:fix/tiering-empty-lake-writer
May 9, 2026
Merged

[flink] Avoid creating empty lake writer in TieringSplitReader#3291
luoyuxia merged 2 commits intoapache:mainfrom
luoyuxia:fix/tiering-empty-lake-writer

Conversation

@luoyuxia
Copy link
Copy Markdown
Contributor

@luoyuxia luoyuxia commented May 9, 2026

Purpose

Linked issue: close #3290

Fix a tiering log reader bug where TieringSplitReader creates a lake writer before it knows whether the current polled batch contains any record below the split stoppingOffset.

Brief change log

  • Delay lake writer creation in TieringSplitReader.forLogRecords until the first record with record.logOffset() < stoppingOffset is encountered.
  • Keep the existing split finishing path unchanged so offset advancement without actual writes no longer produces an empty lake writer commit.
  • Generative AI disclosure: Codex (GPT-5).

Tests

  • Ran ./mvnw -pl fluss-flink/fluss-flink-common -DskipTests spotless:check checkstyle:check.
  • Did not run unit or integration tests, per request.

API and Format

No API or storage format changes.

Documentation

No documentation changes.

Copy link
Copy Markdown
Member

@fresh-borzoni fresh-borzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
nit: just a small regression test will be great, and also I checked quickly SnapshotSplitRecords, it's safe, but relies on BoundedSplitReader logic:

  • If a snapshot has zero records, BoundedSplitReader.readBatch() returns null
  • if Empty iterator -> writer created, but nothing written. Next call writes records to the same writer, so it's safe

Mb we wish to add the same symmetrical fix there or just leave comment, so that it's less brittle

@beryllw
Copy link
Copy Markdown
Contributor

beryllw commented May 9, 2026

LGTM!

Copy link
Copy Markdown
Member

@fresh-borzoni fresh-borzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, ty 👍

@luoyuxia luoyuxia merged commit 682b56e into apache:main May 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[flink] Avoid creating empty lake writer in TieringSplitReader

3 participants