Skip to content

[spark] Support custom Paimon lake table paths - #4230

Open
BackendArchitectX wants to merge 1 commit into
apache:mainfrom
BackendArchitectX:spark-custom-paimon-lake-path-3832
Open

[spark] Support custom Paimon lake table paths#4230
BackendArchitectX wants to merge 1 commit into
apache:mainfrom
BackendArchitectX:spark-custom-paimon-lake-path-3832

Conversation

@BackendArchitectX

Copy link
Copy Markdown

Purpose

Linked issue: close #3832

Support Spark reads from Paimon lake tables whose database and/or table name is configured independently from the Fluss table path.

Previously, Spark rejected lake reads when the resolved Paimon table path differed from the Fluss table path.

Brief change log

  • Resolve the configured lake table path before creating the Spark lake source.
  • Use the resolved Paimon path for lake reads while preserving the original Fluss table path.
  • Enable lake predicate pushdown for tables using custom lake paths.
  • Add coverage for:
    • custom database mapping
    • custom table mapping
    • combined database and table mapping
    • lake-only reads
    • lake + log union reads
    • predicate pushdown with a custom lake path
    • no-lake-snapshot fallback

Tests

Passed:

  • SparkLakePaimonLogTableReadTest
    • 18 tests passed
  • SparkLakePaimonPrimaryKeyTableReadTest
    • 10 tests passed
  • spotless:check for fluss-spark-common and fluss-spark-ut
  • checkstyle:check for fluss-spark-common and fluss-spark-ut
  • git diff --check

API and Format

No public API or storage format changes.

Documentation

No documentation changes are required. This change enables Spark support for the existing custom Paimon lake table path configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[spark] Support custom Paimon lake table paths

1 participant