[722] Implement iceberg versions for TableFormat and HoodieTableMetadata - #723
Open
bvaradar wants to merge 4 commits into
Open
[722] Implement iceberg versions for TableFormat and HoodieTableMetadata#723bvaradar wants to merge 4 commits into
bvaradar wants to merge 4 commits into
Conversation
| <spark.version>3.5.2</spark.version> | ||
| <spark.version.prefix>3.5</spark.version.prefix> | ||
| <iceberg.version>1.5.2</iceberg.version> | ||
| <delta.version>3.0.0</delta.version> |
Contributor
There was a problem hiding this comment.
Can we pull the spark, iceberg, and delta version upgrades into their own PR? It would help shrink the size of this PR
Contributor
There was a problem hiding this comment.
+1, we can split it into version upgrades and iceberg pluggable tf PR's
Contributor
|
Hi @bvaradar, are you still working on this, and are there any in-progress changes on your side? I'm interested in taking this forward. |
Contributor
|
@yihua I will get this PR rebased on top of latest main with hudi 1.2.0 version pinned as I have context on the changes made as part of the 1.x upgrade. After that you can take over. |
The pluggable table format landed in apache/hudi#13216 and ships in Hudi 1.2.0, which main already depends on. The merged SPI differs from the pre-merge API this module was written against: - org.apache.hudi.common.TableFormat is now HoodieTableFormat, so the META-INF/services resource is renamed to match. - TimelineFactory adds an abstract createArchivedTimeline(metaClient, boolean) and drops createCompletionTimeQueryView(metaClient, String). - HoodieTableFormat.archive() takes a Supplier<List<HoodieInstant>>. - HoodieAvroUtils.addMetadataFields moved to HoodieSchemaUtils and now operates on HoodieSchema. Also align the module with current main: the parent POM version, the scala-suffixed artifactId, and the PathBasedPartitionSpecExtractor and PathBasedPartitionValuesExtractor renames. TableSyncMetadata keeps a four-argument of() overload so existing callers do not change. Add delta-core to the module test scope, because the shared HudiTestUtil.getSparkConf registers the Delta catalog and extension.
…ormat The module's integration tests never activated the plugin, because nothing set hoodie.table.format. PR apache#723 worked around this by hardcoding the Iceberg format and disabling the metadata table inside the shared TestAbstractHudiTable, which would have changed every Hudi test table in xtable-core. Add an opt-in hook instead. TestJavaHudiTable.forStandardSchema now takes an optional Properties bag that a single test applies to one table. The bag reaches both hoodie.properties and the write config, so a test can also relax defaults its table format does not support. Three defaults now read from it rather than being hardcoded: the table version, the metadata table, and the column stats index. Existing callers pass an empty bag and are unaffected; xtable-core still passes 494 tests. Three defects surfaced once the hook let the plugin run: - HoodieCommitMetadata.getFullPathToInfo keys its map by the absolute path, but HudiDataFileExtractor.getDiffForCommit looked up the file name and passed the resulting null into HoodieBaseFile. Look up the absolute path, and fail with a clear message rather than a NullPointerException. - ConversionTargetFactory iterated the ServiceLoader directly, so a registered target whose engine is absent from the classpath aborted the lookup with a ServiceConfigurationError. Skip such providers, matching the fix already reviewed on PR apache#843. - IcebergTimelineFactory builds on the v2 timeline, so a table using this format needs table version 8, not the version 6 that xtable-core pins its other Hudi test tables to. Add two tests. TestIcebergTableFormatDiscovery checks that Hudi resolves IcebergTableFormat through the ServiceLoader and defaults to the native format otherwise. ITIcebergPluggableFormatSync proves the end to end contract: a plain Hudi write on a table configured with the Iceberg format produces a readable Iceberg snapshot with a matching row count, and no XTable sync job runs.
vinishjail97
force-pushed
the
iceberg-pluggable-tf-oss
branch
from
August 18, 2026 00:03
13fc241 to
3aac36c
Compare
2 tasks
Contributor
|
Thanks @vinishjail97 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important Read
ISS-722 Add Iceberg Pluggable table format managed directly by Hudi
What is the purpose of the pull request
ISS-722 Add Iceberg Pluggable table format managed directly by Hudi
Feature Request: Support for Pluggable Table Format in Apache Hudi
Apache Hudi is introducing a pluggable table format to enable interoperability with other popular table formats like Iceberg, Delta Lake, and Lance. This allows external engines (e.g., Snowflake) to write to Hudi tables using their native formats while maintaining compatibility. The design focuses on decoupling metadata and storage layers, inspired by Snowflake's metadata architecture. This enhancement supports the modular, composable lakehouse model and promotes broader ecosystem adoption.
This is the first PR
Corresponding Hudi PR : apache/hudi#13216
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)