RFC-0066: Native execution support for Hudi MOR tables - #66
Open
jja725 wants to merge 5 commits into
Open
Conversation
jja725
force-pushed
the
rfc-hudi-mor-native
branch
from
August 17, 2026 21:19
8311e72 to
49eeb94
Compare
jja725
changed the title
jja725
force-pushed
the
rfc-hudi-mor-native
branch
from
August 17, 2026 21:32
49eeb94 to
0508137
Compare
jja725
force-pushed
the
rfc-hudi-mor-native
branch
from
August 17, 2026 22:03
0508137 to
31c8089
Compare
Make the query type a per-query choice via catalog session properties (hudi.query_type = snapshot | read_optimized | incremental), mirroring Spark's hoodie.datasource.query.type, and move incremental queries from the non-goals into scope. Incremental semantics follow Spark's MergeOnReadIncrementalRelationV2: the coordinator resolves the requested range to an explicit instant set (completion-time based on Hudi 1.x timelines), lists the latest merged file slices of the file groups touched by those commits, and the reader merges fully before filtering rows by _hoodie_commit_time membership in the set. Output is latest_state full merged records, not partial images. Validation mirrors the Spark reader's: populated meta fields required, missing begin instant rejected, and ranges broken by cleaning/archiving fail with the earliest servable instant instead of returning an incomplete change set. Since no Presto or Trino incremental behavior exists, the correctness oracle for incremental queries is the Hudi Spark reader; snapshot reads keep the JVM HudiRecordCursors oracle. Time travel and CDC-format output remain non-goals, with the instant-set machinery noted as their base.
Pull incremental queries back out of scope: they return to the non-goals, with the split and reader machinery noted as their natural base and a follow-up RFC as the path. No SQL engine offers Hudi incremental reads today, and carrying their semantics (instant-set resolution, retention validation, a Spark-only correctness oracle) tripled the surface of this proposal. Query-level selection stays, reduced to two values: hudi.query_type = snapshot | read_optimized. The OSS default remains snapshot, matching the JVM presto-hudi behavior this RFC treats as its oracle; a new hudi.default-query-type catalog config lets a deployment default to read_optimized instead - for fleets migrating MOR workloads from Hive-connector setups whose split listing already resolves Hudi tables to base files, so results and cost stay stable through the migration. With incremental gone the query type no longer travels to the worker: read_optimized is fully expressed by a split with an empty log-file list, so the native protocol and reader revert to their original shape.
Answer the "what about Hudi tables read through the Hive connector?" question the way Trino did (hive.hudi-catalog-name, Trino 411+): redirect, don't reimplement. A new section 4 adds an opt-in Hive connector config that redirects Hudi-format tables to the named Hudi catalog at planning time, so existing hive.* SQL reaches the native path unchanged. Extending the Hive connector's own Hudi integration to MOR snapshot becomes an explicit non-goal: the Hive protocol carries log files only as untyped customSplitInfo strings, offers no planning-time hook for compatibility validation, and would double the differential test surface. Redirected queries pick up the full Hudi connector surface, including hudi.query_type and the catalog default, so a deployment can make the redirect behavior-neutral (default-query-type=read_optimized) with snapshot as a per-query opt-in. Without the redirect, native workers keep rejecting Hive splits that carry delta-log paths instead of silently reading base files. Presto has no table redirection today, so this amends the adoption plan: one additive ConnectorMetadata default method modeled on Trino's redirectTable, generic enough for Iceberg and Delta reuse (listed as a follow-up). Adds a redirection risk item (policy evaluation moves to the target catalog), a rollout-order bullet, and redirection tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This RFC proposes native execution (Prestissimo/Velox) support for
SELECTqueries on Hudi MERGE_ON_READ tables, with the query type — snapshot (default) or read-optimized — chosen per query through a catalog session property, mirroring Spark'shoodie.datasource.query.type. The main piece is a native Hudi connector in Velox, built around a C++ HoodieLogFormat reader and record merger, structured like the existing Iceberg connector. It also covers the Presto native protocol, explicit Hudi handle conversion, coordinator-side compatibility validation, and native catalog registration.The revised design treats Java Hudi behavior as the correctness oracle while acknowledging that native semantic parity will be incremental. Each release declares a compatibility matrix for table versions, log formats, merge modes, payloads, key generation, and schema evolution. Unsupported table configurations fail during planning, and unsupported block details discovered on the worker fail the query; there is no JVM fallback from native execution.
Query-level selection needs no SQL grammar changes:
hudi.query_type=snapshot|read_optimized, defaulted by ahudi.default-query-typecatalog config (OSS defaultsnapshot, matching the JVM connector). Read-optimized is expressed entirely at split generation — latest base files, empty log list — so the native protocol and reader are unchanged by it, and no separate_rotable registration is needed. Incremental (change-pulling) queries and CDC output are non-goals: Spark-only today, natural follow-up RFC material on top of this machinery.For Hudi tables read through the Hive connector, the RFC follows Trino's model (
hive.hudi-catalog-name, Trino 411+): an opt-in, planning-time redirect to the Hudi catalog rather than extending the Hive connector's own Hudi integration to MOR snapshot. Redirected queries pick up compatibility validation and the query-type surface; without the redirect, native workers reject Hive splits carrying delta-log paths instead of silently reading base files. Presto has no table redirection today, so this adds one additiveConnectorMetadataSPI default method modeled on Trino'sredirectTable, generic enough for Iceberg and Delta reuse.Other key points:
Related: