Skip to content

[SQL] Reject incompatible Parquet row reader conversions - #58496

Open
Jiayi-Wang-db wants to merge 1 commit into
apache:masterfrom
Jiayi-Wang-db:jiayi-wang-db/fix-parquet-row-reader-type-mismatch
Open

[SQL] Reject incompatible Parquet row reader conversions#58496
Jiayi-Wang-db wants to merge 1 commit into
apache:masterfrom
Jiayi-Wang-db:jiayi-wang-db/fix-parquet-row-reader-type-mismatch

Conversation

@Jiayi-Wang-db

Copy link
Copy Markdown

What changes were proposed in this pull request?

Make the Parquet row-based reader reject two incompatible primitive type conversions that the
vectorized reader already rejects:

  • Parquet FIXED_LEN_BYTE_ARRAY to Spark SQL STRING
  • Parquet INT32 annotated as DATE to Spark SQL DECIMAL

The row converter now accepts INT32 and INT64 as decimals only when their logical annotation
is decimal, absent, or a signed integer annotation. Unsupported conversions raise
SchemaColumnConvertNotSupportedException, consistently with the vectorized reader.

This follows the conversion semantics established by SPARK-34212 and SPARK-40876. The earlier
SPARK-32317 / PR #29600 discussion proposed a broader configurable conversion matrix but was not
merged.

Closes #58495.

Why are the changes needed?

The row-based reader currently ignores incompatible logical and physical type combinations. It
interprets a DATE value as an unannotated integer-backed decimal and interprets arbitrary
fixed-length bytes as UTF-8. This can silently return incorrect data, while reading the same file
and requested schema with the vectorized reader fails with a Parquet column type mismatch.

Both readers should reject these unsupported conversions instead of producing reader-dependent
results.

Does this PR introduce any user-facing change?

Yes. When the vectorized Parquet reader is disabled, the two unsupported schema conversions above
now fail with FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH instead of returning incorrectly
interpreted values. Supported binary-to-string and integer-to-decimal conversions are unchanged.

How was this patch tested?

Added an end-to-end test in ParquetIOSuite that writes raw Parquet schemas for both incompatible
conversions and verifies that the row-based and vectorized readers return the same error condition.

The focused test command was attempted locally:

build/sbt 'sql/core/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite -- -z "Parquet readers reject incompatible primitive type conversions consistently"'

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR @Jiayi-Wang-db, could you please make a Spark Jira ticket for this change? Please see: https://issues.apache.org/jira/projects/SPARK/

Also, could you please enable GHA / GitHub Actions workflows (please see "Pull request" in https://spark.apache.org/contributing.html)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SQL] Row-based Parquet reader silently accepts incompatible primitive type conversions

3 participants