Parquet Variant annotation validation is bypassed when reading as an ordinary struct

[rich7420]

Describe the bug

With Comet enabled on Spark 4.1.3, reading a Parquet VARIANT-annotated field as an ordinary struct<value binary, metadata binary> does not raise the error Spark requires when SQLConf.PARQUET_IGNORE_VARIANT_ANNOTATION is false.

Found while restoring the Variant shredding suites for #5569. The existing Spark test ParquetVariantShreddingSuite / variant logical type annotation - ignore variant annotation fails with:

Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown
(ParquetVariantShreddingSuite.scala:277)

CI log. This is an observed missing exception, not evidence of silent-null corruption in this case.

Steps to reproduce

Use Spark v4.1.3 with the Comet patch, removing the IgnoreCometSuite mixin from ParquetVariantShreddingSuite. The failure was observed in the fork Spark 4.1 CI job, at commit 00fa408852c5fa1714fdd00b3de8212c2f2394ce, based on Comet 7e1984399eb887cd13109698ee55cf2ce150f849.

Run the suite with Comet enabled through the Spark SQL test harness. A focused command after the documented Comet build/install and Spark patch setup is:

NOLINT_ON_COMPILE=true ENABLE_COMET=true ENABLE_COMET_ONHEAP=true \
  build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetVariantShreddingSuite'

The CI reproduction used the full sql_core-1 test selection, not this focused command. Environment: Spark 4.1.3, JDK 17, Linux.

The existing test writes annotated Variant values at the top level and inside structs, arrays and maps. It checks both values of PARQUET_IGNORE_VARIANT_ANNOTATION. The true arm succeeds; the false arm reaches struct_result.collect() without throwing.

Expected behavior

Match Spark's reader policy: reject the incompatible struct read when Variant annotations must be respected, and preserve the successful struct read when ignoring annotations is explicitly enabled.

Spark expects a SparkException caused by an AnalysisException with condition _LEGACY_ERROR_TEMP_3071 and an Invalid Spark read type message.

Additional context

Source tracing points to a different boundary from #4084's shredded-Variant fallback:

• Spark's ParquetToSparkSchemaConverter validates the physical VARIANT annotation against the requested Spark type.
• Comet's scan schema check examines the requested schema. The manually supplied ordinary struct does not carry the per-field Variant metadata used by isVariantStruct, so that fallback does not cover this case.

The failure is confirmed by CI; the routing explanation above is source-derived, without a separately captured execution plan. A fix should validate the file annotation or fall back appropriately, without classifying ordinary structs as Variant solely by their child names.

The other 14 tests in the two restored suites passed. Track this case separately so #5569 can restore that coverage without retaining a whole-suite exclusion. Related: #5438.

[0lai0]

take