Describe the bug
Raised by @viirya in the #5365 review. NativeConfig.extractObjectStoreOptions forwards fs.s3a.* options to the native object store client with no divergence check, but Comet's native client resolves S3 configuration differently from Hadoop's S3AFileSystem in several ways (lookupPassword bucket precedence, JCEKS credential provider aliases, clear text fallback, assumed role session policies, provider class semantics). A plain Parquet table read through the regular native scan can therefore read under a different effective identity or endpoint than Spark would, silently.
The Delta contrib in #5365 carries a divergence comparator that detects these cases and declines to Spark, but the regular native scan path has no equivalent gate. #5658 tracks extracting that comparator into a shared helper; this issue tracks actually applying it (or an equivalent check) to the regular native scan so the gap is closed for every native read, not only Delta.
Steps to reproduce
Configure any setting the two resolvers treat differently, for example a JCEKS-backed fs.s3a.secret.key with hadoop.security.credential.clear-text-fallback=false plus a clear text value, and read a plain Parquet table with the native scan enabled.
Expected behavior
The scan declines to Spark when native would resolve different S3 configuration than Hadoop, or resolves identically.
Additional context
No response
Describe the bug
Raised by @viirya in the #5365 review.
NativeConfig.extractObjectStoreOptionsforwardsfs.s3a.*options to the native object store client with no divergence check, but Comet's native client resolves S3 configuration differently from Hadoop's S3AFileSystem in several ways (lookupPassword bucket precedence, JCEKS credential provider aliases, clear text fallback, assumed role session policies, provider class semantics). A plain Parquet table read through the regular native scan can therefore read under a different effective identity or endpoint than Spark would, silently.The Delta contrib in #5365 carries a divergence comparator that detects these cases and declines to Spark, but the regular native scan path has no equivalent gate. #5658 tracks extracting that comparator into a shared helper; this issue tracks actually applying it (or an equivalent check) to the regular native scan so the gap is closed for every native read, not only Delta.
Steps to reproduce
Configure any setting the two resolvers treat differently, for example a JCEKS-backed fs.s3a.secret.key with hadoop.security.credential.clear-text-fallback=false plus a clear text value, and read a plain Parquet table with the native scan enabled.
Expected behavior
The scan declines to Spark when native would resolve different S3 configuration than Hadoop, or resolves identically.
Additional context
No response