Iceberg scan falls back to Spark on IS NULL/IS NOT NULL over list/map columns (stale complex-type check)

[ErikBPF]

Describe the bug

CometScanRule declines every Iceberg scan whose pushed filters contain an IS NULL/IS NOT NULL predicate on a complex-typed column, with the reason:

IS NULL / IS NOT NULL predicates on complex type columns (struct/array/map) are not yet supported by iceberg-rust

The check (added in #2528) predates the current iceberg-rust pin (bumped in #5651). iceberg-rust's Arrow predicate visitor now evaluates is_null/is_not_null on list and map columns through arrow's native is_null/is_not_null (crates/iceberg/src/arrow/reader/predicate_visitor.rs); only struct columns are rejected (project_column errors on DataType::Struct).

Impact

The check is a false positive for list columns, which is a common real pattern: Spark's optimizer pushes isnotnull(arr) below Generate for explode(arr):

BatchScan local_tpch.orders_nested [COMET: IS NULL / IS NOT NULL predicates on complex type columns ...]

Each declined scan forces the whole plan back to Spark and cascades into further fallbacks (Comet aggregate that merges intermediate buffers requires a Comet child aggregate ...). On a 24-query derived TPC-H benchmark (SF1, 3 runs, cold caches, Spark 4.1.3, Iceberg hadoop catalog, table with array<struct> line-item column):

metric	check active	list false-positive removed
Comet/Iceberg total (24q)	20.63 s	16.93 s (−17.9%)
q1_nested_explode	2.20 s	0.77 s (−65%)
e14_nested	2.03 s	0.86 s (−58%)
Comet/Iceberg vs Comet/Parquet total	+26%	+3.7%

Parity vs vanilla Spark verified on all 24 queries in both configurations.

A fix that narrows the check to struct-only columns will be submitted as a PR.