bug: slice crashes on arrays with a non-nullable element field

[andygrove]

Describe the bug

slice crashes natively when its input array has a non-nullable element field. spark_array_slice passes the input's child field through to its output, but the return type Comet promises at planning time has a nullable element, so the two disagree and execution fails.

Spark declares several expressions with containsNull = false, and any of them feeding slice reproduces this. StringSplit.dataType is ArrayType(StringType, containsNull = false) and Sequence.dataType is ArrayType(start.dataType, containsNull = false), so both are enough on their own.

This fails on release builds too, not just debug. The two builds surface it differently:

• debug: Assertion failed: result_data_type == *expected_type: Function 'spark_array_slice' returned value of type 'List(non-null Utf8)' while the following type was promised at planning time and expected: 'List(Utf8)'.
• release: Invalid argument error: column types must match schema types, expected List(Utf8) but found List(non-null Utf8) at column index 1

This is the same assertion as #4789, but by a route that issue explicitly ruled out. Its body notes the crash "does NOT reproduce over a native Parquet scan (which normalizes children to nullable)". The repro below is over a native Parquet scan, and the non-nullable child comes from the expression rather than from the scan, so the #4789 fix does not cover it.

Steps to reproduce

In a suite extending CometTestBase:

test("slice over a non-nullable element array") {
  val input = Seq((1, "axb", 2), (2, "", 3), (3, "cxd", 2))
  withParquetDataFrame(input) { parquet =>
    withParquetTable(parquet.toDF("id", "s", "n"), "t") {
      // both of these fail
      checkSparkAnswerAndOperator(sql("SELECT id, slice(split(s, 'x'), 1, n) AS a FROM t"))
      checkSparkAnswerAndOperator(sql("SELECT id, slice(sequence(1, n), 1, 2) AS a FROM t"))
    }
  }
}

No generator, no local table scan, and no shuffle is involved. A bare CometProject over CometNativeScan is enough.

Scope

slice is the only affected kernel I found. Other native array kernels handle a non-nullable element field correctly, so this looks specific to how CometSlice derives its return type rather than a general problem with non-nullable children:

expression	result
slice(split(s, 'x'), 1, n)	FAIL
slice(sequence(1, n), 1, 2)	FAIL
slice(concat(split(s, 'x'), array('z')), 1, n)	FAIL
slice(array(1, 2, 3), 1, n) (nullable elements)	OK
slice(arr, 1, n) over a stored array<string>	OK
array_remove(sequence(1, n), 2)	OK
reverse(sequence(1, n))	OK
array_append(sequence(1, n), 9)	OK
concat(sequence(1, n), array(9))	OK
sort_array(sequence(1, n))	OK
size(sequence(1, n))	OK

Setting spark.comet.expression.StringSplit.allowIncompatible=true does not help, so this is not specific to the JVM codegen dispatcher. The native split produces the same non-nullable element field, which is expected given Spark's declared type.

Expected behavior

Same results as Spark, no native crash. Either spark_array_slice should normalize its output child field to match the return type Comet planned, or CometSlice should declare a return type whose element nullability matches the input it will actually receive.

Additional context

Found while reviewing #5737, which is unrelated. That PR only changes the Explode arm of the planner, and the crash reproduces with no generator in the plan at all.

Reproduced on Spark 4.1.3 / JDK 17, macOS aarch64, at 7190df631.