infer_rel_schema takes an Expand switching field's nullability from the first duplicate

[alexandrefimov]

ExpandRel.SwitchingField in algebra.proto states the rule: "All duplicates must return the same type class but may differ in nullability. The effective type of the output field will be nullable if any of the duplicate expressions are nullable."

infer_rel_schema derives the field from the first duplicate alone, at type_inference.py:826-835:

                # All duplicates of a switching field share one type; the first
                # determines the output column type.
                field_types.append(
                    infer_expression_type(
                        duplicates[0],
                        parent_schema,
                        registry=registry,
                        subtrees=subtrees,
                    )
                )

The comment is right about the type class, which the same sentence requires the duplicates to share. Nullability is the half they are allowed to differ on, and taking it from duplicates[0] drops it.

Observed

An ExpandRel over a read of (c0 i64, c1 i64?), with a consistent_field on c0 and a switching_field whose duplicates are c0 and c1. The two plans differ only in the order of those two duplicates:

duplicates	derived	expected
c0, c1	['i64', 'i64', 'i32']	['i64', 'i64?', 'i32']
c1, c0	['i64', 'i64?', 'i32']	['i64', 'i64?', 'i32']

So the answer follows the order the duplicates are written in. The rule is that it should not: the field is nullable because one of the two is, whichever of them comes first.

Reproduced with substrait 0.31.0. main at 2f138c1 carries the same type_inference.py: the one commit between them touches a CI workflow.

This is the same shape as #267 and #268, a required field that should have widened, on a relation neither of them reaches.

Reproducer

from substrait import algebra_pb2 as a, type_pb2 as t
from substrait.type_inference import infer_rel_schema

R, N = t.Type.NULLABILITY_REQUIRED, t.Type.NULLABILITY_NULLABLE

read = a.Rel(read=a.ReadRel(
    named_table=a.ReadRel.NamedTable(names=["t"]),
    base_schema=t.NamedStruct(
        names=["c0", "c1"],
        struct=t.Type.Struct(
            types=[t.Type(i64=t.Type.I64(nullability=R)),
                   t.Type(i64=t.Type.I64(nullability=N))],
            nullability=R,
        ),
    ),
))

def field(i):
    return a.Expression(selection=a.Expression.FieldReference(
        direct_reference=a.Expression.ReferenceSegment(
            struct_field=a.Expression.ReferenceSegment.StructField(field=i)),
        root_reference=a.Expression.FieldReference.RootReference(),
    ))

def render(schema):
    out = []
    for f in schema.types:
        kind = f.WhichOneof("kind")
        n = getattr(f, kind).nullability
        out.append(kind + ("?" if n == N else ""))
    return out

for order in ((0, 1), (1, 0)):
    rel = a.Rel(expand=a.ExpandRel(
        input=read,
        fields=[
            a.ExpandRel.ExpandField(consistent_field=field(0)),
            a.ExpandRel.ExpandField(switching_field=a.ExpandRel.SwitchingField(
                duplicates=[field(order[0]), field(order[1])])),
        ],
    ))
    print("duplicates c%d, c%d ->" % order, render(infer_rel_schema(rel)))

It prints:

duplicates c0, c1 -> ['i64', 'i64', 'i32']
duplicates c1, c0 -> ['i64', 'i64?', 'i32']

The second row is the answer both should give.