Skip to content

isthmus: visit(Aggregate) reads the grouping sets but not the group set, so a Calcite aggregate with a key in no grouping set loses a column #1211

Description

@nielspardon

SubstraitRelVisitor.visit(org.apache.calcite.rel.core.Aggregate) builds the Substrait grouping columns from aggregate.getGroupSets() and never reads getGroupSet(). Calcite emits one output column per bit of getGroupSet()Aggregate.deriveRowType iterates groupSet.asList() — and its constructor only asserts that each grouping set is contained in the group set, never that the union of the sets equals it. A key that appears in the group set but in no grouping set therefore produces a Calcite column the conversion does not account for.

Measured on main at 7310fc8, over a scan foo(a i64, b i64, c string):

LogicalAggregate.create(
    scan, List.of(), ImmutableBitSet.of(0, 1, 2),
    List.of(ImmutableBitSet.of(0, 1), ImmutableBitSet.of(1)), List.of());

That node's row type is RecordType(BIGINT a, BIGINT b, VARCHAR c) — three columns — while the union of its grouping sets is {0, 1}. Converting it fails at the plan root:

java.lang.IllegalArgumentException: Plan.Root names count (3) must match input record type
depth-first named-field count (2)

At the root the mismatch at least surfaces. Deeper in a plan there is no name count to check, so the column is dropped silently and every measure index in the emit mapping shifts by one.

The spec side is unambiguous: "Each of the grouping expressions must occur in at least one of the grouping sets" (logical relations, spec v0.101.0), so there is no Substrait aggregate this Calcite shape maps onto. The fix is to reject it explicitly — naming the group-set keys that appear in no grouping set — rather than converting to a relation that is narrower than the node it came from.

Reachable from a RelBuilder-constructed plan, an optimizer rule, or a non-Isthmus Calcite front end; Calcite's own containment assertion is disabled outside tests, so nothing upstream stops it either. Found while reviewing #1161.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions