Skip to content

Native upsert loses rows when selecting zero-width fixed binary values #1316

Closed

Description

@JingsongLi

Native upsert can lose the row count when selecting non-null FixedSizeBinary(0) values, including values nested in ROW/STRUCT columns.

The native Python CI for apache/paimon#10970 built paimon-rust main at 5bb42a110eeb22229851e9e36205376bedb729f8 and failed four cases in native_update_encoded_inputs_test.py::test_scalar_input_encodings_keep_values: empty_fixed_binary with root/row inputs and batch/stream native upsert. The Python execution path passes.

Failure:

cannot build selected upsert rows: Invalid argument error: all columns in a record batch must have the same length

TableUpsert::selected_rows uses Arrow's take kernel. For zero-width fixed binary, that kernel constructs the result without an explicit length. When the selected values are all valid, the value buffer is empty and there is no null buffer to carry the row count, so a nonempty selection becomes a zero-length array. Enclosing structs can inherit the incorrect length or contain children with inconsistent lengths.

Reproduced locally with the same failing Python test and a Rust regression covering row selections, nulls and nested structs. Affected columns can preserve their row counts by concatenating selected row slices.

CI evidence: https://github.com/apache/paimon/actions/runs/41605967684/job/125817770399

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions