Skip to content

[core] Preserve zero-width binary rows in upsert selection - #1317

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/fix-upsert-empty-fixed-binary
Nov 3, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/fix-upsert-empty-fixed-binary

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Closes #1316. Related CI failure: apache/paimon#10970.

Native upsert fails when selecting non-null FixedSizeBinary(0) values, including values nested in ROW columns. Arrow's take kernel derives the zero-width result length from its null buffer; all-valid selections have no null buffer and become empty arrays. Rebuilding the selected batch then reports inconsistent column lengths.

Brief change log

  • Select affected zero-width binary columns and enclosing structs by concatenating row slices, preserving row counts, order, duplicates and validity.
  • Retain the existing take path for other columns and empty selections.
  • Cover sliced input, empty and single-row selections, repeated/reordered rows, NULLs, sibling fields and two levels of nested structs.

Tests

  • Confirmed the new Rust regression fails before the fix with the same error as the native Python CI.
  • cargo test --locked -p paimon --lib --no-default-features --features storage-memory,storage-fs table_upsert::tests: 18 passed.
  • Built a Python wheel from the patched source. The zero-width binary upsert matrix passed all 16 cases, including the four original CI failures.
  • native_update_encoded_inputs_test.py and native_update_binary_text_test.py, filtered to upsert: 534 passed with the rebuilt native wheel.
  • cargo +1.98.0 fmt --all -- --check and git diff --check passed.
  • cargo +1.98.0 clippy --locked -p paimon --lib --tests -- -D warnings passed.

API and Format

No public API or storage format changes.

Documentation

Existing supported input behavior is restored; no documentation changes are required.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit ac07a25 into apache:main Nov 3, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native upsert loses rows when selecting zero-width fixed binary values

2 participants