Upsert wide table memory

[GaspardMerten]

Rationale for this change

upsert compares the matched rows one cell at a time. For every matched row, it takes a one-row slice of the source and of the target, then calls .as_py() on each non-key column until it finds a difference. That is one PyArrow call per row
and per column, so the cost grows with the size of the table and not with the amount of data that actually changed.

On a table with 200 columns, comparing 20k matched rows takes around 20 seconds before anything is written on a standard computer (mine), while it takes 0.15s if we push this comparison logic to PyArrow Compute (PC).

Two other things come out of that:

• PyArrow cannot compare struct columns, so I implemented nested comparison, still pushing to PC.
• The previous system was casting the source table to the target schema, which in itself is expensive. In addition, when some types such as timestamp[us] were not cast to timestamp[us, UTC] as PyArrow refuses, it reverted back to Python, which made every row reported as changed. This is a small thing we gain.

! Lists and maps have no fields to compare and still go to Python, a slice at a time so the objects of a whole column are never held at once.

The result is the same: two nulls still count as equal, a null and a value still count as a change, and rows whose non-key columns did not change are still skipped.

Are these changes tested?

Yes, 10 tests in tests/table/test_upsert.py.

Eight cover the comparison itself: nulls, a struct that is null, a nested struct, a list column, a column whose type differs from the target, a cast PyArrow refuses, a source missing one of the target columns, and no match at all.

Two cover the change in the way rows are compared, which no assertion on the result can see, because both ways return the same rows. One counts the comparisons and checks there is one per column, whatever the number of rows. The other checks a struct never reaches the Python comparison, with the types the upsert path really produces (a scan reads a string as a large_string).

The existing tests/table/test_upsert.py and tests/table suites pass unchanged.

Are there any user-facing changes?

No API or behaviour change. get_rows_to_update returns the same rows.

One error message changes. A source that does not have every column of the target was rejected by Table.cast with Target schema's field names are not matching the table's field names. It is now rejected by an explicit check, with a message naming the source. Without that check, the missing columns would never be compared and would
be written as null.

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR optimizes upsert row-difference detection for wide tables by moving comparisons into PyArrow Compute (with a structured fallback for complex/nested types), reducing Python scalar materialization and improving performance.

Changes:

• Add column-at-a-time change masking in get_rows_to_update, including nested struct comparison and a sliced Python fallback for non-comparable types.
• Add explicit source/target column-name validation with a new error message.
• Add a focused test suite covering null semantics, nested structs, list/map behavior, casts, and comparison strategy.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
pyiceberg/table/upsert_util.py	Reworks matched-row comparison to use PyArrow Compute masks per column, adds nested-struct handling and a sliced Python fallback, and introduces explicit column-name validation.
tests/table/test_upsert.py	Adds tests verifying new comparison semantics and ensuring comparisons happen per column (and that structs avoid Python comparisons).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.