Skip to content

Arrow: Fix NullabilityHolder.reset() to clear the isNull array#16264

Open
jbbqqf wants to merge 1 commit intoapache:mainfrom
jbbqqf:feat/15808-fix-nullability-holder-reset
Open

Arrow: Fix NullabilityHolder.reset() to clear the isNull array#16264
jbbqqf wants to merge 1 commit intoapache:mainfrom
jbbqqf:feat/15808-fix-nullability-holder-reset

Conversation

@jbbqqf
Copy link
Copy Markdown

@jbbqqf jbbqqf commented May 9, 2026

Summary

NullabilityHolder.reset() previously zeroed numNulls but left the isNull marker array unchanged. Callers that reuse a holder across batches could see hasNulls() == false yet still get isNullAt(i) == 1 for indices set null in a previous batch.

This PR fixes reset() to also clear the isNull array, and adds focused unit tests covering the single-set, bulk-set, and idempotency paths.

Resolves #15808

Context

Today's vectorized read hot-path callers (e.g. the arrow vectorized readers) always overwrite all positions on the next batch via setNulls(start, num) / setNotNulls(start, num), so the latent stale state is never observed in production. The bug is therefore a contract trap rather than an active correctness bug — but the holder's contract documents reset() as restoring a clean initial state, and the original triage in #15808 already proposed the one-line fix.

This PR is a revival of #15809 (closed by stale-bot at 30 days of no review). The diff is unchanged in spirit; this version adds AI disclosure (which a reviewer asked for on the previous attempt) and a slightly broader test class to make the regression-test evidence explicit.

Changes

  • arrow/src/main/java/org/apache/iceberg/arrow/vectorized/NullabilityHolder.javareset() now also calls Arrays.fill(isNull, (byte) 0). A short comment explains the why (so a reviewer reading the diff cold doesn't have to re-derive it from the issue).
  • arrow/src/test/java/org/apache/iceberg/arrow/vectorized/TestNullabilityHolder.java — new test class with three tests: resetClearsIsNullMarkers, resetClearsBulkSetNulls, resetIsIdempotent. All three fail on origin/main and pass with the fix.

The other two arrays in NullabilityHolder (nulls and nonNulls) are immutable arraycopy source buffers and intentionally not cleared — that's documented in the inline comment.

Reproduce BEFORE/AFTER yourself (copy-paste)

# --- one-time setup ---
git clone https://github.com/apache/iceberg.git /tmp/repro && cd /tmp/repro

# --- BEFORE (origin/main) ---
git checkout origin/main
# Pull in only the new test file from the fix branch so the regression
# is visible against unmodified production code:
git fetch https://github.com/jbbqqf/iceberg.git feat/15808-fix-nullability-holder-reset
git checkout FETCH_HEAD -- arrow/src/test/java/org/apache/iceberg/arrow/vectorized/TestNullabilityHolder.java
./gradlew :iceberg-arrow:test --tests "org.apache.iceberg.arrow.vectorized.TestNullabilityHolder"
# Expected: 3 tests, 3 failures — isNullAt() returns the stale 1 from
#           a setNull/setNulls call that preceded reset().

# --- AFTER (this PR) ---
git checkout FETCH_HEAD
./gradlew :iceberg-arrow:test --tests "org.apache.iceberg.arrow.vectorized.TestNullabilityHolder"
# Expected: 3 tests, 3 passes — reset() now zeros isNull as well as numNulls.

The only thing that changes between BEFORE and AFTER is the checked-out ref of the production class; the test file is the same in both runs.

What I ran locally

  • ./gradlew :iceberg-arrow:test --tests "org.apache.iceberg.arrow.vectorized.TestNullabilityHolder" on origin/main with the new test file imported → 3 tests, 3 failures (regression visible).
  • Same command on this branch → 3 tests, 3 passed.
  • ./gradlew :iceberg-arrow:spotlessCheckPASS.
  • Full :iceberg-arrow:test build wrapping the change → BUILD SUCCESSFUL (existing tests untouched).

Edge cases tested

# Scenario Input Expected Verified by
1 Single setNull(idx) then reset() setNull(3), reset() isNullAt(3) == 0, all other indices == 0, numNulls() == 0 resetClearsIsNullMarkers
2 Bulk setNulls(start, num) then reset() setNulls(2, 4), reset() every index == 0, hasNulls() == false resetClearsBulkSetNulls
3 Idempotency — call reset() twice setNull(0), reset(), reset() every index == 0, numNulls() == 0 resetIsIdempotent

Risk / blast radius

Internal: NullabilityHolder is package-private under org.apache.iceberg.arrow.vectorized and only used by the vectorized arrow read path. The fix adds a byte[] zero-fill in reset() (size = batch capacity, typically 4–8 KiB on hot paths). On the dominant hot path the holder is reused across batches and the next bulk setNulls/setNotNulls would have overwritten the array anyway, so the only observable effect is a single extra Arrays.fill per reset() — a few microseconds per batch on typical JVM heuristics.

Additive change; no API surface modified.

Release note

Fix NullabilityHolder.reset() to clear the isNull marker array, restoring the documented clean-state contract for callers that reuse a holder across batches.

PR drafted with assistance from Claude Code. The change was reviewed manually against apache/iceberg's source on main. The reproducer block above was used during development; it is the same one a reviewer can paste verbatim.

reset() previously only zeroed numNulls, leaving the isNull marker
array unchanged. Callers that reused a NullabilityHolder across
batches could observe stale null markers via isNullAt() — the holder
reported hasNulls() == false yet isNullAt(i) returned 1 for indices
that had been set null in a previous batch.

Today's hot-path callers always overwrite all positions on the next
batch via setNulls/setNotNulls, so the bug is latent rather than
observable in production. Fixing reset() to also clear isNull
restores the documented "clean initial state" contract and removes
a trap for future callers.

Adds TestNullabilityHolder covering the single-set, bulk-set, and
idempotency paths. All three tests fail on origin/main and pass
with this change.

Resolves apache#15808

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions github-actions Bot added the arrow label May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NullabilityHolder.reset() does not clear the isNull array

1 participant