Skip to content

fix, perf: Use Arrow comparator for key comparison in map_extract - #24999

Open
neilconway wants to merge 1 commit into
apache:mainfrom
neilconway:neilc/fix-map-extract-comparator
Open

fix, perf: Use Arrow comparator for key comparison in map_extract#24999
neilconway wants to merge 1 commit into
apache:mainfrom
neilconway:neilc/fix-map-extract-comparator

Conversation

@neilconway

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The previous implementation of map_extract did the following for each row:

  1. Create a one-element array slice containing the row's search key
  2. Scan the map's entries. For each entry, create a one-element array slice and compare the two slices using Arrow's array equality
  3. Stop at the first match; if no matches, append a NULL instead

This had three shortcomings:

  1. It was very inefficient, because a lot of allocations are done for every element of every map.
  2. It got the equality semantics wrong for some corner-cases. In particular, maps with dictionary-valued keys might encode a logical NULL in two physically distinct ways (map_extract fails to match equal struct keys with differently encoded dictionary nulls #24983).
  3. It returned [NULL] for missing map keys instead of an empty list (map_extract returns [NULL] instead of [] for missing keys #24981); returning an empty list is what the DataFusion docs claim this function does, and it is the DuckDB behavior.

Instead, we can implement map_extract with a single arrow-ord comparator. This enables comparing the search key with each map element directly by index, without allocating. It also avoids the differences in comparison semantics outlined above.

Finally, this PR fixes the behavior for absent map keys to be consistent with DuckDB.

Benchmark results (M4 Max):

  • int32/first/1024x32, 135.629 µs -> 6.234 µs, -95.40%
  • int32/last/1024x1, 133.110 µs -> 6.157 µs, -95.37%
  • int32/last/1024x32, 3477.393 µs -> 38.562 µs, -98.89%
  • int32/last/1x0, 0.498 µs -> 0.313 µs, -37.23%
  • int32/last/1x1, 0.593 µs -> 0.384 µs, -35.34%
  • int32/missing/1024x32, 3472.698 µs -> 34.889 µs, -99.00%
  • int32/varying/1024x32, 1844.686 µs -> 25.307 µs, -98.63%
  • struct/first/1024x32, 335.421 µs -> 8.769 µs, -97.39%
  • struct/last/1024x1, 335.097 µs -> 8.647 µs, -97.42%
  • struct/last/1024x32, 8325.830 µs -> 71.611 µs, -99.14%
  • struct/last/1x0, 0.466 µs -> 0.242 µs, -48.08%
  • struct/last/1x1, 0.747 µs -> 0.391 µs, -47.67%
  • struct/missing/1024x32, 8451.153 µs -> 61.480 µs, -99.27%
  • struct/varying/1024x32, 4498.016 µs -> 41.343 µs, -99.08%
  • utf8_view/first/1024x32, 218.809 µs -> 9.877 µs, -95.49%
  • utf8_view/last/1024x1, 190.468 µs -> 8.464 µs, -95.56%
  • utf8_view/last/1024x32, 6016.140 µs -> 124.405 µs, -97.93%
  • utf8_view/last/1x0, 0.526 µs -> 0.353 µs, -32.97%
  • utf8_view/last/1x1, 0.762 µs -> 0.523 µs, -31.42%
  • utf8_view/missing/1024x32, 5999.511 µs -> 114.082 µs, -98.10%
  • utf8_view/varying/1024x32, 3226.583 µs -> 71.732 µs, -97.78%

("1024x32" means 1024 rows and each row is a map with 32 entries.)

What changes are included in this PR?

What is the testing strategy for this PR?

Existing tests pass; new tests added. Verified that the new tests fail if the implementation is reverted.

Are there any user-facing changes?

Yes, semantics of map_extract have changed, particularly for absent keys.

The previous implementation of `map_extract` did the following for row:

1. Create a one-element array slice containing the row's search key
2. Scan the map's entries. For each entry, create a one-element array
   slice and compare the two slices using Arrow's array equality
3. Stop at the first match; if no matches, append a NULL instead

This had three shortcomings:

1. It was very inefficient, because a lot of allocations are done for
   every element of every map.
2. It got the equality semantics wrong for some corner-cases. In
   particular, maps with dictionary-valued keys might encode a logical
   NULL in two physically distinct ways (apache#24983). Arrow's array equality
   also considers sparse unions that have different values in unselected
   child fields to be distinct; this is arguably a bug in Arrow though.
3. It returned `[NULL]` for missing map keys instead of an empty list,
   which is the behavior implemented by DuckDB (apache#24981).

Instead, we can implement `map_extract` with a single arrow-ord
comparator. This enables comparing the search key with each map element
directly by index, without allocating. It also avoids the differences in
comparison semantics outlined above.

Finally, this PR fixes the behavior for absent map keys to be consistent
with DuckDB.

Benchmark results (M4 Max):

  - int32/first/1024x32, 135.629 µs -> 6.234 µs, -95.40%
  - int32/last/1024x1, 133.110 µs -> 6.157 µs, -95.37%
  - int32/last/1024x32, 3477.393 µs -> 38.562 µs, -98.89%
  - int32/last/1x0, 0.498 µs -> 0.313 µs, -37.23%
  - int32/last/1x1, 0.593 µs -> 0.384 µs, -35.34%
  - int32/missing/1024x32, 3472.698 µs -> 34.889 µs, -99.00%
  - int32/varying/1024x32, 1844.686 µs -> 25.307 µs, -98.63%
  - struct/first/1024x32, 335.421 µs -> 8.769 µs, -97.39%
  - struct/last/1024x1, 335.097 µs -> 8.647 µs, -97.42%
  - struct/last/1024x32, 8325.830 µs -> 71.611 µs, -99.14%
  - struct/last/1x0, 0.466 µs -> 0.242 µs, -48.08%
  - struct/last/1x1, 0.747 µs -> 0.391 µs, -47.67%
  - struct/missing/1024x32, 8451.153 µs -> 61.480 µs, -99.27%
  - struct/varying/1024x32, 4498.016 µs -> 41.343 µs, -99.08%
  - utf8_view/first/1024x32, 218.809 µs -> 9.877 µs, -95.49%
  - utf8_view/last/1024x1, 190.468 µs -> 8.464 µs, -95.56%
  - utf8_view/last/1024x32, 6016.140 µs -> 124.405 µs, -97.93%
  - utf8_view/last/1x0, 0.526 µs -> 0.353 µs, -32.97%
  - utf8_view/last/1x1, 0.762 µs -> 0.523 µs, -31.42%
  - utf8_view/missing/1024x32, 5999.511 µs -> 114.082 µs, -98.10%
  - utf8_view/varying/1024x32, 3226.583 µs -> 71.732 µs, -97.78%

("1024x32" means 1024 rows and each row is a map with 32 entries.)
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Sep 6, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.58974% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.67%. Comparing base (262936e) to head (feccbd2).

Files with missing lines Patch % Lines
datafusion/functions-nested/src/map_extract.rs 93.58% 0 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24999      +/-   ##
==========================================
- Coverage   81.67%   81.67%   -0.01%     
==========================================
  Files        1126     1126              
  Lines      414842   414903      +61     
  Branches   414842   414903      +61     
==========================================
+ Hits       338841   338888      +47     
- Misses      56070    56078       +8     
- Partials    19931    19937       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

map_extract fails to match equal struct keys with differently encoded dictionary nulls map_extract returns [NULL] instead of [] for missing keys

2 participants