fix, perf: Use Arrow comparator for key comparison in map_extract - #24999
Open
neilconway wants to merge 1 commit into
Open
fix, perf: Use Arrow comparator for key comparison in map_extract#24999neilconway wants to merge 1 commit into
map_extract#24999neilconway wants to merge 1 commit into
Conversation
The previous implementation of `map_extract` did the following for row: 1. Create a one-element array slice containing the row's search key 2. Scan the map's entries. For each entry, create a one-element array slice and compare the two slices using Arrow's array equality 3. Stop at the first match; if no matches, append a NULL instead This had three shortcomings: 1. It was very inefficient, because a lot of allocations are done for every element of every map. 2. It got the equality semantics wrong for some corner-cases. In particular, maps with dictionary-valued keys might encode a logical NULL in two physically distinct ways (apache#24983). Arrow's array equality also considers sparse unions that have different values in unselected child fields to be distinct; this is arguably a bug in Arrow though. 3. It returned `[NULL]` for missing map keys instead of an empty list, which is the behavior implemented by DuckDB (apache#24981). Instead, we can implement `map_extract` with a single arrow-ord comparator. This enables comparing the search key with each map element directly by index, without allocating. It also avoids the differences in comparison semantics outlined above. Finally, this PR fixes the behavior for absent map keys to be consistent with DuckDB. Benchmark results (M4 Max): - int32/first/1024x32, 135.629 µs -> 6.234 µs, -95.40% - int32/last/1024x1, 133.110 µs -> 6.157 µs, -95.37% - int32/last/1024x32, 3477.393 µs -> 38.562 µs, -98.89% - int32/last/1x0, 0.498 µs -> 0.313 µs, -37.23% - int32/last/1x1, 0.593 µs -> 0.384 µs, -35.34% - int32/missing/1024x32, 3472.698 µs -> 34.889 µs, -99.00% - int32/varying/1024x32, 1844.686 µs -> 25.307 µs, -98.63% - struct/first/1024x32, 335.421 µs -> 8.769 µs, -97.39% - struct/last/1024x1, 335.097 µs -> 8.647 µs, -97.42% - struct/last/1024x32, 8325.830 µs -> 71.611 µs, -99.14% - struct/last/1x0, 0.466 µs -> 0.242 µs, -48.08% - struct/last/1x1, 0.747 µs -> 0.391 µs, -47.67% - struct/missing/1024x32, 8451.153 µs -> 61.480 µs, -99.27% - struct/varying/1024x32, 4498.016 µs -> 41.343 µs, -99.08% - utf8_view/first/1024x32, 218.809 µs -> 9.877 µs, -95.49% - utf8_view/last/1024x1, 190.468 µs -> 8.464 µs, -95.56% - utf8_view/last/1024x32, 6016.140 µs -> 124.405 µs, -97.93% - utf8_view/last/1x0, 0.526 µs -> 0.353 µs, -32.97% - utf8_view/last/1x1, 0.762 µs -> 0.523 µs, -31.42% - utf8_view/missing/1024x32, 5999.511 µs -> 114.082 µs, -98.10% - utf8_view/varying/1024x32, 3226.583 µs -> 71.732 µs, -97.78% ("1024x32" means 1024 rows and each row is a map with 32 entries.)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24999 +/- ##
==========================================
- Coverage 81.67% 81.67% -0.01%
==========================================
Files 1126 1126
Lines 414842 414903 +61
Branches 414842 414903 +61
==========================================
+ Hits 338841 338888 +47
- Misses 56070 56078 +8
- Partials 19931 19937 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
map_extractfails to match equal struct keys with differently encoded dictionary nulls #24983map_extractreturns[NULL]instead of[]for missing keys #24981Rationale for this change
The previous implementation of
map_extractdid the following for each row:This had three shortcomings:
map_extractfails to match equal struct keys with differently encoded dictionary nulls #24983).[NULL]for missing map keys instead of an empty list (map_extractreturns[NULL]instead of[]for missing keys #24981); returning an empty list is what the DataFusion docs claim this function does, and it is the DuckDB behavior.Instead, we can implement
map_extractwith a single arrow-ord comparator. This enables comparing the search key with each map element directly by index, without allocating. It also avoids the differences in comparison semantics outlined above.Finally, this PR fixes the behavior for absent map keys to be consistent with DuckDB.
Benchmark results (M4 Max):
("1024x32" means 1024 rows and each row is a map with 32 entries.)
What changes are included in this PR?
map_extractto use a comparatormap_extractmap_extractfails to match equal struct keys with differently encoded dictionary nulls #24983)map_extractreturns[NULL]instead of[]for missing keys #24981)What is the testing strategy for this PR?
Existing tests pass; new tests added. Verified that the new tests fail if the implementation is reverted.
Are there any user-facing changes?
Yes, semantics of
map_extracthave changed, particularly for absent keys.