Skip to content

feat: add vectorized array_reduce - #78

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
array-reduce-higher-order
Sep 10, 2026
Merged

feat: add vectorized array_reduce#78
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
array-reduce-higher-order

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Summary

  • add a native higher-order array_reduce(array, initial, merge) function
  • evaluate each array position as one Arrow batch and compact inactive rows for uneven arrays
  • preserve null-array, null-initial, and terminal null-merge semantics
  • add SQLLogic coverage for numeric/string accumulators, coercion, nulls, uneven arrays, and captured columns
  • add Criterion coverage for uniform and varying-length arrays

Performance

Measured locally with 8,192 rows and 16-element arrays:

  • uniform arrays: ~0.52 ms
  • varying lengths: ~1.19 ms

The lambda expression is never evaluated per row. Uniform arrays use a full-batch fast path; uneven arrays evaluate only active rows.

Verification

  • cargo +1.94.0 test -p datafusion-sqllogictest --test sqllogictests -- array_reduce
  • cargo +1.94.0 clippy -p datafusion-functions-nested --all-targets -- -D warnings
  • cargo +1.94.0 bench -p datafusion-functions-nested --bench array_reduce -- --warm-up-time 1 --measurement-time 2 --sample-size 10

@osipovartem
osipovartem merged commit d75bd8d into embucket-sync-df55.0.0 Sep 10, 2026
7 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant