Skip to content

feat: support filter pushdown through UNNEST operator - #24984

Open
Mark1626 wants to merge 1 commit into
apache:mainfrom
Mark1626:unnest-pushdown
Open

feat: support filter pushdown through UNNEST operator#24984
Mark1626 wants to merge 1 commit into
apache:mainfrom
Mark1626:unnest-pushdown

Conversation

@Mark1626

@Mark1626 Mark1626 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

I have a couple of queries which have Unnest in them, I noticed that the perf of these queries can be improved by pushing down filters to the scan. Filters which are not part of the Unnest operator can be push-down to the scan

What changes are included in this PR?

Implements gather_filters_for_pushdown and handle_child_pushdown_result for the Unnest Operator

What is the testing strategy for this PR?

Unit tests have been added in the PR. I have manually tested this against a couple of queries with SET datafusion.optimizer.enable_dynamic_filter_pushdown = true;

@github-actions github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Sep 6, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.43590% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.67%. Comparing base (33028d5) to head (5bfdf5e).
⚠️ Report is 15 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/unnest.rs 97.43% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24984      +/-   ##
==========================================
+ Coverage   81.61%   81.67%   +0.05%     
==========================================
  Files        1124     1126       +2     
  Lines      412077   414563    +2486     
  Branches   412077   414563    +2486     
==========================================
+ Hits       336318   338581    +2263     
- Misses      55949    56065     +116     
- Partials    19810    19917     +107     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

self.schema
.fields()
.iter()
.position(|output_field| output_field.name() == name)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This name-based lookup can make an unnested output incorrectly eligible when generated and passthrough fields collide. For example, if the input has struct s { f1 } before a passthrough field physically named s.f1, the output can contain two s.f1 fields; position selects the generated one, so a filter on that generated index is accepted and FilterRemapper maps it to the unrelated passthrough input field. Physical expressions are index-based precisely so duplicate names can be represented. Could this mapping be derived positionally from the list/struct expansion (or reject ambiguous matches), with a collision regression test?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants