feat: support filter pushdown through UNNEST operator - #24984
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24984 +/- ##
==========================================
+ Coverage 81.61% 81.67% +0.05%
==========================================
Files 1124 1126 +2
Lines 412077 414563 +2486
Branches 412077 414563 +2486
==========================================
+ Hits 336318 338581 +2263
- Misses 55949 56065 +116
- Partials 19810 19917 +107 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| self.schema | ||
| .fields() | ||
| .iter() | ||
| .position(|output_field| output_field.name() == name) |
There was a problem hiding this comment.
This name-based lookup can make an unnested output incorrectly eligible when generated and passthrough fields collide. For example, if the input has struct s { f1 } before a passthrough field physically named s.f1, the output can contain two s.f1 fields; position selects the generated one, so a filter on that generated index is accepted and FilterRemapper maps it to the unrelated passthrough input field. Physical expressions are index-based precisely so duplicate names can be represented. Could this mapping be derived positionally from the list/struct expansion (or reject ambiguous matches), with a collision regression test?
Rationale for this change
I have a couple of queries which have Unnest in them, I noticed that the perf of these queries can be improved by pushing down filters to the scan. Filters which are not part of the Unnest operator can be push-down to the scan
What changes are included in this PR?
Implements
gather_filters_for_pushdownandhandle_child_pushdown_resultfor the Unnest OperatorWhat is the testing strategy for this PR?
Unit tests have been added in the PR. I have manually tested this against a couple of queries with
SET datafusion.optimizer.enable_dynamic_filter_pushdown = true;