Skip to content

Support predicate subqueries in projections - #24972

Open
osipovartem wants to merge 3 commits into
apache:mainfrom
Embucket:upstream-projection-subquery-expressions
Open

Support predicate subqueries in projections#24972
osipovartem wants to merge 3 commits into
apache:mainfrom
Embucket:upstream-projection-subquery-expressions

Conversation

@osipovartem

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Predicate subqueries such as EXISTS, IN, and NOT IN are valid scalar expressions in a SELECT list, but DataFusion currently only decorrelates them when they appear in filters. Projection queries therefore reach physical planning with an unsupported logical subquery expression.

This continues the work from the now-closed stale #23039 and covers multiple and correlated projection subqueries as well as SQL null semantics.

What changes are included in this PR?

  • decorrelate EXISTS, NOT EXISTS, IN, and NOT IN expressions inside projections into mark joins
  • materialize projected IN/NOT IN values with SQL three-valued logic by tracking matches, nulls, and empty subqueries
  • preserve projection output names and avoid duplicate columns in correlated subquery projections

What is the testing strategy for this PR?

Optimizer plan tests cover projected IN rewriting. New SQLLogicTest cases cover uncorrelated and correlated EXISTS/IN, multiple subqueries in one projection, empty inputs, null inputs, and NOT IN three-valued logic.

Locally validated with:

  • cargo test -p datafusion-optimizer decorrelate_predicate_subquery (53 passed)
  • cargo test -p datafusion-sqllogictest --test sqllogictests -- subquery_projection
  • cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check

Are there any user-facing changes?

Yes. Predicate subqueries can now be evaluated as projected scalar expressions. There are no public API changes.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Sep 6, 2026
@osipovartem
osipovartem force-pushed the upstream-projection-subquery-expressions branch from 0f76189 to edd9e50 Compare September 6, 2026 09:57
@codecov-commenter

codecov-commenter commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.16535% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.67%. Comparing base (262936e) to head (764e4b2).

Files with missing lines Patch % Lines
...on/optimizer/src/decorrelate_predicate_subquery.rs 77.16% 6 Missing and 23 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24972      +/-   ##
==========================================
- Coverage   81.67%   81.67%   -0.01%     
==========================================
  Files        1126     1126              
  Lines      414842   414957     +115     
  Branches   414842   414957     +115     
==========================================
+ Hits       338841   338923      +82     
- Misses      56070    56080      +10     
- Partials    19931    19954      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@osipovartem
osipovartem force-pushed the upstream-projection-subquery-expressions branch from fc6e643 to 707d9df Compare September 6, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Physical plan does not support logical expression Exists

2 participants