transform: detect index lookups from the filter as written - #38717
Draft
frankmcsherry wants to merge 2 commits into
Draft
transform: detect index lookups from the filter as written#38717frankmcsherry wants to merge 2 commits into
frankmcsherry wants to merge 2 commits into
Conversation
LiteralConstraints decided which index to use, and with which values, on a disjunctive normal form of the filter, prepared first and undone afterwards, and the preparation stops at a size guard. Past the guard no lookup was found. Detection now reads the filter as it is: KeyBounds is an exact analysis of the key values a predicate admits, as a disjunction of boxes that AND intersects and OR unions, with no approximation and no budget. Its one limit is on the values enumerated for a lookup, tied to the constant size limit a plan already observes. Preparation, contradictory-disjunct removal, undo_preparation and the DNF-based constraint removal are unchanged and produce the same expressions. Detection therefore has more reach than removal, and a filter can yield a lookup whose constraints removal cannot take out; the filter then stays whole above the lookup. Removal asserted that every disjunct carries a key constraint, which held only because the old detection never let such a shape through; it now declines instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Long IN lists, pair lists on a two-column index, a covered list alongside uncovered ones, a cross product of two covered lists, and two disjoint pair lists (empty, asserted on rows). One existing plan changes: a = NULL OR a = 2 now looks up (2) with the filter kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
|
(Triggered Nightly's optimizer-relevant subset: https://buildkite.com/materialize/nightly/builds/18279) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LiteralConstraintsdecides whether a filter over an indexed relation can be served by looking up literal key values in the index. Today that decision is made on a disjunctive normal form of the filter, prepared first and undone afterwards, and the preparation stops at a size guard. Past the guard the transform finds no lookup: a longINlist combined with another disjunction, a multi-columnINlist, or anINlist on one indexed column next to an unrelated list on another, all plan as full scans (database-issues#1924).This PR makes detection read the filter as written.
key_bounds.rsis an exact analysis of what a predicate implies about a list of key expressions: a disjunction of boxes, each bounding every key field to a set of literals or leaving it unbounded;ANDintersects boxes pairwise,ORunions them.match_indexasks it whether the predicates pin every field of a candidate key and for which values. Nothing is approximated: a predicate the analysis cannot read leaves the key unbounded, and the only limit is on the number of values enumerated for a lookup, tied toFOLD_CONSTANTS_LIMIT, the size of constant collection a plan may carry.Everything else in the transform is unchanged. Preparation, dropping contradictory disjuncts,
undo_preparation, and the DNF-based removal of constraints from the filter are the same code and produce the same expressions. One consequence is stated in the module doc: detection now has more reach than removal, so a filter can yield a lookup whose constraints removal cannot take out, and the filter then stays whole above the lookup. Removal used to assert that every disjunct carried a key constraint; that held only because the old detection never let such a shape through (a = NULL OR a = 2now looks up(2)), and it now declines instead.What changes in plans
Against
main'sliteral_constraints.slt, one plan differs:WHERE a = NULL OR a = 2becomes a lookup of(2)with the filter kept. The new tests in that file are the shapes the transform previously scanned:INlists past the old size guard, pair lists on a two-column index, a covered list alongside uncovered ones, a cross product of two covered lists, and two disjoint pair lists (empty result, asserted on rows). The datadriven transform specs are unchanged.Relation to #38515
The analysis is that PR's; the box budget, widening, the
widenedbit, the contradiction-pruning pass and the removal by per-predicate exactness are deliberately not carried over, so that the transform's simplification and removal behaviour is untouched and the PR has no thresholds or heuristics beyond the existing constant-size limit. Moving contradiction simplification into the fixpoint loop, and deleting the DNF machinery, are follow-ups.