transform: complete partly-literal index keys using join equivalences

[djahandarie]

[Note: recommend reviewing the code via this view of the second commit, as the first commit is just moving some code blocks around.]

Motivation

An arrangement can only be probed with its complete key. When literal equalities on a join input cover only some fields of an index key, LiteralConstraints declines the index and the plan degrades to a full scan, even though the enclosing join's equivalences bind the remaining fields. With an index on t(a, b):

SELECT ... FROM t
WHERE t.a = (SELECT a FROM m WHERE m.k = 1) AND t.b IN (7, 8)

scans t and builds a fresh arrangement on a, and emits a "too wide" notice whose hint (index (b) alone) is usually bad advice. Passing a as a literal instead gives two point lookups. The optimizer has everything it needs to do that itself: the join already performs runtime-keyed probes, and LiteralConstraints already knows how to turn literals into a constant collection. The two passes just never combined.

Several mz_introspection views hit the same shape through worker_id = 0 on (id, worker_id) indexes.

Description

Two commits. The first is a pure move: match_index and undo_preparation become associated fns of LiteralConstraints so a second call site can reuse them. It has no behavior change and can be reviewed on its own.

The second adds a pre-order Join hook to LiteralConstraints::action (complete_join_index_key). It widens the indexed input's projection so the literal-covered key fields are visible to the join, then supplies their values one of two ways:

• One literal per field enters the join as an equivalence {col, lit}. The JoinImplementation orderer treats such a field as bound via a new literal_bound set. This is kept apart from bound, which doubles as the key of arrangements the join builds; merging them would widen those keys and trade existing indexes for new arrangements.
• An IN list of two or more values becomes a constant collection crossed into the one other input that binds the rest of the key, so a single input supplies the whole key. It cannot be a sibling join input: the orderer can only place an input once one of its equivalence classes is active, and a sibling constant only connects to the indexed input itself, so no order ever binds the whole key before placing it.

The literal constraints are removed from the input's MFP in both cases, so the Get case does not report the index as too wide for that read. This is per read: another read of the same index elsewhere in the plan that is still a full scan keeps its notice.

The rewrite pays off only if JoinImplementation can lift the input's MFP and reuse the index, which requires every key it needs on that input to be an index key. Without a guard, a third input reaching the indexed one by a non-index key would leave the MFP unlifted: two fresh arrangements over the now-unfiltered table plus a replicated binder. The rewrite therefore fires only when every input that equates with the indexed one binds all the uncovered key fields, and for a collection only when that input is unique. The NOTE: on complete_join_index_key states the residual cost when the order still declines the index.

Gated by enable_partial_literal_index_lookups, default off in production and on in CI per the project convention for new optimizer paths. Known limits are left as TODO:s: only one input per join is completed, and the number of IN values is not weighed against the binder's cardinality.

Verification

New test/sqllogictest/transform/join_literal_constraints.slt covers both tiers with results asserted against the flag-off plans: NULLs in every key and join column, a duplicate-value IN (7, 7), an empty scalar subquery, a non-unique join key on the binder side (the crossed-in values must not multiply rows), a 3-input delta join in both tiers (the single-literal one fires and drops an arrangement, the IN-list one is declined by the guard and keeps its notice), an explicit flag-off plan, and a maintained index over a rewritten join kept correct under an incremental insert.

catalog_server_explain.slt is rewritten: eight introspection views go from full scan to using their per-worker index as a join arrangement, with no CrossJoin or Constant introduced, since they are all the single-literal case.

No release note: off by default.

🤖 Generated with Claude Code

[djahandarie]

I pushed a third commit, 89138e8, that changes when the join-level key completion is allowed to compete with an index the literals already cover fully. An example of what it fixes:

Take a rollup monthly(id, month, ...) keyed by (id, month), indexed on (month) and on (id, month), joined to a small ids relation on id with month IN ('2026-07-01', '2026-08-01'). Without the new commit the plan looks up the two months through the (month) index, which returns every id's rows for both months, and then arranges all of that by id to meet ids. The (id, month) index would have been two point lookups per id. The narrow index wins because the Get case of LiteralConstraints turns any fully covered index into an IndexedFilter before the join-level completion gets a look.

The commit lets the wider key take precedence, but only when it strictly extends every fully covered index, so no literal is left for the Get case, and contains a unique key of the relation.

Ideally we would weigh |narrow lookup result| against |binder| x |literal values| with statistics, but none exist to estimate either (indexes and materialized views get EmptyStatisticsOracle). Lacking that, the commit falls back on the standard rule-based preference for unique-key lookups, which certifies two things without data: each probe yields at most one row, so |binder| x |literal values| caps the wide plan, and the narrow index is provably not a point lookup. The rule can still lose when a non-key literal is very selective and the binder is large, the bet every rule-based optimizer takes, and it should yield to statistics once column statistics exist. Without a unique key on the relation, behaviour is unchanged.