fix(db): keep residual clauses residual when combining WHERE clauses

.changeset/optimizer-residual-convergence.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/db': patch
+---
+
+Keep pushed outer-join predicates residual across optimizer passes and preserve source-free WHERE clauses on joins. This prevents repeated filters and ensures joined queries still honor constant conditions.

docs/contributing/oracle-reviews/2026-10-01-pr-1977-residual-convergence.md

@@ -0,0 +1,129 @@
+# PR 1977 residual-convergence oracle review
+
+The reviewed baseline is origin/main at
+`18abceee48ebde712e120cbc541289bb83f35d77`. The contributor's supplied
+patch corresponds to proposed head `e6a1e2924d4acb5bb41b0478df1bbbe7171498a9`.
+The reviewed candidate is that baseline plus the uncommitted changes whose
+optimizer file has SHA-256
+`fee8078ebdfc40d9a431a7c48f7061dd1afbc0f7383a95a13f7d87507248bfc3`
+and oracle file has SHA-256
+`bd5b06605814d3f1c190455438974f9112a53219930ec7d8b92673917d3dc56c`.
+The separate task-local external-feedback ledger preserves all 13 PR and bot
+items; this record preserves the oracle and repair evidence.
+The loss-audit follow-up reviewed PR head
+`0da9cb9a0612e0b905723dffd2791c27978c6ec2` against the guide and
+production paths. The code-bearing follow-up head is
+`92490bc7245c734ad547746786ae9c0219fe81db`. This review record is an
+evidence-only successor to that head.
+
+## Contract and checked boundary
+
+The optimizer's predicate-pushdown contract permits a predicate on a
+nonnullable outer-join source to enter that source. The joined row still needs
+the predicate, so its outer copy is residual and must not be pushed again.
+A predicate on a nullable source stays regular. One logical predicate should
+enter each eligible source once, including when optimization runs again.
+The public-row contract requires the same result as applying predicates after
+the join. These rules follow the optimizer's documented pushdown behavior and
+the established outer-join tests.
+
+The finite structural grammar crosses LEFT, RIGHT, INNER, and FULL joins with
+separate or AND-combined clauses, both clause orders, and one or two active
+predicates. Fixed extensions cover a LEFT/INNER chain with two pushed sources,
+same-source OR, cross-source equality, greater-than, nullable-field
+`isUndefined`, and an already residual outer clause. The checkpoint is the
+returned `optimizeQuery` IR, both initially and after re-optimization.
+The oracle counts each predicate in its source and outer WHERE and checks the
+outer residual marker and clause count. Plain-array LEFT, RIGHT, and
+three-source join models separately check the first public snapshot of a
+live-query Collection, with matched and unmatched source rows.
+
+## RED, GREEN, and code weight
+
+| Subject | Result at the intended checkpoint |
+| --- | --- |
+| Original optimizer, full expanded oracle | 21 failed, 28 passed. Source copies multiply and outer residual markers disappear; the new greater-than witness alone fails at source predicate multiplicity. |
+| Contributor's supplied patch, full expanded oracle | 1 failed, 48 passed. A later pass with an existing residual leaves three outer WHERE clauses, while the law expects one regular and one residual group. Predicate multiplicities and markers pass that case. |
+| Candidate partition-and-combine repair, full expanded oracle | 49 passed. The candidate combines each regular/residual group once inside `applyOptimizations` and removes caller-side residual reassembly. |
+| Candidate, optimizer, join, and join-subquery suites | 247 passed after the predicate-kind case was added. |
+| Wrong-design controls | Marking regular clauses residual fails on `member.userId`'s marker. Keeping only the first residual fails on the missing `tag.region` predicate. Both are assertion failures at the structural checkpoint, not setup failures or timeouts. |
+
+At the initial reviewed candidate, the production change added 20 lines and
+removed 28, net **minus eight**. The executable oracle added 533 lines. The
+loss-audit follow-up has separate code-weight figures below.
+
+## Oracle guide audit
+
+| ID | Outcome |
+| --- | --- |
+| ORC-001 | The contract above states the optimizer and outer-join law, authority, finite grammar, and observations. The coverage map owns omitted expressions, join chains, and publication histories. |
+| ORC-002 | The structural expectation is an explicit relational-side table, not the optimizer's nullable-source classifier. The public result models join and filter plain arrays; they import neither optimizer analysis nor compiler evaluation. |
+| ORC-003 | The oracle opening states the contract, model, grammar, driver, and checkpoints. The table and array models give the answer; `optimizeQuery` and `createLiveQueryCollection` are the production drivers; assertions are adjacent to each case. |
+| ORC-004 | The finite matrix claims only its stated input grammar. Reconstruction: each of 4 join kinds × 2 clause forms × 2 clause orders × 2 predicate counts is constructed and run. Ablation: join kind changes nullable-side eligibility; clause form tests residual-marker retention after combination; order checks order independence; the extra predicate checks same-source multiplicity. Range: one or two active predicates, with the FULL no-push case, are the bounds. Exclusion: unknown source aliases are outside this valid-query grammar; UNION, arbitrary functions, and incremental histories are outside the claim. This records the controls even though the guide distinguishes finite enumeration from an important generated property. |
+| ORC-005 | Structural work is observed at the `optimizeQuery` return and after a later pass. The public driver observes exact projected rows at the first synchronous Collection snapshot. The recorder preserves duplicate predicate terms and result rows. |
+| ORC-006 | The original implementation and two wrong designs fail at the intended assertions. The contributor's partial fix also fails the later-pass clause-count assertion. |
+| ORC-007 | Not applicable: there is no important generated property or random campaign. Every enumerated case runs in the package oracle test file. |
+| ORC-008 | Not applicable: the structural table and public array models are stateless; no model state is merged or split. |
+| ORC-009 | Source Collection, live-query Collection, public row, public snapshot, and checkpoint use the glossary meanings. The model's `teamPush` and `memberPush` booleans describe expected source placement, not production states. |
+| ORC-010 | Not applicable: the fixed tests do not shrink, record mutable traces, or perform cleanup that could replace an assertion failure. |
+| ORC-011 | The plain-array public result is a different formulation from the structural predicate-count law. It checks the plausible shared fault that a plan with fewer predicates changed joined rows, including unmatched rows. It does not independently prove optimizer work cost. |
+| ORC-012 | This record scores every applicable requirement and identifies non-applicable triggers. The closure claim is limited to the stated grammar, paths, and checkpoints; remaining cells are assigned in the coverage map. |
+| ORC-013 | The original ten-copy result and the repaired one-copy result distinguish the once-only law from repeated pushdown. LEFT and RIGHT active-side cases distinguish the relational-side boundary from a LEFT-only rule; FULL is the no-push control. The existing-residual case rejects a plausible partial grouping. |
+| ORC-014 | Not applicable to the structural path, which has no controlled provider. The public driver uses synchronous in-memory source Collections and claims only their first snapshot; it does not claim an external-provider handoff. |
+
+This closes the reported repeated-pushdown class for the finite structural
+grammar and initial public-result paths above. Arbitrary expression trees,
+more complex join chains, and incremental source changes or publication
+histories remain open cells for the optimizer-semantics owner. The current
+tests give no wall-clock runtime guarantee.
+
+## Package-wide validation
+
+After building the isolated candidate package, the parent audit ran
+`pnpm exec tsc --noEmit -p packages/db/tsconfig.json --pretty false`
+successfully. The full `packages/db` Vitest suite then passed 229 files and
+7,937 tests with no type errors using `--testTimeout=120000`. This resolves
+the earlier self-package declaration setup errors. The structural and public
+history bounds above remain unchanged.
+
+## Loss-audit follow-up
+
+The audit found a real adjacent predicate-preservation defect: with a join,
+`groupWhereClauses` silently discarded a clause that touched no source. A
+`Value(false)` after an active-side filter therefore vanished, and the first
+public live-query Collection snapshot contained three rows instead of none.
+The structural predicate-preservation assertion and the public snapshot test
+both failed before the repair and passed afterward. The repair retains every
+clause without exactly one pushable source in the outer WHERE. Three older
+optimizer tests expected source-free or malformed clauses to disappear; their
+false-green expectations now assert preservation.
+
+The structural matrix now checks both `optimizedQuery` and
+`sourceWhereClauses`, including nullable-source exclusion. Replacing the latter
+with an empty map failed the left-join matrix case at the intended assertion
+(`[]` versus `['team']`). Fixed extensions check a namespace-only predicate,
+later RIGHT/FULL joins that make earlier aliases nullable, and an ordered
+QueryRef that declines pushdown. Each checks the first and later optimizer
+pass. The source-free case also checks public rows. These extensions do not
+establish public results for the later join chains or all QueryRef refusal
+reasons. UNION remains a separate optimizer path.
+
+The parallel preparation review found that the structural term reader ignored
+predicate values. It could not distinguish `team.active=true` from
+`team.active=false`. The reader now records literal values and rejects
+unexpected compared expressions. A calibration test failed when an equality
+literal was deliberately hidden, then passed with the corrected reader. The
+main matrix also checks those values in pushed and residual predicates.
+The duplicate focused LEFT JOIN test was removed because the matrix receives
+its trace and its original-implementation RED evidence.
+
+The final branch diff against the fetched `origin/main` adds 22 production
+lines and removes 31, net **minus nine**. The executable oracle adds 762 lines;
+the existing optimizer tests add 23 and remove 13. Tests and review documents
+are reported separately from production code.
+
+At the code-bearing head, the full `packages/db/tests` suite passed 197 files
+and 7,675 tests with two Vitest threads and a 120-second test timeout. The
+package TypeScript check, focused ESLint, Prettier, and diff check passed.
+An earlier sandboxed full run could not write Vitest's temporary cache and
+failed its replay calibration; the same suite passed with that cache writable.

packages/db/src/query/optimizer.ts

@@ -435,20 +435,7 @@ function applySingleLevelOptimization(query: QueryIR): QueryIR {
   const groupedClauses = groupWhereClauses(analyzedClauses)
 
   // Step 4: Apply optimizations by lifting single-source clauses into subqueries
-  const optimizedQuery = applyOptimizations(query, groupedClauses)
-
-  // Add back any residual WHERE clauses that were filtered out
-  const residualWhereClauses = query.where.filter((where) =>
-    isResidualWhere(where),
-  )
-  if (residualWhereClauses.length > 0) {
-    optimizedQuery.where = [
-      ...(optimizedQuery.where || []),
-      ...residualWhereClauses,
-    ]
-  }
-
-  return optimizedQuery
+  return applyOptimizations(query, groupedClauses)
 }
 
 /**
@@ -691,11 +678,10 @@ function groupWhereClauses(
         singleSource.set(source, [])
       }
       singleSource.get(source)!.push(clause.expression)
-    } else if (clause.touchedSources.size > 1 || clause.hasNamespaceOnlyRef) {
-      // Multi-source clause or namespace-only reference - must stay in main query
+    } else {
+      // Clauses without one pushable source must stay in the main query.
       multiSource.push(clause.expression)
     }
-    // Skip clauses that touch no sources (constants) - they don't need optimization
   }
 
   // Combine multiple clauses for each source with AND
@@ -766,8 +752,9 @@ function applyOptimizations(
       }))
     : undefined
 
-  // Build the remaining WHERE clauses: multi-source + residual single-source clauses
+  // Keep regular and residual clauses separate so neither loses its marker.
   const remainingWhereClauses: Array<Where> = []
+  const residualWhereClauses = query.where?.filter(isResidualWhere) ?? []
 
   // Add multi-source clauses
   if (groupedClauses.multiSource) {
@@ -784,24 +771,28 @@ function applyOptimizations(
       remainingWhereClauses.push(clause)
     } else if (hasOuterJoins) {
       // Was optimized AND query has outer JOINs - keep as residual WHERE clause
-      remainingWhereClauses.push(createResidualWhere(clause))
+      residualWhereClauses.push(createResidualWhere(clause))
     }
     // If optimized and no outer JOINs - don't keep (original behavior)
   }
 
-  // Combine multiple remaining WHERE clauses into a single clause to avoid
-  // multiple filter operations in the pipeline (performance optimization)
-  // First flatten any nested AND expressions to avoid and(and(...), ...)
-  const finalWhere: Array<Where> =
-    remainingWhereClauses.length > 1
-      ? [
-          combineWithAnd(
-            remainingWhereClauses.flatMap((clause) =>
-              splitAndClausesRecursive(getWhereExpression(clause)),
-            ),
+  // Combine within each group to avoid extra filters and nested ANDs.
+  const combineRemaining = (clauses: Array<Where>): Where | undefined =>
+    clauses.length > 1
+      ? combineWithAnd(
+          clauses.flatMap((clause) =>
+            splitAndClausesRecursive(getWhereExpression(clause)),
           ),
-        ]
-      : remainingWhereClauses
+        )
+      : clauses[0]
+  const regularWhere = combineRemaining(remainingWhereClauses)
+  const residualWhere = combineRemaining(residualWhereClauses)
+  const finalWhere: Array<Where> = [
+    ...(regularWhere ? [regularWhere] : []),
+    ...(residualWhere
+      ? [createResidualWhere(getWhereExpression(residualWhere))]
+      : []),
+  ]
 
   // Preserve untouched query options while replacing the optimized clauses.
   const optimizedQuery: QueryIR = {