Skip to content

Fix FROM regression - #944

Open
rpiaggio wants to merge 2 commits into
typelevel:mainfrom
rpiaggio:reinstate-missing-from
Open

Fix FROM regression#944
rpiaggio wants to merge 2 commits into
typelevel:mainfrom
rpiaggio:reinstate-missing-from

Conversation

@rpiaggio

Copy link
Copy Markdown
Contributor

I tried the latest SNAPSHOT on our codebase and I got some errors that seem to stem from a regression introduced to grackle in 3954c4e.

Honestly, this is beyond my understanding. I asked Claude to fix it and this is what it came up with and what it had to say. I rewrote the comments somewhat to have a vague idea of what's going on and make it less AI-sounding.


Since 3954c4e a nested select with an INNER join beneath a LEFT join is no longer flattened into its parent. That fixed rows being dropped when the nullable parent is absent, but a filter or order path crossing such a select still resolved its columns to the tables inside it. Those names are no longer in scope, so Postgres fails with Missing FROM-clause entry for table "...". This surfaced in lucuma-odb (gemini-hlsw/lucuma-odb#3013) on every WHERE whose path goes through a nullable or list field with a non-null field below it, e.g. observations.program.users.userId.

Two changes in SqlMapping.addFilterOrderByOffsetLimit:

  • Predicate and order columns owned by a path subquery are referenced through it (col.derive(subquery)). The subquery already projects them; only the reference was wrong.
  • SqlJoin.merge replaces .distinct when combining the filter, order and data joins. The filter path and the data selection each nest the same field independently, and once unflattened they arrive as two subqueries under one synthetic alias, which Postgres rejects as a duplicate table name. They are collapsed into one join exposing the union of their columns.

substWhereTables is now an instance of a generic mapWhereColumns.

Tests: two cases added to SqlNullableParentSuite, a filter through a nullable object hop and through a list hop, each onto a non-null field below. Both fail on main with the ODB's error and pass on v0.30.0, so they pin the regression; the suite's existing tests do the reverse, so both directions are now covered. Skunk, doobie-pg and doobie-sqlite suites are green, and the change compiles on Scala 2.13 and 3.

private def sameShape(a: SqlQuery, b: SqlQuery): Boolean =
(a, b) match {
case (a: SqlSelect, b: SqlSelect) =>
a.context == b.context && a.withs == b.withs && sameShape(a.table, b.table) &&

@hugo-vrijswijk hugo-vrijswijk Sep 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't compare the cols property in SqlQuery, is that on purpose? Or are they omitted because we want to know if these two queries are selecting the same rows (but on different columns). And should this be something that is on the equals of SqlQuery (or a Eq typeclass)?

.contextualiseWhereTerms(context, owner, pred)
.map(mapWhereColumns(_)(viaPathSubquery))

def contextualiseOrderTerms[T](

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shadows an outer contextualiseOrderTerms which also applies to the call on :3388 (case 3 outer predQuery) which I don't think should be calling this variant.


the shadowed contextualiseOrderTerms is also applied to the Case 3 outer predQuery, whose table = distSub and joins = Nil, so viaPathSubquery rewrites the ORDER BY column onto a path subquery with no FROM entry. Repro: ds ordered by DType / "es" / "f" / "name"[SQLITE_ERROR] no such column: nullable_parent_e_nullable_parent_f_nested.name. Also as ordered by b/c/name with a limit

val pathSubqueries: List[SubqueryRef] =
(filterJoins ++ orderJoins).map(_.child).collect { case sq: SubqueryRef => sq }

def viaPathSubquery(col: SqlColumn): SqlColumn =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the guard in viaPathSubquery uses sq.owns(col). Through SqlSelect.owns0, that test is true for any column that is in scope inside the subquery. It does not prove that the subquery projects the column. It does not prove that the subquery is joined into the select under construction. The guard needs both conditions, and neither condition covers the other.
Repro: as(cName: "cat-1", order: "mid") { name }, where cName filters on A/b/c/name and mid orders on A/b/name[SQLITE_ERROR] no such column: nullable_parent_b_nullable_parent_c_nested.name_alias_1.
The subquery is in the FROM clause. It projects id_alias_0, id_alias_2 and name_alias_3 only. It does not project b.name. The correct owner,nullable_parent_b, is in the same FROM clause. The order path joined it.
This query takes the plain Case 1 path at line 3276. It never reaches line 3388, so a fix at line 3388 does not repair it. On main the same query fails with no such column: nullable_parent_c.name_alias_3. The query is broken before and after the change, but the new code causes a different mis-rewrite.

Note for the fix: SqlJoin.merge runs at the select-construction sites, after viaPathSubquery reads the raw filterJoins and orderJoins. In the passing probe as(cName: "cat-1", order: "mid") { name b { name c { name } } }, b.name becomes visible only because merge later folds the data subquery into the filter subquery. A plain sq.subquery.cols.contains(col) test breaks that case. Test the projection against the merged join set

@hugo-vrijswijk hugo-vrijswijk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two SqlSelect's at SqlMapping:2977 and SqlMapping:3124 that use the joins = joins, which I believe should also be using the SqlJoin.merge instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants