Skip to content

[SPARK-59147][SQL] Preserve struct nullness when SELECT * EXCEPT drops a nested field - #58477

Open
Vivek1106-04 wants to merge 1 commit into
apache:masterfrom
Vivek1106-04:SPARK-59147-except-null-struct
Open

[SPARK-59147][SQL] Preserve struct nullness when SELECT * EXCEPT drops a nested field#58477
Vivek1106-04 wants to merge 1 commit into
apache:masterfrom
Vivek1106-04:SPARK-59147-except-null-struct

Conversation

@Vivek1106-04

Copy link
Copy Markdown

What changes were proposed in this pull request?

UnresolvedStarExceptOrReplace.filterColumns rewrites a struct column when the EXCEPT list
names a nested field: it extracts the retained fields and wraps them back into a
CreateStruct. CreateStruct resolves to CreateNamedStruct, which is never nullable, so
the rebuilt struct is always non-NULL even when the original struct expression evaluated to
NULL.

This PR guards the reconstruction with If(IsNull(col), Literal(null, dataType), ...) when
the original column is nullable, the same way UpdateFields.evalExpr already does for
DropField.

Why are the changes needed?

Dropping a nested field silently turns a NULL struct into a non-NULL struct whose remaining
fields are NULL. This is a silent correctness bug: nullable structs commonly come from
parsed JSON and from the null-producing side of outer joins, and afterwards IS NULL
checks, filters, joins, COALESCE and serialized output all behave differently.

WITH input AS (
  SELECT id,
    CASE
      WHEN id = 0 THEN CAST(NULL AS STRUCT<a: INT, b: INT>)
      ELSE named_struct('a', id, 'b', id + 10)
    END AS s
  FROM VALUES (0), (1) AS t(id)
),
actual AS (
  SELECT * EXCEPT (s.a) FROM input
)
SELECT i.id, i.s IS NULL AS before_except, a.s IS NULL AS after_except, a.s.b
FROM input i JOIN actual a USING (id) ORDER BY id;

Before:

0  true   false  NULL
1  false  false  11

After:

0  true   true   NULL
1  false  false  11

The same path backs the pipe |> DROP <col>.<field> operator, which had the same problem.

Does this PR introduce any user-facing change?

Yes. SELECT * EXCEPT (col.field) (and |> DROP col.field) now returns NULL for a row whose
col is NULL, instead of a struct of NULL fields. The result type is unchanged. This
corrects wrong results; there is no behavior change for non-nullable structs or for rows
whose struct is not NULL.

How was this patch tested?

Added cases to selectExcept.sql covering a NULL top-level struct, a NULL nested struct, and
struct expansion (SELECT data.* EXCEPT (s2.c)), and regenerated the golden files. The new
cases fail on master and pass with the fix. pipe-operators.sql analyzer results were
regenerated to reflect the added null guard; its query results are unchanged.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

…s a nested field

Nested-field `SELECT * EXCEPT (col.field)` rebuilds the enclosing struct with
`CreateStruct`, which is never nullable. A NULL struct was therefore rewritten
into a non-NULL struct whose remaining fields are NULL, silently changing query
results for `IS NULL` checks, filters, joins and serialized output.

Guard the reconstruction with `If(IsNull(col), Literal(null, dataType), ...)`
when the original column is nullable, matching what `UpdateFields.evalExpr`
already does for `DropField`. The pipe `|> DROP <col>.<field>` operator shares
this code path and is fixed as well.

@tdcmeehan tdcmeehan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix seems correct, minimal, and well-tested to me. Thank you @Vivek1106-04
Please ping @viirya and/or @cloud-fan for additional review!

@Vivek1106-04

Copy link
Copy Markdown
Author

@viirya ! and @cloud-fan - requesting for a review. Thank You

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants