[SPARK-59170][SQL] Align SQL function groupings across APIs and documentation - #58468
Draft
zhengruifeng wants to merge 4 commits into
Draft
[SPARK-59170][SQL] Align SQL function groupings across APIs and documentation#58468zhengruifeng wants to merge 4 commits into
zhengruifeng wants to merge 4 commits into
Conversation
This reverts commit 5df40d2.
HyukjinKwon
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aligns SQL function grouping across the Catalyst function registry, the Scala API, and the
PySpark API reference.
It uses
ExpressionInfo.getGroupas the canonical taxonomy because the values are validated byExpressionInfo.validGroupsand consumed by the SQL function documentation generator. Specifically,this PR:
ExpressionInfogroups;try_avgandtry_sumfrommath_funcstoagg_funcs;bitmap_and_aggandcollect_unionentries to the PySpark function reference;and
Scala
@grouptags and PySpark's grouped__all__remain the authoritative public API grouping.The registry remains split into helper methods to control bytecode size. The largest helper after
regrouping is
aggregateExpressions, with 88 entries and 2,485 bytes of uninstrumented bytecode,leaving substantial headroom for coverage instrumentation.
Why are the changes needed?
The registry, Scala API, PySpark exports, and API documentation previously used inconsistent group
organization. This made it difficult to determine where a function belongs and easy for the
different surfaces to drift further apart. Aligning the group-defining surfaces gives contributors
one consistent taxonomy, while removing implementation headings that implied a conflicting one.
Does this PR introduce any user-facing change?
No. The function implementations and runtime behavior are unchanged. The Scala and PySpark API
documentation grouping becomes consistent with the SQL function metadata.
How was this patch tested?
build/sbt 'sql-api/compile' 'catalyst/Test/compile'build/sbt 'sql-api/scalastyle' 'sql-api/Test/scalastyle'build/sbt 'catalyst/testOnly *LookupFunctionsSuite'build/sbt 'sql/testOnly *ExpressionInfoSuite'conda run -n spark-dev-313 python -m py_compile python/pyspark/sql/functions/__init__.py python/pyspark/sql/functions/builtin.pyExpressionInfogroups and compared theoverlapping Scala, PySpark export, and RST groups; no mismatches were found.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)