Skip to content

[SPARK-59170][SQL] Align SQL function groupings across APIs and documentation - #58468

Draft
zhengruifeng wants to merge 4 commits into
apache:masterfrom
zhengruifeng:sql-unify-function-groups-dev-4
Draft

[SPARK-59170][SQL] Align SQL function groupings across APIs and documentation#58468
zhengruifeng wants to merge 4 commits into
apache:masterfrom
zhengruifeng:sql-unify-function-groups-dev-4

Conversation

@zhengruifeng

@zhengruifeng zhengruifeng commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR aligns SQL function grouping across the Catalyst function registry, the Scala API, and the
PySpark API reference.

It uses ExpressionInfo.getGroup as the canonical taxonomy because the values are validated by
ExpressionInfo.validGroups and consumed by the SQL function documentation generator. Specifically,
this PR:

  • regroups all 594 Catalyst registry entries into helpers matching their ExpressionInfo groups;
  • aligns the Scala group ordering and moves try_avg and try_sum from math_funcs to
    agg_funcs;
  • adds the missing bitmap_and_agg and collect_union entries to the PySpark function reference;
    and
  • removes misleading physical section headings from the Scala and Python implementation files;
    Scala @group tags and PySpark's grouped __all__ remain the authoritative public API grouping.

The registry remains split into helper methods to control bytecode size. The largest helper after
regrouping is aggregateExpressions, with 88 entries and 2,485 bytes of uninstrumented bytecode,
leaving substantial headroom for coverage instrumentation.

Why are the changes needed?

The registry, Scala API, PySpark exports, and API documentation previously used inconsistent group
organization. This made it difficult to determine where a function belongs and easy for the
different surfaces to drift further apart. Aligning the group-defining surfaces gives contributors
one consistent taxonomy, while removing implementation headings that implied a conflicting one.

Does this PR introduce any user-facing change?

No. The function implementations and runtime behavior are unchanged. The Scala and PySpark API
documentation grouping becomes consistent with the SQL function metadata.

How was this patch tested?

  • build/sbt 'sql-api/compile' 'catalyst/Test/compile'
  • build/sbt 'sql-api/scalastyle' 'sql-api/Test/scalastyle'
  • build/sbt 'catalyst/testOnly *LookupFunctionsSuite'
  • build/sbt 'sql/testOnly *ExpressionInfoSuite'
  • conda run -n spark-dev-313 python -m py_compile python/pyspark/sql/functions/__init__.py python/pyspark/sql/functions/builtin.py
  • Audited all 594 registry entries against their runtime ExpressionInfo groups and compared the
    overlapping Scala, PySpark export, and RST groups; no mismatches were found.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants