Add support for LISTOF function, array return, and EXPLODE operator in PyDough by knassre-bodo · Pull Request #523 · bodo-ai/PyDough

knassre-bodo · 2026-05-26T17:22:30Z

Linked ticket

Closes #521
Closes #522

Type of change

Bug fix
New feature
Refactor
Docs / config

What changed and why?

Adds the LISTOF function to PyDough to aggregate data into arrays, and the EXPLODE operator to flatten such arrays into multiple rows of table data. Also added support for DataFrame collections containing arrays, which become array literals in the SQL.

All of these features are not supported for ANSI or SQLITE because those dialects do not have those capabilities.
Array literals are not supported for Snowflake because the dialect does not have that capability.
EXPLODE is not yet supported currently for MySQL or Oracle due to additional complications with those dialects.

How I tested this?

Multiple new tests on TPCH data with the name prefix array_data_ to test the LISTOF and array literal features.
Multiple new tests on TPCH data with the name prefix explode_ to test different variations of the EXPLODE operator, both on arrays and strings. Variations include:
- Regular collection.EXPLODE(data)
- Inside of a child context collection.CALCULATE(x=COUNT(EXPLODE(data)))
- Exploding array data vs string data
- Different cardinality/distinctness information about the exploded data
- Exploded data being used for grouping and for window operations
- Nested explosions (e.g. splitting an already split string with collection.EXPLODE(...).EXPLODE(...))
- Splitting strings on an empty delimiter
- DataFrame tables with varying data-types, empty arrays, nulls
Error tests for incorrect uses of the EXPLODE operator (e.g. bad arguments)

Notes for reviewers

A few changes to how the database connections are handled to account for difference in how the connectors return arrays.
Several of the tests have dialect-specific skip marks. Some of these are truly unsupported features, and some are just for now unsupported.

knassre-bodo · 2026-08-17T17:32:52Z

-    def user_collection(self) -> PyDoughUserGeneratedCollection:
-        """The wrapped user-generated collection."""
-        return self._parcel[0]


This is not a thing that should be allowed with unqualified nodes, since any method or property name you add becomes something that can no longer be used as a PyDough term name.

…exploded data columns [RUN DIALECTS][RUN CI]

john-sanchez31

Great job Kian, almost there. Left some comments below :)

john-sanchez31 · 2026-08-19T21:35:56Z

+- `name` (required): the name of the collection created from the explosion operation (similar to `PARTITION`).
+- `value_name` (required): a string literal declaring the name of the new column that will be used to store the exploded data.
+- `index_name` (optional): a string literal declaring the name of the new column that will be used to store the indices of the exploded data.  If not provided, this column is not generated. The `index_name` is required if `is_distinct` is False. The indices are 0-indexed.
+- `version` (optional, default=`"array"`): either `"array"` or `"string"`, stating whether the data to explode is an array being flattened or a string being split on a delimiter.


Is this argument really needed? Can't we just check the type of data?

No because our typing system is not fully robust/reliable

john-sanchez31 · 2026-08-19T21:44:13Z

+- `index_name` (optional): a string literal declaring the name of the new column that will be used to store the indices of the exploded data.  If not provided, this column is not generated. The `index_name` is required if `is_distinct` is False. The indices are 0-indexed.
+- `version` (optional, default=`"array"`): either `"array"` or `"string"`, stating whether the data to explode is an array being flattened or a string being split on a delimiter.
+- `delimiter` (optional): a string literal indicating the delimiter that should be used to split up the string if `version="string"`. If `delimiter` is an empty string, the string will be split into individual characters.
+- `filtering` (optional, default=`True`): `True` if it is possible for not every row in the original collection to be preserved in the exploded sub-collection (i.e. if one of the arrays is empty), and `False` otherwise.


I find this one kind of confusing. Will this filter on certain conditions or will be more like filtering Empty/None values? Assuming the second one, the description could be something as follow:

Suggested change

- `filtering` (optional, default=`True`): `True` if it is possible for not every row in the original collection to be preserved in the exploded sub-collection (i.e. if one of the arrays is empty), and `False` otherwise.

`filtering_empty` (optional, default=`True`): `True` will filter all empty or None values from the original collection to the exploded sub-collection, and `False` otherwise.

That's not what this does. It is just a boolean telling PyDough whether the operation will potentially remove rows (e.g., CAN any of the rows from the original be an empty array.

john-sanchez31 · 2026-08-19T21:48:35Z

 infinity value with `DatabaseDiatect.MYSQL` an error will be raised.

+> [!IMPORTANT]
+> `ArrayType` is only supported for certain dialects: Trino, Postgres, DuckDB, Databricks.


What about Snowflake? I saw it among the supported dialects for the EXPLODE operator

Array literals aren't a thing in Snowflake; you cannot put arrays inside VALUES

john-sanchez31 · 2026-08-19T21:50:19Z

+
+```py
+# For each region, list the names of all nations inside that region
+Regions.CALCULATE(region_name=name, nation_names=LISTOF(nations.name))


Can we add how the result for each of the good example would look like?

For the regions, sure. The other one is too big.

john-sanchez31 · 2026-08-20T14:05:48Z

+        idx_index: int | None,
+        lateral_alias: str,
+        subquery_alias: str,
+    ) -> SQLGlotExpression:


Can we add a docstring describing this implementation? What would be the final SQL for every dialect ?

john-sanchez31 · 2026-08-20T14:16:50Z

+                    inner_term = "CAST('1970-01-01' AS TIMESTAMP)"
+                case _:
+                    raise ValueError(
+                        f"Cannot support empty array of type {inner_type} in Postgres."


Also can we add in the documentation what exactly is supported for Postgres as well?

Will do. To be clear, this is a weird edge case where Postgres struggles to handle empty array literals, and this is a workaround that only works for certain types.

john-sanchez31 · 2026-08-20T14:47:35Z

        if self.skip_sql:
            pytest.skip(f"Skipping SQL text test for {self.test_name}")

+        if (


I like this new functionality. Can you check if the skip of MySQL for infinity values, BodoSQL with to_table and DuckDb/Databricks with keywords can be skipped with this new property?

knassre-bodo added 30 commits May 13, 2026 09:52

Start of explode tests, need to fill out rest

3420f45

Adding LISTOF operator WIP

a9a5c6c

Fixed snowflake array data returning via json.loads

c7ec035

Merge branch 'main' into kian/explode

e61265b

WIP

baf47cb

Experimenting with ordering

91a18cc

Merge branch 'main' into kian/explode

766f2ea

Fixing array_agg testing for dialects

6a1d179

WIP array literal generation on dialects

6c89f4e

Array literal dialect WIP

4ecdaf4

Array data tests handled except for ORACLE

f6f0877

Testing WIP

99098a0

Test run [RUN ORACLE]

4120c9b

Test run oracle 2 [RUN ORACLE]

df06954

Early testing and unqualified nodes WIP

3be7bd3

Resolving conflicts

d2057e4

WIP hybrid/relational translation of EXPLODE operator

3f6fb6f

Switched to ExplodeSpec handling of misc args

2bdf031

WIP Relational steps

d0473e5

Continued work on relational handling

ac158cb

WIP relational handling and qdag collection access

e72bb2f

Skipped accessing sub-collections, for now

10ac9d9

Minor testing patch

df4308e

Three explode tests working in Snowflake

c727d6b

Adding POSTGRES support

ac53caf

Added generated alias, fixed some of problems with multi-explode

799534d

Added more tests

cea2df7

Added more tests, fixed filter pushdown bugs

2462791

Added databricks support

314c7d4

Merging conflcits, adding DuckDB support

9edcbf2

knassre-bodo added 3 commits August 12, 2026 10:58

Added duckdb explode support

f375231

Adding trino support

24de864

Added more tests, need to fix lateral ON condition [RUN DIALECTS]

4a077e5

knassre-bodo requested review from a team, hadia206, john-sanchez31 and juankx-bodo and removed request for a team August 17, 2026 17:26

knassre-bodo commented Aug 17, 2026

View reviewed changes

knassre-bodo added 4 commits August 17, 2026 11:08

Fixing LATERAL ON bug

4910fc5

Added documentation, fixed bugs with name collisions by qualifying un…

dafd2c1

…exploded data columns [RUN DIALECTS][RUN CI]

Merge branch 'main' into kian/explode

84037af

[RUN CI][RUN DIALECTS]

51517f6

john-sanchez31 reviewed Aug 20, 2026

View reviewed changes

knassre-bodo added 11 commits August 20, 2026 10:14

Fixing test bugs [RUN CI][RUN DIALECTS]

069ad2e

Merge branch 'main' into kian/explode

8c57b74

Revisions

ad51c98

Adding explode literal array test

ce0979d

Added exploding string literal test

2d60647

Revisions, testing fixes [RUN CI][RUN DIALECTS]

4ff2736

Fixed bugs, added docs, test run to epxpose trino issue [RUN TRINO]

2adbb90

Fixing trino bug, adding BodoSQL test [RUN CI][RUN DIALECTS]

c8f706b

Explode test fix [RUN CI][RUN DIALECTS]

f6b2431

Explode test fix [RUN CI][RUN DIALECTS]

c80670c

Explode test fix [RUN CI][RUN DIALECTS]

4871bf9

	- `filtering` (optional, default=`True`): `True` if it is possible for not every row in the original collection to be preserved in the exploded sub-collection (i.e. if one of the arrays is empty), and `False` otherwise.
	`filtering_empty` (optional, default=`True`): `True` will filter all empty or None values from the original collection to the exploded sub-collection, and `False` otherwise.

Uh oh!

Conversation

knassre-bodo commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked ticket

Type of change

What changed and why?

How I tested this?

Notes for reviewers

Uh oh!

Choose a reason for hiding this comment

Uh oh!

john-sanchez31 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knassre-bodo Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

knassre-bodo commented May 26, 2026 •

edited

Loading

knassre-bodo Aug 20, 2026 •

edited

Loading

knassre-bodo Aug 20, 2026 •

edited

Loading

knassre-bodo Aug 26, 2026 •

edited

Loading