minor: remove deprecated interfaces by timsaucer · Pull Request #1481 · apache/datafusion-python

timsaucer · 2026-04-07T18:57:43Z

Which issue does this PR close?

None

Rationale for this change

We have a bunch of stale interfaces, some that have been marked as deprecated for over 10 releases. This removes any that have been deprecated for >3.

What changes are included in this PR?

Remove deprecated interfaces.

Are there any user-facing changes?

Yes, these old interfaces are no longer available.

…been deprecated since DF48.

Copilot

Pull request overview

This PR removes a set of long-deprecated Python and Rust-facing interfaces in the DataFusion Python bindings, and updates examples/docs to use the supported APIs (notably migrating window usage to Expr.over(Window(...))).

Changes:

Remove deprecated Python modules/methods/classes (e.g., datafusion.html_formatter, datafusion.udf module alias, Expr.display_name, functions.window, SessionContext.tables, Catalog.database, etc.).
Remove deprecated Rust/PyO3 bindings that backed those Python APIs (e.g., the window() pyfunction and deprecated DataFrame methods).
Update docs and TPCH examples to use the non-deprecated window/formatter APIs.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
python/tests/test_expr.py	Removes test coverage for the deleted `Expr.display_name()` deprecation path.
python/datafusion/udf.py	Deletes deprecated `datafusion.udf` module shim.
python/datafusion/substrait.py	Removes deprecated lowercase alias classes (`plan`, `serde`, `producer`, `consumer`).
python/datafusion/html_formatter.py	Deletes deprecated `datafusion.html_formatter` module shim.
python/datafusion/functions.py	Removes deprecated `functions.window()` API and related exports/imports.
python/datafusion/expr.py	Removes deprecated `Expr.display_name()` method and decorator import usage.
python/datafusion/dataframe.py	Removes deprecated `select_columns()` and `unnest_column()` Python methods.
python/datafusion/dataframe_formatter.py	Updates docstring examples to use `datafusion.dataframe_formatter` instead of `datafusion.html_formatter`.
python/datafusion/context.py	Removes deprecated `RuntimeConfig`, `from_arrow_table()`, and `tables()` Python APIs.
python/datafusion/catalog.py	Removes deprecated `Catalog.database()` and `Database` alias class.
python/datafusion/init.py	Stops exporting `Database` from the public top-level API.
examples/tpch/q22_global_sales_opportunity.py	Migrates aggregate window usage from `F.window(...)` to `F.avg(...).over(Window(...))`.
examples/tpch/q17_small_quantity_order.py	Migrates aggregate window usage from `F.window(...)` to `F.avg(...).over(Window(...))`.
examples/tpch/q15_top_supplier.py	Migrates aggregate window usage from `F.window(...)` to `F.max(...).over(Window(...))`.
examples/tpch/q11_important_stock_identification.py	Migrates aggregate window usage from `F.window(...)` to `F.sum(...).over(Window(...))`.
examples/tpch/q02_minimum_cost_supplier.py	Migrates aggregate window usage from `F.window(...)` to `F.min(...).over(Window(...))`.
docs/source/user-guide/common-operations/windows.rst	Updates documentation to demonstrate windowing aggregates via `.over(Window(...))` rather than `functions.window()`.
crates/core/src/functions.rs	Removes deprecated `window` PyO3 function and its helper lookup logic.
crates/core/src/dataframe.rs	Removes deprecated `select_columns`/`unnest_column` bindings; switches `__getitem__` to `select_exprs`.
crates/core/src/context.rs	Removes deprecated `tables()` binding implementation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-08T17:33:07Z

python/datafusion/dataframe_formatter.py

    Example:
-        >>> from datafusion.html_formatter import get_formatter
+        >>> from datafusion.dataframe_formatter import get_formatter
        >>> formatter = get_formatter()
        >>> formatter.max_cell_length = 50  # Increase cell length
    """


The docs still reference the removed datafusion.html_formatter module (e.g. docs/source/user-guide/dataframe/rendering.rst), but this PR deletes python/datafusion/html_formatter.py. Update those documentation imports/usages to datafusion.dataframe_formatter (or keep a small compatibility shim) to avoid broken docs/examples.

Here is the referenced page: https://datafusion.apache.org/python/user-guide/dataframe/rendering.html

ntjohnson1

I basically just verified everything removed said deprecated. I didn't check the exact deprecation date

nuno-faria

Thanks @timsaucer.

nuno-faria · 2026-04-08T20:09:27Z

crates/core/src/dataframe.rs

-    #[pyo3(signature = (*args))]
-    fn select_columns(&self, args: Vec<PyBackedStr>) -> PyDataFusionResult<Self> {
-        let args = args.iter().map(|s| s.as_ref()).collect::<Vec<&str>>();
-        let df = self.df.as_ref().clone().select_columns(&args)?;
-        Ok(Self::new(df))
-    }
-


Strange that this is deprecated in datafusion-python but not in datafusion: https://docs.rs/datafusion/53.0.0/datafusion/dataframe/struct.DataFrame.html#method.select_columns

Because we can take both arguments in select vs select_columns, we deprecated select_columns a long time ago because it's more pythonic to have just one interface.

nuno-faria · 2026-04-08T20:10:10Z

docs/source/user-guide/common-operations/windows.rst

+        f.avg(col('"Attack"')).over(
+            Window(
+                window_frame=WindowFrame("rows", None, None),
+                partition_by=[col('"Type 1"')],
+                order_by=[col('"Speed"')],
+                null_treatment=NullTreatment.IGNORE_NULLS,
+            )
+        ).alias("Average Attack"),


I don't think the order_by, window_frame, or null_treatment make sense here.

The window_frame is necessary so that we get the same avg across the entire partition. Otherwise we'd need a sort on it afterwards to make sure it shows up in the "running avg" and I think using the frame is more easy to understand. But you're right about the null treatment and order_by.

…he deprecated html_formatter

timsaucer added 10 commits April 7, 2026 14:14

udf module has been deprecated since DF47. html_formatter module has …

98da313

…been deprecated since DF48.

database has been deprecated since DF48

bdf8c89

select_columns has been deprecated since DF43

9d28c0b

unnest_column has been deprecated since DF42

1b77551

display_name has been deprecated since DF42

e70cd28

window() has been deprecated since DF50

e6e6459

serde functions have been deprecated since DF42

599ef60

from_arrow_table and tables have been deprecated since DF42

d5283ef

RuntimeConfig has been deprecated since DF44

2ab0698

Update user documentation to remove deprecated function

885502b

timsaucer self-assigned this Apr 7, 2026

timsaucer added the api change label Apr 7, 2026

update tpch examples for latest function uses

d8d6441

timsaucer marked this pull request as ready for review April 8, 2026 13:33

Merge branch 'main' into minor/remove-deprecated

07502c1

timsaucer requested a review from Copilot April 8, 2026 17:25

Copilot started reviewing on behalf of timsaucer April 8, 2026 17:25 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

timsaucer mentioned this pull request Apr 8, 2026

Release DataFusion-Python 53.0.0 #1442

Open

14 tasks

ntjohnson1 approved these changes Apr 8, 2026

View reviewed changes

nuno-faria approved these changes Apr 8, 2026

View reviewed changes

timsaucer added 2 commits April 8, 2026 17:30

Remove unnecessary options in example

a185cfb

update rendering for the most recent dataframe_formatter instead of t…

8f7bd79

…he deprecated html_formatter

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

minor: remove deprecated interfaces#1481

minor: remove deprecated interfaces#1481
timsaucer wants to merge 14 commits intoapache:mainfrom
timsaucer:minor/remove-deprecated

timsaucer commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

nuno-faria Apr 8, 2026

Uh oh!

ntjohnson1 left a comment

Uh oh!

nuno-faria left a comment

Uh oh!

nuno-faria Apr 8, 2026

Uh oh!

timsaucer Apr 8, 2026

Uh oh!

nuno-faria Apr 8, 2026

Uh oh!

timsaucer Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

timsaucer commented Apr 7, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

nuno-faria Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

ntjohnson1 left a comment

Choose a reason for hiding this comment

Uh oh!

nuno-faria left a comment

Choose a reason for hiding this comment

Uh oh!

nuno-faria Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

nuno-faria Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants