GH-51302: [Python] Avoid deprecated .values call in pandas->pyarrow conversion

[jorisvandenbossche]

Rationale for this change

Pandas has deprecated certain aspects of .values (when it looses information), which our pandas->pyarrow conversion currently relies upon. So updating that code path to avoid this usage.

What changes are included in this PR?

Removed usage of .values for the deprecated cases (period, interval, datetimetz). In the custom get_values, we already avoided this for period and interval, added a special case for converting datetimetz to a numpy array. And ensured that other cases were we did a direct .values goes through this get_values helper.

Are these changes tested?

Yes, I added more tests first to ensure full coverage of all cases that might run into the deprecation (e.g. the Schema.from_pandas(df) or pa.array(EA) with period/datetimetz dtypes were not currently covered)

Are there any user-facing changes?

No

Was AI used for this PR?

No

• GitHub Issue: [Python] Update pandas->pyarrow conversion for the .values deprecation in pandas #51302

[jorisvandenbossche]

@github-actions crossbow submit test-conda-python-3.14-pandas-nightly-numpy-nightly

[github-actions]

Revision: 800fb5a

Submitted crossbow builds: ursacomputing/crossbow @ actions-883d546e1c

Task	Status
test-conda-python-3.14-pandas-nightly-numpy-nightly

[Copilot]

Copilot review overview

🟡 Changes recommended

The threaded DataFrame path still accesses .values for timezone-aware columns, and tests do not assert warning-free behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Default DataFrame conversion still triggers DatetimeTZdtype warning · New

What changed in this PR

This PR updates pandas-to-PyArrow conversion to avoid deprecated .values access for timezone-aware, period, and interval dtypes.

Changes:

• Added dtype-aware pandas value extraction.
• Updated array conversion and DataFrame type inference.
• Expanded pandas and schema tests.

File	Summary
python/​pyarrow/​tests/​test_schema.py	Adds schema coverage across supported pandas types.
python/​pyarrow/​tests/​test_pandas.py	Adds conversion and round-trip tests.
python/​pyarrow/​pandas-shim.pxi	Adds timezone-aware datetime handling.
python/​pyarrow/​pandas_compat.py	Updates DataFrame type inference.
python/​pyarrow/​array.pxi	Routes array-like inputs through shared value extraction.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jorisvandenbossche]

The nightly crossbow build now shows no related warnings anymore

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved review issues remain.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Default DataFrame conversion still triggers DatetimeTZdtype warning

[raulcd]

@jorisvandenbossche this is a release blocker, right? I just have one question even though I am not an expert on the area, @AlenkaF do you want to take a look?

[AlenkaF]

LGTM. Happy to merge if my understanding of the .values call in pandas_compat.py is correct.