[SPARK-59219][PYTHON] Avoid copying the Series to name it in convert_numpy

[Spenserrrr]

What changes were proposed in this pull request?

ArrowArrayToPandasConversion.convert_numpy ended with series.rename(ser_name), but Series.rename is not an in-place rename: pandas implements it as self.copy(deep=False) followed by setting the name. The Series is created inside convert_numpy, so this names it in place instead.

This also adds python/benchmarks/bench_arrow_to_pandas.py, since nothing in the committed ASV suite exercised the Arrow-to-pandas converter.

Why are the changes needed?

The copy runs per column per batch and is fixed cost that does not scale with row count, so it matters most for wide or small batches. For 50 int64 columns at 10k rows the path costs 2.71ms, against 1.11ms for a bare per-column to_pandas() loop; this removes ~21us of that ~32us per-column gap. (The bare loop does not set the Series name, so it is a floor this path cannot reach.)

bench_arrow_to_pandas.py, master vs this PR:

n_rows	n_cols	long	timestamp	string
128	1	60.7 -> 41.3us (1.47x)	196 -> 139us (1.41x)	160 -> 157us
128	50	2.75 -> 1.61ms (1.71x)	7.05 -> 5.41ms (1.30x)	6.34 -> 6.32ms
10000	1	63.3 -> 42.3us (1.50x)	732 -> 690us (1.06x)	199 -> 196us
10000	50	2.64 -> 1.66ms (1.59x)	35.6 -> 33.5ms (1.06x)	7.18 -> 7.07ms

long and timestamp take convert_numpy; string takes convert_legacy and is unaffected. Since the saving is fixed per column, the ratio shrinks as per-row work grows: timestamp is 1.30x at 128 rows but 1.06x at 10k.

Does this PR introduce any user-facing change?

No. The returned Series is identical.

How was this patch tested?

Existing test_conversion.py and test_pandas_udf_scalar.py pass, plus a new test pinning that convert_numpy reads the Arrow field name before preprocess_time, whose pa.compute kernels return a new array without it; moving that capture below preprocess_time fails only the new test.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[Spenserrrr]

Hi @Yicong-Huang! This is part of the effort to speed up the arrow-to-Pandas conversion. I also added a benchmark file so you can produce the results if you want. Could you take a look when you have time? Thanks!
cc @zhengruifeng

[Yicong-Huang]

thanks, can you also post benchmark results on PR description, so we can see the improvement?

though from the current description seems the benefit is not very large.

[Spenserrrr]

Thanks for the feedback! I added the number to the PR description. And yes it is not a big improvement on its own. It is saving ~20us of fixed cost per column, so it only shows as a ratio when per-row work is small: 1.7x on 50 long columns, but 1.06x on timestamp at 10k rows.

[zhengruifeng]

Merge Summary:

• merged into master 7b60b3e
• merged into branch-4.x e94c638

Posted by merge_spark_pr.py