perf: Optimize prefix-group processing in PartialSortExec

[lyne7-sc]

Which issue does this PR close?

• Closes #.

Rationale for this change

PartialSortStream concatenates each incoming batch with the buffered incomplete prefix before looking for a completed prefix boundary.

When one prefix spans multiple batches, this repeatedly copies rows accumulated from earlier batches. When a batch contains multiple completed prefix groups, the completed region is also sorted using the full ordering even though the prefix ordering is already satisfied.

This adds unnecessary copying and sorting work, especially for wide batches and workloads containing multiple prefix groups per batch.

What changes are included in this PR?

• Keep the unfinished trailing prefix as a list of RecordBatch slices instead of concatenating it with every incoming batch.
• Detect completed prefix groups within each batch and compare prefix expression results directly across batch boundaries.
• Sort completed groups only by the remaining suffix ordering.
• Concatenate a single completed prefix at most once.
• For multiple completed prefixes, evaluate suffix expressions once per source batch and materialize the final output with one interleave operation.
• Preserve fetch handling, input release, empty-batch handling, and zero-column batch semantics.

What is the testing strategy for this PR?

The implementation is covered by the existing PartialSortExec unit tests and the additional SQL logic test cases in group_by.slt.

Are there any user-facing changes?

No. This is an internal performance improvement.

Benchmark

group                                 main                                   optimized
-----                                 ----                                   ---------
partial_sort/rows_per_prefix/100      3.10     12.1±0.13ms        ? ?/sec    1.00      3.9±0.05ms        ? ?/sec
partial_sort/rows_per_prefix/1000     4.17     13.2±0.07ms        ? ?/sec    1.00      3.2±0.14ms        ? ?/sec
partial_sort/rows_per_prefix/10000    1.30      2.8±0.02ms        ? ?/sec    1.00      2.1±0.07ms        ? ?/sec
partial_sort/rows_per_prefix/20000    1.41      3.0±0.22ms        ? ?/sec    1.00      2.1±0.15ms        ? ?/sec
partial_sort/rows_per_prefix/5000     3.85     12.4±0.27ms        ? ?/sec    1.00      3.2±0.03ms        ? ?/sec
partial_sort/rows_per_prefix/8192     1.56      2.8±0.05ms        ? ?/sec    1.00  1780.8±34.70µs        ? ?/sec

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.97059% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.67%. Comparing base (20d1c56) to head (ddd03a0).
⚠️ Report is 30 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/sorts/partial_sort.rs	88.97%	19 Missing and 11 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24979      +/-   ##
==========================================
+ Coverage   81.64%   81.67%   +0.03%     
==========================================
  Files        1123     1126       +3     
  Lines      410248   414772    +4524     
  Branches   410248   414772    +4524     
==========================================
+ Hits       334940   338770    +3830     
- Misses      55617    56082     +465     
- Partials    19691    19920     +229     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ryux1]

Traced the stream state across continued prefixes, batch-boundary changes, multiple completed prefixes, end-of-input, and fetch exhaustion. The logical prefix offsets and batch-local fragment mapping are consistent; first-occurrence prefix order is preserved, suffix ordering is applied independently per completed group, and fetch is consumed across groups before the next poll closes/releases the input. The explicit zero-column materialization also preserves row counts. The added SQL cases cover the important cross-batch paths, and the reported CI matrix is green.