Fix gather_for_metrics bias in trainer metrics

[RerankerGuo]

What does this PR do?

Fixes #7010.

Several trainers reduced per-example tensors to a rank-local scalar with .mean() before passing them to Accelerator.gather_for_metrics. The gather then had no batch dimension left to de-duplicate, so on the last batch of an epoch the duplicated example was folded into the rank-local mean and the padding that accelerate adds to balance rank sizes carried into the reported metric. On a nine-example run split across two ranks, the metric drifted by 10% on that batch, with the sign set by which element accelerate chose to repeat.

The fix moves the per-example reduction after the gather at every site that reduces first:

• trl/experimental/cpo/cpo_trainer.py: logits/rejected, logits/chosen
• trl/experimental/orpo/orpo_trainer.py: logits/rejected, logits/chosen
• trl/experimental/online_dpo/online_dpo_trainer.py: objective/scores_margin, objective/scores, objective/kl, objective/non_score_reward, objective/entropy
• trl/trainer/reward_trainer.py: accuracy, margin

The TPO entropy site is left as-is: the quantity is global (averaged over all masked tokens), not per-example, so the gather ordering does not matter.

Before submitting

• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? — Metrics are reduced to a rank-local scalar before gather_for_metrics, biasing the final batch #7010.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

• AI-assisted: the diff and PR description were drafted with AI assistance and reviewed by a human.

---

Note

Medium Risk
Changes only metric aggregation in distributed runs; logged values may shift on multi-GPU uneven batches but training objectives are untouched.

Overview
Fixes biased logged metrics in multi-process training when trainers called .mean() (or other reductions) before Accelerator.gather_for_metrics, leaving no per-example batch dimension for Accelerate to de-duplicate padded/repeated samples on uneven last batches.

CPO and ORPO now reduce logits per sequence with mean(dim=(-1, -2)) and only average after gather. Online DPO gathers per-example scores, scores_margin, KL, non-score reward, and entropy vectors before the final mean. RewardTrainer does the same for accuracy and margin.

pyproject.toml also lists several previously implicit runtime deps (huggingface_hub, numpy, pandas, psutil, pyarrow, pyyaml) with inline call-site comments. Training loss paths are unchanged; only reported metrics should become correct under distributed uneven batches.

^{Reviewed by Cursor Bugbot for commit 81639ad. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a04650c. Configure here.}

[qgallouedec]

Closing: #7010 was closed by its reporter, as it was found by reading the code rather than in a real run (see #6862).