[behroozazarkhalili]

GRPO gave longer gradient accumulation windows more weight when averaging loss metrics. I changed GRPO and its GMPO and GSPO-token copies to record one value per optimizer window, and tests for equal and uneven windows pass. This overlaps #7042 on the same lines and will need a small conflict resolution.
Fixes #7047.

---

Note

Low Risk
Changes only how training metrics are aggregated for logging; optimization math is explicitly unchanged. Clip-ratio and other per-micro-batch diagnostics are untouched.

Overview
Fixes incorrect logged policy_loss, entropy, kl, and aux_loss when gradient accumulation uses uneven windows (e.g. a shortened last window), while leaving the actual training loss and gradients unchanged.

In train mode, micro-batch metric values are now scaled by 1 / current_gradient_accumulation_steps, summed in _loss_window_metrics, and appended to _metrics only when accelerator.sync_gradients is true—so log()’s mean is one value per optimizer step with equal weight per window. Eval still logs per batch. train() clears the accumulator at the start to avoid stale partial sums after an interrupted run. The same logging block is applied in the GMPO and GSPO-token GRPO trainer copies.

Adds test_policy_loss_is_averaged_over_accumulation_windows to assert window-weighted averaging vs the old flat micro-batch mean.

^{Reviewed by Cursor Bugbot for commit 0c3d4dc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[behroozazarkhalili]

Closing this one myself. It came from reading the code, not from a failure in a real training run, and per the discussion on #6862 that isn't the bar for this repo. If someone hits this in practice, the branch is still here and I'm happy to reopen it with a reproduction.