[albertvillanova]

This PR declares a torch floor of 2.6.0 and removes the three fallbacks that existed for older torch.

Part of #7644.

Depends on

• torch_compile=True trains with zero loss and NaN gradients on torch < 2.11 with the Triton loss kernels #7687
  • Fix silently wrong torch.compile results in the Triton loss kernels on torch < 2.11 #7688
• GRPO VLM training fails for LlavaNext with torch 2.6.0: ValueError: Image features and image tokens do not match: tokens: 1873, features 1872 #7692
  • Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation #7695
• CheckpointError and wrong gradients in later training tests after a training error on the same test worker #7714: nothing to fix in TRL, it only follows another training failure, so it goes away with Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation #7695 corrected: the leak is in accelerate below 1.11 itself, so it survives Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation #7695
• torch.compile(fullgraph=True) fails for selective_log_softmax_and_entropy on torch 2.6: torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode #7715
  • Fix torch.compile(fullgraph=True) of selective_log_softmax_and_entropy on torch 2.6 #7719
• Saving a checkpoint fails on torch 2.6 when generating the model card: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 #7720

Motivation

TRL names no torch version at all, so the effective floor is whatever the stack enforces: torch>=2.2 from transformers 4.56.2, since accelerate only asks for 2.0.0. torch 2.2 is 32 months old, so under the criterion in #7116 it can be dropped without further justification, and 2.6.0 is 20 months old, well clear of the 1-year rule. transformers on main already requires torch>=2.6.

Three branches exist only to support below that:

• torch.accelerator was introduced in 2.6, with the older path hard-coding "cuda"
• FSDPModule became public as torch.distributed.fsdp in 2.6, previously torch.distributed._composable.fsdp
• the same for MixedPrecisionPolicy and fully_shard

Hard-coding "cuda" is the one with user-visible consequences: on torch below 2.6 an XPU or other accelerator was reported as CUDA.

Changes

• Add torch>=2.6.0 to the project dependencies
• Import FSDPModule, MixedPrecisionPolicy and fully_shard from the public torch.distributed.fsdp unconditionally
• Read the accelerator type from torch.accelerator unconditionally in the experimental online DPO trainer
• Read torch.accelerator without a version check when listing Neuron devices in trl env

The torch>=2.7.0 and >=2.8.0 guards stay, since both are above this floor.

---

Note

Medium Risk
Dependency and distributed-training path changes (FSDP2, activation offloading, vLLM device detection) can break installs below 2.6 and affect non-CUDA accelerators if torch.accelerator behavior differs from the old fallbacks.

Overview
Raises the PyTorch floor to >=2.6.0 in pyproject.toml and pins torch==2.6.0 in the CI “minimum versions” job so supported stacks are exercised against that baseline.

With that floor in place, the PR removes version-gated compatibility paths that targeted older PyTorch: vLLM setup in online DPO always uses torch.accelerator for device type (no CUDA hard-code); FSDP2 code always imports FSDPModule, fully_shard, and MixedPrecisionPolicy from the public torch.distributed.fsdp module; activation offloading always uses torch.accelerator / torch.get_device_module, imports DTensor from torch.distributed.tensor, and drops is_torch_version checks and related doc caveats; trl env lists Neuron devices via torch.accelerator without a 2.6 guard.

^{Reviewed by Cursor Bugbot for commit 0ed7eca. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Thanks. A few things:

• trl/models/activation_offloading.py still has pre-2.6 fallbacks:

  • hasattr(torch, "accelerator") else "cuda" (L152)
  • the is_torch_version(">=", "2.5.0") DTensor import (L41)
  • hasattr(torch.nn, "Buffer") (L240, 2.5+)

• The min-versions job installs torch 2.14.1, so the floor isn't tested. Could you pin torch==2.6.0 there? (like peft in Require peft>=0.17.0 and drop the target_parameters guards #7640)

• I ran the suite with torch 2.6.0 and the min pins: the kernel torch.compile tests in tests/test_kernels.py fail:

  • The chunked log-prob/divergence ones give NaN on 2.6 through 2.9.1 and pass on 2.14.1
  • selective_log_softmax_and_entropy fullgraph fails on 2.6 only.

  So they need a skip below the version that works, or the floor is not what we test.

[albertvillanova]

Thanks, all three confirmed and addressed:

• activation_offloading.py: dropped the torch.accelerator, DTensor and torch.nn.Buffer fallbacks, plus the "Requires a torch build after torch-2.5.0" notes in the docstrings.

• The minimum versions job now pins torch==2.6.0.

• Kernel torch.compile tests: I bisected them and they are two separate issues:

  • The chunked log-prob/divergence tests fail below torch 2.11 (2.6 through 2.10.0), not only up to 2.9.1. Before 2.11, Inductor's analysis of which tensors a Triton kernel mutates traces the kernel with the Python float arguments as constexpr/float, so the .to(tl.float32) added in Keep the Triton loss kernels in fp32 under torch.compile #7600 raises. Inductor then falls back to "every input is mutated" and the compiled forward and backward are wrong, not just NaN. This is a real bug for users with torch_compile=True on torch < 2.11, independent of this PR. Replacing .to(tl.float32) with tl.cast(..., tl.float32) fixes it on 2.6 through 2.14.1. I'll open an issue and a dedicated PR for it.
  • selective_log_softmax_and_entropy fullgraph fails on 2.6 only: Dynamo can't trace tensor.stride() in the autograd.Function backward with fullgraph=True. Without fullgraph it compiles and matches eager on 2.6, and TRL doesn't use fullgraph.

  For now both are marked xfail(strict=True) below the version that works (2.11 and 2.7), so the floor is tested; the fix PR will remove the 2.11 one.

[albertvillanova]

Tracking issue: #7687; dedicated PR this PR depends on:

• Fix silently wrong torch.compile results in the Triton loss kernels on torch < 2.11 #7688

[albertvillanova]

CI is red: "Tests with minimum versions" (torch 2.6.0) fails with 45 failed tests: https://github.com/huggingface/trl/actions/runs/38386202899/job/115258503262

I am investigating.

[albertvillanova]

The following failure is tracked in #7692 and addressed in #7695:

FAILED tests/test_grpo_trainer.py::TestGRPOTrainerVLM::test_train_vlm[trl-internal-testing/tiny-LlavaNextForConditionalGeneration] - ValueError: Image features and image tokens do not match: tokens: 1873, features 1872

This PR depends on:

• Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation #7695

[qgallouedec]

Thanks, LGTM. Let's merge once the minimum versions job is green: on top of #7688 and #7695, the 32 UnicodeDecodeError: 'ascii' reading the model-card template and the 8 CheckpointError (recomputed tensor metadata) only show up on 2.6 and aren't tracked yet.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4cda0de. Configure here.}