[albertvillanova]

This PR makes the trainers raise a clear error when a LoRA config would silently not train the routed experts of an MoE model, which happens with transformers v5 and peft<0.19.

Closes #7765.

Motivation

With transformers v5, MoE routed experts are fused into 3D parameters, and PEFT only applies target_modules like ["gate_proj", "up_proj", "down_proj"] to them from peft 0.19.0. With older versions, still in TRL's supported range, the routed experts are silently not trained on models with a dense first layer or shared experts (GLM-4-MoE, DeepSeek-V3, ...), and other MoE models fail with a "target modules not found" error. See #7765 for the root cause.

Solution

Before creating the PEFT model, the trainers raise a ValueError when all of these hold:

• the config is a LoraConfig, with transformers>=5.0.0 and peft<0.19.0
• the model has fused routed experts (3D parameters under .experts.)
• target_modules would have reached the experts ("all-linear", or a leaf name among gate_proj, up_proj, down_proj, w1, w2, w3)
• target_parameters is not set, so the workaround from LoRA on MoE models does not train the routed experts with transformers v5 and peft<0.19 #7765 keeps working

The error suggests upgrading peft, or adding target_parameters for the fused experts. LoRA on other modules of MoE models, like attention only, is not affected.

No test is added: no CI job combines transformers v5 with peft<0.19, so the error cannot trigger in CI, as with the existing prefix tuning guard. Verified locally with SFT on transformers 5.19.0:

peft	GLM-4-MoE MLP targets	all-linear	attention only	Qwen3-MoE MLP targets	dense model	with target_parameters
0.17.0, 0.18.1	error	error	ok	error	ok	ok
0.21.2	ok	ok	ok	ok	ok	ok

A regex in target_modules, such as an attention-only one, does not raise. On transformers 4.56.2 (peft 0.17.0), nothing raises. DPO and Reward were also checked with peft 0.18.1 (error) and 0.21.2 (ok).

Changes

• Raise a ValueError before get_peft_model in the SFT, DPO, KTO, GRPO, RLOO, Reward and Distillation trainers, with the same block in each
• Import LoraConfig in the Reward, SFT and Distillation trainers, and peft in the Distillation trainer

---

Note

Low Risk
Startup-only validation before PEFT wrapping; no change to training loops or loss computation for valid configurations.

Overview
Adds a pre-get_peft_model guard in SFT, DPO, KTO, GRPO, RLOO, Reward, and Distillation trainers so LoRA setups that would skip fused MoE routed experts fail fast instead of training silently.

The check runs when transformers ≥ 5, peft < 0.19, the model has 3D .experts. parameters, target_parameters is unset, and target_modules is "all-linear" or names MLP expert leaves (gate_proj, up_proj, etc.). It raises a ValueError pointing users to upgrade peft or set target_parameters for fused expert weights (issue #7765). Attention-only or non-MoE LoRA configs are unchanged; configs that already use target_parameters are allowed through.

^{Reviewed by Cursor Bugbot for commit 9c569cc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Thanks. One false positive: any string target_modules is treated as targeting the experts, so an attention-only regex raises. On tiny GLM-4-MoE with peft reported as 0.18.1, r".*\.(q_proj|v_proj)" raises while ["q_proj", "v_proj"] passes. Checking peft_config.target_modules == "all-linear" instead of isinstance(peft_config.target_modules, str) would fix it, same condition in #7771.

[albertvillanova]

Thanks, good catch! Fixed in both PRs: only "all-linear" is now treated as reaching the experts, so an attention-only regex no longer raises (checked with peft 0.17.0 and 0.18.1). A regex targeting the MLP projections is no longer caught either, which seems the right trade-off for an error. I also merged main and dropped the peft<0.17 case, now that #7640 raised the floor.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit f73a5ef. Configure here.}

[cursor]

Guard misses string LoRA targets

Medium Severity

The new guard iterates target_modules as a collection of names, but PEFT treats a string as a regex. Walking that string character by character never matches leaf names such as gate_proj, so regex configs and single-module strings still skip fused MoE experts silently. ModelConfig unwraps a one-item CLI list into a string, which hits this path.

Additional Locations (2)

• trl/trainer/dpo_trainer.py#L671-L676
• trl/trainer/distillation_trainer.py#L546-L551

 

^{Reviewed by Cursor Bugbot for commit f73a5ef. Configure here.}

[albertvillanova]

This is intended. For PEFT, a string target_modules is a regex matched against full module names, and only "all-linear" is treated as reaching the experts, following the review above: an attention-only regex must not raise, so a regex targeting the MLP projections is not caught either. A single module name passed as a string, such as what ModelConfig produces for --lora_target_modules down_proj, matches no module at all, so PEFT already raises that the target modules were not found (checked with peft 0.17.0, 0.18.1 and 0.21.2).

[albertvillanova]

The requested change is addressed in f73a5ef: only "all-linear" is treated as reaching the experts, so an attention-only regex no longer raises, the same change as in #7771. CI is green, so merging.