[RLlib] Fix IQL beta validation rejecting paper-default values - #65895
[RLlib] Fix IQL beta validation rejecting paper-default values#65895Swigler wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request modifies the validate method in IQLConfig to temporarily override self.beta during the parent class's validation, bypassing its parameter cap. The review feedback suggests wrapping this temporary state mutation in a try...finally block to ensure self.beta is safely restored even if an exception is raised during validation.
| saved_beta = self.beta | ||
| self.beta = 0.5 | ||
| super().validate() | ||
| self.beta = saved_beta |
There was a problem hiding this comment.
If super().validate() raises an exception (for example, due to other invalid configuration parameters), self.beta will not be restored to its original value, leaving the configuration object in a mutated/corrupted state. Wrapping the temporary state mutation in a try...finally block ensures that self.beta is always safely restored.
| saved_beta = self.beta | |
| self.beta = 0.5 | |
| super().validate() | |
| self.beta = saved_beta | |
| saved_beta = self.beta | |
| try: | |
| self.beta = 0.5 | |
| super().validate() | |
| finally: | |
| self.beta = saved_beta |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 09c0f1a. Configure here.
09c0f1a to
804776a
Compare
…roject#64954) IQLConfig inherits MARWILConfig.validate() which rejects beta > 1.0. IQL uses beta as an inverse temperature (paper default: 3.0), not as a [0, 1] weight like MARWIL, so the inherited cap makes the algorithm degenerate into near-uniform behavioral cloning. Bypass MARWIL's beta range check in IQLConfig.validate() by temporarily setting beta to a safe value during the parent call, then restoring it and applying IQL's own constraint (beta > 0). Uses try/finally to ensure beta is restored even if the parent validation raises. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Radu Swigler <radu.porumba@gmail.com>
804776a to
bcda735
Compare
In `IQLLearner.add_module()`, the expectile tensor was incorrectly initialized with `config.beta` instead of `config.expectile` — a copy-paste error from the temperature block below it. The `build()` method had it correct; only the dynamic `add_module()` path (multi-agent setups) was affected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Radu Swigler <radu.porumba@gmail.com>
0087157 to
d2a965b
Compare

Summary
Fixes #64954.
IQLConfiginheritsMARWILConfig.validate()which rejectsbeta > 1.0. IQL uses beta as an inverse temperature (paper default: 3.0), not as a[0, 1]weight like MARWIL. The inherited cap makes the algorithm degenerate into near-uniform behavioral cloning — you literally cannot use IQL as designed.Fix: Bypass MARWIL's beta range check in
IQLConfig.validate()by temporarily setting beta to a safe value during the parent call, then restoring it and applying IQL's own constraint (beta > 0).Test plan
IQLConfig(beta=3.0).validate()no longer raises (was:ValueError: beta must be within 0.0 and 1.0)IQLConfig(beta=0.0).validate()still raises (IQL's own check: beta must be > 0)IQLConfig(beta=-1.0).validate()still raises🤖 Generated with Claude Code