Skip to content

[RLlib] Fix IQL beta validation rejecting paper-default values - #65895

Open
Swigler wants to merge 2 commits into
ray-project:masterfrom
Swigler:fix/64954-iql-beta-validation
Open

[RLlib] Fix IQL beta validation rejecting paper-default values#65895
Swigler wants to merge 2 commits into
ray-project:masterfrom
Swigler:fix/64954-iql-beta-validation

Conversation

@Swigler

@Swigler Swigler commented Sep 3, 2026

Copy link
Copy Markdown

Summary

Fixes #64954.

IQLConfig inherits MARWILConfig.validate() which rejects beta > 1.0. IQL uses beta as an inverse temperature (paper default: 3.0), not as a [0, 1] weight like MARWIL. The inherited cap makes the algorithm degenerate into near-uniform behavioral cloning — you literally cannot use IQL as designed.

Fix: Bypass MARWIL's beta range check in IQLConfig.validate() by temporarily setting beta to a safe value during the parent call, then restoring it and applying IQL's own constraint (beta > 0).

Test plan

  • IQLConfig(beta=3.0).validate() no longer raises (was: ValueError: beta must be within 0.0 and 1.0)
  • IQLConfig(beta=0.0).validate() still raises (IQL's own check: beta must be > 0)
  • IQLConfig(beta=-1.0).validate() still raises
  • Existing RLlib IQL tests pass

🤖 Generated with Claude Code

@Swigler
Swigler requested a review from a team as a code owner September 3, 2026 14:25

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request modifies the validate method in IQLConfig to temporarily override self.beta during the parent class's validation, bypassing its parameter cap. The review feedback suggests wrapping this temporary state mutation in a try...finally block to ensure self.beta is safely restored even if an exception is raised during validation.

Comment thread rllib/algorithms/iql/iql.py Outdated
Comment on lines +203 to +206
saved_beta = self.beta
self.beta = 0.5
super().validate()
self.beta = saved_beta

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If super().validate() raises an exception (for example, due to other invalid configuration parameters), self.beta will not be restored to its original value, leaving the configuration object in a mutated/corrupted state. Wrapping the temporary state mutation in a try...finally block ensures that self.beta is always safely restored.

Suggested change
saved_beta = self.beta
self.beta = 0.5
super().validate()
self.beta = saved_beta
saved_beta = self.beta
try:
self.beta = 0.5
super().validate()
finally:
self.beta = saved_beta

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 09c0f1a. Configure here.

Comment thread rllib/algorithms/iql/iql.py Outdated
@Swigler
Swigler force-pushed the fix/64954-iql-beta-validation branch from 09c0f1a to 804776a Compare September 3, 2026 17:56
@ray-gardener ray-gardener Bot added rllib RLlib related issues community-contribution Contributed by the community labels Sep 3, 2026
…roject#64954)

IQLConfig inherits MARWILConfig.validate() which rejects beta > 1.0.
IQL uses beta as an inverse temperature (paper default: 3.0), not as a
[0, 1] weight like MARWIL, so the inherited cap makes the algorithm
degenerate into near-uniform behavioral cloning.

Bypass MARWIL's beta range check in IQLConfig.validate() by temporarily
setting beta to a safe value during the parent call, then restoring it
and applying IQL's own constraint (beta > 0). Uses try/finally to
ensure beta is restored even if the parent validation raises.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Radu Swigler <radu.porumba@gmail.com>
@Swigler
Swigler force-pushed the fix/64954-iql-beta-validation branch from 804776a to bcda735 Compare September 4, 2026 12:12
In `IQLLearner.add_module()`, the expectile tensor was incorrectly
initialized with `config.beta` instead of `config.expectile` — a
copy-paste error from the temperature block below it. The `build()`
method had it correct; only the dynamic `add_module()` path (multi-agent
setups) was affected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Radu Swigler <radu.porumba@gmail.com>
@Swigler
Swigler force-pushed the fix/64954-iql-beta-validation branch from 0087157 to d2a965b Compare September 5, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community rllib RLlib related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RLlib] IQL inherits MARWIL's beta <= 1.0 validation, preventing use of higher and paper-recommended values

1 participant