fix(rewards): make DAPO zero-penalty region inclusive so cache_length=0 no longer divides by zero

[Linxiushen]

Problem

MathDAPORewardFn.compute_overlong_penalty() returns 0 only while response_len < max_response_length - cache_length, and otherwise divides by cache_length:

if response_len < expected_len:
    return 0.0
elif response_len > self.max_response_length:
    return -self.penalty_factor
else:
    return (expected_len - response_len) / self.cache_length * self.penalty_factor

With cache_length: 0 (no soft window, only the hard cut) and a response whose length is exactly max_response_length, which is the usual length of a rollout truncated at max_tokens, the last branch divides by zero:

File "trinity/common/rewards/dapo_reward.py", line 97, in compute_overlong_penalty
    return (expected_len - response_len) / self.cache_length * self.penalty_factor
ZeroDivisionError: division by zero

The workflow runner catches the exception, logs it and returns an empty result, so the practical effect is that every truncated rollout's experience is silently dropped for that configuration, plus an error log line. The training job itself does not crash. Reachability, stated plainly: examples/dapo_math/dapo.yaml uses cache_length: 4096, so the default path is unaffected; the two existing assertions in __init__ accept 0.

Fix

Use <= for the zero-penalty region. This is the closed interval of the overlong shaping definition in the DAPO paper (Section 2.4) that the module docstring references, and for cache_length > 0 the value at response_len == expected_len was already 0.0 from the linear branch, so results there are unchanged.

I checked that with a differential over 738 combinations of max_response_length x cache_length x penalty_factor x response length: the only cells that differ are the 9 cache_length = 0 cells at response_len == max_response_length, which go from ZeroDivisionError to 0.0.

Tests

• test_overlong_penalty_piecewise covers the three regions of the penalty with a soft window (cache_length=20). It passes before and after; there was no test of the piecewise function before.
• test_overlong_penalty_without_soft_window covers cache_length=0. It fails with the ZeroDivisionError on main and passes with the fix.

pre-commit pins (black 23.7.0, isort 5.12.0, flake8 6.1.0) pass on both files.

---

This fix was developed with AI assistance (Claude); the change and the tests were reviewed and verified locally before submission.