Commit 0f2c4fc
authored
SDPA: use exp_u20 for the softmax exponential (pytorch#22082)
Summary:
Switch the softmax exponential in the flash-attention custom SDPA
kernel (`op_sdpa_impl.h`) from `Vectorized::exp()` to
`Vectorized::exp_u20()`. The file already carried this as a TODO.
This is not bit-exact. `exp_u20` is a ULP-20 approximation, so it differs
from `exp` outright, and fourteen autoregressive layers on that is enough
to flip a near-tie argmax.
Reviewed By: JakeStevens
Differential Revision: D117198987
Pull Request resolved: pytorch#220821 parent 96c621d commit 0f2c4fc
1 file changed
Lines changed: 1 addition & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
593 | 593 | | |
594 | 594 | | |
595 | 595 | | |
596 | | - | |
597 | | - | |
598 | | - | |
| 596 | + | |
599 | 597 | | |
600 | 598 | | |
601 | 599 | | |
| |||
0 commit comments