models: fix GDN normalization from max to rsqrt - #28068
Conversation
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
Correctness: all reference implementations including Qwen's own FlashQLA use x*rsqrt(sum(x^2)+eps); llama.cpp used the max-form. Affects every token through the 36 GDN layers.
|
I'm not sure the difference of ±sqrt(1e-6) is worth this? |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
@CISC You're not wrong, but the numerics do somewhat change a bit and transformers had to be patched 4 days after their impl for Qwen3.5 last year It's more just a reconciliation between the upstream impl and the llama.cpp impl |
|
FLA is being weird though, the eps is there to avoid division by zero, not to offset the value. As this may mess with fusion (not sure it actually does) is it worth enough to have its own op? |
|
https://godbolt.org/z/nWa7se5da for CPU: vsqrtss xmm1, xmm1, xmm1
vmovsd xmm0, QWORD PTR .LC3[rip]
vcvtss2sd xmm1, xmm1, xmm1
vmaxsd xmm1, xmm1, QWORD PTR .LC2[rip] **
vdivsd xmm0, xmm0, xmm1
vcvtsd2ss xmm0, xmm0, xmm0vs vsqrtss xmm1, xmm1, xmm1
vmovsd xmm0, QWORD PTR .LC3[rip]
vcvtss2sd xmm1, xmm1, xmm1
vaddsd xmm1, xmm1, QWORD PTR .LC2[rip] **
vdivsd xmm0, xmm0, xmm1
vcvtsd2ss xmm0, xmm0, xmm0So I guess identical haha except for the max and add op and https://uops.info/table.html - interestingly vmaxsd is actually slower in latency than vaddsd (same throughput though) |
|
I can ask the Qwen team if that helps to reconcile which impl is in fact correct if that helps? Maybe FLA itself is wrong haha |
This comment was marked as spam.
This comment was marked as spam.
Even if FLA is technically wrong, they used FLA during training, so it's still right. :) |
|
I noticed this discrepancy way back during the implementation of Qwen3-Next and we had a discussion back then. I did some measurements and the difference turned out to be completely insignificant for implementation faithfulness, so just dropping the note here :) |
|
@pwilkin nice work as well! Ye i doubt this will change things that much |
|
I'm leaning towards matching the reference implementation, regardless if the numerical impact is small. |
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
5 participants
GDN normalizes q and k with
x * rsqrt(sum(x^2) + eps).llama.cpp uses
x / max(sqrt(sum(x^2)), eps)All references uses the non max form:
sum_sq = (x_f32*x_f32).sum(-1) + eps; rstd = rsqrt(sum_sq), eps default 1e-6.F.normalizefor Qwen3-Next on 2025-09-09 and corrected it three days later in Fix the misalignment between the l2norm in GDN of Qwen3-Next and the implementation in the FLA library. huggingface/transformers#40842-ub 128control-ub 128controlAffects qwen35, qwen35moe, qwen3next, qwen4exp, kimi-linear, kimi-k3, bailingmoe3
ggml_l2_norm unchanged, rwkv7 untouched
AI Usage
Used Claude and Local Models for testing, iteration and code design - manual verification of model / PR usage