Skip to content

feat(BACKEND-ROCM): register kFusedNormRope, so GLM-5.3 stops taking the path a keep-quant MLA checkpoint cannot survive - #2572

Merged
mudler merged 5 commits into
mainfrom
row/BACKEND-ROCM-FUSED-NORM-ROPE
Sep 2, 2026
Merged

feat(BACKEND-ROCM): register kFusedNormRope, so GLM-5.3 stops taking the path a keep-quant MLA checkpoint cannot survive#2572
mudler merged 5 commits into
mainfrom
row/BACKEND-ROCM-FUSED-NORM-ROPE

Conversation

@localai-org-maint-bot

@localai-org-maint-bot localai-org-maint-bot commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

GLM-5.3 loads on strix:gpu0 and emits nothing. The first forward throws in the
MLA block, and the cause is not the op call -- it is the branch above it.

mla_attention.cpp's fused_nr predicate ends in
vt::OpRegistered(kFusedNormRope, device). That is a native-only probe by
design (src/vt/op_provider.cpp:779-803), and ROCm registered no such kernel.
So with every environment variable unset the predicate was false on gfx1151,
the split A-projection path was taken, and it row-slices kv_a_proj_with_mqa --
which this checkpoint's q8_0 weight has no way to provide, because a quant
block spans whole rows.

That makes kFusedNormRope unlike the seven other MLA/DSA ops ROCm still lacks.
Each of those has a CPU registration, so on this host-addressable board each
falls through to the portable reference tier and merely runs on the host. This
one is consulted BEFORE the call, so its absence is a refusal rather than a
slowdown, and it was the only thing between this model and a token.

Why this repair and not the other two

#2564 prices three. This takes the first and says why it declines the others.

Widening OpRegistered to see the reference tier would change what "available"
means at a shared seam for every backend and every op, and op_provider.cpp
states the contract it would break: a unified accelerator would report every op
registered the moment its fallback installed, and the fused-recipe ladder would
stop choosing its portable composite path. It also keys naturally on
host-addressability, and the two host-addressability predicates disagree on this
board -- DeviceMemoryIsHostAddressable() is true, which is what makes the
reference tier eligible at all, while HostMemoryIsDeviceAddressable() is false
because gfx1151 reports pageableMemoryAccess=0 (#2515, measured twice on
hardware). A predicate written against the wrong one reads plausible and answers
backwards on the only board that can test it.

Teaching the split path to slice a block-quantized row is the largest of the
three and repairs a fallback nobody wants taken, since the fused arm is
bit-identical and one launch cheaper. It stays owed in the spec.

What the kernel is

The composition of two kernels this backend already had natively:
rocm_rmsnorm.hip's shared-memory tree reduce, which uses no warp-level
primitive and is therefore wavefront-width agnostic, and
rocm_dense_basic.hip's cache-read rotation. kBlock stays 256 for the reason
rocm_rmsnorm.hip:41 already gives: four whole wavefronts, and the same
reduction ORDER as the CUDA and CPU siblings, which is what makes an NMSE bar
mean anything.

vLLM has no fused kernel for the pair, so upstream is the behavioural reference
and the in-tree CUDA sibling is the structural one. Read on the pin
5559679229: mla.py:164-165 splits the merged row and norms the latent half,
mla.py:175-177 ropes the trailing half, and deepseek_v2.py:512-518 fixes the
[kv_lora_rank + qk_rope_head_dim, hidden_size] shape both arms assume. The two
halves address disjoint dims, which is why fusing them is arithmetically inert.
One briefing anchor was wrong and is corrected in the spec:
GlmMoeDsaForCausalLM is at deepseek_v2.py:1930 at this pin, not in
deepseek_v32.py.

The message stopped naming one cause when there are two

The throw's comment enumerated the backends that HAVE the op -- CPU and CUDA --
and forgot the ones that do not, so it told a reader the only way there was
VT_MLA_FUSED_NORM_ROPE=0. It now reads the predicate terms back and says which
one was false, so a reader is not sent looking for a variable nobody set.

Evidence

strix:gpu0 (gfx1151, Radeon 8060S, ROCm 7.2.53211-97f5574fe2), under rc
leases, tree b413e323b. The tarball sha256 was read on both host and worker
and the two agree.

Build: ninja rc=0, [555/577] Building HIP object rocm_mla_fused_norm_rope.hip.o -- the first compile this TU has ever had.

GLM-5.3 e2e, through the production entry point (vllm-cli ->
vllm_engine_load -> ModelRegistry::Forward), greedy, rc=0,
prompt_tokens=5 completion_tokens=4 finish_reason=length:

 Paris, which is

op=114 device=5 selected=vt-native in that run -- op 114 is kFusedNormRope,
device 5 is kROCM. The id mapping is cross-checked three ways against the run's
own named lines (29/ConcatAndCacheMla, 99/ConcatMlaNopeRope,
33/MlaPrefillAttention).

Five ops served from the portable CPU reference tier in that run --
ConcatAndCacheMla, ConcatMlaNopeRope, MlaPrefillAttention,
BatchedMatmul, MlaDecodeAttention -- and kFusedNormRope is not among them.
No speed number is admissible from this run and none is offered.
docs/ROCM.md disqualifies a performance result with a non-zero reference-tier
hit count. The 3516.719 s for four tokens is the cost of a host-tier MLA arm,
not throughput.

Focused numeric gate: 1 test case | 1 passed | 0 failed,
20 assertions | 20 passed | 0 failed.

The assertion count is the discriminator, not a grep of the trace. CAPTURE
prints only on failure, so grepping a passing run for ROCM returns 0 and proves
nothing; that instrument is recorded as useless rather than quietly dropped. The
same case on a CPU-only build with no accelerator registered runs 2
assertions. 20 against 2 is the ROCm arm executing.

Mutation ladder, each rebuilt (a mutation the compiler kills is not a killed
mutation) and each restored before the next:

Mutation Build Result
M1: delete the RegisterOp(kFusedNormRope, kROCM) line rc=0 LEG rc=1, the #2564 throw reproduced verbatim. KILLED
M2: drop the sin term from the rope half rc=0 GATE1 rc=1, 1 case failed, 20 assertions / 18 passed / 2 failed. KILLED
restored control rc=0 pre.sha == post.sha byte-for-byte on both files; GATE1 rc=0, 1 case / 20 assertions

M1 is both the RED and the reachability proof: the mutated tree is behaviourally
the base tree at the branch this repairs, and deleting the production call site
reds the production gate, which a by-hand construction cannot show. Its throw
also reads back the corrected message -- "the fused path was not taken because
this backend (rocm) registers NO NATIVE vt::FusedNormRope kernel"
.

Full cross-device suite: 29 cases | 28 passed | 1 failed,
80296 assertions | 80295 passed | 1 failed. The single failure is
MoeSiluMul's bit-exact bf16 CHECK(got == ref_b) at 1 ULP, which is the
standing red #1954 already tracks on gfx1200; the gfx1151 reading is recorded
on that issue. This change adds one TU and one registration and touches no MoE
path. It is not measured at the base commit on this board, so "pre-existing"
is argued from the absence of code-path overlap, not from an A/B.

Host-side control (CPU-only build, same tree): test_mla_attention_block
21 cases / 2,282,067 assertions green; whole cross-device suite 28 cases / 13
assertions green.

What is still owed

The seven other MLA/DSA ops on ROCm. Each has a CPU registration, so each serves
from the reference tier on this host-addressable board and none refuses -- they
are what makes a speed result inadmissible here, not what blocks a token. Listed
under ## Owed in the spec and owned by BACKEND-ROCM.

Closes #2564

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…hat needs it asks a native-only question

GLM-5.3 loads on `strix:gpu0` and its first forward throws (#2564). The MLA
block's `fused_nr` predicate ends in `vt::OpRegistered(kFusedNormRope,
device)`, which is a native-only probe by design, and ROCm registers no such
op. With every environment variable unset the predicate is false, the split
A-projection path row-slices `kv_a_proj_with_mqa`, and this checkpoint's q8_0
weight has no row slice.

This spec takes the first of the three repairs #2564 prices, and says why it
rejects the other two. Both halves of the composite are already native ROCm
kernels -- `rocm_rmsnorm.hip`'s wavefront-agnostic tree reduce and
`rocm_dense_basic.hip`'s cache-read rotation -- so the port is the composition
CUDA already makes rather than new arithmetic. Widening `OpRegistered` to see
the reference tier would instead change what "available" means for every
backend and every op, and it keys naturally on a host-addressability predicate
that answers backwards on this board: `gfx1151` reports
`pageableMemoryAccess=0`, so the two host-addressability predicates disagree
(#2515).

The upstream anchors are read on the pin rather than copied from the briefing,
and one of them was wrong: `GlmMoeDsaForCausalLM` lives at `deepseek_v2.py:1930`
at `5559679229`, not in `deepseek_v32.py`.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…MLA checkpoint stops taking the path it cannot survive

GLM-5.3 loads on `strix:gpu0` and emits nothing: the first forward throws in the
MLA block (#2564). The cause is not the op call, it is the BRANCH above it.
`mla_attention.cpp`'s `fused_nr` ends in `vt::OpRegistered(kFusedNormRope,
device)`, a native-only probe by design, and ROCm registered no such kernel. So
with every environment variable unset the split A-projection path was taken, and
it row-slices `kv_a_proj_with_mqa` -- which a q8_0 weight has no way to provide,
because a quant block spans whole rows.

That makes this op unlike the seven other MLA ops ROCm still lacks. Those fall
through to the portable reference tier and merely run on the host. This one is
consulted BEFORE the call, so its absence is a refusal.

The kernel is the composition of two kernels this backend already had natively:
`rocm_rmsnorm.hip`'s shared-memory tree reduce, which uses no warp-level
primitive and is therefore wavefront-width agnostic, and `rocm_dense_basic.hip`'s
cache-read rotation. `kBlock` stays 256 for the reason `rocm_rmsnorm.hip:41`
already gives -- four whole wavefronts, and the same reduction ORDER as the CUDA
and CPU siblings, which is what makes an NMSE bar mean anything.

The refusal message no longer names one cause when there are two. It reads the
predicate terms back: either the environment override is set, or this backend
registers no native kernel and `OpRegistered` cannot see the reference tier. The
previous text enumerated the backends that HAVE the op and forgot the ones that
do not, so it sent a reader looking for a variable nobody had set.

Gated by a new FusedNormRope arm in the cross-device harness, at NMSE <= 5e-4
against the CPU oracle in both rope styles, with `off` deliberately not a
multiple of the block width and the outputs pre-seeded to a value the kernel
must overwrite.

Refs #2564

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…wrapped argument list

Found while adding the `kFusedNormRope` registration this branch carries. The
new call wraps after `RegisterOp(`, the way eight of its neighbours already do,
and the recount command this document prescribes did not see it -- the count read
the same before and after adding an operation.

`grep` is line-based, so `RegisterOp(OpId::[A-Za-z0-9_]*` never matches a call
whose argument list breaks after the paren. A contributor who runs the documented
command and quotes its answer quotes an undercount, and the command cannot detect
that it undercounted. The recorded 44 beside it was stale by eight regardless.

The replacement reads the file whole with `-z` and names the `DeviceType` in the
pattern, because the old one also counted every backend's registrations in the
directory rather than the one a reader asked about. The cell now says what the
table IS. `AGENTS.md` forbids storing a measurement of one file inside another,
and a live count in a projection document is exactly that shape -- so the fix is
to remove the number, not to correct it.

Closes #2573

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…eld has

`RmsNormArgs::eps` is a `float`. The case declared its constant `double` and
assigned it, which is a narrowing conversion the compiler is free to warn about
under a stricter flag set than this tree carries today, and which states the
wrong thing about the value either way: the number that reaches the kernel is
the float, so that is the number the test should name.

No behaviour changes -- 1e-6 is exactly representable either way.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
… serve it are on the host

The checkpoint row said this artifact loads and emits nothing. It now emits
` Paris, which is` on `strix:gpu0` through the shipped entry point, so the row
says that instead, with the five reference-tier ops beside it -- because
configuration alone cannot tell a device run from a host fallback, which is the
failure #2505 names.

No speed number is offered. `docs/ROCM.md` disqualifies a performance result
from a run with a non-zero reference-tier hit count, and this run has five. The
3516.719 s the harness printed is recorded as the cost of a host-tier MLA arm
and explicitly not as throughput.

The spec's evidence section carries the measurements: the build, both assertion
counts on the focused gate, the CPU-only control that makes those counts mean
something, the op-id mapping cross-checked against the run's own named lines,
and the mutation ladder with its restore proof.

One instrument is recorded as useless rather than quietly dropped. Grepping a
passing doctest run for the device name returns zero, because `CAPTURE` prints
only on failure. The assertion count is what discriminates a device arm that ran
from one that skipped: 20 on the board, 2 on a CPU-only build of the same tree.

Refs #2564

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@mudler
mudler merged commit 9f3e6e2 into main Sep 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants