feat(BACKEND-ROCM): register kFusedNormRope, so GLM-5.3 stops taking the path a keep-quant MLA checkpoint cannot survive - #2572
Merged
Conversation
…hat needs it asks a native-only question GLM-5.3 loads on `strix:gpu0` and its first forward throws (#2564). The MLA block's `fused_nr` predicate ends in `vt::OpRegistered(kFusedNormRope, device)`, which is a native-only probe by design, and ROCm registers no such op. With every environment variable unset the predicate is false, the split A-projection path row-slices `kv_a_proj_with_mqa`, and this checkpoint's q8_0 weight has no row slice. This spec takes the first of the three repairs #2564 prices, and says why it rejects the other two. Both halves of the composite are already native ROCm kernels -- `rocm_rmsnorm.hip`'s wavefront-agnostic tree reduce and `rocm_dense_basic.hip`'s cache-read rotation -- so the port is the composition CUDA already makes rather than new arithmetic. Widening `OpRegistered` to see the reference tier would instead change what "available" means for every backend and every op, and it keys naturally on a host-addressability predicate that answers backwards on this board: `gfx1151` reports `pageableMemoryAccess=0`, so the two host-addressability predicates disagree (#2515). The upstream anchors are read on the pin rather than copied from the briefing, and one of them was wrong: `GlmMoeDsaForCausalLM` lives at `deepseek_v2.py:1930` at `5559679229`, not in `deepseek_v32.py`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…MLA checkpoint stops taking the path it cannot survive GLM-5.3 loads on `strix:gpu0` and emits nothing: the first forward throws in the MLA block (#2564). The cause is not the op call, it is the BRANCH above it. `mla_attention.cpp`'s `fused_nr` ends in `vt::OpRegistered(kFusedNormRope, device)`, a native-only probe by design, and ROCm registered no such kernel. So with every environment variable unset the split A-projection path was taken, and it row-slices `kv_a_proj_with_mqa` -- which a q8_0 weight has no way to provide, because a quant block spans whole rows. That makes this op unlike the seven other MLA ops ROCm still lacks. Those fall through to the portable reference tier and merely run on the host. This one is consulted BEFORE the call, so its absence is a refusal. The kernel is the composition of two kernels this backend already had natively: `rocm_rmsnorm.hip`'s shared-memory tree reduce, which uses no warp-level primitive and is therefore wavefront-width agnostic, and `rocm_dense_basic.hip`'s cache-read rotation. `kBlock` stays 256 for the reason `rocm_rmsnorm.hip:41` already gives -- four whole wavefronts, and the same reduction ORDER as the CUDA and CPU siblings, which is what makes an NMSE bar mean anything. The refusal message no longer names one cause when there are two. It reads the predicate terms back: either the environment override is set, or this backend registers no native kernel and `OpRegistered` cannot see the reference tier. The previous text enumerated the backends that HAVE the op and forgot the ones that do not, so it sent a reader looking for a variable nobody had set. Gated by a new FusedNormRope arm in the cross-device harness, at NMSE <= 5e-4 against the CPU oracle in both rope styles, with `off` deliberately not a multiple of the block width and the outputs pre-seeded to a value the kernel must overwrite. Refs #2564 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…wrapped argument list Found while adding the `kFusedNormRope` registration this branch carries. The new call wraps after `RegisterOp(`, the way eight of its neighbours already do, and the recount command this document prescribes did not see it -- the count read the same before and after adding an operation. `grep` is line-based, so `RegisterOp(OpId::[A-Za-z0-9_]*` never matches a call whose argument list breaks after the paren. A contributor who runs the documented command and quotes its answer quotes an undercount, and the command cannot detect that it undercounted. The recorded 44 beside it was stale by eight regardless. The replacement reads the file whole with `-z` and names the `DeviceType` in the pattern, because the old one also counted every backend's registrations in the directory rather than the one a reader asked about. The cell now says what the table IS. `AGENTS.md` forbids storing a measurement of one file inside another, and a live count in a projection document is exactly that shape -- so the fix is to remove the number, not to correct it. Closes #2573 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…eld has `RmsNormArgs::eps` is a `float`. The case declared its constant `double` and assigned it, which is a narrowing conversion the compiler is free to warn about under a stricter flag set than this tree carries today, and which states the wrong thing about the value either way: the number that reaches the kernel is the float, so that is the number the test should name. No behaviour changes -- 1e-6 is exactly representable either way. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… serve it are on the host The checkpoint row said this artifact loads and emits nothing. It now emits ` Paris, which is` on `strix:gpu0` through the shipped entry point, so the row says that instead, with the five reference-tier ops beside it -- because configuration alone cannot tell a device run from a host fallback, which is the failure #2505 names. No speed number is offered. `docs/ROCM.md` disqualifies a performance result from a run with a non-zero reference-tier hit count, and this run has five. The 3516.719 s the harness printed is recorded as the cost of a host-tier MLA arm and explicitly not as throughput. The spec's evidence section carries the measurements: the build, both assertion counts on the focused gate, the CPU-only control that makes those counts mean something, the op-id mapping cross-checked against the run's own named lines, and the mutation ladder with its restore proof. One instrument is recorded as useless rather than quietly dropped. Grepping a passing doctest run for the device name returns zero, because `CAPTURE` prints only on failure. The assertion count is what discriminates a device arm that ran from one that skipped: 20 on the board, 2 on a CPU-only build of the same tree. Refs #2564 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.3 loads on
strix:gpu0and emits nothing. The first forward throws in theMLA block, and the cause is not the op call -- it is the branch above it.
mla_attention.cpp'sfused_nrpredicate ends invt::OpRegistered(kFusedNormRope, device). That is a native-only probe bydesign (
src/vt/op_provider.cpp:779-803), and ROCm registered no such kernel.So with every environment variable unset the predicate was false on
gfx1151,the split A-projection path was taken, and it row-slices
kv_a_proj_with_mqa--which this checkpoint's
q8_0weight has no way to provide, because a quantblock spans whole rows.
That makes
kFusedNormRopeunlike the seven other MLA/DSA ops ROCm still lacks.Each of those has a CPU registration, so on this host-addressable board each
falls through to the portable reference tier and merely runs on the host. This
one is consulted BEFORE the call, so its absence is a refusal rather than a
slowdown, and it was the only thing between this model and a token.
Why this repair and not the other two
#2564 prices three. This takes the first and says why it declines the others.
Widening
OpRegisteredto see the reference tier would change what "available"means at a shared seam for every backend and every op, and
op_provider.cppstates the contract it would break: a unified accelerator would report every op
registered the moment its fallback installed, and the fused-recipe ladder would
stop choosing its portable composite path. It also keys naturally on
host-addressability, and the two host-addressability predicates disagree on this
board --
DeviceMemoryIsHostAddressable()is true, which is what makes thereference tier eligible at all, while
HostMemoryIsDeviceAddressable()is falsebecause
gfx1151reportspageableMemoryAccess=0(#2515, measured twice onhardware). A predicate written against the wrong one reads plausible and answers
backwards on the only board that can test it.
Teaching the split path to slice a block-quantized row is the largest of the
three and repairs a fallback nobody wants taken, since the fused arm is
bit-identical and one launch cheaper. It stays owed in the spec.
What the kernel is
The composition of two kernels this backend already had natively:
rocm_rmsnorm.hip's shared-memory tree reduce, which uses no warp-levelprimitive and is therefore wavefront-width agnostic, and
rocm_dense_basic.hip's cache-read rotation.kBlockstays 256 for the reasonrocm_rmsnorm.hip:41already gives: four whole wavefronts, and the samereduction ORDER as the CUDA and CPU siblings, which is what makes an NMSE bar
mean anything.
vLLM has no fused kernel for the pair, so upstream is the behavioural reference
and the in-tree CUDA sibling is the structural one. Read on the pin
5559679229:mla.py:164-165splits the merged row and norms the latent half,mla.py:175-177ropes the trailing half, anddeepseek_v2.py:512-518fixes the[kv_lora_rank + qk_rope_head_dim, hidden_size]shape both arms assume. The twohalves address disjoint dims, which is why fusing them is arithmetically inert.
One briefing anchor was wrong and is corrected in the spec:
GlmMoeDsaForCausalLMis atdeepseek_v2.py:1930at this pin, not indeepseek_v32.py.The message stopped naming one cause when there are two
The throw's comment enumerated the backends that HAVE the op -- CPU and CUDA --
and forgot the ones that do not, so it told a reader the only way there was
VT_MLA_FUSED_NORM_ROPE=0. It now reads the predicate terms back and says whichone was false, so a reader is not sent looking for a variable nobody set.
Evidence
strix:gpu0(gfx1151, Radeon 8060S, ROCm7.2.53211-97f5574fe2), underrcleases, tree
b413e323b. The tarball sha256 was read on both host and workerand the two agree.
Build:
ninja rc=0,[555/577] Building HIP object rocm_mla_fused_norm_rope.hip.o-- the first compile this TU has ever had.GLM-5.3 e2e, through the production entry point (
vllm-cli->vllm_engine_load->ModelRegistry::Forward), greedy,rc=0,prompt_tokens=5 completion_tokens=4 finish_reason=length:op=114 device=5 selected=vt-nativein that run -- op 114 iskFusedNormRope,device 5 is
kROCM. The id mapping is cross-checked three ways against the run'sown named lines (29/
ConcatAndCacheMla, 99/ConcatMlaNopeRope,33/
MlaPrefillAttention).Five ops served from the portable CPU reference tier in that run --
ConcatAndCacheMla,ConcatMlaNopeRope,MlaPrefillAttention,BatchedMatmul,MlaDecodeAttention-- andkFusedNormRopeis not among them.No speed number is admissible from this run and none is offered.
docs/ROCM.mddisqualifies a performance result with a non-zero reference-tierhit count. The 3516.719 s for four tokens is the cost of a host-tier MLA arm,
not throughput.
Focused numeric gate:
1 test case | 1 passed | 0 failed,20 assertions | 20 passed | 0 failed.The assertion count is the discriminator, not a grep of the trace.
CAPTUREprints only on failure, so grepping a passing run for
ROCMreturns 0 and provesnothing; that instrument is recorded as useless rather than quietly dropped. The
same case on a CPU-only build with no accelerator registered runs 2
assertions. 20 against 2 is the ROCm arm executing.
Mutation ladder, each rebuilt (a mutation the compiler kills is not a killed
mutation) and each restored before the next:
RegisterOp(kFusedNormRope, kROCM)lineLEG rc=1, the #2564 throw reproduced verbatim. KILLEDGATE1 rc=1, 1 case failed,20 assertions / 18 passed / 2 failed. KILLEDpre.sha == post.shabyte-for-byte on both files;GATE1 rc=0, 1 case / 20 assertionsM1 is both the RED and the reachability proof: the mutated tree is behaviourally
the base tree at the branch this repairs, and deleting the production call site
reds the production gate, which a by-hand construction cannot show. Its throw
also reads back the corrected message -- "the fused path was not taken because
this backend (rocm) registers NO NATIVE vt::FusedNormRope kernel".
Full cross-device suite:
29 cases | 28 passed | 1 failed,80296 assertions | 80295 passed | 1 failed. The single failure isMoeSiluMul's bit-exact bf16CHECK(got == ref_b)at 1 ULP, which is thestanding red #1954 already tracks on
gfx1200; the gfx1151 reading is recordedon that issue. This change adds one TU and one registration and touches no MoE
path. It is not measured at the base commit on this board, so "pre-existing"
is argued from the absence of code-path overlap, not from an A/B.
Host-side control (CPU-only build, same tree):
test_mla_attention_block21 cases / 2,282,067 assertions green; whole cross-device suite 28 cases / 13
assertions green.
What is still owed
The seven other MLA/DSA ops on ROCm. Each has a CPU registration, so each serves
from the reference tier on this host-addressable board and none refuses -- they
are what makes a speed result inadmissible here, not what blocks a token. Listed
under
## Owedin the spec and owned byBACKEND-ROCM.Closes #2564
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]