@bri-prism — your megakernel/rmsnorm-qmv-fuse branch has been quiet since July 8, and after measuring dispatch cost on gfx1201 I think the GDN state ladder on it deserves reviving. I ported it to HIP and ran it on the R9700 box. Two rungs transfer, one inverts, and one turns out to be already shipped on the CUDA/HIP side. Opening this as its own issue deliberately, so it does not pile onto #116's review.
The motivation, quantified first. On gfx1201, Bonsai-27B Q1_0 decode spends its 15.5 ms token across 1837 single-stream dispatches; injecting null dispatches into the live engine prices the per-dispatch wall cost at low-single-digit microseconds, and HIP graph replay does not reduce it. So your instinct to fold dispatches was right, and it is worth more on HIP than the launch tax alone (below).
A reframe before the numbers: the write side of your ladder is already shipped here. ggml-cuda.cu carries an unconditional graph-fusion pass (ggml_cuda_try_gdn_cache_fusion) that folds the GDN snapshot-cpy into the kernel — I instrumented it and it fires 144/144 on real Bonsai-27B decode. Metal's write-side kernel change has no HIP equivalent to write. So the port is the read side plus your beta/alpha glue.
Per-rung, tg128, Bonsai-27B, gfx1201, r5 x 2 rotated rounds (baseline = prism-v7 + the RDNA4 mmvq tuning from the #116 thread):
| rung |
Q1_0 |
PQ2_0 |
vs baseline |
| baseline |
65.0 |
51.97 |
— |
| f16 recurrent state |
63.2 |
50.70 |
−2.6% — inverts on HIP |
| in-place (read-view) |
67.68 |
54.17 |
+4.1 / +4.2% |
| fused beta/alpha glue |
67.12 |
53.81 |
+3.3 / +3.5% |
| in-place + glue stacked |
69.36 |
55.70 |
+6.7% / +7.2% |
Dispatch removal is exactly additive across the two winning rungs (48.0 + 139.6 = 187.6/token). The glue rung's measured gain matches a pure launch-tax model almost exactly; the in-place rung measures ~4x the launch-tax prediction — the eliminated gather/scatter was moving real DRAM bytes, so on HIP that rung is worth more than dispatch count suggests.
The f16-state inversion is an interaction, not a contradiction of your Metal result: with GGML_RECURRENT_STATE_F16=1, the pre-existing write-fusion's match requires dst->type == F32, so it stops firing entirely (instrumented: 144 hits -> 0). The read-side byte saving loses to that. On Metal, where the write path is your new kernel anyway, the tradeoff does not exist.
Correctness, held to your own "byte-identical" bar: greedy 128-token byte-identity vs baseline for every flag combination on both models; MTP spec-decode (--spec-type draft-mtp, the trained Bonsai MTP head, --temp 0 --seed 42) byte-identical; pp512 flat; test-backend-ops MUL_MAT 1238/1238, MUL_MAT_ID 944/944, GATED_DELTA_NET 36/36. The fused-BA path is gated to n_seq_tokens==1 && n_rs_seq==0, matching the in-place rung's discipline, so rollback-active paths never see it.
Two standalone bugs found while porting, independent of this ladder — happy to file each separately:
ggml_cuda_op_scale hard-asserts F32-only, so any op that scales an f16 tensor crashes; fixed generically via the existing ggml_cuda_cast<> helper.- CUDA
supports_op for GATED_DELTA_NET rejects any op with src[6] set (written for the unimplemented rows-mode). Anything else that populates src[6] silently falls back to the CPU reference kernel — which produced correct token 1 and degenerate output from token 2 before I disambiguated on op_params[1]. Silent-fallback-to-wrong is the worst failure shape; worth a look independent of my usage.
The port is 6 attributable commits on a worktree here (your mechanism, my port, per-rung numbers in each message), rebased on prism-v7 at 3c12dbd. Say the word and I will open it as a PR — or cherry-pick whatever is useful into the branch directly; either works. The R9700 box remains available for anything you want re-run.
@bri-prism — your
megakernel/rmsnorm-qmv-fusebranch has been quiet since July 8, and after measuring dispatch cost on gfx1201 I think the GDN state ladder on it deserves reviving. I ported it to HIP and ran it on the R9700 box. Two rungs transfer, one inverts, and one turns out to be already shipped on the CUDA/HIP side. Opening this as its own issue deliberately, so it does not pile onto #116's review.The motivation, quantified first. On gfx1201, Bonsai-27B Q1_0 decode spends its 15.5 ms token across 1837 single-stream dispatches; injecting null dispatches into the live engine prices the per-dispatch wall cost at low-single-digit microseconds, and HIP graph replay does not reduce it. So your instinct to fold dispatches was right, and it is worth more on HIP than the launch tax alone (below).
A reframe before the numbers: the write side of your ladder is already shipped here.
ggml-cuda.cucarries an unconditional graph-fusion pass (ggml_cuda_try_gdn_cache_fusion) that folds the GDN snapshot-cpy into the kernel — I instrumented it and it fires 144/144 on real Bonsai-27B decode. Metal's write-side kernel change has no HIP equivalent to write. So the port is the read side plus your beta/alpha glue.Per-rung, tg128, Bonsai-27B, gfx1201, r5 x 2 rotated rounds (baseline = prism-v7 + the RDNA4 mmvq tuning from the #116 thread):
Dispatch removal is exactly additive across the two winning rungs (48.0 + 139.6 = 187.6/token). The glue rung's measured gain matches a pure launch-tax model almost exactly; the in-place rung measures ~4x the launch-tax prediction — the eliminated gather/scatter was moving real DRAM bytes, so on HIP that rung is worth more than dispatch count suggests.
The f16-state inversion is an interaction, not a contradiction of your Metal result: with
GGML_RECURRENT_STATE_F16=1, the pre-existing write-fusion's match requiresdst->type == F32, so it stops firing entirely (instrumented: 144 hits -> 0). The read-side byte saving loses to that. On Metal, where the write path is your new kernel anyway, the tradeoff does not exist.Correctness, held to your own "byte-identical" bar: greedy 128-token byte-identity vs baseline for every flag combination on both models; MTP spec-decode (
--spec-type draft-mtp, the trained Bonsai MTP head,--temp 0 --seed 42) byte-identical; pp512 flat;test-backend-opsMUL_MAT 1238/1238, MUL_MAT_ID 944/944, GATED_DELTA_NET 36/36. The fused-BA path is gated ton_seq_tokens==1 && n_rs_seq==0, matching the in-place rung's discipline, so rollback-active paths never see it.Two standalone bugs found while porting, independent of this ladder — happy to file each separately:
ggml_cuda_op_scalehard-asserts F32-only, so any op that scales an f16 tensor crashes; fixed generically via the existingggml_cuda_cast<>helper.supports_opforGATED_DELTA_NETrejects any op withsrc[6]set (written for the unimplemented rows-mode). Anything else that populatessrc[6]silently falls back to the CPU reference kernel — which produced correct token 1 and degenerate output from token 2 before I disambiguated onop_params[1]. Silent-fallback-to-wrong is the worst failure shape; worth a look independent of my usage.The port is 6 attributable commits on a worktree here (your mechanism, my port, per-rung numbers in each message), rebased on
prism-v7at 3c12dbd. Say the word and I will open it as a PR — or cherry-pick whatever is useful into the branch directly; either works. The R9700 box remains available for anything you want re-run.