vulkan: fuse GATED_DELTA_NET state write into recurrent cache - #27973
vulkan: fuse GATED_DELTA_NET state write into recurrent cache#27973PrajwalMukatti wants to merge 1 commit into
Conversation
Closes ggml-org#27193. The CUDA and SYCL backends already fuse the GATED_DELTA_NET -> CPY pattern: instead of writing the recurrent-state snapshots into dst and then copying them into the recurrent (KV) cache with a separate CPY node, they detect the pattern at graph-build time and write the state straight into the cache buffer from the op itself (ggml_cuda_try_gdn_cache_fusion / ggml_sycl_try_gdn_cache_fusion). Vulkan was the only major backend missing this, so every decode step ran an extra CPY per GDN layer (48 per token for Qwen3-Next / Qwen3.8). This adds the same fusion to the Vulkan backend: - ggml_vk_try_gdn_cache_fusion() detects a GATED_DELTA_NET immediately followed by a CPY of its state-snapshot tail into a recurrent-cache view (same checks as the CUDA/SYCL versions), and returns how many nodes to skip. - The gated_delta_net shader gains a second storage buffer (binding 7, CacheBuf) and a state_out_off push constant. When state_out_off > 0 the shader writes the state snapshots directly into the cache at that offset; when 0 it keeps the original behaviour (write into dst at s_off). A +1 sentinel is used so offset 0 is unambiguous. - ggml_vk_gated_delta_net() binds the cache buffer and sets the offset when the fusion fired; otherwise it binds dst as a dummy and leaves state_out_off = 0 (non-fused path unchanged). - GGML_VK_DISABLE_FUSION=1 forces the non-fused path, mirroring GGML_CUDA_DISABLE_FUSION, for A/B verification and as a driver kill-switch. DRAFT — verification status: - Builds cleanly at master (CI). - test-backend-ops GATED_DELTA_NET passes on CPU (CI). - Vulkan-device A/B (fused vs GGML_VK_DISABLE_FUSION=1) NOT yet run on real hardware — the CI runner has no Vulkan device. Requesting review of the approach and help verifying on real GPUs before this leaves draft. Developed/tested informally on a Radeon 890M (gfx1150).
|
Hi @PrajwalMukatti, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
| { | ||
| vk_gdn_fused_cache fc; | ||
| const int skip = ggml_vk_try_gdn_cache_fusion(cgraph, node_idx, fc); | ||
| if (skip > 0) ctx->num_additional_fused_ops = skip; |
There was a problem hiding this comment.
you should do this in a way that's more consistent with the existing fusions.
ggml-org#25144 speculative: fix MTP draft crash on vision inputs ggml-org#27879 Qwen4exp correctness fixes ggml-org#27897 speculative: fix combined draft-mtp + external draft (-md) init crash ggml-org#27973 vulkan: fuse GATED_DELTA_NET state write into recurrent cache 27836 (qwen4exp NextN/MTP draft head) is left out: it overlaps 27879 in qwen4exp.cpp and cannot apply alongside it. 27879 is the correctness set, and we already have a working MTP head, so it wins the conflict. 25144 and 27897 both cover configurations we run -- mmproj alongside MTP, and -md with --spec-type draft-mtp. 27973 is the Gated DeltaNet path, 36 of Flash-Next's 48 layers.
|
Real-hardware test on Strix Halo (AMD Ryzen AI Max+ 395, Radeon 8060S, gfx1151), since this branch specifically wants that. Built from this PR vs a
No measurable win or loss here — all deltas are within noise on this quant/workload. Happy to test a different model/quant or config if useful. |
Closes #27193.
What
The CUDA and SYCL backends already fuse the
GATED_DELTA_NET → CPYpattern: rather than writing the recurrent-state snapshots intodstand then copying them into the recurrent cache with a separateCPYnode, they detect the pattern at graph-build time and write the state straight into the cache buffer from the op itself (ggml_cuda_try_gdn_cache_fusion/ggml_sycl_try_gdn_cache_fusion).Vulkan was the only major backend missing this, so every decode step runs an extra
CPYper GDN layer (48 per token for Qwen3-Next / Qwen3.8-class models). This PR adds the same fusion to the Vulkan backend.How
ggml_vk_try_gdn_cache_fusion()— detects aGATED_DELTA_NETimmediately followed by aCPYof its state-snapshot tail into a recurrent-cache view, using the same structural checks as the CUDA/SYCL versions, and returns how many nodes to skip.gated_delta_net.comp— gains a second storage buffer (binding = 7,CacheBuf) and astate_out_offpush constant. Whenstate_out_off > 0the shader writes the state snapshots directly into the cache at that offset; when0it keeps the original behaviour (write intodstats_off). A+1sentinel keeps offset0unambiguous.ggml_vk_gated_delta_net()— binds the cache buffer and sets the offset when the fusion fired; otherwise bindsdstas a dummy and leavesstate_out_off = 0(non-fused path unchanged).GGML_VK_DISABLE_FUSION=1forces the non-fused path, mirroringGGML_CUDA_DISABLE_FUSION, for A/B verification and as a driver kill-switch.The diff is 73 lines across the two files; the non-fused path is byte-for-byte unchanged when the fusion doesn't fire.
Verification status (why this is a draft)
master.test-backend-ops -o GATED_DELTA_NETpasses on the CPU backend.GGML_VK_DISABLE_FUSION=1) has not yet been run on real hardware in CI — the CI runner exposes no Vulkan device (lavapipe did not initialise), sotest-backend-opsfell back to CPU. I'm opening this as a draft to (a) get review of the approach, and (b) ask for help verifying on real GPUs. It was developed against a Radeon 890M (gfx1150 / RDNA3.5) but I'd like independent confirmation on other Vulkan drivers/vendors before it leaves draft.Feedback on the shader binding approach and the sentinel scheme especially welcome.