Skip to content

vulkan: fuse GATED_DELTA_NET state write into recurrent cache - #27973

Draft
PrajwalMukatti wants to merge 1 commit into
ggml-org:masterfrom
PrajwalMukatti:vulkan-gdn-cache-fusion
Draft

vulkan: fuse GATED_DELTA_NET state write into recurrent cache#27973
PrajwalMukatti wants to merge 1 commit into
ggml-org:masterfrom
PrajwalMukatti:vulkan-gdn-cache-fusion

Conversation

@PrajwalMukatti

Copy link
Copy Markdown

Closes #27193.

What

The CUDA and SYCL backends already fuse the GATED_DELTA_NET → CPY pattern: rather than writing the recurrent-state snapshots into dst and then copying them into the recurrent cache with a separate CPY node, they detect the pattern at graph-build time and write the state straight into the cache buffer from the op itself (ggml_cuda_try_gdn_cache_fusion / ggml_sycl_try_gdn_cache_fusion).

Vulkan was the only major backend missing this, so every decode step runs an extra CPY per GDN layer (48 per token for Qwen3-Next / Qwen3.8-class models). This PR adds the same fusion to the Vulkan backend.

How

  • ggml_vk_try_gdn_cache_fusion() — detects a GATED_DELTA_NET immediately followed by a CPY of its state-snapshot tail into a recurrent-cache view, using the same structural checks as the CUDA/SYCL versions, and returns how many nodes to skip.
  • gated_delta_net.comp — gains a second storage buffer (binding = 7, CacheBuf) and a state_out_off push constant. When state_out_off > 0 the shader writes the state snapshots directly into the cache at that offset; when 0 it keeps the original behaviour (write into dst at s_off). A +1 sentinel keeps offset 0 unambiguous.
  • ggml_vk_gated_delta_net() — binds the cache buffer and sets the offset when the fusion fired; otherwise binds dst as a dummy and leaves state_out_off = 0 (non-fused path unchanged).
  • GGML_VK_DISABLE_FUSION=1 forces the non-fused path, mirroring GGML_CUDA_DISABLE_FUSION, for A/B verification and as a driver kill-switch.

The diff is 73 lines across the two files; the non-fused path is byte-for-byte unchanged when the fusion doesn't fire.

Verification status (why this is a draft)

  • ✅ Builds cleanly at master.
  • test-backend-ops -o GATED_DELTA_NET passes on the CPU backend.
  • ⚠️ The Vulkan-device A/B (fused vs GGML_VK_DISABLE_FUSION=1) has not yet been run on real hardware in CI — the CI runner exposes no Vulkan device (lavapipe did not initialise), so test-backend-ops fell back to CPU. I'm opening this as a draft to (a) get review of the approach, and (b) ask for help verifying on real GPUs. It was developed against a Radeon 890M (gfx1150 / RDNA3.5) but I'd like independent confirmation on other Vulkan drivers/vendors before it leaves draft.

Feedback on the shader binding approach and the sentinel scheme especially welcome.

Closes ggml-org#27193.

The CUDA and SYCL backends already fuse the GATED_DELTA_NET -> CPY
pattern: instead of writing the recurrent-state snapshots into dst and
then copying them into the recurrent (KV) cache with a separate CPY node,
they detect the pattern at graph-build time and write the state straight
into the cache buffer from the op itself (ggml_cuda_try_gdn_cache_fusion /
ggml_sycl_try_gdn_cache_fusion). Vulkan was the only major backend
missing this, so every decode step ran an extra CPY per GDN layer
(48 per token for Qwen3-Next / Qwen3.8).

This adds the same fusion to the Vulkan backend:

- ggml_vk_try_gdn_cache_fusion() detects a GATED_DELTA_NET immediately
  followed by a CPY of its state-snapshot tail into a recurrent-cache
  view (same checks as the CUDA/SYCL versions), and returns how many
  nodes to skip.
- The gated_delta_net shader gains a second storage buffer (binding 7,
  CacheBuf) and a state_out_off push constant. When state_out_off > 0
  the shader writes the state snapshots directly into the cache at that
  offset; when 0 it keeps the original behaviour (write into dst at
  s_off). A +1 sentinel is used so offset 0 is unambiguous.
- ggml_vk_gated_delta_net() binds the cache buffer and sets the offset
  when the fusion fired; otherwise it binds dst as a dummy and leaves
  state_out_off = 0 (non-fused path unchanged).
- GGML_VK_DISABLE_FUSION=1 forces the non-fused path, mirroring
  GGML_CUDA_DISABLE_FUSION, for A/B verification and as a driver
  kill-switch.

DRAFT — verification status:
- Builds cleanly at master (CI).
- test-backend-ops GATED_DELTA_NET passes on CPU (CI).
- Vulkan-device A/B (fused vs GGML_VK_DISABLE_FUSION=1) NOT yet run on
  real hardware — the CI runner has no Vulkan device. Requesting review
  of the approach and help verifying on real GPUs before this leaves
  draft. Developed/tested informally on a Radeon 890M (gfx1150).
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 29, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

Hi @PrajwalMukatti, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

  • AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

{
vk_gdn_fused_cache fc;
const int skip = ggml_vk_try_gdn_cache_fusion(cgraph, node_idx, fc);
if (skip > 0) ctx->num_additional_fused_ops = skip;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should do this in a way that's more consistent with the existing fusions.

dzannotti added a commit to dzannotti/strix-halo-llama.cpp that referenced this pull request Aug 30, 2026
  ggml-org#25144  speculative: fix MTP draft crash on vision inputs
  ggml-org#27879  Qwen4exp correctness fixes
  ggml-org#27897  speculative: fix combined draft-mtp + external draft (-md) init crash
  ggml-org#27973  vulkan: fuse GATED_DELTA_NET state write into recurrent cache

27836 (qwen4exp NextN/MTP draft head) is left out: it overlaps 27879 in
qwen4exp.cpp and cannot apply alongside it. 27879 is the correctness set, and we
already have a working MTP head, so it wins the conflict.

25144 and 27897 both cover configurations we run -- mmproj alongside MTP, and -md
with --spec-type draft-mtp. 27973 is the Gated DeltaNet path, 36 of Flash-Next's
48 layers.
@randomm

randomm commented Aug 30, 2026

Copy link
Copy Markdown

Real-hardware test on Strix Halo (AMD Ryzen AI Max+ 395, Radeon 8060S, gfx1151), since this branch specifically wants that.

Built from this PR vs a master baseline build, bench-sweep -r 3 (llama-bench), model Qwen3.8-Flash-Next UD-IQ3_XXS, -ub 2048 -fa 1 --no-mmap -ngl 999:

depth metric master this PR delta
0 pp 431.82 ± 2.30 422.01 ± 6.27 -2.3%
0 tg 30.54 ± 0.02 30.42 ± 0.11 -0.4%
4096 pp 377.07 ± 0.14 372.10 ± 4.20 -1.3%
4096 tg 24.19 ± 0.04 24.27 ± 0.06 +0.3%

No measurable win or loss here — all deltas are within noise on this quant/workload. Happy to test a different model/quant or config if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: vulkan: fuse GATED_DELTA_NET state write into recurrent cache (skip following CPY)

3 participants