vulkan: top_k radix select for k >= 1024 for Qwen 3.8 Flash Next - #28032
Merged
Conversation
github-actions
Bot
added
testing
Contributor
|
I've also been poking around at a larger top_k today, I hadn't quite finished reviewing but I'll share it in a moment. Mine does a radix/histogram in global memory and tries to spread each row across multiple workgroups. |
CISC
approved these changes
Aug 30, 2026
jeffbolznv
approved these changes
Aug 30, 2026
| // Don't pull forward nodes from fusion patterns | ||
| // Protect every interior QSA node (not just the start): the mask branch is | ||
| // independent, so it gets pulled out and breaks keep_pattern otherwise. | ||
| auto const &in_qsa_pattern = [&](int n) -> bool { |
Contributor
There was a problem hiding this comment.
I had to have codex explain this to me, but it makes sense now.
0cc4m
changed the title
amangupta-tether
pushed a commit
to amangupta-tether/qvac-fabric-llm.cpp
that referenced
this pull request
Aug 31, 2026
…l-org#28032) * vulkan: add top-k radix sort shader for k >= 1024 * add Qwen 3.8 Flash Next top-k tests * add top-k qsa fusion * clean up code
4 tasks
wdenejko
added a commit
to wdenejko/llama.cpp
that referenced
this pull request
Aug 31, 2026
54 upstream commits since 1844325. Conflict resolutions keep the fork's validated mmid stack authoritative and graft upstream additions alongside: - mul_mat_id pc: union of our padded_N/use_row_lists/fusion_flags/tile_list_base and upstream's n_experts/hoist_row_ids (ggml-org#26686); mmq declares tile_list_base as layout filler so both shader blocks match the one host struct. - row-id loading: our row-lists prepass first, upstream's hoisted path second (their gate is n_as <= 256, so qwen4exp n_as=512 never hoists), ballot scan as the fallback; hoisting yields the prepass buffer to our row lists. - K-vs-N padding (ggml-org#27925): took upstream's y_staged_row_stride/K-pad scheme; padded_N stays as a pc-layout filler (no merged shader reads it). - Strix mat-vec tuning (ggml-org#27909): took upstream's rm_id/rm_int_n RDNA3 lambdas alongside our GGML_VK_MMV_RM_* sweep knobs. - kept our epilog-fusion ctx fields next to upstream's fused_topk_qsa (ggml-org#28032). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gagallo7
pushed a commit
to gagallo7/qvac-fabric-llm.cpp
that referenced
this pull request
Aug 31, 2026
…l-org#28032) * vulkan: add top-k radix sort shader for k >= 1024 * add Qwen 3.8 Flash Next top-k tests * add top-k qsa fusion * clean up code (cherry picked from commit 65c7c96)
Nathanw1014
pushed a commit
to Nathanw1014/llama.cpp
that referenced
this pull request
Aug 31, 2026
…l-org#28032) * vulkan: add top-k radix sort shader for k >= 1024 * add Qwen 3.8 Flash Next top-k tests * add top-k qsa fusion * clean up code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Qwen 3.8 Flash Next requires top-k with k 2048 and higher, which Vulkan's current shader does not support. Add a radix sort for top-k with k 1024 or higher.
I also fused the QSA indexer pattern the Qwen model uses into it, which gives a few percent more performance. Not sure if this is premature and the pattern will change. It adds a few percent performance on top.
Supersedes #28005, with significantly higher performance than using argsort here.
AMD 8060S (Strix Halo)
DGX Spark (GB10)
Requirements