llama : enable recurrent-state rollback for qwen3next / qwen4exp - #28120
Closed
vahpetr wants to merge 1 commit into
Closed
llama : enable recurrent-state rollback for qwen3next / qwen4exp#28120vahpetr wants to merge 1 commit into
vahpetr wants to merge 1 commit into
Conversation
qwen3next and qwen4exp are GDN-based hybrids with the same recurrent-state design as qwen35 / qwen35moe, which already support rollback. Adding them to llm_arch_supports_rs_rollback lets speculative decoding roll back the recurrent state natively (n_rs_seq widens the state to 1 + n_rs_seq groups) instead of falling back to SEQ_RM_TYPE_FULL, which serializes the whole recurrent state to host on every speculative round. Measured on AMD Strix Halo gfx1151 / Vulkan+RADV, Qwen3.8-Flash-Next Q4_K_M, -ctk q8_0 -ctv q8_0, temp 0, -np 1, MTP draft: no draft: 32.4 t/s MTP, SEQ_RM_TYPE_FULL: 6.2 t/s (host checkpoint every round, 5x loss) MTP, this change (RS): 43.6 t/s (+35% over no-draft; acceptance 0.89) Output stays greedy-equivalent to no-draft. Validated with -np 1 and short/medium generations; the widened-state rollback across deep rejections, long context and multi-slot could use a maintainer eye on the qwen4exp graph.
|
Hi @vahpetr, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
removed
the
draft
vahpetr
marked this pull request as ready for review
August 31, 2026 21:29
Member
|
See #28123 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
qwen3next and qwen4exp use the same GDN recurrent state as qwen35 / qwen35moe,
which are already in
llm_arch_supports_rs_rollback. They were just missing fromthat list, so speculative decoding fell back to
SEQ_RM_TYPE_FULLandcheckpointed the whole recurrent state to host on every round.
Adding the two arches lets the recurrent memory roll back natively (via
n_rs_seq), so no checkpointing is needed.Numbers on a Strix Halo (gfx1151, Vulkan/RADV), Qwen3.8-Flash-Next Q4_K_M,
temp 0,
-np 1, MTP draft head:Greedy output matches the no-draft run. This came out of the discussion in
#28118, where @ggerganov pointed out that rollback is the right fix instead of
speeding up the checkpoint.
I only tested
-np 1with short/medium generations, so an extra look at theqwen4exp graph for deep rejections, long context and multiple slots would be
good.