qwen4exp : add NextN/MTP draft head (--spec-type draft-mtp) for Qwen3.8-Flash-Next - #27836
qwen4exp : add NextN/MTP draft head (--spec-type draft-mtp) for Qwen3.8-Flash-Next#27836rmonsurate wants to merge 3 commits into
Conversation
Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.
Adds --spec-type draft-mtp support for Qwen3.8-Flash-Next. The MTP head folds the next token's embedding into the trunk's wide hyper-connection residual, runs one trunk-style block (dense attention + MoE) over it, and collapses the result with its own mixer before reusing the trunk's LM head. - read nextn_predict_layers so n_layer() excludes the MTP block - load the trailing block through the existing trunk path: is_recr() and is_ple() are already false past the trunk, so it needs no special casing - eh_proj fuses the checkpoint's fc_embedding and fc_hidden side by side, so one matmul computes fc_embedding@e + fc_hidden@h - the head carries its own hyper-connection mixer, mirroring the trunk's hc_head_*, which stands in for the output norm qwen4exp does not have - export the wide pre-collapse residual as t_h_nextn from both graphs, so the driver can feed it back for the next draft step - route MTP contexts to a plain KV cache filtered to the trailing layer The draft block attends densely for now: the trunk's QSA only prunes context past a 2048-token budget, so dense is a numerical superset and drafts are verified either way. Indexer tensors are still loaded.
The MTP block is one trunk-shaped block (dense attention + MoE wrapped in hyper-connections) plus a head-level combiner, so once _QwenMtpMixin renames mtp.layers.0.* to the trailing block index its tensors ride the existing qwen4exp mappings unchanged. Two head-level pieces need handling: - fc_embedding and fc_hidden fuse into the eh_proj the shared NextN code expects, since W_e@e + W_h@h == [W_e|W_h] @ concat(e, h) - mtp.hyper_connection_mixer.* is the head's own copy of the trunk's hc_head_* output mixer, unindexed in the checkpoint and per-block in the GGUF compress_ratios is read with length block_count, so it gains a trailing 0 for the MTP block, which attends densely. --no-nextn drops the head; --mtp exports it on its own.
|
Hi @rmonsurate, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
removed
the
draft
AMD HIP validation + tuning report (2× Radeon AI PRO R9700, gfx1201, TheRock 7.14)Thanks for this PR — it fixes the exact decode crash we hit on the #27742 merge (the MTP works on AMD HIP. Load ( Notes for the converter / file formatThe only community export we could find ( Benchmarks (Q4_K_XL 5-shard, q8 KV, 16K ctx, temp 0, thinking off, 3 reps, harness-measured)
Happy to share the full per-rep matrix or the graft script if useful. Thanks again for the fix. |
|
I could not get this to work, so i asked qwen 3.8 to help me a bit. Now it is working for me, see https://github.com/freqmod/llama.cpp/tree/qwen4exp-mtp if you want to be inspired. i got it to patch the source code and the script to build the files so i don't have to download the whole model to convert it NB: this is posted here mostly if anyone tinkering wants to see and use it while the pr gets approved. It is not following the llama.cpp ai policy as it is mostly done by qwen 3.8 27t so it is probably not useful for merging into the review. I quickly skimmed trough the diff and it looks reasonable to me. also it works on my computer whatever that is worth. |
|
Can't get this to work either, it seems like the GGUFs do not come with MTP heads enabled or something, anybody knows which GGUF I should download to get this working ? |
Vulkan validation report (gfx1151 / Strix Halo, RADV) — works, but draft path is a net loss on this backendBuilt What works:
What doesn't (Vulkan-specific):
Question: is the draft context expected to checkpoint/reuse the recurrent state across draft steps on rejection, or is the full replay by design for hybrid archs? Happy to run additional Vulkan datapoints if useful. |
|
Would the work this guy is doing be of any interest https://github.com/Nathanw1014/llama.cpp/commits/strix-halo-vulkan It just works out of the box, I'm getting 21-30t/s with MTP on a EVO X2 Windows 11 128GB unified memory with Vulkan. Quite impressive as I wasn't expecting to get that performance at all. I am using unsloth/Qwen3.8-Flash-Next-UD-Q4_K_XL with a separate MTP model Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf I tried using this PR but couldn't even load the MTP draft |
|
This doesn't work for me, but the reason is trivial: it looks for blk.0, whereas all GGUFs on hf (which correctly follow convention) ship blk.38. This patch fixes it (cherry-pick this commit): crusaderky@a82a58a (beware: unreviewed AI slop). This branch + the above commit work for me with on CUDA RTX3090, using this presets.ini: [Qwen3.8-Flash-Next]
hf = unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS
ngl = 99
n-cpu-moe = 48
load-mode = mmap
jinja = true
ctx-size = 262144
flash-attn = on
kv-unified = true
cache-type-k = q8_0
cache-type-v = q8_0
# MTP drafter
spec-draft-hf = agentionai/Qwen3.8-Flash-Next-MTP-Q8_0-GGUF:Q8_0
spec-type = draft-mtp
spec-draft-ngl = 99
spec-draft-n-max = 3
spec-draft-p-min = 0.6
# Thinking mode
temperature = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0 |
|
Data point supporting your combiner warning, from Strix Halo (Ryzen AI Max+ 395 / gfx1151, ROCm 7.1): We tested an independent MTP port for qwen4exp (
Identical at Since that branch's issues are disabled, leaving the repro here as it seems relevant to "the combiner must be run per hyper-connection stream... If you do mean pooling first, the acceptance rate drops catastrophically." Happy to run this PR on gfx1151 (128 GB Strix Halo, UD-IQ4_XS) and report accept rates / tok/s / long-prompt correctness if useful — that hardware currently has no published numbers. |
|
Follow-up from the gfx1151 side: I tried to produce and run a standalone draft sidecar with this PR's Setup: fetched only the MTP-relevant tensors from Issue 1 — keep = name in (
"model.embed_tokens.weight", "model.norm.weight", "lm_head.weight",
"embed_tokens.weight", "norm.weight",
) or name.startswith("model.hyper_connection_mixer.")(The checkpoint names it Issue 2 — the exported sidecar can't be loaded standalone. The export writes
Question: what is the intended usage — was your M3 Max test run with the draft head still merged in the target gguf (self-speculative, no Happy to test any layout on Strix Halo — target UD-IQ4_XS, 128 GB, ROCm 7.1. The ngram-mod baseline here is 33 tok/s on file-edit workloads, so an MTP acceptance rate like your 89% would be a big deal on this hardware. |
unsloth ships this checkpoint at 83.8 gib (ud-q3_k_xl) / 87.2 gib (ud-iq4_xs) vs our 115.5 gib iq4_xs from the same f16. the gap is the 51b n-gram embedding table, which llama-quantize keeps at high precision. that, not the engine or the context size, is what made mtp not fit in 128g. correction posted to the engramhalo thread. also records that qwen4exp mtp now exists upstream (ggml-org/llama.cpp#27836, apepojken/llama.cpp@32af70900) and that our sidecar renumbering used the wrong contract.
|
Delivering the promised gfx1151/ROCm numbers — this PR + @crusaderky's loader fix works on Strix Halo, and it composes with ngram-mod and with the hipCUB TOP_K path. Setup: Ryzen AI Max+ 395 / Radeon 8060S, ROCm 7.1, target Correctness first: greedy output at 2.7k-token prompts is clean in every configuration (this is where the earlier independent MTP port degenerated into multilingual noise), and arithmetic/tool-calling/vision all check out at 24k-token context. Decode tok/s on real coding workloads (llama-server, greedy, 718-line Python file; A = rewrite file with small edit, B = targeted bugfix, C = write new code, D = prose):
Baseline without any speculation is 16.8 tok/s, so the full stack is 2.8x on file rewrites at 8k and +58% on code-writing at 24k. (hipCUB runs with B's variance across runs (24.7–40.2) tracks generation-content differences between builds at temp 0, not a regression — flagging it so nobody reads the last column pair as a CUB penalty. This configuration is now our production stack. Thanks @rmonsurate and @crusaderky — happy to run any follow-up variant on this hardware. |
|
Follow-up to my Vulkan report above — we've now run the same grafted model through HIP/ROCm 7.2.3 on the same gfx1151 box (containerized, MTP works great on ROCm (same spot where Vulkan collapsed):
Two operational notes for anyone reproducing on Strix Halo:
One honest caveat: in our end-to-end agentic workload (browser automation + tool calls, 97-99% prompt cache hits), total run time was statistically identical between ROCm+MTP and Vulkan no-spec at equal power limits — the LLM-side gains dilute into tool/navigation time. The +17-39% is real for LLM-bound workloads (batch, long prefills, sustained generation), which is where this PR shines. Thanks again for the head — acceptance numbers held up beautifully across two backends. |
|
For anyone wanting to run this without assembling the pieces by hand, the working combination is now published:
@gabrielfreire this combination is what got the draft loading on our end — the load failure you hit is the detached-head issue crusaderky's commit fixes. |
|
Follow-up datapoint from the Windows/CUDA side (RTX 3090 24 GB, 96 GB DDR4, The combiner fix is confirmed on CUDA. Draft acceptance: 0.84–0.91 on code, 0.75–0.82 on step-by-step reasoning, 0.62 on German prose — versus 0.33–0.40 we measured earlier on a mean-pooling-combiner fork (same GGUF, same prompts). Keeping the hyper-connection streams distinct is clearly the difference. But on a 24 GB card the speedup mostly evaporates: with It composes beautifully with timadinorth's MoE expert-residency split (timadinorth#1 — hot experts in VRAM instead of whole layers): with ~74% of routed traffic served from VRAM, the verify batches get cheap and the same box reaches 24.1–26.4 t/s on code (+38–52% over baseline) at 0.86–0.91 acceptance. Combined branch for reproduction: https://github.com/mjungnickel18/llama.cpp/tree/qwen4exp-mtp-plus-moe-residency (MTP branch + residency commit cherry-picked, flags in the branch's service script: Happy to re-test on this setup once the head lands here upstream. |
|
Adding some data for Apple Silicon. M5 Max 128GB, Unsloth UD-IQ4_XS with the standalone MTP head GGUF merged into the target's file set (nextn tensors as a 4th split, since current converts drop them). Prompts are C++ source-code continuation; prose lands a few t/s lower with the same ordering.
Short context: 41.6 t/s no spec → ~70 with One finding: combining with draftless ngram ( |
|
@sammcj thanks — ran your exact question as an A/B on Strix Halo (gfx1151/ROCm, UD-IQ4_XS, same coding workloads as before, greedy):
So your displacement finding partially replicates on ROCm: MTP-alone wins bugfix (+13%) and novel-code-at-depth (+13%) — the ngram matches do crowd out higher-acceptance MTP drafts there. But on echo-heavy file rewrites the combination still wins big here (45.4 vs 33.2), opposite of your Metal result. Plausibly backend-dependent: on this box ngram's long free drafts amortize better against the wide-residual verify cost than they do against Metal's. Your |
Squashed so the set reverts as one on rebase. Takes the line descending from the merged ggml-org#27742: ggml-org#27836 (NextN/MTP draft head) and ggml-org#27941 (danielhanchen's follow-up fixes), plus ggml-org#27977 and fifteen others. Drops ggml-org#27879 (third-party correctness fixes) -- it conflicts with ggml-org#27836 in qwen4exp.cpp, and MTP is not optional here. ggml-org#27836 supplies everything the MTP head needs: it reads nextn_predict_layers, loads the nextn block behind ml.load_mtp, and adds the MTP graph. Earlier local patches for those are dropped as redundant. Still ours: the two --ngram-on-disk bugs from halo-box#8, which live in code these PRs do not touch. ple_head_offsets was applied twice, and can_reuse dereferenced rows -- null whenever the table is on disk.
Squashed so the set reverts as one on rebase. Takes the line descending from the merged ggml-org#27742: ggml-org#27836 (NextN/MTP draft head) and ggml-org#27941 (danielhanchen's follow-up fixes), plus ggml-org#27977 and fifteen others. Drops ggml-org#27879 (third-party correctness fixes) -- it conflicts with ggml-org#27836 in qwen4exp.cpp, and MTP is not optional here. ggml-org#27836 supplies everything the MTP head needs: it reads nextn_predict_layers, loads the nextn block behind ml.load_mtp, and adds the MTP graph. Earlier local patches for those are dropped as redundant. Still ours: the two --ngram-on-disk bugs from halo-box#8, which live in code these PRs do not touch. ple_head_offsets was applied twice, and can_reuse dereferenced rows -- null whenever the table is on disk.
Squashed so the set reverts as one on rebase. Takes the line descending from the merged ggml-org#27742: ggml-org#27836 (NextN/MTP draft head) and ggml-org#27941 (danielhanchen's follow-up fixes), plus ggml-org#27977 and fifteen others. Drops ggml-org#27879 (third-party correctness fixes) -- it conflicts with ggml-org#27836 in qwen4exp.cpp, and MTP is not optional here. ggml-org#27836 supplies everything the MTP head needs: it reads nextn_predict_layers, loads the nextn block behind ml.load_mtp, and adds the MTP graph. Earlier local patches for those are dropped as redundant. Still ours: the two --ngram-on-disk bugs from halo-box#8, which live in code these PRs do not touch. ple_head_offsets was applied twice, and can_reuse dereferenced rows -- null whenever the table is on disk.
Squashed so the set reverts as one on rebase. Takes the line descending from the merged ggml-org#27742: ggml-org#27836 (NextN/MTP draft head) and ggml-org#27941 (danielhanchen's follow-up fixes), plus ggml-org#27977 and fifteen others. Drops ggml-org#27879 (third-party correctness fixes) -- it conflicts with ggml-org#27836 in qwen4exp.cpp, and MTP is not optional here. ggml-org#27836 supplies everything the MTP head needs: it reads nextn_predict_layers, loads the nextn block behind ml.load_mtp, and adds the MTP graph. Earlier local patches for those are dropped as redundant. Still ours: the two --ngram-on-disk bugs from halo-box#8, which live in code these PRs do not touch. ple_head_offsets was applied twice, and can_reuse dereferenced rows -- null whenever the table is on disk.
Squashed so the set reverts as one on rebase. Takes the line descending from the merged ggml-org#27742: ggml-org#27836 (NextN/MTP draft head) and ggml-org#27941 (danielhanchen's follow-up fixes), plus ggml-org#27977 and fifteen others. Drops ggml-org#27879 (third-party correctness fixes) -- it conflicts with ggml-org#27836 in qwen4exp.cpp, and MTP is not optional here. ggml-org#27836 supplies everything the MTP head needs: it reads nextn_predict_layers, loads the nextn block behind ml.load_mtp, and adds the MTP graph. Earlier local patches for those are dropped as redundant. Still ours: the two --ngram-on-disk bugs from halo-box#8, which live in code these PRs do not touch. ple_head_offsets was applied twice, and can_reuse dereferenced rows -- null whenever the table is on disk.
Apple Silicon M5 Pro validation report (Metal, 64 GB): acceptance matches, throughput regresses, small-batch kernel selection looks like the causeThanks for the PR. The draft head works on Metal and acceptance matches your numbers, but on this box MTP is a net loss at every draft depth I tried. I think I found where the time goes, details below. Setup
Results (median of 3)
Acceptance is right where your M3 Max numbers are, so the head and the graft path are fine. The loss is entirely in verification. Where the time goesPer-step decode cost, computed as (predicted_ms minus draft dur) / draft rounds, against the no-spec per-token cost, medians of 3:
Batch 3 costs 2.9x batch 1, so at that size essentially nothing is batching and the drafted tokens are pure overhead. The step from 3 to 4 costs only 17 ms, which made me look at the kernel selection on this branch:
So on this hardware the speedup seems gated on small-batch coverage (an Greedy identitydn=2 is token-identical to no-spec over the full 200-token run, in all 3 runs. dn=3 diverges mid-thought around generated char 130 ("Need brief explanation" vs "Need concise explanation") and stays diverged, and the divergence is deterministic: all 3 dn=3 runs produce the same alternate text, byte for byte. Acceptance still reads 100% per step in the logs. So on Metal the temp-0 identity you measured holds at dn=2 but not at dn=3 here, which lines up with the HIP report above seeing divergence too. The determinism makes me suspect verify-batch numerics rather than a race: logits computed in a batch can sum in a different order than the batch-1 baseline and flip a near-tie at argmax, but I have not confirmed that. Raw statistics lines from run 1: |
Port of ggml-org/llama.cpp#27836 (3 commits) plus crusaderky's detached -head loader fix. Also skips the MoE hot-expert validation for detached draft heads (no trunk exps tensors) and caps the MTP draft context batch/ubatch at 1024 so its compute reserve fits alongside the target. Measured on the dual-GPU CUDA0+ROCm0 rig with the agentionai Q8_0 sidecar: draft acceptance 77.5% (mean 2.55 tok) but decode is a net 34.9 vs 42.8 tok/s without speculation - the hybrid DeltaNet state rollback per rejected draft plus multi-token verify outweigh the saved passes when the target already decodes fast. Disabled unless --spec-type draft-mtp is passed; kept for future single-device or improved-state-handling scenarios. Assisted-by: Sisyphus
Port of ggml-org/llama.cpp#27836 (3 commits) plus crusaderky's detached -head loader fix. Also skips the MoE hot-expert validation for detached draft heads (no trunk exps tensors) and caps the MTP draft context batch/ubatch at 1024 so its compute reserve fits alongside the target. Measured on the dual-GPU CUDA0+ROCm0 rig with the agentionai Q8_0 sidecar: draft acceptance 77.5% (mean 2.55 tok) but decode is a net 34.9 vs 42.8 tok/s without speculation - the hybrid DeltaNet state rollback per rejected draft plus multi-token verify outweigh the saved passes when the target already decodes fast. Disabled unless --spec-type draft-mtp is passed; kept for future single-device or improved-state-handling scenarios. Assisted-by: Sisyphus
First vision-capable expert-offloaded MoE on this stack, and the fastest model on it (39.28 narr / 38.57 code canonical) — the reason is expert BYTE-SIZE (1,638,400 weights per ffn_up vs GLM's 8,388,608), not the quant. Composes: dual is the validated source; multi4/multi8 are derived by a 5-line delta (service key, container_name, port, device_ids, -ts) and are NEVER BOOTED — this rig has 2 GPUs, so those are community-validated by design, not a TODO. All three carry a TUNING block and mark every inherited 2-card constant as inherited. q8_0 KV (KV_TYPE knob): MEASURED KV -47%, expert-cache slots 9,435 -> 12,437 (+31.8%), decode neutral, VRAM balanced (22608/22640). Boots, serves, answers correctly. q8 is the serving-grade floor on this stack. Projector on the HOST (-mmdev none): on CUDA0 it cost 1,114 MiB and loads AFTER the pool is sized, so nothing budgets for it — 830 MiB headroom with a drafter active was a latent OOM on the first image request. Host placement costs ZERO cache slots (9,435 both ways) and GPU1 was byte-identical. Sampler: the card's two rows live in the registry as sampler_profiles and are DERIVED, not hardcoded. INSTRUCT ships as the default (matching qwen3.8-27b) because this template defaults enable_thinking=true AND reasoning_effort=xhigh. THINKING=1 selects the thinking row and --reasoning on. Validated on two real boots: default -> reasoning_content len 0; THINKING -> len 97. NO drafter machinery: the GGUF carries no MTP head (nextn_predict_layers absent, verified) and none exists for qwen4exp; engine support is an open draft PR (ggml-org/llama.cpp#27836). The vestigial -devd/SPEC_N block was removed rather than left inert. Catalog-count guards moved with the catalog (registry 103->106, composes 104->107, models 19->20, preflight slug->compose map). test-compose- registry-disk's message strings said 100/101 while asserting 103/104 — corrected. verify-full 10/10, bench-agentic 2x12 turns to 35,254 prompt tok with TTFT sub-linear (8.0x vs 25.1x context), vision 4/4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Validation report: 2× RTX A6000 48GB (CUDA, Windows, WDDM) — head works, acceptance matches, throughput regresses ~2.2×TL;DR: the draft head loads and drafts correctly on dual-GPU CUDA (acceptance 82–95%, matching the M3 Max numbers), but net throughput drops from ~44–46 t/s to ~19–22 t/s at every draft depth tried. Same direction as @ovidiu-morar's M5 Pro report; opposite of the unified-memory platforms. Detailed config and elimination steps below. Configuration
MethodA/B on the same build and machine: real coding prompt (write a complete Python module — thinking model, ~220 prompt tokens after warm cache), 300 completion tokens, warmup request on both arms before the measured run. MTP arm adds Results
What we ruled out
AnalysisWith 85% acceptance and ~3.4 tokens advanced per step, the theory says ~2–3× speedup; we observe ~0.47×. On this machine a single-token decode step (21.6 ms at 46 t/s) is mostly fixed cost — 48 layers × kernel-launch/graph overhead × a PCIe sync at each layer boundary across the two GPUs — while actual weight reads (~2.5 GB of active MoE params at Q3_K) account for only ~3–4 ms of it at 768 GB/s. The verify step inherits the full fixed cost while its weight-read amortization doesn't help on MoE (n+1 tokens routing to distinct experts multiply expert reads anyway), and the draft's autoregressive steps pay their own per-step fixed costs on top. WDDM's higher launch latency likely makes all of this worse than an equivalent Linux/TCC setup. Net: on platforms where decode is launch/sync-bound rather than bandwidth-bound, MTP's theoretical win is eaten twice — once by verify's fixed cost, once by the draft's own steps. That would explain the split between unified-memory reports (win) and this + the M5 Pro report (loss). Happy to re-run with a profiling build ( Verdict here: functionally correct, not worth enabling for throughput on dual-GPU CUDA + this quant today. Thanks for the PR, and to @drluoto for the packaged branch + verified sidecar — bring-up was genuinely smooth. |
|
Tried this on a DGX Spark (GB10, sm_121, 128GB unified) and hit a tensor-index mismatch with the community MTP draft head. Setup: merged this PR cleanly onto master 41ef91f (b10717), built with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121a. Draft head from dzannotti/Qwen3.8-Flash-Next-MTP-GGUF (Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf). Failure: llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.hc_attn_norm.weight' not found Reproduces with both unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS and argyelan/Qwen3.8-Flash-Next-GGUF IQ4XS-NGQ5 as the target model. Both targets load fine without -md. Cause: the draft head's hyper-connection tensors are present, but indexed at blk.48 — its absolute position in the full model — rather than blk.0: $ strings Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf | grep -i "hc_" | head The target model has the same tensors at blk.0, so the loader appears to be applying the target's indexing convention to the draft. Not sure whether the right fix is here or in the export — flagging in case the --mtp converter in this PR emits relative indices and third-party heads exported against #27739 use absolute. Command: LLAMA_ATTN_ROT_DISABLE=1 llama-server Worth cross-posting a short version to the dzannotti repo's discussions too, since the fix may belong there. |
Repin llama-cpp-qwen4exp to Nathanw1014/llama.cpp strix-halo-vulkan (ad914eb): PR-27742 arch support plus a working NextN/MTP draft path on Vulkan (upstream PR: ggml-org/llama.cpp#27836 is ROCm-first and loses throughput under RADV; this fork wins). Measured on mango (greedy, 32K ctx, UD-IQ4_XS + drluoto Q8_0 sidecar): file rewrite 22.0 -> 53.4 t/s, new code 27.5 -> 46.8, prose +12%, 7K-prompt +39%; temp-1.0 sampling holds up, greedy output is prefix-identical to no-spec. n-max 6 beat 3; adding ngram-mod displaced higher-acceptance MTP drafts, so MTP runs alone. The sidecar ships the head mixer under both PR-27836 names (output_hc_*) and legacy names (blk.48.nextn.hc_head_*); the fork loader rejects the 3 unconsumed duplicates, so the deployed file is a gguf-py copy with them stripped (see Zulip: #topic-ai > MTP for Qwen on FW desktop). At 262K the sidecar overcommits GTT: RADV fails command submission and can wedge llama-server in unkillable D-state (amdgpu/TTM livelock). MTP therefore only runs on the default entry at 131K; the new :iq4_xs-262k entry keeps the full native context without MTP.
|
Data point that may be directly useful here: the ~4.7 tok/s at 90% acceptance reported on this PR looks like it may not be the draft graph at all. I built this PR (plus @crusaderky's detached-head loader fix) on an AMD Strix Halo (gfx1151), Vulkan/RADV, with Qwen3.8-Flash-Next UD-Q4_K_XL and reproduced the profile almost exactly: 4.7 tok/s against a 25.1 baseline at ~90% draft acceptance — 4x slower than no speculation. Matches @flobob45's report above. Separately I ported @Nathanw1014's MTP implementation onto current master (#28104) and initially got the same profile: 4.77 tok/s, 70-80% acceptance. Two different graphs, same slowdown, which suggested the cost was somewhere shared rather than in either implementation. It was. qwen4exp is a recurrent hybrid, so the target context cannot partially The cost is flat in context and dominates everything else — about 600 ms of each 825 ms round, i.e. 201 ms/token decode. It also inflates the reported prompt eval time, which runs to first token and so absorbs the first save: I was seeing 827 ms of "prefill" for an 11-token prompt. Requesting
Acceptance unchanged at 70-80%. The flag and mechanism are Gaetan Puleo's ( Worth testing on this branch — it is a six-line server change, independent of the graph work here, and if it is the same bottleneck it would move this PR from "4x slower" to a substantial win without touching the implementation. Two other things from building this branch, in case they save anyone time:
Environment for all numbers: gfx1151 / RADV, 128 GB unified, |
Correction to my report above: with draft gating, one configuration is slightly net-positive on M5 ProI wrote "MTP is a net loss at every draft depth I tried". That was true of the flat settings I tried, and it does not survive the tuning suggested later in this thread. Retested with @sammcj's flags on the same box, model and prompt as my report (M5 Pro 64 GB, Metal, UD-IQ3_XXS grafted,
Two observations:
So the corrected summary for M5 Pro-class machines: slightly net-positive is achievable with gating at n-max 3, and the temp-0 divergence I reported for n-max ≥3 is still present in every speculative configuration tested, including these. Whether +5% is worth a non-identical greedy stream is a per-use call; for my use it is not. |
|
@JayToltTech tested your six-line
All within noise, generated text byte-identical per arm across the two builds. I verified the flag is not a no-op here (the So the two datasets compose rather than conflict: on unified memory the host-path state save is close to free and the remaining cost is per-token verify work (small-batch kernel coverage, per my report); on backends with expensive readback your checkpoint path dominates everything and the six-line change is transformative. Worth keeping both in mind when triaging the platform table on this thread, since the same symptom (fixed per-round cost, flat in draft width) has two different causes on different backends. |
|
any ETA on when will this makes its way to master? |
CUDA/Windows datapoint for the
|
| decode, tok/s | no spec | MTP, before | MTP, after |
|---|---|---|---|
| code gen (400 tok) | 17.0 | 11.0 | 17.7 |
| 20k-context summary (200 tok) | 15.1 | 14.0 | 15.4 |
Draft acceptance is unchanged by the flag (0.84 code / 0.79 long), as expected.
Two things worth separating out:
1. The fix reproduces on CUDA, clearly. +61% on the short-context code arm. So the host round-trip is real on this backend too, and this is now Vulkan ✅ / CUDA ✅ / Metal ❌ (where @ovidiu-morar's ~112 ms round leaves no room for it) — consistent with it being a host-transfer cost that only shows up where the transfer is expensive.
2. But the "4x slowdown" does not reproduce here — we saw 17.0 → 11.0, a 35% loss, not 4x. Worth flagging because it may narrow the mechanism: with --n-cpu-moe 40 this box only keeps 8 of 48 expert layers on the GPU, so a decode step is dominated by host-side expert matmuls. Our whole speculative round is ~450 ms, i.e. there is no room for a 600 ms checkpoint — the checkpoint we are paying must be substantially cheaper than the one you measured on gfx1151. If the cost scales with device-resident state, an offloaded target would naturally see less of it. That would also explain why @llbzow's 2x A6000 (96 GB, nothing offloaded) is at the severe end of the range while we are at the mild end, both on CUDA/WDDM.
Net effect on this box: MTP goes from "not worth enabling" (11.0 vs 17.0 plain) to "roughly break-even, slightly ahead" (17.7 vs 17.0). Modest, but the right side of zero, and the long-context arm gains too.
For completeness, on the same hardware we get a real MTP win by attacking the verify cost instead: pairing the draft head with the hot/cold expert split from timadinorth/llama.cpp#1 puts ~74% of routed expert traffic back in VRAM and lands 24.1-26.4 tok/s on code at 0.86-0.91 acceptance (vs 17.4-17.5 unsplit, no spec). The two are complementary: this flag removes the per-round host checkpoint, residency makes the verify batch itself cheap. Combined branch: https://github.com/mjungnickel18/llama.cpp/tree/qwen4exp-mtp-plus-moe-residency
One note for anyone reproducing on this PR: sidecars are not interchangeable. A head exported by apepojken's converter fails here with done_getting_tensors: wrong number of tensors; expected 35, got 30 — same class of failure as @tbtlr's DGX Spark report. The dzannotti Q4_K_M head loads cleanly.
|
ON_DEVICE change is on my fork: JayToltTech#1. Six lines in server-context.cpp on current master. Three backends now. Vulkan gfx1151: 4.33 → 16.08 tok/s at 70k. @mjungnickel18 CUDA: +61%. @ovidiu-morar Metal: neutral, flag verified not a no-op. Win where per-round host round-trips are expensive, neutral where they aren't, no regression anywhere tested. Not submitting it upstream myself. Anyone who wants it, take it as your own — no credit needed. |
|
Cherry-picked the three commits here onto current master ( MTP now works on current master, but it needed one fix that lives outside this PR — flagging it here since the rebased version will likely hit the same thing. Root cause of the
|
Qwen3.8-Flash-Next ships an MTP block in the checkpoint that the converter was dropping, so the model had no speculative path at all. Three pieces: - ggml-org#27836: the qwen4exp NextN/MTP draft head. Converter export (fc_embedding|fc_hidden fuse into the single eh_proj the shared NextN code expects), the nextn.hc_head_* tensors that stand in for the output norm qwen4exp does not have, and the LLM_GRAPH_TYPE_DECODER_MTP graph. Resolved against the fork's per-layer n_ff_exp accessor. - ggml-org#27210: adaptive draft depth, --spec-type draft-mtp-adaptive. Carries a delta-net fix that matters well beyond the adaptive path: build_conv_state was emitting a snapshot slot for every one of the n_rs_seq + 1 rollback depths, including the ones no rollback inside the batch can reach. Decode is one token, so all but one slot repeated the pre-batch state; the bound turns n_rs_seq into free headroom instead of a per-layer kernel-launch tax. - [fork] mtp_only/trunk_only probing in qwen4exp's load_arch_tensors, following the bailingmoe3/deepseek2 pattern. The draft head can now ship as its own GGUF: quantized apart from the trunk and pinned to the head GPU with --model-draft + --device-draft, rather than riding the trunk's tensor-split out onto the RPC fabric. Trunk-only tensors (hc_head_*, the PLE table, blk.0..n-1) become NOT_REQUIRED when the file has no blk.0, and the nextn block likewise when the file has no eh_proj. token_embd and output stay required in both halves, since qwen4exp sets mtp_use_dedicated_embeddings=false and the draft graph reuses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U3H5motr51eTWujztSXykc
17 participants
Overview
Adds the NextN/MTP draft head for qwen4exp. Tested in the Fabric harness using Qwen3.8-Flash-Next. This is a follow-up PR to #27742.
The combiner must be run per hyper-connection stream on the wide hidden state. If you do mean pooling first, the acceptance rate drops catastrophically — other people might hit this, so just leaving this here.
Converter support is included (
--mtp).Additional information
Measured on M3 Max 128GB, UD-IQ4_XS (temp-0 output byte-identical with MTP on/off):
--spec-draft-n-max 2--spec-draft-n-max 3Prior art: #27739.
Requirements