spec : add adaptive MTP draft depth (draft-mtp-adaptive) - #27210
Conversation
|
Hi! I've also created a fork with adaptive length for mtp and dflash with a rolling window based heuristic. https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window and this PR #25726 with the groundwork. Would you like to colab? |
I corrected a different MTP issue in this commit to my personal fork's working branch here: stew675@b2655d3 I really should file a PR for that, but its exposure range is pretty narrow. My fix targets which custom kernel runs for F32-activation small batches. It fixes a corner case for the handling of BF16 KV caches though (most of the work in my branch there targets adding BF16 KV to the ROCm backend), and my code exposed that latent bug. It appears there is a handful of inconsistencies in the mainline MTP code for certain corner cases and you have found another. I do see that your PR got closed. Did you ever work on a more targetted fix as per the reviewer's suggestion? |
Thank you for bringing your PR up. I searched for similar PR's and I don't know why yours didn't show up. I've looked over your branch and groundwork PR. There's a number of conceptual similarities, both being counting based heuristics. Both are sliding window approach, with yours being explicit, and mine being implicit. Both employ a form of "depth stickiness". Yours has a lot more knobs where I made a conscious effort to keep the knobs to a minimum. I do like your Bias knob though. That could be a genuinely useful knob for users to tweak for certain models to adjust the level of depth stickiness. The falling back approaches do differ significantly in concept. I spent quite a bit of time trying difference approaches, and then dialing in the one I settled on. Your personal branch seems to target DFlash too, right? I'm fine with us combining efforts, but we'd need to settle on what the UX approach should be. I'm a big fan of KISS. Years ago I would've had knobs for everything, but nowadays I'm a firm believer of use as few knobs as possible. Do you have any performance comparisons of your implementation and how it affects both prose and code? I tried pretty hard to ensure that prose performance was affected as little as possible, so I'm curious to see results on how you solved that issue. At the end of the day though, we both need one of the maintainers to offer some guidance on what they'd like to see. That will ultimately set the direction on what the next steps will be here. Edited to correct my earlier statement of both being implicit sliding windows. That was my mistake earlier. |
Chained MTP drafting (PR ggml-org#27173 backport): - All N draft tokens produced in one fused GPU decode via in-graph argmax - Deferred catch-up rows merged into first draft decode - --spec-chain N flag enables chain mode and sets depth (default: off) - New llama_set_mtp_chain() API for graph mode switching - Per-shape scheduler pool (LLAMA_SPEC_CHAIN env var still works) Adaptive MTP draft depth (PR ggml-org#27210 backport): - --spec-type draft-mtp-adaptive with hysteresis state machine - Depth climbs after consecutive full accepts, drops on misses - --spec-draft-n-min-adaptive for floor depth (default: 3) Results on RTX 5090, Qwen3.8-27B Q4_K_P: - Code: 206 t/s (chain n=8) vs 156 t/s (MTP n=3) vs 69 t/s (no MTP) - Chain delivers 3.0x over no-MTP, +31% over standard MTP on code Assisted-by: Claude
|
Numbers from our side supporting adaptive draft depth, measured on a Strix Halo APU (gfx1151, Ryzen AI Max+ 395) serving Qwen3.8-27B with draft-mtp + ngram-mod speculative decoding (datapost: #27154): At
The second lane replicates independently on a different Strix Halo system: acceptance "way below 50%" under xhigh-reasoning coding traffic, with n-max 12 regressing generation (peak ~26 t/s, dips into the ~5-6 t/s range) versus lower n-max values — that user is now moving down to n-max 2 (comment: #27154 (comment)). So a fixed n-max serves one lane and starves the other: deep enough for the ~0.96-acceptance lane, or shallow enough to cap wasted verification in the ~0.5-acceptance lane, but not both. Acceptance-driven draft depth is the knob that spans both, which is exactly what this PR adds. |
This comment was marked as abuse.
This comment was marked as abuse.
|
I went over all the check failures and it appears that they are all pre-existing/known flaky tests unrelated to my code changes. Now, I do have another commit that's almost ready which actually solves the 3% performance regression on hard prose, and actually boosts normal prose. The question I have is shall I wait for this PR to go ahead and file a followup, or shall I attach that change to this same PR. I'm fine either way. |
How does this compare with draft-mtp and spec-draft-p-min? I've been playing with pretty deep n-max values (8-16) and p-min>0.5 with good results. |
This comment was marked as low quality.
This comment was marked as low quality.
Yeah, I'll cover it in a followup PR. That will give me more time to refine it further. |
|
Getting this error with plain draft-mtp: It occurs whenever the effective n_max < 3 (e.g. --spec-draft-n-max 2). The range check added in |
Thank you for catching this. I'll be pushing a fix shortly. |
This comment was marked as low quality.
This comment was marked as low quality.
You've provided both a broad range (8-16) and an ambiguous range of p-min > 0.5 I'm guessing you meant --spec-draft-p-min, and not --min-p? Just to be clear, what's your ask here? For me to test all depths from 8-16 and vaguely a number of --spec-draft-p-min values as a matrix? Wouldn't that be highly dependent on what exactly it is you're doing? It's kind of the point of this PR that you don't need to be fiddling about trying to dial in the perfect depth and p-value each time. It's also been my experience that raising --spec-draft-p-min, whether it be normal or adaptive MTP, absolutely destroys prose performance. With the rise of deep thinking models (like our good friend Qwen3.8 here) there's likely to be significant amounts of time spent generating prose before code gets written. This is why I've been focusing on preventing/minimising harm to prose performance. We don't want trash that thinking performance just to get some more t/s on code generation and end up slower overall. |
No ask, just curious because I had your exact though this weekend "man, we need an adaptive n-max or something." and then found spec-draft-p-min and it gave me the sort of results I was looking for, with Qwen3.8.
Your PR is on my list to test. |
|
Tested the fixed-depth equivalent of this on the target hardware/backend this PR doesn't cover yet — Intel Arc Pro B70 (BMG G31, 32GB), llama.cpp SYCL via LocalAI
Draft acceptance 75–90% (mean accepted length 2.5–4.7) across runs. Two observations that support the adaptive approach directly:
Happy to run the adaptive build against the same 700-token workload on this B70 once it's testable and post the side-by-side. |
I ran my heuristic against static length in speedbench with qwen 3.6 35B and mtp roleplay category was used for prose *note: on my hardware (rtx 4080 super + rx 7900 xtx, both using vulkan backend), I noticed the maximum draft length that doesn't hurt my performance was 5, and the minimum draft length that didn't hurt it was 2. draft length 2 was used for prose, and draft length 5 was used for coding. got very similar results in both the non adaptive case and the adaptive case. Speedbench, qwen 3.6 35B UD-Q4_K_XL, using mtp: coding, draft length 5: 157.70 tps roleplay, draft length 2: 147.71 tps In my personal experience with my fork, the draft length of my heuristic usually stabilizes at 2 tokens for prose, Tell me if there are additional files/logs you would like to see.
Yes. By targeting Dflash I meant I copy pasted the code from my mtp implementation into Dflash - the heuristic works exactly the same in both cases. In the code, Because the draft length update is checked on every accept(), which every draft spec implementation has, it can absolutely be ported to draft-simple and draft-eagle3, I don't use them so I didn't apply the heuristic to them. I now also deduplicated code in my fork by using your approach of putting the heuristic in a header, and added a short explanation on the algorithm I used, so it should be easier to read and trivial to port to other draft implementations. I looked at your code, and it seems like adding the same heuristic you implemented in mtp to Dflash (and other spec types) should be simple, and I think adding your heuristic to Dflash as well is a good experiment because even with Dflash generating all tokens at once, the target model can still try to verify "useless tokens" (In my experiments, my heuristic worked well on Dflash)
I agree I that have too many knobs, and when I use my fork I always use --spec-adaptive-length-default (sets my knobs to params that worked well for me) However, I still think that 1-3 knobs of configuration is still good to have, to let people experiment. I think a good approach would be like with n-gram speculative decoding - configurable but includes a parameter that sets sane defaults (--spec-default). |
This comment was marked as low quality.
This comment was marked as low quality.
Ooh, good catch! Thank you for that. I see that another work-flow hasn't been approved yet, so now's a good time to merge that in. |
This comment was marked as low quality.
This comment was marked as low quality.
I did not test for this. Adaptive MTP more or less just sits on top of normal MTP. It pretty much just acts as a way to dynamically limit the maximum draft depth based upon prior acceptance behaviour, so I don't expect that it would behave any differently with respect to parallel sessions than regular MTP does today. |
|
I have some data to present to you. I started to try to implement your suggestion, but then I decided to stop and check the raw data on if there's any patterns to follow at all, and developed a "best choice that can be made Oracle" script that analyses historical runs and figures out what is the best depth to choose at any moment. The following is (hindsight derived) optimal draft depth for reasoning/prose from my tests: So, 1,2,8,6,6,1,1,8,1,1,8,8 and so on. It's basically just statistical white-noise. Choosing a fixed draft depth of 2 results in 50.0t/s and a fixed draft depth of 3 results in ~51.6 t/s (on my hardware. On different hardware a depth of 2 may win over 3). The Oracle predicted, even if we chose every draft depth perfectly with foresight, that the best Reasoning speed would be 57.2t/s (up from 51.6), however the data shows that there's absolutely no pattern or way to predict what depth to choose next. Also keep in mind that the predicted 57.2t/s does not take into account the performance hit of rapidly changing the draft depth, which would likely incur a 2-3t/s performance hit. The data is stating that the best that you can practically do is just pick the best floor for your hardware and stay there until enough patterns form to suggest it's safe to go higher and this is exactly what I found, which is to stay at 3 and only climb when something REALLY proves itself. While the average may be 2.57 or whatever, that single value doesn't tell the full story, which is the sheer range of values that makes up that 2.57. Where the chance for true wins exists is in the semi-predictable content like Coding, for which Adaptive MTP already tracks 5% higher than the best single fixed depth that can be chosen (which is 5 by the way), but as we all know, a fixed static depth of 5 kills reasoning and prose performance, and Adaptive will fall down to the floor to track the best prose depths. The Oracle predicts that the best t/s with the coding dataset that could be achieved with perfect foresight is 84.5 t/s, and adaptive hits ~78t/s. That last 6 or 7t/s is subject to the same issues that reasoning has in getting from 51 up to a perfect 57. That 6t/s gap is just random jittery unpredictable noise. It may be possible to eke out another 2t/s but it's more likely to be pure luck. At this stage, it seems to me that the only real way to do much better at all would be to create a better MTP draft model head. |
|
Follow-up to my benchmark above, this time a real multi-turn agentic session instead of synthetic filler. Same stack: Qwen3.8-27B UD-Q4_K_XL, q8_0 KV, adaptive 2-6, Intel SYCL on oneAPI 2026.1. The session ran tool calls, JSON, and prose, growing from empty context out past 100k tokens.
Acceptance tracks content, not depth. It holds the same 0.76-0.99 band from 40k all the way to 102k of context, only oscillating with what's being generated. Structured stretches (tool calls, JSON) sit at 0.88-0.99 with mean draft length 4-5; prose sits at roughly 0.78 with mean length about 3. On predictable output the controller climbs to about 4.7 depth with a 0.988 peak, and on prose it backs off to about 3.0. The whole-run cumulative is about 0.79 at about 3.0 depth. The adaptivity works in both directions. Decode: 30.9 t/s shallow, down to 13.5 t/s at 102k context. That run used temperature 0.95 and got attention-bound past about 60k, so I read acceptance as the signal here, not absolute throughput. The decode drop is the usual attention cost at depth. |
|
Thank you for the results there @bucknova Your results are basically affirming the implicit "sliding window" design of the algorithm which in practise rarely extends beyond 50 tokens of memory for prose, or beyond ~200 tokens for coding, even if that "memory" is an emergent property of the current depth's "window". Whatever happened over 200 tokens ago has almost zero impact on the now. The "Oracle algorithm" actually predicts that the memory should be even shorter than what it's currently set at, but then we start running into the hardware impact where too much churn is bad, so the window is larger than is algorithmically optimal, but no larger than it needs to be to remain balanced with what llama.cpp wants to do with the hardware. |
|
@stew675 Firstly, some very interesting discussions!. I’d push back on one specific claim: "Whatever happened over 200 tokens ago has almost zero impact on the now." I think there is an important distinction between the effective memory observed by the adaptive MTP controller and the actual dependency structure of the model’s conditional distribution. A trivial counterexample is a book where the current section is influenced directly by something introduced an arbitrary number of tokens earlier: a chapter title, a section heading, a list of constraints established at the beginning of a task, a variable definition, etc. Those tokens can be extremely sparse in the sequence while still having a profound influence on what the model should generate now (Dirac-like behavior). What your measurements show, I think, is narrower and more defensible: for the workloads tested, deeper speculative drafting based on longer history provides little incremental predictive value to the MTP verifier. That is a statement about the draft/verifier interaction, not about whether the target model’s probability distribution has negligible dependence beyond 200 tokens. MTP acceptance is not a direct impulse-response measurement of model memory. It conflates draft-target agreement, local predictability, entropy of the current generation, and the controller’s own hysteresis/window. So even if acceptance becomes insensitive to history beyond ~200 tokens, that does not mean the underlying language model has no meaningful dependencies beyond that distance. This closely matches something I’ve observed in a different problem: codec inversion. Temporal interaction structure there tends to have three regimes: Language models likely exhibit exactly this mixture: mostly local syntactic/semantic continuation, some exponentially decaying dependencies, and sparse high-information anchors that remain relevant for arbitrarily long distances. In other words, “most of the probability mass is local” does not imply “all meaningful dependencies are local.” So I think the result is stronger when phrased as: adaptive MTP learns an effective speculative memory horizon of ~50 tokens for prose and ~200 tokens for coding under these workloads. That’s a useful and interesting finding. I just wouldn’t take it as evidence that information older than 200 tokens has “almost zero impact” in general. Separately, there’s also the statistical robustness question. The current evidence is based on a single model (Qwen3.8-27B Q8_0), one hardware setup, one sampling configuration, and only 2 repeats per cell with mean tok/s reported. That’s useful for demonstrating the controller works on that setup, but it’s not strong enough to support a general claim about a 200-token memory horizon, nor to validate the Oracle analysis, which inherits the same limited data and lacks variance estimates. Single prompts per workload category, one backend, and no confidence intervals make the “200 tokens” more of a point estimate from one environment than a robust property. |
|
Thank you for the detailed response. You're drawing the correct distinction here and To clarify: my "~50–200 tokens" was about the controller's own state. The depth and The data supports your conflation point. The same model and backend show a floor of 3 Regarding the Oracle: it's an offline perfect-foresight replay for ranking controller With regards to controller memory though it predicts an even shorter algorithmic optimum. On the topic of robustness, I do agree. It's just one model, one backend, and one So your framing is correct. Adaptive MTP learns an effective speculative memory horizon |
|
It looks like the 5 failed checks are all known flakes independent of this PR's changes. I don't intend to work on this PR further. Barring blocking issues being found, if any followup work is needed they should be handled in follow-up PR's. |
|
Forced pushed in order to rebase against current master tip and resolve a handful of merge conflicts due to drift. |
Evaluate the algorithmic DFlash2 selector boundary without importing an untrained convolution layer. Prior-art: ggml-org/llama.cpp#27210; ggml-org/llama.cpp#27342 Signed-off-by: Codex <codex@openai.com> (fak model)
|
@am17an sorry to bug you again as I know the upstream developers are all busy working on exciting new model support, but I'd like to know if this PR here of any interest to the llama.cpp maintainers? I'm reaching out to you since you were responsible for most of the excellent MTP work that this is building upon. I'd just like to know if I should keep holding out hope for this PR, or if the maintainers have other plans instead? A good number of people have independently proven that this PR does exactly what it says on the tin, so to speak, across a number of architectures. It just seems like a free 10% boost and an improved user configuration experience is a lot to sleep on. |
|
I've been testing this Adaptive MTP implementation for a week, and the results are excellent. |
|
Long-context data point: on a single RTX 3090 (Qwen3.6-27B UD-Q3_K_XL, GPU KV q4_0, FA on) we compared fixed draft depths across 20K-120K context on a diverse corpus. n-max=4 beat n-max=3 at every level above 55K (+19% at 65K, +23% at 75K) and only lost at short context. So the drop-pressure side of the state machine should be conservative at long context: in our measurements the deeper draft keeps paying off as context grows. |
Hysteresis state machine with a climb counter and a weighted drop-pressure accumulator. The depth climbs one step after 5 consecutive verifies that accepted every drafted token; any miss adds (N - acceptance) to the drop pressure and the depth drops one step once it exceeds 30. High depths fall quickly (a total miss adds N), low depths hold, and at the floor no pressure accumulates at all. The floor is max(1, n_min) and the ceiling is n_max, so --spec-draft-n-min/--spec-draft-n-max bound the adaptive range and the cold-start depth is 3. Assisted-by: pi
Adaptive MTP starts at the floor of --spec-draft-n-min-adaptive (default 3) and adjusts its own depth: consecutive full accepts climb one step, with a cost table that rises fast to depth 3, blocks 3->4 (where marginal content collapses), and climbs fast at depth; a drop-pressure accumulator of n_draft - n_accepted, with a budget of max(depth * 5, 20), lowers it. Fully accepted but truncated drafts count as full accepts, and only drafts this implementation actually produced update the controller. The depth is independent of --spec-draft-n-min, which keeps its usual meaning of a minimum draft length to verify for the non-adaptive spec types. Extracted the controller into a standalone struct and added unit test cases for adaptive MTP. Assisted-by: pi
n_min_adaptive only applies to draft-mtp-adaptive, but the range check ran unconditionally in the shared MTP ctor, so plain draft-mtp aborted whenever the effective n_max was below the default floor of 3 (e.g. --spec-draft-n-max 2). Gate the check on adaptive mode, and when the chain_heads clamp capped n_max at the model MTP layer count, say so in the abort message. Assisted-by: pi
Assisted-by: pi
climb_threshold: the 3->4 barrier is hardened to 10 consecutive full accepts so prose/reasoning stay pinned at the floor; 4->5 raised to 6; 5->6 and 6->7 lowered to 3 and 2 so code accelerates to the deep hold without over-drafting into the marginal depths. recurrent snapshot fix: the conv-state loop wrote n_rs_seq + 1 copies per layer per round, but a rollback can only reach n_seq_tokens - 1 slots back, so copies beyond the batch were dead work (~2.6% per-round overhead at n_rs_seq=10 on shallow verifies). Start the loop at max(1, K - n_seq_tokens + 1). tests: move the #undef NDEBUG before <cassert> so the asserts actually run in Release builds (they were silent no-ops), and re-derive the climb expectations for the new table. Assisted-by: Pi
I rebased the code against the latest master tip and corrected a handful of outdated comments
b26c775 to
45e3d26
Compare
|
I merged PR #27210 into the latest master branch. Adaptive MTP works correctly on models like Gemma-4, but Qwen3.8-27B fails to load with an access violation crash. I'd like to report this issue. Build steps Configuration Error log when loading Qwen3.8-27B The crash occurs at ggml_mul_mat() + 0x6, which suggests a null or invalid tensor pointer is being passed — possibly the MTP tensor is not being resolved correctly during common_fit_params for this model architecture. |
|
Does this still happen with The backtrace indicates that this is happening even before the model weights have loaded. Another thing to try is to lower Basically the backtrace has failed at a point before the Adaptive MTP stuff even really kicks in. This is the pre-allocation part of the setup, so my immediate guess is a memory buffer overflow from a too large n-max value, or the good old: "You must use --fit off with MTP" issue. If it still persists after these changes then let me know. |
|
Thanks for the quick response. I tried both --fit on and lowering --spec-draft-n-max to 5, but unfortunately the crash persists with the same backtrace. Here's the error log: |
Qwen3.8-Flash-Next ships an MTP block in the checkpoint that the converter was dropping, so the model had no speculative path at all. Three pieces: - ggml-org#27836: the qwen4exp NextN/MTP draft head. Converter export (fc_embedding|fc_hidden fuse into the single eh_proj the shared NextN code expects), the nextn.hc_head_* tensors that stand in for the output norm qwen4exp does not have, and the LLM_GRAPH_TYPE_DECODER_MTP graph. Resolved against the fork's per-layer n_ff_exp accessor. - ggml-org#27210: adaptive draft depth, --spec-type draft-mtp-adaptive. Carries a delta-net fix that matters well beyond the adaptive path: build_conv_state was emitting a snapshot slot for every one of the n_rs_seq + 1 rollback depths, including the ones no rollback inside the batch can reach. Decode is one token, so all but one slot repeated the pre-batch state; the bound turns n_rs_seq into free headroom instead of a per-layer kernel-launch tax. - [fork] mtp_only/trunk_only probing in qwen4exp's load_arch_tensors, following the bailingmoe3/deepseek2 pattern. The draft head can now ship as its own GGUF: quantized apart from the trunk and pinned to the head GPU with --model-draft + --device-draft, rather than riding the trunk's tensor-split out onto the RPC fabric. Trunk-only tensors (hc_head_*, the PLE table, blk.0..n-1) become NOT_REQUIRED when the file has no blk.0, and the nextn block likewise when the file has no eh_proj. token_embd and output stay required in both halves, since qwen4exp sets mtp_use_dedicated_embeddings=false and the draft graph reuses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U3H5motr51eTWujztSXykc
|
Sorry, my mistake — I had --fit on when I tested earlier. With --fit off, the model loads without any issues. |
--spec-type draft-mtp-adaptive with a separate MTP-head GGUF segfaulted in ggml_is_empty before the model finished loading. common_init_result runs the --fit estimation pass (on by default) before the real load, and it fits the draft model alongside the main one. It decided whether the draft context is an MTP context by matching only COMMON_SPECULATIVE_TYPE_DRAFT_MTP, so the adaptive type left ctx_type at LLAMA_CONTEXT_TYPE_DEFAULT. The fit pass then built the *trunk* graph for a draft-only file, where the trunk tensors are deliberately absent, and build_hc_mix passed a null hc_attn_norm to ggml_mul -> ggml_can_repeat -> ggml_is_empty. The fault address is tensor->ne[0], and the crash lands before tensor loading, which is why the log tail is empty: the threaded logger never flushed. ggml-org#27210 added the adaptive type and updated four copies of this predicate but not the fifth. Collapse all of them onto one common_spec_has_mtp() so the next type cannot drift again. That also fixes a second live instance of the same gap: the mtp_dev pin in common_model_params_to_llama was skipped for adaptive, so an embedded-MTP adaptive run would silently lose its --device-draft placement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U3H5motr51eTWujztSXykc
All good. I'm just glad we got you sorted out in the end. |
|
Follow-up from our August 18 datapost — same Strix Halo box, but this time at the PR head (45e3d26) with the embedded-MTP draft and the adaptive controller, not ngram-mod. Qwen3.8-27B UD-Q4_K_XL from unsloth,
Code is a wash on this run — the edge adaptive shows there is smaller than the run-to-run spread we see with this setup, so I'm not claiming anything from it. Prose is where it shows: adaptive is ~6% faster than fixed-4 and accepts 6.5 points more of what it drafts. The fixed-12 row is the one I keep looking at. 13.2% acceptance means most of those deep drafts were verification work that went nowhere, and on this chip drafts share memory bandwidth with the target so that work isn't free. This is the lane I called out in the August post — deep drafting into low-acceptance traffic — showing up in one run. We ran Happy to share the raw per-request jsonl, or run whatever config anyone wants on this box. |
|
Been using --spec-type draft-mtp-adaptive --spec-draft-n-max 12 for about 2 weeks now with Qwen3.8-27B-UD-Q4_K_XL on the R9700 on Linux/ROCm. It's one of the few methods that speed up both coding and prose in a single setting without sacrificing either. Getting roughly 34 tok/s on prose, 72 tok/s on code. As a comparison, before that i used a static n-max of 3 as a middle ground which gave about also 34 tok/s for prose yet 53 tok/s for code. Would be great to see this land in main. @CISC @ggerganov |
17 participants
Overview
Implements adaptive mtp with a new option
--spec-type draft-mtp-adaptiveSuggested configuration to use is:
--spec-type draft-mtp-adaptive --spec-draft-n-max 12The algorithm is a counting based state machine with a climb counter and a weighted drop-pressure accumulator. The depth climbs one step after a number consecutive verifies that accepted every drafted token; any miss adds (N - acceptance) to the drop pressure and the depth drops one step once it exceeds a certain amount. High depths fall quickly (a total miss adds N), low depths hold, and at the floor no pressure accumulates at all. The specific climb difficulty and drop pressure values were empirically determined over a wide range of tests.
--spec-draft-n-maxbounds the upper adaptive range and both the floor and the cold-start depth default to 3. The default floor of 3 may be adjusted with--spec-draft-n-min-adaptive, although in testing 3 seems to be the best value to choose here.Overview of the climb-cost algorithm
It was experimentally determined that a draft MTP depth of 2 or 3 is close to optimal for reasoning and prose.
More typically an MTP depth of 3 was found to be optimal, and so this was chosen as the baseline default.
When the depth is less than three the algorithm allows for an easy transition from a depth of 1 to 2.
A slightly higher cost from 2 to 3 is used to minimise the depth oscillating between two and three.
To climb past a depth of three requires passing a hardened barrier. The hardened barrier is in place to
minimise easily reaching a depth of four which is known to be harmful to performance for reasoning and prose.
Beyond a depth of four the barrier to climb higher is gradually reduced.
This allows for predictive runs to quickly ascend as the admission rates indicate that this is useful.
Latest Testing and Performance Results
(Updated 23 Aug 2026)
Latest performance results can be seen here: #27210 (comment)
Extended discussion of results can be seen here: #27210 (comment)
Test Setup
Qwen3.8-27B-Q8_0.gguf9950X3D2, ROCm,
HIP_VISIBLE_DEVICES=0,2,GGML_CUDA_DISABLE_GRAPHS=0llama-serverfrom branchadaptive-mtp(build 10459, 0c6426b,freshly rebuilt)
ctx 262144, f16 KV,
--parallel 1, mlock, reasoning on(
--reasoning-budget 65536 --reasoning-preserve)timings.predicted_per_second(generation only, >= 4000 tokens per request)
Configurations Tested
--spec-draft-n-max 3--spec-draft-n-max 12 --spec-draft-p-min 0.75--spec-draft-n-min-adaptive 3 --spec-draft-n-max 12--spec-draft-p-min 0.75--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64Results
Additional information
The total diff size is +451/-26
This PR touches on 10 files which seems large at first, but this mostly arises from integrating the new command line options. The main functional changes take place within a single file (speculative.cpp).
218 lines are for the unit test file. 92 lines are for the header file that contains the algorithm itself.
The rest of the line changes are the wiring of the algorithm into the llama.cpp code base, and the command line option handling.
I tried to keep the size of the functional changes to llama.cpp itself as small as possible, and I tried to keep this diff as independent as I could so that unless it's specifically activated, it won't interfere with normal llama.cpp operations.
Related Issues/PRs
I searched known issues and open PRs and did not find anything precisely mentioning this sort of feature at first.
@Us5rName pointed out in the comments below that they have a similar feature PR here: https://github.com/Us5rName/llama.cpp/tree/draft-adaptive-length-rolling-window
Requirements