Eval bug: Severe speed drop from 50 t/s to 30 t/s in recent updates on RTX 5060 Ti

[ethernidee]

Name and Version

beellama v0.4.4
commit 287da7c (HEAD -> v0.4.4, origin/v0.4.4)
Author: Anbeeld anbeeld@gmail.com
Date: Fri Aug 28 01:25:13 2026 +0200
Update CHANGELOG.md

Operating systems

Windows

GGML backends

CUDA

Hardware

Ryzen 7 5700X + RTX 5060 Ti 16GB + 128 GB RAM

Models

Qwen3.8-27B-UD-Q3_K_XL.gguf by unsloth, tried IQ4 and mixed quants too.

Problem description & steps to reproduce

I had 50-55 t/s MTP speed for Qwen 3.8 27B. At least on earlier 0.4.3 and 0.4.4 version. After recent update I got 30-33 t/s both with MTP and dflash2, with kvarn6, kvarn4, kvarn8, q6_0 KV cache.

e:\beellama\build\bin\Release\llama-server.exe ^
  -m "e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf" ^
  --alias "Qwen3.8 27B MTP" ^
  --host 127.0.0.1 --port 8001 ^
  --n-gpu-layers 999 ^
  --gpu-layers-draft 999 ^
  --ctx-size 60000 ^
  --fit off ^
  --min-p 0.0 ^
  --temp 0.6 --top-p 0.95 --top-k 20 ^
  --repeat-penalty 1.03 ^
  -ctk q6_0 -ctv q6_0 --kv-tail-tokens 1024 ^
  --flash-attn on ^
  --batch-size 128 --ubatch-size 128 ^
  --threads 8 --threads-batch 8 ^
  --no-mmap --mlock ^
  --spec-type draft-mtp --spec-draft-n-max 3 ^
  --parallel 1 ^
  --log-colors off ^
  --kv-unified ^
  --metrics ^
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
  --jinja

Built via:

cmake .. ^
  -DCMAKE_BUILD_TYPE=Release ^
  -DGGML_CUDA=ON ^
  -DGGML_NATIVE=ON ^
  -DGGML_CUDA_GRAPHS=ON ^
  -DGGML_CUDA_F16=ON ^
  -DGGML_CUDA_NO_PEER_COPY=ON ^
  -DGGML_CUDA_FP4=ON ^
  -DGGML_CUDA_FA_ALL_QUANTS=ON ^
  -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build . --config Release --target llama-server llama-quantize llama-bench llama-perplexity --parallel

First Bad Commit

No response

Relevant log output

Logs

e:\llama.cpp\build\bin\Release>e:\beellama\build\bin\Release\llama-server.exe   -m "e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf"   --alias "Qwen3.8 27B MTP"   --host 127.0.0.1 --port 8001   --n-gpu-layers 999   --gpu-layers-draft 999   --ctx-size 60000   --fit off   --min-p 0.0   --temp 0.6 --top-p 0.95 --top-k 20   --repeat-penalty 1.03   -ctk q6_0 -ctv q6_0 --kv-tail-tokens 1024   --flash-attn on   --batch-size 128 --ubatch-size 128   --threads 8 --threads-batch 8   --no-mmap --mlock   --spec-type draft-mtp --spec-draft-n-max 3   --parallel 1   --log-colors off   --kv-unified   --metrics   --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"   --jinja
�[34m0.00.034.806�[0m �[35mW DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead
�[0m�[34m0.00.034.812�[0m �[35mW DEPRECATED: --mlock is deprecated. use --load-mode mlock instead
�[0m0.00.035.438 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.035.672 W srv  llama_server: -----------------
0.00.035.676 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.035.681 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.035.683 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.035.684 W srv  llama_server: -----------------
0.00.044.843 I srv    load_model: loading model 'e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf'
0.05.230.187 I cmn          init: llama threadpool init, n_threads = 8
0.05.334.566 I common_speculative_init_result: creating MTP draft context against the target model 'e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf'
0.05.345.758 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 60160, kv_unified = 'true'
0.05.352.932 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.05.352.946 I srv  llama_server: model loaded
0.05.352.951 I srv  llama_server: listening on http://127.0.0.1:8001
0.16.253.613 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.16.253.729 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.20.303.828 I slot print_timing: id  0 | task 0 | n_gen =    103, tg =  33.98 t/s, tg_3s =  34.31 t/s
0.23.372.525 I slot print_timing: id  0 | task 0 | n_gen =    202, tg =  33.11 t/s, tg_3s =  32.26 t/s
0.26.438.205 I slot print_timing: id  0 | task 0 | n_gen =    305, tg =  33.27 t/s, tg_3s =  33.60 t/s
0.29.485.206 I slot print_timing: id  0 | task 0 | n_gen =    405, tg =  33.16 t/s, tg_3s =  32.82 t/s
0.32.531.383 I slot print_timing: id  0 | task 0 | n_gen =    506, tg =  33.16 t/s, tg_3s =  33.16 t/s
0.35.558.491 I slot print_timing: id  0 | task 0 | n_gen =    598, tg =  32.70 t/s, tg_3s =  30.39 t/s
0.38.616.017 I slot print_timing: id  0 | task 0 | n_gen =    691, tg =  32.37 t/s, tg_3s =  30.42 t/s
0.41.670.634 I slot print_timing: id  0 | task 0 | n_gen =    800, tg =  32.79 t/s, tg_3s =  35.68 t/s
0.44.699.564 I slot print_timing: id  0 | task 0 | n_gen =    901, tg =  32.85 t/s, tg_3s =  33.35 t/s
0.47.768.127 I slot print_timing: id  0 | task 0 | n_gen =    989, tg =  32.43 t/s, tg_3s =  28.68 t/s
0.50.821.897 I slot print_timing: id  0 | task 0 | n_gen =   1082, tg =  32.25 t/s, tg_3s =  30.45 t/s
0.53.831.208 I slot print_timing: id  0 | task 0 | n_gen =   1165, tg =  31.87 t/s, tg_3s =  27.58 t/s
0.56.865.386 I slot print_timing: id  0 | task 0 | n_gen =   1247, tg =  31.49 t/s, tg_3s =  27.03 t/s
0.59.913.022 I slot print_timing: id  0 | task 0 | n_gen =   1362, tg =  31.94 t/s, tg_3s =  37.73 t/s
1.02.936.742 I slot print_timing: id  0 | task 0 | n_gen =   1455, tg =  31.86 t/s, tg_3s =  30.76 t/s
1.05.999.930 I slot print_timing: id  0 | task 0 | n_gen =   1575, tg =  32.32 t/s, tg_3s =  39.17 t/s
1.09.055.718 I slot print_timing: id  0 | task 0 | n_gen =   1697, tg =  32.77 t/s, tg_3s =  39.92 t/s
1.12.060.849 I slot print_timing: id  0 | task 0 | n_gen =   1833, tg =  33.46 t/s, tg_3s =  45.26 t/s
1.12.850.473 W srv          stop: cancel task, id_task = 0
1.12.914.335 I slot      release: id  0 | task 0 | stop processing: n_tokens = 2263, truncated = 0

[Anbeeld]

Thanks, this looks like a real performance regression.

I found a potentially related upstream report under the DFlash2 PR: on an RTX 5080, MTP decode dropped from 183.7 to ~160 t/s with essentially unchanged acceptance. They traced it to changes on the DFlash2 branch itself, not an upstream rebase, and already ruled out the new top-k kernel, draft sampling, "p_min" behavior, and CUDA graphs.

Could you A/B these official upstream Windows CUDA 13.3 builds with the same target model, prompt/workload and MTP settings? Use standard upstream KV since KVarN/tail flags don't exist there.

• b10644 — earlier pre-merge control:
  https://github.com/ggml-org/llama.cpp/releases/download/b10644/llama-b10644-bin-win-cuda-13.3-x64.zip

• b10657 — immediately before DFlash2 merge:
  https://github.com/ggml-org/llama.cpp/releases/download/b10657/llama-b10657-bin-win-cuda-13.3-x64.zip

• b10658 — first build containing DFlash2:
  https://github.com/ggml-org/llama.cpp/releases/download/b10658/llama-b10658-bin-win-cuda-13.3-x64.zip

Please post generation t/s + acceptance for each.

This should narrow it down pretty cleanly:

• b10644 fast, b10657 slow → regression is in upstream changes before the DFlash2 merge.
• b10657 fast, b10658 slow → very likely the same DFlash2-branch/shared speculative regression.
• all three upstream builds fast → likely BeeLlama-specific integration or KV precision-tail path.

Current hypothesis is a shared graph/backend execution regression. The upstream Blackwell report is especially suspicious because it affects MTP with unchanged acceptance, although their measured regression was smaller and NVFP4-specific, so it may only be part of what we're seeing here.

Upstream investigation: ggml-org#27342 (comment)

[Anbeeld]

One more useful test: please compare the release build against a local MSVC + CUDA build from the exact same BeeLlama commit.

There is a related upstream Blackwell issue where the official Windows CUDA binaries were much slower, but rebuilding the same/newer source with MSVC + CUDA completely removed the regression: ggml-org#26918 (comment)

BeeLlama's current Windows release pipeline follows a similar split to upstream: the Windows executable/CPU side is built with the LLVM toolchain, while "ggml-cuda.dll" is built separately with CUDA.

So if possible, please test:

• current "v0.4.4" release binary
• the exact same "v0.4.4" commit built locally with normal MSVC + CUDA

Same model, prompt and flags.

If the local MSVC build restores the old ~50 t/s performance, that would point at the Windows release toolchain rather than the DFlash2/KV changes themselves.

[ethernidee]

Ok, thanks, I'm trying different configurations currently.

[Anbeeld]

Also found a much closer upstream match to your report:

ggml-org#25489 (comment)

That user is also on an RTX 5060 Ti 16 GB and reports that current llama.cpp MTP performance drops hard once context reaches ~60k+, averaging around 30 t/s. Rebuilding an older upstream commit (8a091c47abe67e0a03b85bc7c9eee8bdb9b14b05) restores ~41 t/s average and much more stable performance.

Importantly, these are locally built binaries, so this regression exists independently of the Windows release build/toolchain issue mentioned above.

Could you also try building upstream commit: 8a091c47abe67e0a03b85bc7c9eee8bdb9b14b05 with MSVC + CUDA and run the same ~60k-context MTP workload?

If that restores the old performance on your 5060 Ti too, this is very likely the same upstream regression and gives us a much better range to bisect.

[ethernidee]

By the way, it seemed to be a bug in upstream DFlash2 implementation, causing hard speed drop when using tool calls.
There is a fork with bugfix: https://github.com/HermiG/llama.cpp/tree/fix/dflash2-tool-tg-collapse

fix: avoid DFlash2 grammar scan collapse with tool calls

common : add --spec-draft-ubatch-size argument to control block draft…

[Anbeeld]

By the way, it seemed to be a bug in upstream DFlash2 implementation, causing hard speed drop when using tool calls. There is a fork with bugfix: https://github.com/HermiG/llama.cpp/tree/fix/dflash2-tool-tg-collapse

fix: avoid DFlash2 grammar scan collapse with tool calls

common : add --spec-draft-ubatch-size argument to control block draft…

Have you tried it, does it resolve the issue for you?

[Anbeeld]

One clarification after looking into those commits more closely: that HermiG branch is based on an earlier DFlash2 development path, not the final DFlash2 implementation that was merged upstream and then into BeeLlama.

So I would not cherry-pick those commits directly yet.

What may still be very relevant is the trigger they identified: tool calls / active grammar causing a large TG collapse.

[ethernidee]

Sorry, seems I was wrong making short tests in a hurry. The speed starts at 30-35 t/s, but later growth on non-thinking and approaches 47 t/s.
I made multiple tests with upstream llama and latest beellama, everything rebuilt from scratch with CUDA_ARCHITECTURE=120. So the issue may be closed, I suppose.
Here are the results:

SYSTEM PROMPT

Write a single-file HTML/JS/CSS Windows-style desktop application with six apps, a start menu, a task manager, and a console.
No external dependencies. The whole desktop and each application must be ui friendly ,beautiful, fully functional,
meaningful and useful, including all tabls and settings.

------------------------------

llama-server-upstream.exe ^
  -m "e:\LMStudio_Models\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf" ^
  --alias "Qwen3.8 27B MTP" ^
  --host 127.0.0.1 --port 8001 ^
  --n-gpu-layers 999 ^
  --ctx-size 60000 ^
  --fit off ^
  --min-p 0.0 ^
  --temp 0.6 --top-p 0.95 --top-k 20 ^
  --repeat-penalty 1.03 ^
  -ctk q4_1 -ctv q4_1 ^
  --flash-attn on ^
  --batch-size 128 --ubatch-size 128 ^
  --threads 8 --threads-batch 8 ^
  --no-mmap --mlock ^
  --spec-type draft-mtp --spec-draft-n-max 3 ^
  --parallel 1 ^
  --log-colors off ^
  --kv-unified ^
  --metrics ^
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
  --jinja

17571 tokens ; 6min 10s ; 47.45 t/s ; draft acceptance = 0.70436 ( 9175 accepted / 13026 generated), mean len =  3.11

------------------------------

[NON-MTP] beellama-server.exe ^
  -ctk q4_1 -ctv q4_1
  ...

17002 tokens; 11min 15s ; 25.17 t/s (note, this is NON-MTP run)

------------------------------

beellama-server.exe ^
  -ctk q4_1 -ctv q4_1
  ...

17452 tokens; 5.92 min ; 49.13 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn8 -ctv kvarn6 --kv-tail-tokens 1024 ^
  ...

19453 tokens ; 7min 24s ; 43.80 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn6 -ctv kvarn4 --kv-tail-tokens 1024 
  ...

19302 tokens ; 8min 25s ; 38.21 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn6 -ctv kvarn4 --kv-tail-tokens 1024 
  ...

16410 tokens; 6min 10s ; 44.34 t/s

------------------------------

beellama-server.exe ^
  -ctk kvarn4 -ctv kvarn4 --kv-tail-tokens 1024 ^
  ...

15963 tokens ; 5min 59s ; 44.37 t/s

------------------------------

beellama-server.exe ^
  -md "Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf" ^
  --spec-type draft-dflash --spec-draft-n-max 5 ^
  -ctk q4_1 -ctv q4_1 --kv-tail-tokens 1024 ^

15789 tokens; 5min 30s; 47.77 t/s ; draft acceptance = 0.48516 (11178 accepted / 23040 generated), mean len =  3.43

------------------------------

beellama-server.exe ^
  -md "Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf" ^
  --spec-type draft-dflash --spec-draft-n-max 4 ^
  -ctk q4_1 -ctv q4_1 --kv-tail-tokens 1024 ^

14121 tokens; 5min 4s; 46.30 t/s ; draft acceptance = 0.57774 ( 9854 accepted / 17056 generated), mean len =  3.31

------------------------------

beellama-server.exe ^
  -md "Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf" ^
  --spec-type draft-dflash --spec-draft-n-max 6 ^
  -ctk q4_1 -ctv q4_1 --kv-tail-tokens 1024 ^

15315 tokens; 5min 32s; 46.06 t/s ; draft acceptance = 0.40785 (10870 accepted / 26652 generated), mean len =  3.45

[Anbeeld]

No problem, thank you for keeping an eye on Bee!