Gemma4 Assistant Draft Model Support

[lyrakat]

koboldcpp fails to load draft models for Gemma4

Attempting to load draft model for speculative decoding. It will be fully offloaded if possible. Vocab must match the main model.
llama_model_loader: loaded meta data with 50 key-value pairs and 49 tensors from C:/Users/kat/AI/Models/Gemma4/gemma-4-26b-A4B-it-assistant-QAT-Q4_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 291.21 MiB (5.82 BPW)
llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant'
llama_model_load_from_file_impl: failed to load model
llama_init_from_model: model cannot be NULL
Error: failed to load speculative decoding draft model 'C:/Users/kat/AI/Models/Gemma4/gemma-4-26b-A4B-it-assistant-QAT-Q4_0.gguf'
Speculative Decoding will not be used!
Starting model warm up, please wait a moment...
Load Text Model OK: True

Additional Information:

OS: Windows 11
CPU: ryzen 2700x
gpu: nvidia rtx 4060ti
KoboldCPP version: 1.114.1

[LostRuins]

Hello, can you try it at with the rolling builds at https://github.com/LostRuins/koboldcpp/releases/tag/rolling

[lyrakat]

getting this now

llama_context: constructing llama_context
llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
Error: failed to load speculative decoding draft model 'C:/Users/kat/AI/Models/Gemma4/gemma-4-26b-A4B-it-assistant-QAT-Q4_0.gguf'
Speculative Decoding will not be used!
Starting model warm up, please wait a moment...
Load Text Model OK: True

[Limezero]

Did you reuse the main model as the draft one by mistake? The MTP draft model should be a separate file called something like gemma-4-26B-A4B-it-Q4_0-MTP.gguf.

[lyrakat]

I don't think so. I'll include the full log
Here's where the assistant model is from in case i'm using the wrong thing: https://huggingface.co/RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf

***
Welcome to KoboldCpp - Version 1.115
For command line arguments, please run --help in the terminal.
Note: The GUI mode is not accessible to screen readers.
***
Auto Selected CUDA Backend (flag=0)

KoboldCpp Proxy starting on port 5001, forwarding to port 15001
Loading Chat Completions Adapter: C:\Users\USERNAME\AppData\Local\Temp\_MEI433162\kcpp_adapters\Gemma-4-Thinking.json
Chat Completions Adapter Loaded
Using custom Jinja template: C:/Users/USERNAME/AI/Models/Gemma4/chat_template.jinja
Forced autofit is selected, moecpu and overridetensors will be set automatically.
Auto Recommended GPU Layers: 0
System: Windows 10.0.26200 AMD64 AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD
Detected Available GPU Memory: 16380 MB
Detected Available RAM: 48014 MB
Initializing dynamic library: koboldcpp_cublas.dll
==========
Namespace(admin=True, admindir='C:/Users/USERNAME/AI/KoboldAI', adminpassword='PASSWORD', adminunloadtimeout=0, analyze='', autofit=True, autofitpadding=1024, autoswapmode=False, baseconfig='', batchsize=512, benchmark=None, blasthreads=None, chatcompletionsadapter='C:/Users/USERNAME/AppData/Local/Temp/_MEI247322/kcpp_adapters/Gemma-4-Thinking.json', cli=False, config=None, contextsize=65536, debugmode=0, defaultgenamt=4096, device='', downloaddir='', draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel='C:/Users/USERNAME/AI/Models/Gemma4/gemma-4-26b-A4B-it-assistant-QAT-Q4_0.gguf', embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='C:/Users/USERNAME/AI/Models/Embeddings/KaLM-Embedding-Gemma3-12B-2511.Q5_K_M.gguf', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=False, forceversion=False, foreground=False, gendefaults='', gendefaultsoverwrite=False, genlimit=0, gpulayers=0, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='', ignoremissing=False, jinja=True, jinja_kwargs='{"enable_thinking": true}', jinja_tools=True, jinjatemplate='C:/Users/USERNAME/AI/Models/Gemma4/chat_template.jinja', jinjathink='default', launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mcpfile=None, mmproj=None, mmprojcpu=False, model=[], model_param='C:/Users/USERNAME/AI/Models/Gemma4/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=10, musicdiffusion='', musicembeddings='', musicllm='', musiclowvram=False, musicvae='', noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, noflashattention=False, nommap=False, nommq=True, nomodel=False, nopipelineparallel=False, noshift=False, noswa=False, onready='', overridekv=None, overridenativecontext=0, overridetensors='', parallelrequests=1, password=None, pipelineparallel=False, port=15001, port_param=15001, preloadstory=None, prompt='', proxy_port=5001, quantkv='f16', quiet=False, ratelimit=0, remotetunnel=False, reqtimeout=600, ropeconfig=[0.0, 10000.0], routermode=True, rpcdevice='', rpchost='0.0.0.0', rpcmode='disabled', rpcport=5551, rpctargets='', savedatafile=None, sdaudiovae='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipdevice=-2, sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults=False, sdlora=[], sdloramult=[1.0], sdmaingpu=-1, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=7, sdtiledvae=768, sdupscaler='', sdvae='', sdvaeauto=False, sdvaecpu=False, sdvaedevice=-1, sdvramlimit=0, showgui=False, singleinstance=False, skiplauncher=False, smartcache=5, smartcontext=False, splitmode='layer', ssl=None, swapadding=0, tensor_split=None, testmemory=False, threads=8, ttsdir='', ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=7, ttswavtokenizer='', unpack='', usecpu=False, usecuda=['normal', '0'], usemlock=False, usemmap=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, visionmaxtokens=-1, visionmintokens=-1, websearch=False, whispermodel='')
==========
Loading Text Model: C:\Users\USERNAME\AI\Models\Gemma4\gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf

The reported GGUF Arch is: gemma4
Arch Category: 49

---
Identified as GGUF model.
Attempting to Load...
---
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16379 MiB):
  Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes, VRAM: 16379 MiB
Number of Backends: 3
Backend 0: CUDA
Backend 1: RPC
Backend 2: CPU
CUDA MMQ: False
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---

Attempting to use llama.cpp's automating fitting code. This will override all your layer configs, may or may not work!
Autofit Reserve Space: 1483 MB
Autofit Success: 1, Autofit Result: -c 65664 -ngl 31 -ot blk\.25\.ffn_(gate|gate_up|down).*=CPU,blk\.26\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU,blk\.27\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU,blk\.28\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU,blk\.29\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU,blk\.30\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU
llama_model_loader: loaded meta data with 50 key-value pairs and 658 tensors from C:\Users\USERNAME\AI\Models\Gemma4\gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 13.26 GiB (4.51 BPW)
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4060 Ti) (0000:0a:00.0) - 15233 MiB free
init_tokenizer: initializing tokenizer for type 2
load: override 'tokenizer.ggml.add_bos_token' to 'true' for Gemma4
load: 0 unused tokens
load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 50 ('<|tool_response>')
load:   - 106 ('<turn|>')
load:   - 212 ('</s>')
load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
load: special tokens cache size = 24
load: token to piece cache size = 1.9445 MB
print_info: arch                  = gemma4
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 262144
print_info: n_embd_inp            = 2816
print_info: n_embd                = 2816
print_info: n_embd_out            = 2816
print_info: n_layer               = 30
print_info: n_layer_all           = 30
print_info: n_head                = 16
print_info: n_head_kv             = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2]
print_info: n_rot                 = 512
print_info: n_swa                 = 1024
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 512
print_info: n_embd_head_v         = 512
print_info: n_gqa                 = [2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8]
print_info: n_embd_k_gqa          = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: n_embd_v_gqa          = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 1.0e+00
print_info: f_attn_value_scale    = 0.0000
print_info: n_ff                  = 2112
print_info: n_expert              = 128
print_info: n_expert_used         = 8
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_embd_head_k_swa     = 256
print_info: n_embd_head_v_swa     = 256
print_info: n_rot_swa             = 256
print_info: n_ctx_orig_yarn       = 262144
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 26B.A4B
print_info: model params          = 25.23 B
print_info: general.name          = Gemma-4 26B-A4B IT (smart Q4_0, QAT-lossless)
print_info: vocab type            = BPE
print_info: n_vocab               = 262144
print_info: n_merges              = 514906
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: EOT token             = 106 '<turn|>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: MASK token            = 4 '<mask>'
print_info: LF token              = 107 '
'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 50 '<|tool_response>'
print_info: EOG token             = 106 '<turn|>'
print_info: max token length      = 93
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
tensor blk.25.ffn_gate.weight (3 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.25.ffn_down.weight (3 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.25.ffn_gate_inp.weight (1 MiB f32) buffer type overridden to CUDA_Host
tensor blk.25.ffn_gate_inp.scale (0 MiB f32) buffer type overridden to CUDA_Host
tensor blk.25.ffn_gate_up_exps.weight (272 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.25.ffn_down_exps.weight (136 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.26.ffn_gate_up_exps.weight (272 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.26.ffn_down_exps.weight (136 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.27.ffn_gate_up_exps.weight (272 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.27.ffn_down_exps.weight (136 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.28.ffn_gate_up_exps.weight (272 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.28.ffn_down_exps.weight (136 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.29.ffn_gate_up_exps.weight (272 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.29.ffn_down_exps.weight (136 MiB q4_0) buffer type overridden to CUDA_Host
tensor blk.25.ffn_down_exps.scale (0 MiB f32) buffer type overridden to CUDA_Host
tensor blk.26.ffn_down_exps.scale (0 MiB f32) buffer type overridden to CUDA_Host
tensor blk.27.ffn_down_exps.scale (0 MiB f32) buffer type overridden to CUDA_Host
tensor blk.28.ffn_down_exps.scale (0 MiB f32) buffer type overridden to CUDA_Host
tensor blk.29.ffn_down_exps.scale (0 MiB f32) buffer type overridden to CUDA_Host
done_getting_tensors: tensor 'blk.25.ffn_gate.weight' (q4_0) (and 18 others) moved from CUDA0, using CUDA_Host instead
load_tensors: offloading output layer to GPU
load_tensors: offloading 29 repeating layers to GPU
load_tensors: offloaded 31/31 layers to GPU
load_tensors:        CUDA0 model buffer size = 11524.27 MiB
load_tensors:    CUDA_Host model buffer size =  2445.64 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
.......................................................................

SWA Mode is ENABLED!
Note that using SWA Mode cannot be used with Context Shifting!
Context shifting is DISABLED!
SmartCache: Prepared 5 KV slots
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 65792
llama_context: n_ctx_seq     = 65792
llama_context: n_batch       = 1024
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_rs_seq      = 0
llama_context: n_outputs_max = 1024
llama_context: n_ctx_seq (65792) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     1.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 65792 cells
llama_kv_cache: reusing layers:
llama_kv_cache:      CUDA0 KV buffer size =  1285.00 MiB
llama_kv_cache: size = 1285.00 MiB ( 65792 cells,   5 layers,  1/1 seqs), K (f16):  642.50 MiB, V (f16):  642.50 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
llama_kv_cache_iswa: creating     SWA KV cache, size = 1664 cells
llama_kv_cache: reusing layers:
llama_kv_cache:      CUDA0 KV buffer size =   325.00 MiB
llama_kv_cache: size =  325.00 MiB (  1664 cells,  25 layers,  1/1 seqs), K (f16):  162.50 MiB, V (f16):  162.50 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
sched_reserve: reserving ...
sched_reserve: max_nodes = 5272
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:      CUDA0 compute buffer size =   532.88 MiB
sched_reserve:  CUDA_Host compute buffer size =    76.90 MiB
sched_reserve: graph nodes  = 2644
sched_reserve: graph splits = 16 (with bs=512), 18 (with bs=1)
sched_reserve: reserve took 34.79 ms, sched copies = 1
Threadpool set to 8 threads and 8 blasthreads...
attach_threadpool: call

Attempting to load draft model for speculative decoding. It will be fully offloaded if possible. Vocab must match the main model.
llama_model_loader: loaded meta data with 50 key-value pairs and 49 tensors from C:/Users/USERNAME/AI/Models/Gemma4/gemma-4-26b-A4B-it-assistant-QAT-Q4_0.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 291.21 MiB (5.82 BPW)
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4060 Ti) (0000:0a:00.0) - 1071 MiB free
init_tokenizer: initializing tokenizer for type 2
load: 0 unused tokens
load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 50 ('<|tool_response>')
load:   - 106 ('<turn|>')
load:   - 212 ('</s>')
load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
load: special tokens cache size = 23
load: token to piece cache size = 1.9445 MB
print_info: arch                  = gemma4-assistant
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 131072
print_info: n_embd_inp            = 2816
print_info: n_embd                = 1024
print_info: n_embd_out            = 2816
print_info: n_layer               = 0
print_info: n_layer_all           = 4
print_info: n_head                = 16
print_info: n_head_kv             = [8, 8, 8, 2]
print_info: n_rot                 = 512
print_info: n_swa                 = 1024
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 512
print_info: n_embd_head_v         = 512
print_info: n_gqa                 = [2, 2, 2, 8]
print_info: n_embd_k_gqa          = [2048, 2048, 2048, 1024]
print_info: n_embd_v_gqa          = [2048, 2048, 2048, 1024]
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 1.0e+00
print_info: f_attn_value_scale    = 0.0000
print_info: n_ff                  = 8192
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_embd_head_k_swa     = 256
print_info: n_embd_head_v_swa     = 256
print_info: n_rot_swa             = 256
print_info: n_ctx_orig_yarn       = 131072
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = ?B
print_info: model params          = 419.71 M
print_info: general.name          = Assistant
print_info: vocab type            = BPE
print_info: n_vocab               = 262144
print_info: n_merges              = 514906
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: MASK token            = 4 '<mask>'
print_info: LF token              = 107 '
'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 50 '<|tool_response>'
print_info: EOG token             = 106 '<turn|>'
print_info: max token length      = 93
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
done_getting_tensors: tensor 'token_embd.weight' (q6_K) (and 0 others) moved from CUDA_Host, using CPU instead
load_tensors: offloading output layer to GPU
load_tensors: offloading 3 repeating layers to GPU
load_tensors: offloaded 5/5 layers to GPU
load_tensors:          CPU model buffer size =   210.00 MiB
load_tensors:        CUDA0 model buffer size =   291.21 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
...................
llama_context: constructing llama_context
llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
Error: failed to load speculative decoding draft model 'C:/Users/USERNAME/AI/Models/Gemma4/gemma-4-26b-A4B-it-assistant-QAT-Q4_0.gguf'
Speculative Decoding will not be used!
Starting model warm up, please wait a moment...
Load Text Model OK: True

Loading Embeddings Model: C:\Users\USERNAME\AI\Models\Embeddings\KaLM-Embedding-Gemma3-12B-2511.Q5_K_M.gguf
llama_model_loader: loaded meta data with 53 key-value pairs and 626 tensors from C:\Users\USERNAME\AI\Models\Embeddings\KaLM-Embedding-Gemma3-12B-2511.Q5_K_M.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 7.86 GiB (5.74 BPW)
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4060 Ti) (0000:0a:00.0) - 658 MiB free
init_tokenizer: initializing tokenizer for type 1
load: 6242 unused tokens
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 106 ('<end_of_turn>')
load:   - 212 ('</s>')
load: special tokens cache size = 6415
load: token to piece cache size = 1.9446 MB
print_info: arch                  = gemma-embedding
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 131072
print_info: n_embd_inp            = 3840
print_info: n_embd                = 3840
print_info: n_embd_out            = 3840
print_info: n_layer               = 48
print_info: n_layer_all           = 48
print_info: n_head                = 16
print_info: n_head_kv             = 8
print_info: n_rot                 = 256
print_info: n_swa                 = 1024
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 256
print_info: n_embd_head_v         = 256
print_info: n_gqa                 = 2
print_info: n_embd_k_gqa          = 2048
print_info: n_embd_v_gqa          = 2048
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 6.2e-02
print_info: f_attn_value_scale    = 0.0000
print_info: n_ff                  = 15360
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 0
print_info: pooling type          = 3
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 0.125
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_embd_head_k_swa     = 256
print_info: n_embd_head_v_swa     = 256
print_info: n_rot_swa             = 256
print_info: n_ctx_orig_yarn       = 131072
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = ?B
print_info: model params          = 11.77 B
print_info: general.name          = KaLM Embedding Gemma3 12B 2511
print_info: vocab type            = SPM
print_info: n_vocab               = 262208
print_info: n_merges              = 0
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: EOT token             = 106 '<end_of_turn>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: LF token              = 248 '<0x0A>'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 106 '<end_of_turn>'
print_info: EOG token             = 212 '</s>'
print_info: max token length      = 48
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
done_getting_tensors: tensor 'token_embd.weight' (q6_K) (and 0 others) moved from CUDA_Host, using CPU instead
load_tensors: offloading 0 repeating layers to GPU
load_tensors: offloaded 0/49 layers to GPU
load_tensors:          CPU model buffer size =   787.69 MiB
load_tensors:    CUDA_Host model buffer size =  8047.57 MiB
.....................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 65536
llama_context: n_ctx_seq     = 65536
llama_context: n_batch       = 65536
llama_context: n_ubatch      = 65536
llama_context: causal_attn   = 0
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 0.125
llama_context: n_rs_seq      = 0
llama_context: n_outputs_max = 65536
llama_context: n_ctx_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context:        CPU  output buffer size =     1.01 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
sched_reserve: reserving ...
sched_reserve: max_nodes = 5016

Embeddings Model Load Complete.
Load Embeddings Model OK: True
Embedded KoboldAI Lite loaded.
Embedded API docs loaded.
Llama.cpp UI loaded.
======
Active Modules: TextGeneration VectorEmbeddings AdminControl RouterMode
Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech MCPBridge MusicGen
Enabled APIs: KoboldCppApi OpenAiApi OllamaApi AnthropicApi
Note: For third party Ollama API Emulation, you should set the port to 11434.
Starting Kobold API on port 5001 at http://localhost:5001/api/
Starting OpenAI Compatible API on port 5001 at http://localhost:5001/v1/
Starting llama.cpp secondary WebUI at http://localhost:5001/lcpp/
======
Please connect to custom endpoint at http://localhost:5001

[Limezero]

Ah, okay. That seems right based on the filesize, Google just named their official MTP repo weird. "assistant" is apparently their terminology for a draft model.

If you're already using Unsloth's quants (UD) though, it might be worth using their MTP model as well, just to maximize compatibility.

https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF/tree/main/MTP

I know for a fact that one works in upstream llama.cpp. I haven't tested it in KoboldCpp yet since I was waiting for a stable release.

[lyrakat]

Grabbed theirs but I'm getting the same thing

llama_context: constructing llama_context
llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
Error: failed to load speculative decoding draft model 'C:/Users/kat/AI/Models/Gemma4/gemma-4-26B-A4B-it-Q4_0-MTP.gguf'
Speculative Decoding will not be used!
Starting model warm up, please wait a moment...
Load Text Model OK: True

[Limezero]

Yeah, same here with both the 12B and 26B.

Error: failed to load speculative decoding draft model 'D:/LLM/gemma-4-12B-it-mtp-Q8_0.gguf'
Speculative Decoding will not be used!

Tested on the latest rolling release build from 47 minutes ago, definitely a bug on Kobold's end. It's strange since it gets all the way to putting the model weights into the GPU before deciding that something is wrong.

[Skydea777]

Have to agree, also tried it to see what the Fuzz is about, sadly not working right now, same error.

[LostRuins]

Should be working in the latest rolling now

[lyrakat]

seems to successfully load. it also seems to massively slow down prompt processing speed without any gain in T/S output

the lack of speed gain may be related to MoE models getting limited benefit from speculative drafting, as per: https://ai.google.dev/gemma/docs/mtp/overview

So i'm not really sure beyond that

[Khrspk]

I'm still getting the "failed to load speculative decoding draft model" error on the latest rolling build. (unknown model architecture)

[Limezero]

It doesn't seem to be working on my end, at least not the way it should. Gemma 12B QAT, generating exactly 1500 tokens on a simple prompt. The models do load, but performance is much worse across the board, and drafting only slows generations down.

llama.cpp b9596 CUDA 12.4:

no draft:     25.21 t/s
draft-max 0:  31.31 t/s (0.54 draft acceptance)
draft-max 1:  31.90 t/s (0.55 draft acceptance)
draft-max 2:  28.00 t/s (0.43 draft acceptance)
draft-max 3:  23.64 t/s (0.33 draft acceptance)
draft-max 4:  22.03 t/s (0.27 draft acceptance)
draft-max 8:  16.81 t/s (0.14 draft acceptance)

koboldcpp rolling CUDA:

no draft:     16.22 t/s
draft-max 0:  13.73 t/s
draft-max 1:  13.72 t/s
draft-max 2:  16.36 t/s
draft-max 3:  15.84 t/s
draft-max 4:  13.57 t/s
draft-max 8:  9.58 t/s

Commands used:

llama-server.exe
  --fit off
  -m "D:/LLM/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf"
  -ngl 99
  -c 16384
  -t 4
  -md "D:/LLM/gemma-4-12b-it-Q8_0-MTP.gguf"
  -ngld 99
  --spec-type draft-mtp
  --spec-draft-n-max 4

koboldcpp-launcher.exe
  --usecuda
  -m "D:/LLM/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf"
  -ngl 99
  -c 16384
  -t 4
  -md "D:/LLM/gemma-4-12b-it-Q8_0-MTP.gguf"
  -ngld 99
  --usemtp
  --draft-max 4

[LostRuins]

Drafting definitely works for me, although performance is very prompt dependent. On simple prompts I get almost 2x speedup.

To see if the drafting triggers correctly, you can use --debugmode although for actual timings you'd want to turn that off as it slows generation down.

First off, are you sure you are loading the right model @Limezero ? The MTP assistant is only about 400mb, you might be loading the "combined version" by mistake - I noticed mine was named gemma-4-12B-it-qat-assistant-MTP-Q8_0 with the assistant word. Also, are you by any chance near the VRAM limit? The draft model could be forcing you to use less layers.

Anyway, I tried the MTP assistant model from https://huggingface.co/Janvitos/gemma-4-12B-it-qat-assistant-MTP-Q8_0-GGUF/tree/main along with my base model gemma-4-12B-it-Q4_K_M.gguf.

My prompt is: Give me the first 100 integers

Without drafting:
Processed:20 in 0.01s (1538.46T/s), Generated:403/2048 in 10.01s (40.24T/s), Total:10.04s

With drafting (default launch settings):
Processed:20 in 0.06s (363.64T/s), Generated:407/2048 in 5.10s (79.82T/s), Total:5.17s

In debug mode, checking my hitrate:
(Draft Results - Success:368, Failure:38) (do share your own hitrate)

---

I also tried gemma-4-26B-A4B with the same good results. I used an even smaller MTP draft head gemma-4-26b-A4B-it-assistant-Q4_0.gguf, https://huggingface.co/RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf/resolve/main/gemma-4-26b-A4B-it-assistant-Q4_0.gguf

And then combined with the normal gemma-4-26B-A4B-it-UD-Q4_K_M.gguf

This time I will use a pure CLI command for benchmarking

Without drafting:

python .\koboldcpp.py -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --usevulkan --fit --benchmark --prompt "The first 100 integers are: 1, 2, 3"

ProcessingTime: 87.934s
ProcessingSpeed: 185.19T/s
GenerationTime: 3.072s
GenerationSpeed: 32.55T/s
TotalTime: 91.006s
Output: , 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,

With drafting:

python .\koboldcpp.py -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf -md gemma-4-26b-A4B-it-assistant-Q4_0.gguf --usevulkan --fit --benchmark --prompt "The first 100 integers are: 1, 2, 3"

ProcessingTime: 92.819s
ProcessingSpeed: 175.44T/s
GenerationTime: 1.718s
GenerationSpeed: 58.21T/s
TotalTime: 94.537s
Output: , 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,

So you can see, my generation speed has increased greatly.

[LostRuins]

@Khrspk did you download from https://github.com/LostRuins/koboldcpp/releases/tag/rolling ? Can you share your launch commands and model urls?

[LostRuins]

The simple "generate first 100 integers" test is the easiest way to see if drafting works, because the draft model is smart enough to do it. If it works you WILL see massive generation speed increase.