MTP: separate sidecar draft model, llama.cpp b10435 (Qwen 3.8)

CHANGELOG.md

@@ -6,9 +6,13 @@ DGX Spark (GB10) support: a silent ARM code-generation bug fixed, the ggml RPC
 backend wired up so a model can span two machines, and a measured runbook for
 both configurations in [docs/dgx-spark.md](docs/dgx-spark.md).
 
-Verified on macOS (Metal, no RPC): **423 passed, 143 excluded**. On a DGX Spark
-(CUDA 13.0, `sm_121a`): **423 passed**, and **424 with `--include rpc_live`**
-against a live two-node worker.
+llama.cpp bumped to [`b10435`](https://github.com/ggml-org/llama.cpp/releases/tag/b10435)
+(`9e40df63b`), which brings Qwen 3.8 in under the existing `qwen35`
+architecture, and MTP support for its target/sidecar split (see Added).
+
+Verified on macOS (Metal, no RPC): **428 passed, 149 excluded**, plus
+**6 passed** with `--include mtp_sidecar` against Qwen3.8-27B-Q4_K_M and its
+`mtp-*-Q4_0` head. On a DGX Spark (CUDA 13.0, `sm_121a`): **428 passed**.
 
 ### Fixed
 
@@ -57,6 +61,31 @@ against a live two-node worker.
 
 ### Added
 
+- **`LlamaCppEx.MTP.init/2` accepts a separate `:draft_model`.** The MTP head no
+  longer has to live inside the target GGUF. Qwen 3.8 is why: `ggml-org/Qwen3.8-27B-GGUF`
+  ships `Qwen3.8-27B-Q4_K_M.gguf` with *zero* nextn layers and the head alone in
+  `mtp-Qwen3.8-27B-Q4_0.gguf`, so the old single-file path refused the pair
+  outright with "this GGUF contains no MTP head". This is the binding's
+  equivalent of upstream's `-hf <target> -hfd <draft> --spec-type draft-mtp`.
+  The NIF already took two independent contexts; only the Elixir side was tying
+  them to one model.
+
+  Mismatched pairings are refused before any context is built, including a
+  target/draft hidden-width mismatch — upstream compares those with a
+  `GGML_ASSERT`, which is an unconditional `ggml_abort` and would take the VM
+  down instead of returning an error.
+- **`stats/1` reports `timing_us.ckpt`.** Recurrent-state save/restore, which
+  only hybrid models pay, was previously folded into `:other` — a bucket whose
+  documented cause is Metal GPU-sync waits. On Qwen 3.8 (48 SSM layers to 16
+  attention ones, ~150 MiB of state snapshotted every iteration) it is the term
+  that decides whether speculation helps at all: 6.9 s of a 16.3 s M1 Max run at
+  `n_draft: 3`. Attributing it correctly dropped `:other` for that run from
+  8.7 s to 0.17 s.
+- **`LlamaCppEx.Model.n_embd_out/1` and `n_layer_nextn/1`** — the two numbers
+  that decide whether a GGUF can serve as an MTP target or head. `n_layer_nextn`
+  wraps a NIF that already existed but was only reachable through
+  `LlamaCppEx.NIF`.
+
 - **`LlamaCppEx.RPC`** — register a remote machine's devices into the local
   device registry so a model's layers can live on another host.
   `add_server/1`, `add_servers/1`, `devices/0`, `ping/1`, `supported?/0`.

Makefile

@@ -36,7 +36,7 @@ endif
 # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST
 # match the vendor/llama.cpp submodule; bump both together, see
 # docs/release-guide.md. Override to build the NIF against another revision.
-LLAMA_COMMIT ?= a94d563ed801d1da1b8c2432946de07d0231bb3d
+LLAMA_COMMIT ?= 9e40df63ba151d771d8b247ac4011cf203337e99
 
 # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT
 # following it, and the build has to key off what is really there.

README.md

@@ -610,14 +610,42 @@ Multi-Token Prediction speculative decoding (upstream PR [#22673](https://github
 > `n_draft: 3` as the datacenter default the upstream 2× assumes, not as a value
 > that transfers.
 
+> **Performance note: Qwen 3.8 27B (hybrid SSM, sidecar head).** This one is
+> shaped by a cost the models above do not pay. Qwen 3.8 puts 48 SSM layers
+> beside 16 attention ones, and a recurrent layer cannot be rolled back to an
+> arbitrary position, so every speculative iteration snapshots and restores the
+> whole recurrent state — 150 MiB of it at these sizes. `stats/1` reports that
+> separately as `timing_us.ckpt`. Q4_K_M target + Q4_0 sidecar head, 120-token
+> greedy generations:
+>
+> | `n_draft` | acceptance | M1 Max (Metal) | GB10 (DGX Spark) |
+> |---|---|---|---|
+> | 1 | 75.0% | 0.89× | **1.24×** |
+> | 2 | 54–59% | 0.66× | 1.17× |
+> | 3 | 40–44% | 0.68× | 1.09× |
+> | 4 | 32% | — | 0.96× |
+> | 5 | 30% | 0.56× | — |
+>
+> On Metal MTP is a net loss at every draft length: `ckpt` alone was 1.8 s of a
+> 12.5 s run at `n_draft: 1` and 6.9 s of 16.3 s at `n_draft: 3`. On GB10 the
+> same snapshot is cheap enough that `n_draft: 1` wins, and — unlike the MoE
+> above, whose optimum was 2 — the optimum here is 1, monotonically decreasing
+> after it. Measure `ckpt` against `total` before trusting speculation on any
+> hybrid model.
+>
+> Both arms of the GB10 column are warm-cache numbers. A first run off cold page
+> cache reads ~19 GB and reported a 4.18 tok/s baseline against 10.83 tok/s warm,
+> which inverts the comparison entirely.
+
 ### Other speculative types (EAGLE-3, DFlash, n-gram)
 
-Upstream llama.cpp implements more speculative types behind the same `common_speculative` API — `draft-eagle3`, `draft-dflash` (block-diffusion drafting via a separate drafter GGUF), and several n-gram self-speculation modes. **This binding currently exposes only MTP**: `MTP.init/2` pins `COMMON_SPECULATIVE_TYPE_DRAFT_MTP` and builds both contexts from the same model, so there is no way to load a separate drafter model yet.
+Upstream llama.cpp implements more speculative types behind the same `common_speculative` API — `draft-eagle3`, `draft-dflash` (block-diffusion drafting via a separate drafter GGUF), and several n-gram self-speculation modes, plus `--spec-default`, which stacks n-gram speculation on top of a model-based drafter. **This binding currently exposes only MTP**: `MTP.init/2` pins `COMMON_SPECULATIVE_TYPE_DRAFT_MTP`, so the other types and the combinations are not reachable from here. The draft *model* is no longer tied to the target, though — see `:draft_model` below for the target/sidecar split.
 
 > **DFlash status (July 2026, llama.cpp b9932).** DFlash runs end-to-end on Metal via upstream `llama-cli`/`llama-server`, but we measured it *slower* than plain decoding on Apple Silicon at small target sizes: Qwen3.5-4B target + z-lab 0.6B drafter on M4 Max reached 42 tok/s with DFlash vs 85 tok/s plain (greedy sampling; 30% draft acceptance, mean accepted run 2.8 — and stochastic sampling at `temp 0.8` collapses acceptance to ~7%). The Metal economics are the same as the MTP note above (wide verify batches are expensive), and the community drafter-GGUF conversions are still churning: of three third-party Qwen 4B drafter repos tested, only one loads with current upstream (the others hit the `dflash-draft` arch mismatch [#25116](https://github.com/ggml-org/llama.cpp/issues/25116) or lack the `target_layers` metadata added by the conversion refactor [#25110](https://github.com/ggml-org/llama.cpp/pull/25110)). Worth revisiting when the drafter format settles; the natural entry point is a `spec_type` + drafter-model option on `speculative_init`.
 
 ### Models with MTP heads
 
+- [`ggml-org/Qwen3.8-27B-GGUF`](https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF) — **sidecar layout**: the target (`Qwen3.8-27B-Q4_K_M.gguf`, ~18 GB) carries *no* head, and `mtp-Qwen3.8-27B-Q4_0.gguf` (~1.6 GB) carries nothing else. Load both and pass the head as `draft_model:`.
 - [`ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-35B-A3B-MTP-GGUF) (recommended: `Q4_K_M`, ~21 GB)
 - [`ggml-org/Qwen3.6-27B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-27B-MTP-GGUF)
 - [`unsloth/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)
@@ -632,7 +660,7 @@ Acceptance on a 0.8B target is not representative of production throughput —
 drafting is nearly as expensive as decoding at that size — so use these to
 exercise the path, not to measure it.
 
-A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the GGUF must contain the MTP head's tensors. To check a file before loading it, look for a `*.nextn_predict_layers` key and `blk.N.nextn.*` tensors in its metadata.
+A regular (non-MTP) quant will fail at `LlamaCppEx.MTP.init/2` — some GGUF in the pair must contain the MTP head's tensors. To check a file before loading it, look for a `*.nextn_predict_layers` key and `blk.N.nextn.*` tensors in its metadata. When the publisher ships the head separately (Qwen 3.8), that sidecar is the file with those tensors and the target legitimately has none; pass it as `draft_model:` rather than looking for a combined build.
 
 The model must also be loaded with `load_mtp: true` (see below). Upstream gates those tensors behind a load-time flag that defaults to off, and they cannot be attached afterwards, so `MTP.init/2` refuses a model loaded without it rather than letting the omission surface later as `verify decode failed: code=-1`.
 
@@ -670,6 +698,49 @@ IO.puts("\nacceptance: #{Float.round(stats.acceptance_rate * 100, 1)}%  " <>
         "throughput: #{Float.round(stats.tokens_per_sec, 1)} tok/s")
 ```
 
+#### Sidecar head: Qwen 3.8 (`-hf` target + `-hfd` draft)
+
+Same session API, two files. This is what upstream's
+`llama serve -hf ggml-org/Qwen3.8-27B-GGUF --spec-type draft-mtp` resolves to
+once it has downloaded the pair — the flag makes upstream fetch the `mtp-*`
+sidecar and build the draft context against it instead of against the target.
+
+```elixir
+:ok = LlamaCppEx.init()
+
+# The target carries no MTP head at all: Model.n_layer_nextn/1 returns 0 for it.
+{:ok, target} =
+  LlamaCppEx.load_model(
+    Path.expand("~/Downloads/Qwen3.8-27B-Q4_K_M.gguf"),
+    n_gpu_layers: 999,
+    load_mtp: true
+  )
+
+# The sidecar carries nothing *but* the head — ~1.6 GB against the target's 18.
+{:ok, head} =
+  LlamaCppEx.load_model(
+    Path.expand("~/Downloads/mtp-Qwen3.8-27B-Q4_0.gguf"),
+    n_gpu_layers: 999,
+    load_mtp: true
+  )
+
+# n_draft: 1 — see the Qwen 3.8 performance note above. Acceptance is 75% here
+# and falls off fast, and every iteration pays a recurrent-state snapshot.
+{:ok, mtp} = LlamaCppEx.MTP.init(target, draft_model: head, n_draft: 1, n_ctx: 8192)
+
+{:ok, text} = LlamaCppEx.MTP.generate(mtp, "Explain MTP in one paragraph.", max_tokens: 200)
+
+stats = LlamaCppEx.MTP.stats(mtp)
+IO.puts("acceptance: #{Float.round(stats.acceptance_rate * 100, 1)}%  " <>
+        "ckpt: #{div(stats.timing_us.ckpt, 1000)}ms of #{div(stats.timing_us.total, 1000)}ms")
+```
+
+`MTP.init/2` refuses the mismatched pairings before building anything: a sidecar
+loaded without `load_mtp: true`, an ordinary model passed as `:draft_model`, and
+a head whose hidden width does not match the target's — that last one because
+upstream compares the two with a `GGML_ASSERT`, which aborts the VM rather than
+failing the call.
+
 #### Synchronous generate (collect to a string)
 
 ```elixir

c_src/llama_cpp_ex/llama_nif.cpp

@@ -662,6 +662,18 @@ int64_t model_n_embd(ErlNifEnv* env, fine::ResourcePtr<LlamaModel> model) {
 }
 FINE_NIF(model_n_embd, 0);
 
+// Output-side embedding width, which is what the MTP draft head consumes. It is
+// `n_embd` for every architecture in tree today, but upstream reads this one
+// (speculative.cpp: "MTP input row width must match the target h_nextn width")
+// and enforces the target/draft match with a GGML_ASSERT — an unconditional
+// ggml_abort that takes the whole VM down rather than failing the call. A
+// separate drafter GGUF is the only way to reach that assert, so MTP.init/2
+// compares this across the two models before it builds anything.
+int64_t model_n_embd_out(ErlNifEnv* env, fine::ResourcePtr<LlamaModel> model) {
+    return llama_model_n_embd_out(model->model);
+}
+FINE_NIF(model_n_embd_out, 0);
+
 // Number of MTP / "next-N" prediction layers the checkpoint carries. Zero means
 // the GGUF has no MTP head at all, which is a different situation from a model
 // loaded with load_mtp: false: no flag can recover it, only a different file.
@@ -2027,28 +2039,31 @@ static ERL_NIF_TERM build_mtp_stats_map(ErlNifEnv* env, const LlamaSpeculative&
     uint64_t udraft  = s.us_draft.load(std::memory_order_relaxed);
     uint64_t uverify = s.us_verify.load(std::memory_order_relaxed);
     uint64_t usample = s.us_sample.load(std::memory_order_relaxed);
+    uint64_t uckpt   = s.us_ckpt.load(std::memory_order_relaxed);
     uint64_t uother  = s.us_other.load(std::memory_order_relaxed);
     uint64_t utotal  = s.us_total.load(std::memory_order_relaxed);
 
     double acceptance_rate = dgen > 0 ? (double)dacc / (double)dgen : 0.0;
     double tokens_per_sec  = utotal > 0 ? (double)emitted * 1e6 / (double)utotal : 0.0;
 
-    ERL_NIF_TERM tk[5] = {
+    ERL_NIF_TERM tk[6] = {
         enif_make_atom(env, "draft"),
         enif_make_atom(env, "verify"),
         enif_make_atom(env, "sample"),
+        enif_make_atom(env, "ckpt"),
         enif_make_atom(env, "other"),
         enif_make_atom(env, "total"),
     };
-    ERL_NIF_TERM tv[5] = {
+    ERL_NIF_TERM tv[6] = {
         enif_make_uint64(env, udraft),
         enif_make_uint64(env, uverify),
         enif_make_uint64(env, usample),
+        enif_make_uint64(env, uckpt),
         enif_make_uint64(env, uother),
         enif_make_uint64(env, utotal),
     };
     ERL_NIF_TERM timing;
-    enif_make_map_from_arrays(env, tk, tv, 5, &timing);
+    enif_make_map_from_arrays(env, tk, tv, 6, &timing);
 
     ERL_NIF_TERM keys[8] = {
         enif_make_atom(env, "iters"),
@@ -2323,6 +2338,7 @@ fine::Ok<> generate_mtp_tokens(
         // PART at init time, so llama_memory_seq_rm handles partial rejection
         // natively and the checkpoint would be pure overhead.
         if (sp.needs_ckpt) {
+            auto t_ck0 = std::chrono::steady_clock::now();
             size_t sz_tgt = llama_state_seq_get_size_ext(ctx_tgt, seq_id, ckpt_flags);
             ckpt_tgt.resize(sz_tgt);
             if (sz_tgt > 0) {
@@ -2333,6 +2349,13 @@ fine::Ok<> generate_mtp_tokens(
             if (sz_dft > 0) {
                 llama_state_seq_get_data_ext(ctx_dft, ckpt_dft.data(), sz_dft, seq_id, ckpt_flags);
             }
+            // Bill this to us_ckpt, then slide the anchor past it so the same
+            // microseconds are not also counted as unaccounted "other".
+            auto t_ck1 = std::chrono::steady_clock::now();
+            sp.us_ckpt.fetch_add(
+                std::chrono::duration_cast<std::chrono::microseconds>(t_ck1 - t_ck0).count(),
+                std::memory_order_relaxed);
+            t_anchor += (t_ck1 - t_ck0);
         }
 
         // 1. Generate drafts from the MTP head's current state.
@@ -2468,6 +2491,7 @@ fine::Ok<> generate_mtp_tokens(
                 // Hybrid model: partial seq_rm isn't supported, so restore
                 // both contexts from the pre-iteration recurrent-state
                 // snapshot and re-decode just the accepted prefix.
+                auto t_rs0 = std::chrono::steady_clock::now();
                 if (!ckpt_tgt.empty()) {
                     llama_state_seq_set_data_ext(ctx_tgt, ckpt_tgt.data(),
                                                   ckpt_tgt.size(), seq_id, ckpt_flags);
@@ -2479,6 +2503,14 @@ fine::Ok<> generate_mtp_tokens(
                                                   ckpt_dft.size(), seq_id, ckpt_flags);
                 }
                 soft_seq_rm(ctx_dft, seq_id, n_past);
+                auto t_rs1 = std::chrono::steady_clock::now();
+                sp.us_ckpt.fetch_add(
+                    std::chrono::duration_cast<std::chrono::microseconds>(t_rs1 - t_rs0).count(),
+                    std::memory_order_relaxed);
+                // Same anchor slide as the save above: us_other is closed out
+                // from t_anchor at the end of the iter, so without this the
+                // restore would be billed twice.
+                t_anchor += (t_rs1 - t_rs0);
 
                 // Re-decode the accepted tokens on the target so the next
                 // iteration's draft starts from a consistent state.

c_src/llama_cpp_ex/llama_nif.h

@@ -194,6 +194,13 @@ class LlamaSpeculative {
     std::atomic<uint64_t> us_draft{0};
     std::atomic<uint64_t> us_verify{0};
     std::atomic<uint64_t> us_sample{0};
+    // Recurrent-state save/restore, which only hybrid models pay (needs_ckpt).
+    // Broken out of us_other because on a model like Qwen 3.8 — 48 SSM layers
+    // beside 16 attention ones — the snapshot is over a hundred MiB and taken
+    // every iteration, which is enough on its own to make speculation a net
+    // loss. Attributing it to "other" hid that behind a bucket whose documented
+    // cause is GPU-sync waits.
+    std::atomic<uint64_t> us_ckpt{0};
     // Everything in the speculative iter NOT inside the three hot-path
     // timers above. On Metal this is dominated by implicit GPU-sync waits
     // from the previous iter's async verify decode (llama_decode returns

lib/llama_cpp_ex/model.ex

@@ -197,6 +197,24 @@ defmodule LlamaCppEx.Model do
   @spec n_embd(t()) :: integer()
   def n_embd(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_embd(ref)
 
+  @doc """
+  Returns the output-side embedding width — the row width an MTP draft head
+  consumes. Equal to `n_embd/1` for every architecture currently in tree; it is
+  a distinct number because `LlamaCppEx.MTP` matches it across the target and a
+  separate drafter GGUF.
+  """
+  @spec n_embd_out(t()) :: integer()
+  def n_embd_out(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_embd_out(ref)
+
+  @doc """
+  Returns the number of MTP / "next-N" prediction layers in the checkpoint, or
+  `0` when it carries no MTP head. Note this reports what the *file* contains;
+  the layers are only actually loaded when the model was opened with
+  `load_mtp: true`.
+  """
+  @spec n_layer_nextn(t()) :: non_neg_integer()
+  def n_layer_nextn(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_layer_nextn(ref)
+
   @doc "Returns a human-readable description of the model."
   @spec desc(t()) :: String.t()
   def desc(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_desc(ref)