From 2e2c530cfbddac30ff34e698fedbf22879e60450 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Fri, 14 Aug 2026 23:04:02 -0400 Subject: [PATCH 1/2] MTP: separate sidecar draft model, llama.cpp b10435 (Qwen 3.8) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Qwen 3.8 ships MTP as a sidecar: ggml-org/Qwen3.8-27B-GGUF pairs Qwen3.8-27B-Q4_K_M.gguf, which has zero nextn layers, with the head alone in mtp-Qwen3.8-27B-Q4_0.gguf. MTP.init/2 built both contexts from the one model, so it refused that pair outright with "this GGUF contains no MTP head". The NIF already took two independent contexts and upstream's draft-mtp impl only requires the hidden widths to agree, so the restriction was purely on the Elixir side. MTP.init/2 now takes :draft_model — the equivalent of upstream's -hf -hfd --spec-type draft-mtp. No new arch work was needed: Qwen 3.8 loads under the existing qwen35 architecture. Mismatched pairings are refused before any context is built, including a target/draft hidden-width mismatch. That one matters because upstream compares the widths with a GGML_ASSERT, an unconditional ggml_abort that would take the VM down rather than return an error, so Model.n_embd_out/1 is exposed to check it first. stats/1 gains timing_us.ckpt. A hybrid target cannot roll back part of a sequence, so every speculative iteration snapshots the whole recurrent state — 48 SSM layers beside 16 attention ones on Qwen 3.8, ~150 MiB. That cost was folded into :other, whose documented cause is Metal GPU-sync waits. Splitting it out dropped :other for one M1 Max run from 8.7s to 0.17s and showed ckpt was 6.9s of the 16.3s total. Measured, Q4_K_M target + Q4_0 head, 120-token greedy: n_draft acceptance M1 Max GB10 1 75.0% 0.89x 1.24x 3 40-44% 0.68x 1.09x 4-5 30-32% 0.56x 0.96x So MTP is a net loss on Metal for this model and a win on GB10 at n_draft: 1 — unlike the 35B MoE, whose optimum was 2. Check ckpt against total before trusting speculation on any hybrid model. The GB10 column is warm-cache; a cold first run read ~19GB and reported a 4.18 tok/s baseline against 10.83 warm, which inverts the comparison. Two things deliberately not changed: - --spec-default, which stacks n-gram speculation on top of a model-based drafter, stays unreachable: the NIF pins COMMON_SPECULATIVE_TYPE_DRAFT_MTP. - At n_draft >= 4 greedy output diverges from baseline, deterministically and always at the same token. This is not a rollback bug. Plain greedy decode with no speculation flips at the same position purely from batch row count (1-3 rows " computational", 4+ " latency"): ggml-metal-ops.cpp takes the mul_mv_ext path for Q4_K only at ne11 >= 4 and picks r1ptg by ne11, and the verify batch is 1 + n_draft rows. It is a near-tie decided by reduction order. Confirmed on Metal only. llama.cpp a94d563 (b10423) -> 9e40df6 (b10435), LLAMA_COMMIT bumped to match. Verified on macOS (Metal): 428 passed, 149 excluded, plus 6 passed with --include mtp_sidecar against the Qwen 3.8 pair. On a DGX Spark (CUDA 13.0, sm_121a): 428 passed. mix format, --warnings-as-errors and credo --strict clean. --- CHANGELOG.md | 35 +++++++- Makefile | 2 +- README.md | 75 ++++++++++++++++- c_src/llama_cpp_ex/llama_nif.cpp | 38 ++++++++- c_src/llama_cpp_ex/llama_nif.h | 7 ++ lib/llama_cpp_ex/model.ex | 18 ++++ lib/llama_cpp_ex/mtp.ex | 140 +++++++++++++++++++++++++------ lib/llama_cpp_ex/nif.ex | 1 + test/mtp_model_test.exs | 123 +++++++++++++++++++++++++++ test/mtp_test.exs | 43 ++++++++++ test/support/test_models.exs | 9 +- test/test_helper.exs | 14 +++- vendor/llama.cpp | 2 +- 13 files changed, 466 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 277b0fe..de68812 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,13 @@ DGX Spark (GB10) support: a silent ARM code-generation bug fixed, the ggml RPC backend wired up so a model can span two machines, and a measured runbook for both configurations in [docs/dgx-spark.md](docs/dgx-spark.md). -Verified on macOS (Metal, no RPC): **423 passed, 143 excluded**. On a DGX Spark -(CUDA 13.0, `sm_121a`): **423 passed**, and **424 with `--include rpc_live`** -against a live two-node worker. +llama.cpp bumped to [`b10435`](https://github.com/ggml-org/llama.cpp/releases/tag/b10435) +(`9e40df63b`), which brings Qwen 3.8 in under the existing `qwen35` +architecture, and MTP support for its target/sidecar split (see Added). + +Verified on macOS (Metal, no RPC): **428 passed, 149 excluded**, plus +**6 passed** with `--include mtp_sidecar` against Qwen3.8-27B-Q4_K_M and its +`mtp-*-Q4_0` head. On a DGX Spark (CUDA 13.0, `sm_121a`): **428 passed**. ### Fixed @@ -57,6 +61,31 @@ against a live two-node worker. ### Added +- **`LlamaCppEx.MTP.init/2` accepts a separate `:draft_model`.** The MTP head no + longer has to live inside the target GGUF. Qwen 3.8 is why: `ggml-org/Qwen3.8-27B-GGUF` + ships `Qwen3.8-27B-Q4_K_M.gguf` with *zero* nextn layers and the head alone in + `mtp-Qwen3.8-27B-Q4_0.gguf`, so the old single-file path refused the pair + outright with "this GGUF contains no MTP head". This is the binding's + equivalent of upstream's `-hf -hfd --spec-type draft-mtp`. + The NIF already took two independent contexts; only the Elixir side was tying + them to one model. + + Mismatched pairings are refused before any context is built, including a + target/draft hidden-width mismatch — upstream compares those with a + `GGML_ASSERT`, which is an unconditional `ggml_abort` and would take the VM + down instead of returning an error. +- **`stats/1` reports `timing_us.ckpt`.** Recurrent-state save/restore, which + only hybrid models pay, was previously folded into `:other` — a bucket whose + documented cause is Metal GPU-sync waits. On Qwen 3.8 (48 SSM layers to 16 + attention ones, ~150 MiB of state snapshotted every iteration) it is the term + that decides whether speculation helps at all: 6.9 s of a 16.3 s M1 Max run at + `n_draft: 3`. Attributing it correctly dropped `:other` for that run from + 8.7 s to 0.17 s. +- **`LlamaCppEx.Model.n_embd_out/1` and `n_layer_nextn/1`** — the two numbers + that decide whether a GGUF can serve as an MTP target or head. `n_layer_nextn` + wraps a NIF that already existed but was only reachable through + `LlamaCppEx.NIF`. + - **`LlamaCppEx.RPC`** — register a remote machine's devices into the local device registry so a model's layers can live on another host. `add_server/1`, `add_servers/1`, `devices/0`, `ping/1`, `supported?/0`. diff --git a/Makefile b/Makefile index f7a96d4..44a8a4a 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= a94d563ed801d1da1b8c2432946de07d0231bb3d +LLAMA_COMMIT ?= 9e40df63ba151d771d8b247ac4011cf203337e99 # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. diff --git a/README.md b/README.md index 0948f35..1451e87 100644 --- a/README.md +++ b/README.md @@ -610,14 +610,42 @@ Multi-Token Prediction speculative decoding (upstream PR [#22673](https://github > `n_draft: 3` as the datacenter default the upstream 2× assumes, not as a value > that transfers. +> **Performance note: Qwen 3.8 27B (hybrid SSM, sidecar head).** This one is +> shaped by a cost the models above do not pay. Qwen 3.8 puts 48 SSM layers +> beside 16 attention ones, and a recurrent layer cannot be rolled back to an +> arbitrary position, so every speculative iteration snapshots and restores the +> whole recurrent state — 150 MiB of it at these sizes. `stats/1` reports that +> separately as `timing_us.ckpt`. Q4_K_M target + Q4_0 sidecar head, 120-token +> greedy generations: +> +> | `n_draft` | acceptance | M1 Max (Metal) | GB10 (DGX Spark) | +> |---|---|---|---| +> | 1 | 75.0% | 0.89× | **1.24×** | +> | 2 | 54–59% | 0.66× | 1.17× | +> | 3 | 40–44% | 0.68× | 1.09× | +> | 4 | 32% | — | 0.96× | +> | 5 | 30% | 0.56× | — | +> +> On Metal MTP is a net loss at every draft length: `ckpt` alone was 1.8 s of a +> 12.5 s run at `n_draft: 1` and 6.9 s of 16.3 s at `n_draft: 3`. On GB10 the +> same snapshot is cheap enough that `n_draft: 1` wins, and — unlike the MoE +> above, whose optimum was 2 — the optimum here is 1, monotonically decreasing +> after it. Measure `ckpt` against `total` before trusting speculation on any +> hybrid model. +> +> Both arms of the GB10 column are warm-cache numbers. A first run off cold page +> cache reads ~19 GB and reported a 4.18 tok/s baseline against 10.83 tok/s warm, +> which inverts the comparison entirely. + ### Other speculative types (EAGLE-3, DFlash, n-gram) -Upstream llama.cpp implements more speculative types behind the same `common_speculative` API — `draft-eagle3`, `draft-dflash` (block-diffusion drafting via a separate drafter GGUF), and several n-gram self-speculation modes. **This binding currently exposes only MTP**: `MTP.init/2` pins `COMMON_SPECULATIVE_TYPE_DRAFT_MTP` and builds both contexts from the same model, so there is no way to load a separate drafter model yet. +Upstream llama.cpp implements more speculative types behind the same `common_speculative` API — `draft-eagle3`, `draft-dflash` (block-diffusion drafting via a separate drafter GGUF), and several n-gram self-speculation modes, plus `--spec-default`, which stacks n-gram speculation on top of a model-based drafter. **This binding currently exposes only MTP**: `MTP.init/2` pins `COMMON_SPECULATIVE_TYPE_DRAFT_MTP`, so the other types and the combinations are not reachable from here. The draft *model* is no longer tied to the target, though — see `:draft_model` below for the target/sidecar split. > **DFlash status (July 2026, llama.cpp b9932).** DFlash runs end-to-end on Metal via upstream `llama-cli`/`llama-server`, but we measured it *slower* than plain decoding on Apple Silicon at small target sizes: Qwen3.5-4B target + z-lab 0.6B drafter on M4 Max reached 42 tok/s with DFlash vs 85 tok/s plain (greedy sampling; 30% draft acceptance, mean accepted run 2.8 — and stochastic sampling at `temp 0.8` collapses acceptance to ~7%). The Metal economics are the same as the MTP note above (wide verify batches are expensive), and the community drafter-GGUF conversions are still churning: of three third-party Qwen 4B drafter repos tested, only one loads with current upstream (the others hit the `dflash-draft` arch mismatch [#25116](https://github.com/ggml-org/llama.cpp/issues/25116) or lack the `target_layers` metadata added by the conversion refactor [#25110](https://github.com/ggml-org/llama.cpp/pull/25110)). Worth revisiting when the drafter format settles; the natural entry point is a `spec_type` + drafter-model option on `speculative_init`. ### Models with MTP heads +- [`ggml-org/Qwen3.8-27B-GGUF`](https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF) — **sidecar layout**: the target (`Qwen3.8-27B-Q4_K_M.gguf`, ~18 GB) carries *no* head, and `mtp-Qwen3.8-27B-Q4_0.gguf` (~1.6 GB) carries nothing else. Load both and pass the head as `draft_model:`. - [`ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-35B-A3B-MTP-GGUF) (recommended: `Q4_K_M`, ~21 GB) - [`ggml-org/Qwen3.6-27B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-27B-MTP-GGUF) - [`unsloth/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) @@ -632,7 +660,7 @@ Acceptance on a 0.8B target is not representative of production throughput — drafting is nearly as expensive as decoding at that size — so use these to exercise the path, not to measure it. -A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the GGUF must contain the MTP head's tensors. To check a file before loading it, look for a `*.nextn_predict_layers` key and `blk.N.nextn.*` tensors in its metadata. +A regular (non-MTP) quant will fail at `LlamaCppEx.MTP.init/2` — some GGUF in the pair must contain the MTP head's tensors. To check a file before loading it, look for a `*.nextn_predict_layers` key and `blk.N.nextn.*` tensors in its metadata. When the publisher ships the head separately (Qwen 3.8), that sidecar is the file with those tensors and the target legitimately has none; pass it as `draft_model:` rather than looking for a combined build. The model must also be loaded with `load_mtp: true` (see below). Upstream gates those tensors behind a load-time flag that defaults to off, and they cannot be attached afterwards, so `MTP.init/2` refuses a model loaded without it rather than letting the omission surface later as `verify decode failed: code=-1`. @@ -670,6 +698,49 @@ IO.puts("\nacceptance: #{Float.round(stats.acceptance_rate * 100, 1)}% " <> "throughput: #{Float.round(stats.tokens_per_sec, 1)} tok/s") ``` +#### Sidecar head: Qwen 3.8 (`-hf` target + `-hfd` draft) + +Same session API, two files. This is what upstream's +`llama serve -hf ggml-org/Qwen3.8-27B-GGUF --spec-type draft-mtp` resolves to +once it has downloaded the pair — the flag makes upstream fetch the `mtp-*` +sidecar and build the draft context against it instead of against the target. + +```elixir +:ok = LlamaCppEx.init() + +# The target carries no MTP head at all: Model.n_layer_nextn/1 returns 0 for it. +{:ok, target} = + LlamaCppEx.load_model( + Path.expand("~/Downloads/Qwen3.8-27B-Q4_K_M.gguf"), + n_gpu_layers: 999, + load_mtp: true + ) + +# The sidecar carries nothing *but* the head — ~1.6 GB against the target's 18. +{:ok, head} = + LlamaCppEx.load_model( + Path.expand("~/Downloads/mtp-Qwen3.8-27B-Q4_0.gguf"), + n_gpu_layers: 999, + load_mtp: true + ) + +# n_draft: 1 — see the Qwen 3.8 performance note above. Acceptance is 75% here +# and falls off fast, and every iteration pays a recurrent-state snapshot. +{:ok, mtp} = LlamaCppEx.MTP.init(target, draft_model: head, n_draft: 1, n_ctx: 8192) + +{:ok, text} = LlamaCppEx.MTP.generate(mtp, "Explain MTP in one paragraph.", max_tokens: 200) + +stats = LlamaCppEx.MTP.stats(mtp) +IO.puts("acceptance: #{Float.round(stats.acceptance_rate * 100, 1)}% " <> + "ckpt: #{div(stats.timing_us.ckpt, 1000)}ms of #{div(stats.timing_us.total, 1000)}ms") +``` + +`MTP.init/2` refuses the mismatched pairings before building anything: a sidecar +loaded without `load_mtp: true`, an ordinary model passed as `:draft_model`, and +a head whose hidden width does not match the target's — that last one because +upstream compares the two with a `GGML_ASSERT`, which aborts the VM rather than +failing the call. + #### Synchronous generate (collect to a string) ```elixir diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 7a6bb9f..cbcf932 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -662,6 +662,18 @@ int64_t model_n_embd(ErlNifEnv* env, fine::ResourcePtr model) { } FINE_NIF(model_n_embd, 0); +// Output-side embedding width, which is what the MTP draft head consumes. It is +// `n_embd` for every architecture in tree today, but upstream reads this one +// (speculative.cpp: "MTP input row width must match the target h_nextn width") +// and enforces the target/draft match with a GGML_ASSERT — an unconditional +// ggml_abort that takes the whole VM down rather than failing the call. A +// separate drafter GGUF is the only way to reach that assert, so MTP.init/2 +// compares this across the two models before it builds anything. +int64_t model_n_embd_out(ErlNifEnv* env, fine::ResourcePtr model) { + return llama_model_n_embd_out(model->model); +} +FINE_NIF(model_n_embd_out, 0); + // Number of MTP / "next-N" prediction layers the checkpoint carries. Zero means // the GGUF has no MTP head at all, which is a different situation from a model // loaded with load_mtp: false: no flag can recover it, only a different file. @@ -2027,28 +2039,31 @@ static ERL_NIF_TERM build_mtp_stats_map(ErlNifEnv* env, const LlamaSpeculative& uint64_t udraft = s.us_draft.load(std::memory_order_relaxed); uint64_t uverify = s.us_verify.load(std::memory_order_relaxed); uint64_t usample = s.us_sample.load(std::memory_order_relaxed); + uint64_t uckpt = s.us_ckpt.load(std::memory_order_relaxed); uint64_t uother = s.us_other.load(std::memory_order_relaxed); uint64_t utotal = s.us_total.load(std::memory_order_relaxed); double acceptance_rate = dgen > 0 ? (double)dacc / (double)dgen : 0.0; double tokens_per_sec = utotal > 0 ? (double)emitted * 1e6 / (double)utotal : 0.0; - ERL_NIF_TERM tk[5] = { + ERL_NIF_TERM tk[6] = { enif_make_atom(env, "draft"), enif_make_atom(env, "verify"), enif_make_atom(env, "sample"), + enif_make_atom(env, "ckpt"), enif_make_atom(env, "other"), enif_make_atom(env, "total"), }; - ERL_NIF_TERM tv[5] = { + ERL_NIF_TERM tv[6] = { enif_make_uint64(env, udraft), enif_make_uint64(env, uverify), enif_make_uint64(env, usample), + enif_make_uint64(env, uckpt), enif_make_uint64(env, uother), enif_make_uint64(env, utotal), }; ERL_NIF_TERM timing; - enif_make_map_from_arrays(env, tk, tv, 5, &timing); + enif_make_map_from_arrays(env, tk, tv, 6, &timing); ERL_NIF_TERM keys[8] = { enif_make_atom(env, "iters"), @@ -2323,6 +2338,7 @@ fine::Ok<> generate_mtp_tokens( // PART at init time, so llama_memory_seq_rm handles partial rejection // natively and the checkpoint would be pure overhead. if (sp.needs_ckpt) { + auto t_ck0 = std::chrono::steady_clock::now(); size_t sz_tgt = llama_state_seq_get_size_ext(ctx_tgt, seq_id, ckpt_flags); ckpt_tgt.resize(sz_tgt); if (sz_tgt > 0) { @@ -2333,6 +2349,13 @@ fine::Ok<> generate_mtp_tokens( if (sz_dft > 0) { llama_state_seq_get_data_ext(ctx_dft, ckpt_dft.data(), sz_dft, seq_id, ckpt_flags); } + // Bill this to us_ckpt, then slide the anchor past it so the same + // microseconds are not also counted as unaccounted "other". + auto t_ck1 = std::chrono::steady_clock::now(); + sp.us_ckpt.fetch_add( + std::chrono::duration_cast(t_ck1 - t_ck0).count(), + std::memory_order_relaxed); + t_anchor += (t_ck1 - t_ck0); } // 1. Generate drafts from the MTP head's current state. @@ -2468,6 +2491,7 @@ fine::Ok<> generate_mtp_tokens( // Hybrid model: partial seq_rm isn't supported, so restore // both contexts from the pre-iteration recurrent-state // snapshot and re-decode just the accepted prefix. + auto t_rs0 = std::chrono::steady_clock::now(); if (!ckpt_tgt.empty()) { llama_state_seq_set_data_ext(ctx_tgt, ckpt_tgt.data(), ckpt_tgt.size(), seq_id, ckpt_flags); @@ -2479,6 +2503,14 @@ fine::Ok<> generate_mtp_tokens( ckpt_dft.size(), seq_id, ckpt_flags); } soft_seq_rm(ctx_dft, seq_id, n_past); + auto t_rs1 = std::chrono::steady_clock::now(); + sp.us_ckpt.fetch_add( + std::chrono::duration_cast(t_rs1 - t_rs0).count(), + std::memory_order_relaxed); + // Same anchor slide as the save above: us_other is closed out + // from t_anchor at the end of the iter, so without this the + // restore would be billed twice. + t_anchor += (t_rs1 - t_rs0); // Re-decode the accepted tokens on the target so the next // iteration's draft starts from a consistent state. diff --git a/c_src/llama_cpp_ex/llama_nif.h b/c_src/llama_cpp_ex/llama_nif.h index 11bc844..11481b0 100644 --- a/c_src/llama_cpp_ex/llama_nif.h +++ b/c_src/llama_cpp_ex/llama_nif.h @@ -194,6 +194,13 @@ class LlamaSpeculative { std::atomic us_draft{0}; std::atomic us_verify{0}; std::atomic us_sample{0}; + // Recurrent-state save/restore, which only hybrid models pay (needs_ckpt). + // Broken out of us_other because on a model like Qwen 3.8 — 48 SSM layers + // beside 16 attention ones — the snapshot is over a hundred MiB and taken + // every iteration, which is enough on its own to make speculation a net + // loss. Attributing it to "other" hid that behind a bucket whose documented + // cause is GPU-sync waits. + std::atomic us_ckpt{0}; // Everything in the speculative iter NOT inside the three hot-path // timers above. On Metal this is dominated by implicit GPU-sync waits // from the previous iter's async verify decode (llama_decode returns diff --git a/lib/llama_cpp_ex/model.ex b/lib/llama_cpp_ex/model.ex index 4ac5b67..4044208 100644 --- a/lib/llama_cpp_ex/model.ex +++ b/lib/llama_cpp_ex/model.ex @@ -197,6 +197,24 @@ defmodule LlamaCppEx.Model do @spec n_embd(t()) :: integer() def n_embd(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_embd(ref) + @doc """ + Returns the output-side embedding width — the row width an MTP draft head + consumes. Equal to `n_embd/1` for every architecture currently in tree; it is + a distinct number because `LlamaCppEx.MTP` matches it across the target and a + separate drafter GGUF. + """ + @spec n_embd_out(t()) :: integer() + def n_embd_out(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_embd_out(ref) + + @doc """ + Returns the number of MTP / "next-N" prediction layers in the checkpoint, or + `0` when it carries no MTP head. Note this reports what the *file* contains; + the layers are only actually loaded when the model was opened with + `load_mtp: true`. + """ + @spec n_layer_nextn(t()) :: non_neg_integer() + def n_layer_nextn(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_n_layer_nextn(ref) + @doc "Returns a human-readable description of the model." @spec desc(t()) :: String.t() def desc(%__MODULE__{ref: ref}), do: LlamaCppEx.NIF.model_desc(ref) diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index fdbf2ad..1fa7304 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -2,9 +2,10 @@ defmodule LlamaCppEx.MTP do @moduledoc """ Multi-Token Prediction (MTP) speculative decoding. - Drives a target/draft speculative loop where the draft model is the MTP head - embedded in the same GGUF as the target. On Qwen 3.6 with `n_draft: 3` this - typically yields ~2x token-generation throughput at ~75% draft acceptance. + Drives a target/draft speculative loop where the draft model is an MTP head — + either embedded in the target GGUF, or shipped beside it as a sidecar file. On + Qwen 3.6 with `n_draft: 3` this typically yields ~2x token-generation + throughput at ~75% draft acceptance. ## Usage @@ -22,12 +23,40 @@ defmodule LlamaCppEx.MTP do stats = LlamaCppEx.MTP.stats(mtp) IO.puts("acceptance: \#{Float.round(stats.acceptance_rate * 100, 1)}%") - The model GGUF must contain MTP head layers (e.g. - `ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`) — look for `*.nextn_predict_layers` in its - metadata — **and** must be loaded with `load_mtp: true`. Upstream defaults that - flag to `false` so non-speculative callers do not pay for the head's tensors, - and the layers cannot be attached afterwards, so `init/2` refuses a model - loaded without it. + ## Where the MTP head lives + + The head is a set of `*.nextn_predict_layers` and comes in two shapes, and + either way the file carrying it must be loaded with `load_mtp: true`. Upstream + defaults that flag to `false` so non-speculative callers do not pay for the + head's tensors, and the layers cannot be attached afterwards, so `init/2` + refuses a model loaded without it. + + **In the target GGUF** (e.g. `ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`) — pass just + the model, as above. + + **In a sidecar GGUF** — pass it as `:draft_model`. This is how Qwen 3.8 ships: + `Qwen3.8-27B-Q4_K_M.gguf` carries no head at all (`n_layer_nextn == 0`) and + `mtp-Qwen3.8-27B-Q4_0.gguf` carries nothing else. It is the binding's + equivalent of upstream's `-hf -hfd --spec-type draft-mtp`. + + {:ok, target} = LlamaCppEx.load_model("Qwen3.8-27B-Q4_K_M.gguf", + n_gpu_layers: 999, load_mtp: true) + {:ok, head} = LlamaCppEx.load_model("mtp-Qwen3.8-27B-Q4_0.gguf", + n_gpu_layers: 999, load_mtp: true) + + {:ok, mtp} = LlamaCppEx.MTP.init(target, draft_model: head, n_draft: 1) + + > #### Speculation is not always a win on hybrid models {: .warning} + > + > A model that mixes recurrent (SSM) layers with attention ones — Qwen 3.8 is + > 48 SSM layers to 16 attention layers — cannot roll back part of a sequence + > natively, so every speculative iteration snapshots and restores the whole + > recurrent state. That state is over 100 MiB at Qwen 3.8's sizes, and the + > cost lands in `stats/1`'s `timing_us.ckpt`. Measured on an M1 Max (Metal, + > Q4_K_M target + Q4_0 head), MTP was a net *slowdown* at every draft length: + > 0.89x at `n_draft: 1` (75% acceptance) falling to 0.56x at `n_draft: 5` + > (30%). Check `timing_us.ckpt` against `timing_us.total` before assuming + > speculation is helping, and prefer small `n_draft` when it is not. Upstream currently requires `n_parallel = 1` for MTP. This module reflects that — a single MTP session decodes one sequence at a time. Reuse the same @@ -46,12 +75,12 @@ defmodule LlamaCppEx.MTP do > a fresh session with `init/2`, before using the session again. MTP is the only speculative type this binding exposes. Upstream llama.cpp - also implements EAGLE-3, DFlash (block-diffusion drafting via a separate - drafter GGUF), and n-gram self-speculation behind the same - `common_speculative` API, but the NIF pins the MTP type and both contexts - are built from the same model, so a separate drafter model cannot be loaded - yet. See the "Speculative decoding" section of the README for the current - status of DFlash on Apple Silicon. + also implements EAGLE-3, DFlash (block-diffusion drafting), n-gram + self-speculation and combinations of them behind the same + `common_speculative` API; the NIF pins the MTP type, so `--spec-default`-style + stacking of n-gram speculation on top of MTP is not reachable from here. See + the "Speculative decoding" section of the README for the current status of + DFlash on Apple Silicon. """ alias LlamaCppEx.{Context, Model, Sampler, Tokenizer} @@ -81,6 +110,12 @@ defmodule LlamaCppEx.MTP do ## Options + * `:draft_model` - A separate `LlamaCppEx.Model` holding the MTP head, for + checkpoints that ship it as a sidecar GGUF rather than inside the target + file (Qwen 3.8 is the current example: `Qwen3.8-27B-Q4_K_M.gguf` plus + `mtp-Qwen3.8-27B-Q4_0.gguf`). It must be loaded with `load_mtp: true`. + Defaults to `nil`, meaning the head is expected inside the target model + and the draft context is built against it. * `:n_draft` - Max draft tokens generated per iteration. Defaults to `3`. Larger values mean fewer model forward passes but lower per-iteration acceptance; 2–4 is the sweet spot in practice. @@ -94,12 +129,23 @@ defmodule LlamaCppEx.MTP do @spec init(Model.t(), keyword()) :: {:ok, t()} | {:error, term()} def init(%Model{} = model, opts \\ []) do n_draft = Keyword.get(opts, :n_draft, 3) + draft_model = Keyword.get(opts, :draft_model) - cond do - not (is_integer(n_draft) and n_draft > 0) -> - {:error, ":n_draft must be a positive integer"} + with :ok <- validate_n_draft(n_draft), + {:ok, head_model} <- validate_head(model, draft_model) do + do_init(model, head_model, opts, n_draft) + end + end + + defp validate_n_draft(n) when is_integer(n) and n > 0, do: :ok + defp validate_n_draft(_), do: {:error, ":n_draft must be a positive integer"} - not model.load_mtp -> + # Which model the draft context is built from, and whether it can actually + # serve as one. Two shapes reach this: the head inside the target GGUF (no + # `:draft_model`), and the head in a sidecar GGUF alongside it. + defp validate_head(%Model{} = target, nil) do + cond do + not target.load_mtp -> # Upstream gates the MTP head's tensors behind a load-time flag that # defaults to false (#26296), and nothing downstream notices they are # missing: both contexts build and `common_speculative_init` returns ok, @@ -110,25 +156,60 @@ defmodule LlamaCppEx.MTP do "model was loaded without load_mtp: true, so its MTP head layers are " <> "absent; reload it with LlamaCppEx.load_model(path, load_mtp: true)"} - LlamaCppEx.NIF.model_n_layer_nextn(model.ref) == 0 -> + Model.n_layer_nextn(target) == 0 -> # Distinct from the branch above and not fixable by any flag: the # checkpoint simply has no MTP head. llama.cpp logs "context type MTP # requested but model doesn't contain MTP layers" and returns null, which # reaches the caller as a bare "failed to create context" with the real # reason buried in engine output the caller may not even be showing. # Most GGUF conversions of an MTP-capable model drop the head; the - # publisher usually ships it as a separate `-MTP` repository. + # publisher usually ships it as a separate repository or sidecar file, + # which is what `:draft_model` is for. {:error, "this GGUF contains no MTP head (0 nextn layers), so MTP speculative " <> - "decoding is unavailable for it; use an MTP-preserving conversion of " <> - "the model, which publishers typically ship as a separate -MTP build"} + "decoding is unavailable for it; either use an MTP-preserving " <> + "conversion of the model, or pass the publisher's sidecar MTP GGUF " <> + "as draft_model: (loaded with load_mtp: true)"} true -> - do_init(model, opts, n_draft) + {:ok, target} end end - defp do_init(model, opts, n_draft) do + defp validate_head(%Model{} = target, %Model{} = draft) do + cond do + not draft.load_mtp -> + {:error, + "draft_model was loaded without load_mtp: true, so the sidecar's MTP " <> + "head layers are absent; reload it with " <> + "LlamaCppEx.load_model(path, load_mtp: true)"} + + Model.n_layer_nextn(draft) == 0 -> + {:error, + "draft_model contains no MTP head (0 nextn layers) — it is an ordinary " <> + "model, not an MTP sidecar; pass the publisher's mtp-* GGUF instead"} + + # Upstream compares these two widths with a GGML_ASSERT in the draft-mtp + # constructor, and GGML_ASSERT is an unconditional ggml_abort: a mismatched + # pair would take the VM down instead of returning an error. Only a + # separate drafter can be mismatched, so this is checked on this branch + # alone, and before any context is built. + Model.n_embd_out(draft) != Model.n_embd_out(target) -> + {:error, + "draft_model hidden width #{Model.n_embd_out(draft)} does not match the " <> + "target's #{Model.n_embd_out(target)}; the sidecar belongs to a " <> + "different model than the one it was paired with"} + + true -> + {:ok, draft} + end + end + + defp validate_head(_target, other) do + {:error, ":draft_model must be a LlamaCppEx.Model, got: #{inspect(other)}"} + end + + defp do_init(model, head_model, opts, n_draft) do base_ctx_opts = forwardable_context_opts(opts) main_opts = Keyword.merge(base_ctx_opts, ctx_type: :default) # Match upstream server: MTP draft context is created with n_rs_seq=0. @@ -137,7 +218,7 @@ defmodule LlamaCppEx.MTP do draft_opts = Keyword.merge(base_ctx_opts, ctx_type: :mtp, n_rs_seq: 0) with {:ok, main_ctx} <- Context.create(model, main_opts), - {:ok, mtp_ctx} <- Context.create(model, draft_opts), + {:ok, mtp_ctx} <- Context.create(head_model, draft_opts), {:ok, spec_ref} <- LlamaCppEx.NIF.speculative_init(main_ctx.ref, mtp_ctx.ref, n_draft) do {:ok, @@ -314,7 +395,12 @@ defmodule LlamaCppEx.MTP do * `:acceptance_rate` - `drafts_accepted / drafts_generated` (0.0–1.0) * `:tokens_emitted` - tokens streamed back to the caller * `:tokens_per_sec` - throughput over the active generation window - * `:timing_us` - `%{draft: μs, verify: μs, sample: μs, total: μs}` + * `:timing_us` - `%{draft: μs, verify: μs, sample: μs, ckpt: μs, other: μs, + total: μs}`. `:ckpt` is the recurrent-state save/restore that only hybrid + models pay and is zero elsewhere; on Qwen 3.8 it is large enough to decide + whether speculation helps at all. `:other` is whatever falls outside the + named buckets, dominated on Metal by GPU-sync waits from the previous + iteration's async verify decode. * `:n_draft` - max draft length configured at init Counters are cumulative across all `stream/3` / `generate/3` calls on this diff --git a/lib/llama_cpp_ex/nif.ex b/lib/llama_cpp_ex/nif.ex index b816e7f..99f6363 100644 --- a/lib/llama_cpp_ex/nif.ex +++ b/lib/llama_cpp_ex/nif.ex @@ -51,6 +51,7 @@ defmodule LlamaCppEx.NIF do def model_n_ctx_train(_model), do: :erlang.nif_error(:not_loaded) def model_n_embd(_model), do: :erlang.nif_error(:not_loaded) + def model_n_embd_out(_model), do: :erlang.nif_error(:not_loaded) def model_n_layer_nextn(_model), do: :erlang.nif_error(:not_loaded) def model_desc(_model), do: :erlang.nif_error(:not_loaded) def model_size(_model), do: :erlang.nif_error(:not_loaded) diff --git a/test/mtp_model_test.exs b/test/mtp_model_test.exs index fa0e0a7..ac2196a 100644 --- a/test/mtp_model_test.exs +++ b/test/mtp_model_test.exs @@ -201,3 +201,126 @@ defmodule LlamaCppEx.MTPCancelTest do assert {:ok, _} = MTP.generate(mtp, "2 + 2 =", max_tokens: 4, temp: 0.0) end end + +defmodule LlamaCppEx.MTPSidecarTest do + # The target/draft split, which is how Qwen 3.8 ships MTP: the target GGUF + # carries zero nextn layers and a separate `mtp-*.gguf` carries the head. This + # needs two files rather than one, so it gates on its own tag and its own env + # var — `--include mtp` has only the single-file model: + # + # GGML_METAL_NO_RESIDENCY=1 LLAMA_BACKEND=auto \ + # LLAMA_SMOKE_MTP_MODEL=/path/to/Qwen3.8-27B-Q4_K_M.gguf \ + # LLAMA_SMOKE_MTP_DRAFT_MODEL=/path/to/mtp-Qwen3.8-27B-Q4_0.gguf \ + # mix test --include mtp_sidecar + # + # async: false, and one session for the module, for the same reason + # MTPModelTest says: one GPU, and each session reserves two contexts. + use ExUnit.Case, async: false + + alias LlamaCppEx.MTP + + @moduletag :mtp_sidecar + @moduletag timeout: 300_000 + + setup_all do + :ok = LlamaCppEx.init() + + {:ok, target} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp), + n_gpu_layers: -1, + load_mtp: true + ) + + {:ok, draft} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp_draft), + n_gpu_layers: -1, + load_mtp: true + ) + + {:ok, session} = MTP.init(target, draft_model: draft, n_ctx: 2048, n_draft: 1) + + %{target: target, draft: draft, session: session} + end + + test "the sidecar carries the head and the target does not", %{target: t, draft: d} do + # The premise of the whole pairing. If a future conversion moves the head + # into the target this test is the thing that notices. + assert LlamaCppEx.Model.n_layer_nextn(d) > 0 + assert LlamaCppEx.Model.n_layer_nextn(t) == 0 + + # Upstream compares these two with a GGML_ASSERT, which aborts the VM rather + # than failing a call, so the pair must agree before init/2 builds anything. + assert LlamaCppEx.Model.n_embd_out(d) == LlamaCppEx.Model.n_embd_out(t) + end + + test "init/2 builds a session from the pair", %{session: session} do + assert %LlamaCppEx.Context{} = session.main_ctx + assert %LlamaCppEx.Context{} = session.mtp_ctx + assert is_reference(session.spec_ref) + end + + test "the same target is refused without the sidecar", %{target: t} do + # Not a redundant restatement of the unit test: there the nextn count comes + # from a hand-built struct, here it is read off the real file. + assert {:error, message} = MTP.init(t, n_ctx: 512) + assert message =~ "no MTP head" + assert message =~ "draft_model" + end + + test "generate/3 produces text and the head actually drafts", %{session: session} do + before = MTP.stats(session) + + assert {:ok, text} = MTP.generate(session, "2 + 2 =", max_tokens: 16, temp: 0.0) + assert is_binary(text) and text != "" + + now = MTP.stats(session) + + # Deltas, not absolutes: the session is shared across this module's tests. + assert now.drafts_generated > before.drafts_generated, + "the sidecar head proposed no drafts at all" + + assert now.tokens_emitted > before.tokens_emitted + end + + # A hybrid target (Qwen 3.8: SSM layers beside attention ones) cannot roll back + # part of a sequence natively, so the loop snapshots the recurrent state every + # iteration. That cost is the difference between speculation paying off and not, + # so it gets its own bucket rather than hiding inside :other. + test "timing_us reports a ckpt bucket", %{session: session} do + assert {:ok, _} = MTP.generate(session, "Count to ten:", max_tokens: 24, temp: 0.0) + + timing = MTP.stats(session).timing_us + + for key <- [:draft, :verify, :sample, :ckpt, :other, :total] do + assert Map.has_key?(timing, key), "timing_us is missing #{inspect(key)}" + end + + assert timing.ckpt > 0, + "a hybrid target should have paid for at least one recurrent-state snapshot" + + # The named buckets are carved out of total, never billed twice on top of it. + assert timing.draft + timing.verify + timing.sample + timing.ckpt <= timing.total + end + + test "greedy output matches plain greedy decode on the target", %{ + target: target, + session: session + } do + # Speculation is exactness-preserving in principle. In practice ggml selects + # a different matmul kernel by batch row count — ggml-metal-ops.cpp picks the + # mul_mv_ext path for Q4_K only at ne11 >= 4, and its r1ptg by ne11 — and the + # MTP verify batch is 1 + n_draft rows wide, so at a position where the top + # two logits are within rounding error the argmax can differ from a 1-row + # plain decode. That is not a bug and it is demonstrable *without* MTP: on + # Qwen 3.8 / M1 Max, feeding one fixed prefix to plain greedy decode gives + # " computational" at 1-3 rows and " latency" at 4+. So compare a short + # continuation, where no such near-tie has come up. + prompt = "The capital of France is" + opts = [max_tokens: 8, temp: 0.0] + + assert {:ok, spec} = MTP.generate(session, prompt, opts) + assert {:ok, plain} = LlamaCppEx.generate(target, prompt, opts ++ [n_ctx: 2048]) + + assert spec == plain + end +end diff --git a/test/mtp_test.exs b/test/mtp_test.exs index b55f8d6..43ceb36 100644 --- a/test/mtp_test.exs +++ b/test/mtp_test.exs @@ -58,6 +58,49 @@ defmodule LlamaCppEx.MTPTest do end end + # Qwen 3.8 ships the MTP head as a sidecar GGUF: the target carries zero nextn + # layers and the head file carries nothing else, so the pair only works if the + # draft context can be built from a *different* model than the target. These + # guards all run before any context is created, so a nil ref never reaches the + # NIF — the ones that must read model metadata (nextn count, hidden width) need + # a real file and live in MTPModelTest. + describe "init/2 with a separate :draft_model" do + @unloaded %LlamaCppEx.Model{ref: nil} + @loaded_mtp %LlamaCppEx.Model{ref: nil, load_mtp: true} + + test "refuses a sidecar loaded without load_mtp: true, naming the remedy" do + assert {:error, message} = MTP.init(@loaded_mtp, draft_model: @unloaded) + assert message =~ "draft_model was loaded without load_mtp: true" + assert message =~ "reload it" + end + + test "the sidecar's flag is what matters, not the target's" do + # The target legitimately has no head of its own in this shape, so its own + # load_mtp is not what gates the session — a target without the flag and a + # sidecar with it must get past the flag check rather than be refused for + # the target's sake. Reaching the metadata read (which a nil ref cannot + # survive) is the proof it got that far. + assert catch_error(MTP.init(@unloaded, draft_model: @loaded_mtp)) + end + + test "rejects a :draft_model that is not a Model" do + for bad <- ["mtp.gguf", :mtp, 42, %{ref: nil}] do + assert {:error, message} = MTP.init(@loaded_mtp, draft_model: bad) + assert message =~ ":draft_model must be a LlamaCppEx.Model" + end + end + + test "nil :draft_model is the in-target-head path, not a bad argument" do + # Explicitly passing nil must behave exactly like omitting the option. + assert MTP.init(@unloaded, draft_model: nil) == MTP.init(@unloaded) + end + + test "n_draft is still validated first, so its error is not masked" do + assert MTP.init(@loaded_mtp, draft_model: @unloaded, n_draft: 0) == + {:error, ":n_draft must be a positive integer"} + end + end + # A checkpoint with no MTP head is a different failure from a model loaded # without the flag, and no flag recovers it. Most GGUF conversions of an # MTP-capable model drop the head — unsloth's Qwen3.6-35B-A3B-UD-Q4_K_XL has diff --git a/test/support/test_models.exs b/test/support/test_models.exs index 1fe964e..1d4ec8a 100644 --- a/test/support/test_models.exs +++ b/test/support/test_models.exs @@ -10,12 +10,17 @@ defmodule LlamaCppEx.TestModels do @vars %{ gen: {"LLAMA_SMOKE_GEN_MODEL", "a chat/instruct"}, emb: {"LLAMA_SMOKE_EMB_MODEL", "an embedding"}, - mtp: {"LLAMA_SMOKE_MTP_MODEL", "an MTP-enabled"} + mtp: {"LLAMA_SMOKE_MTP_MODEL", "an MTP-enabled"}, + # The head-only sidecar half of a target/draft pair, e.g. Qwen 3.8's + # mtp-Qwen3.8-27B-Q4_0.gguf. Its own env var rather than a second use of + # :mtp because the two files are provisioned independently and the sidecar is + # useless without the target it was built for. + mtp_draft: {"LLAMA_SMOKE_MTP_DRAFT_MODEL", "an MTP sidecar (head-only)"} } @kinds Map.keys(@vars) - @type kind :: :gen | :emb | :mtp + @type kind :: :gen | :emb | :mtp | :mtp_draft @doc "Name of the environment variable holding the model path for `kind`." @spec var(kind()) :: String.t() diff --git a/test/test_helper.exs b/test/test_helper.exs index 1279d69..95a815a 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -1,9 +1,14 @@ # Tests that load real GGUF models and run inference are excluded by default, -# behind four opt-in tags. Each tag names the model it needs: +# behind a set of opt-in tags. Each tag names the model it needs: # # :smoke — generation/chat/grammar/server paths; needs LLAMA_SMOKE_GEN_MODEL # :embeddings — embedding paths; needs LLAMA_SMOKE_EMB_MODEL # :mtp — MTP speculative decoding; needs LLAMA_SMOKE_MTP_MODEL +# :mtp_sidecar — MTP with the head in a *separate* sidecar GGUF (Qwen 3.8's +# shape), so it needs a pair: LLAMA_SMOKE_MTP_MODEL for the +# target and LLAMA_SMOKE_MTP_DRAFT_MODEL for the head. Its own +# tag rather than `:mtp` because that tag's single-file model +# cannot satisfy it. # :mtp_cancel — one known-broken MTP test, excluded on its own tag so that # `--include mtp` is green. It does not fail, it aborts the VM: # cancelling an MTP stream is fire-and-forget, so reusing the @@ -39,6 +44,11 @@ # LLAMA_SMOKE_MTP_MODEL=/path/to/mtp-model.gguf \ # mix test --include mtp # +# GGML_METAL_NO_RESIDENCY=1 \ +# LLAMA_SMOKE_MTP_MODEL=/path/to/Qwen3.8-27B-Q4_K_M.gguf \ +# LLAMA_SMOKE_MTP_DRAFT_MODEL=/path/to/mtp-Qwen3.8-27B-Q4_0.gguf \ +# mix test --include mtp_sidecar +# # LLAMA_RPC=1 mix compile # LLAMA_RPC_ENDPOINT=10.100.64.2:50052 mix test --include rpc_live # @@ -67,4 +77,4 @@ Code.require_file("support/test_slots.exs", __DIR__) # default run quiet, and rpc_test.exs additionally carries a compile-time `skip:` # so an explicit `--include rpc_live` without a worker skips rather than fails # (`--include` beats `--exclude`, so the exclusion alone cannot do that). -ExUnit.start(exclude: [:smoke, :embeddings, :slow, :mtp, :mtp_cancel, :rpc_live]) +ExUnit.start(exclude: [:smoke, :embeddings, :slow, :mtp, :mtp_cancel, :mtp_sidecar, :rpc_live]) diff --git a/vendor/llama.cpp b/vendor/llama.cpp index a94d563..9e40df6 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit a94d563ed801d1da1b8c2432946de07d0231bb3d +Subproject commit 9e40df63ba151d771d8b247ac4011cf203337e99 From fb7d275783201b979076682d345a9dca3bfeb313 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Fri, 14 Aug 2026 23:26:38 -0400 Subject: [PATCH 2/2] MTP: fix the no-head test's assertion on the reworded remedy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `:smoke` guard test asserted `message =~ "-MTP"`, which was the old message's way of naming the remedy ("publishers typically ship as a separate -MTP build"). Adding `:draft_model` reworded that message to offer both routes out — an MTP-preserving conversion, or the publisher's head-only sidecar — and dropped the bare "-MTP" token, so the assertion failed in CI while the guard itself was working correctly. Assert on what the test is actually for: that the message names the remedies rather than only the diagnosis. Both routes are now checked, so neither can be dropped from the message unnoticed, and the assertion no longer pins one incidental phrasing. Reproduced with the CI job's own models and flags (SmolLM2-135M-Instruct-Q8_0, `--include smoke --include embeddings --include slow`): 556/558 before, 557/558 after. The remaining local failure is `embed_batch matches per-text embed`, and it is neither new nor CI's. It reports byte-identical 0.0024415459483861923 against a 1.0e-3 tolerance on a Metal build, on a CPU build, and with vendor/llama.cpp reverted to the pre-bump b10423 — so it is not backend numerics and not the submodule bump. macOS compiles ggml-blas against Accelerate for every backend, so batched embeddings go through Accelerate sgemm while the single-text reference path uses ggml's own kernel; Linux CI has no Accelerate and takes the same kernel on both sides, which is why CI reported one failure and not two. --- test/mtp_test.exs | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/test/mtp_test.exs b/test/mtp_test.exs index 43ceb36..58ca1d6 100644 --- a/test/mtp_test.exs +++ b/test/mtp_test.exs @@ -102,10 +102,12 @@ defmodule LlamaCppEx.MTPTest do end # A checkpoint with no MTP head is a different failure from a model loaded - # without the flag, and no flag recovers it. Most GGUF conversions of an - # MTP-capable model drop the head — unsloth's Qwen3.6-35B-A3B-UD-Q4_K_XL has - # zero nextn layers while their separate -MTP build of the same model has - # them — so this is the case a user actually lands on first. + # without the flag, and no flag recovers it — only a different file. There are + # two such files, and the message has to name both: an MTP-preserving + # conversion of the whole model (unsloth's Qwen3.6-35B-A3B-UD-Q4_K_XL has zero + # nextn layers while their separate -MTP build of it has them), or the + # publisher's head-only sidecar passed as `:draft_model` (Qwen 3.8 ships only + # that shape). This is the case a user actually lands on first. # # One gate tag (`:smoke`), never `:mtp` as well: the generation model is an # ordinary checkpoint, which is exactly what makes it the right fixture here. @@ -125,7 +127,10 @@ defmodule LlamaCppEx.MTPTest do test "refuses with the reason, not 'failed to create context'", %{model: model} do assert {:error, message} = MTP.init(model, n_draft: 3) assert message =~ "no MTP head" - assert message =~ "-MTP" + # Naming the remedy is the point of the guard, so assert on both routes out + # rather than on any one phrasing of the diagnosis. + assert message =~ "MTP-preserving conversion" + assert message =~ "draft_model" # The bare context error is what this guard exists to replace. refute message =~ "failed to create context" end