bench(mtp): add standalone MTP self-speculative decode benchmark harness (#253)

[jamesburton]

Adds `mtp-bench`, a standalone CPU benchmark harness (`benchmarks/DotLLM.Benchmarks/Profile/MtpBenchProfile.cs`) that drives `MtpSpeculativeDecoder` directly against a real loaded model. Needed because MTP has no CLI/server wiring yet (tracked follow-up in `docs/SPECULATIVE.md`) — this exists purely for real-model measurement without touching `TextGenerator`'s production decode loops.

Runs interleaved A/B trials (baseline greedy decode vs. MTP self-speculative decode, alternating per round to share thermal/scheduling conditions), reports median tok/s and MTP acceptance rate. Gracefully falls back to baseline-only reporting when the loaded GGUF has no MTP head — verified via a smoke test against a non-MTP model (SmolLM2-135M): correct detection, correct fallback message, correct baseline numbers, no crash.

Real-model results produced with this harness (first-ever validation of dotLLM's MTP against a genuine Qwen3.6-27B-MTP-GGUF)

Full comparison against llama.cpp (mainline, commit `3653e6d`, 2026-08-07) — fixture `froggeric/Qwen3.6-27B-MTP-GGUF` Q4_K_M:

Configuration	Decode tok/s	vs baseline	Notes
llama.cpp CPU, baseline	0.343	1.00x
llama.cpp CPU, MTP	0.318	0.93x	accept ~94%
llama.cpp CUDA, baseline	0.742	1.00x	ngl=37/66
llama.cpp CUDA, MTP	0.785	1.058x	accept ~94%, ngl=37/66
dotLLM CPU, baseline	0.13	1.00x	via this harness
dotLLM CPU, MTP	0.07	0.57x	accept 92.6%
dotLLM CUDA, baseline (no MTP)	0.12	—	degraded, see caveat

Headline finding: dotLLM's MTP acceptance rate (92.6%) closely tracks llama.cpp's own (~94%) on the identical model/prompt — strong cross-validation that the MTP head math (#285/#286) is correct. Neither engine shows a CPU win on this specific hardware (an old SSE-only Xeon Westmere, compute- not bandwidth-bound at decode — MTP's core assumption doesn't hold there); llama.cpp shows a modest real CUDA win (1.058x, diluted by partial GPU offload).

Caveats: single-run (not the project's usual interleaved-repeat convention — this CPU is too slow, 3-14s/token, to make that practical), prompt token counts not perfectly matched between engines (llama.cpp's new chat-first CLI wraps in a chat template, dotLLM's harness uses raw completion), dotLLM CUDA baseline number is real but comes from a degraded path (see the filed follow-up issue below) — not a fair GPU comparison yet.

Real bug found while getting even the baseline CUDA number: `HybridTransformerModel.LoadFromGguf` throws `KeyNotFoundException` for `Qwen3HybridDense` on the partial-GPU-layer-offload path — not architecture-aware the way the CPU and full-offload CUDA paths are. Worked around with full 64-layer offload, which only "succeeds" because Windows WDDM silently overcommits the 16.8GB weights past the 3060's 12GB VRAM. Filing separately.

Refs #253.

[Copilot]

Pull request overview

Adds a new standalone benchmark entrypoint (mtp-bench) to measure Multi-Token Prediction (MTP) self-speculative decoding directly against a real GGUF-loaded CPU model, without integrating into the production TextGenerator loops.

Changes:

• Adds an mtp-bench subcommand routing in the benchmarks Program.
• Introduces MtpBenchProfile, an interleaved A/B harness (baseline greedy vs. MTP) that reports median tok/s and an acceptance proxy, with baseline-only behavior when MTP is unsupported.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
benchmarks/DotLLM.Benchmarks/Program.cs	Adds mtp-bench subcommand routing to the new benchmark harness.
benchmarks/DotLLM.Benchmarks/Profile/MtpBenchProfile.cs	Implements the standalone MTP benchmark runner with interleaved trials, summary reporting, and model loading/tokenization.

Suppressed comments (3)

benchmarks/DotLLM.Benchmarks/Profile/MtpBenchProfile.cs:78

• If tokenization yields 0 tokens (e.g., empty --prompt) or --prefill is 0, prefillLen becomes 0 and the subsequent Argmax call uses row (prefillLen - 1) = -1, which will compute an invalid pointer offset. Guard against empty prompts/prefill before slicing and running the model.

        int[] promptTokens = tokenizer.Encode(prompt).ToArray();
        int prefillLen = Math.Min(promptTokens.Length, prefillTokens);
        int[] prefill = promptTokens[..prefillLen];

benchmarks/DotLLM.Benchmarks/Profile/MtpBenchProfile.cs:200

• Same as baseline prefill: this prefill forward pass only needs the last row's logits, but it currently uses the non-hinted overload. Consider calling Forward(..., lastTokenLogitsOnly: true) and selecting the last row via t.Shape[0]-1 so the code works whether or not the model honors the hint.

        using (var t = model.Forward(prefill, prefillPositions, deviceId: -1, kv))
            lastToken = ArgmaxFirstRow(t, prefillLen - 1, config.VocabSize);

benchmarks/DotLLM.Benchmarks/Profile/MtpBenchProfile.cs:123

• The summary computes speedup as mtpMedian / baseMedian without guarding against baseMedian being 0/NaN (e.g., --decode 0 yields 0 tok/s, or repeats=0 would leave baselineResults empty). This can print Infinity/NaN or throw in downstream formatting. Consider printing "N/A" when the baseline median is not positive.

            double mtpMedian = Median(mtpResults.Select(r => r.TokPerSec).ToArray());
            double mtpPrefillMedian = Median(mtpResults.Select(r => r.PrefillTokPerSec).ToArray());
            double acceptMedian = Median(mtpResults.Select(r => r.AcceptanceRate).ToArray());
            Console.WriteLine($"  MTP decode                = {mtpMedian,7:F2} tok/s   prefill = {mtpPrefillMedian,7:F1} tok/s   speedup = {mtpMedian / baseMedian,5:F2}x   acceptance = {acceptMedian:P1}");

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.