docs: record MTP milestone in README News and ROADMAP (#253) - #304
Merged
Conversation
MTP self-speculative decoding (CPU+CUDA+CLI+server) landed across several PRs this session (#253/#286/#295) but only SPECULATIVE.md was updated at the time; add the README News entry and ROADMAP step-43 addendum per the project's milestone-doc-sync convention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Documentation-only PR that records the Multi-Token Prediction (MTP) self-speculative decoding milestone in the project’s public-facing progress surfaces (README News + ROADMAP), aligning with the repo’s “sync README/ROADMAP for significant milestones” convention.
Changes:
- Add a new README News entry for MTP self-speculative decoding (CPU/CUDA + CLI/server wiring + related engine fix).
- Add a Step 43 (“Speculative decoding”) addendum in
docs/ROADMAP.mdreferencing the same follow-up work.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| README.md | Adds a new News bullet documenting the MTP milestone and associated wiring/fixes. |
| docs/ROADMAP.md | Extends Step 43 with an addendum noting MTP self-speculative decoding and related rollback fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| ## News | ||
|
|
||
| - **2026-08** — **MTP (Multi-Token Prediction) self-speculative decoding — CPU, CUDA, CLI, and server.** ([#253](https://github.com/jamesburton/dotLLM/issues/253)) Extends the existing two-model speculative-decode infrastructure with single-model self-speculation: a small MTP head predicts several future tokens per step, verified in one batched forward against the base model, reusing the DeepSeek-V3 "NextN" GGUF tensor convention (`blk.{n}.nextn.{eh_proj,enorm,hnorm,embed_tokens}`) — first seen at scale in `froggeric/Qwen3.6-27B-MTP-GGUF`, sharing `Architecture.Qwen3HybridDense` with PrismML's Bonsai-27B. CUDA (`CudaMtpState`) composes entirely from already-proven kernels (RmsNorm, RoPE, attention, SwiGLU, the `Gemm` dispatcher) rather than new kernel work. CLI wiring: `--no-mtp` opt-out for `run`/`chat` (auto-detects and enables by default); server wiring: `--mtp` opt-in for `serve` (disables the continuous-batch scheduler while active, mirroring `--speculative-model`'s existing restriction), surfaced via `mtp_active` in `/props`. Along the way, fixed a correctness gap in GDN (Gated DeltaNet) recurrent architectures: unlike KV-cache, GDN's recurrent state has no position addressing, so a rejected draft token's effect couldn't be "un-reached" by truncation alone ([#287](https://github.com/jamesburton/dotLLM/issues/287)) — new `IModel.CheckpointRecurrentState()`/`RestoreRecurrentState()` (CPU + CUDA) make rollback safe for both `SpeculativeDecoder` and `MtpSpeculativeDecoder` on hybrid GDN/attention models. Real end-to-end validation via `dotllm run` against the real `froggeric/Qwen3.6-27B-MTP-GGUF` (Q4_K_M) checkpoint. |
| | 55 | **Native AOT (experimental)** :white_check_mark: | Trimming-safe deployment via .NET Native AOT. Audit `[DynamicallyAccessedMembers]` annotations, source-generated JSON serialization, replace reflection-based patterns. `rd.xml` for preserved types. Goal: single-file `dotllm` binary with instant startup (~50ms vs ~500ms JIT). Mark experimental — some features (runtime LoRA loading, source generators) may require JIT fallback. | 34 | | ||
| | 36 | **Paged KV-cache** :white_check_mark: | Block-based KV-cache memory management: block pool, block tables, free list, reference counting, copy-on-write. Staging-buffer gather for attention kernel compatibility (not PagedAttention — kernels still operate on contiguous buffers). Foundation for prefix sharing (37, Phase 9) and speculative decoding (43, benefits from cheap rollback/fork but not strictly required). `--paged` (opt-in for CLI), `--no-paged` (opt-out for serve). `--no-ui` for API-only hosting. | 7 | | ||
| | 43 | **Speculative decoding** :white_check_mark: | `ISpeculativeDecoder`. Draft-verify-accept loop with modified rejection sampling. KV-cache rollback. Constraint state rollback via `IDecodingConstraint.Clone()`. Benefits from paged KV (36) for cheap block-level rollback and CoW fork for draft branching, but can fall back to `SimpleKvCache.SetCurrentLength()` truncation. | 36 | | ||
| | 43 | **Speculative decoding** :white_check_mark: | `ISpeculativeDecoder`. Draft-verify-accept loop with modified rejection sampling. KV-cache rollback. Constraint state rollback via `IDecodingConstraint.Clone()`. Benefits from paged KV (36) for cheap block-level rollback and CoW fork for draft branching, but can fall back to `SimpleKvCache.SetCurrentLength()` truncation. **Follow-up (#253):** single-model self-speculative decoding via MTP (Multi-Token Prediction) heads — `MtpSpeculativeDecoder`, CPU + CUDA (`CudaMtpState`), CLI (`--no-mtp` opt-out for `run`/`chat`) and server (`--mtp` opt-in for `serve`, since it disables continuous-batch scheduling) wiring. GDN recurrent-state checkpoint/restore (#287) makes rollback safe for hybrid GDN/attention architectures where KV-cache truncation alone doesn't cover recurrent state. | 36 | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Docs-only, per CLAUDE.md rule 7 (README/ROADMAP sync for significant milestones) — this was missed at the time since only docs/SPECULATIVE.md was updated when the feature landed.
Test plan