multi-token-prediction

llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).

DMax: Aggressive Parallel Decoding for dLLMs

Fused TBQ4 Flash Attention + MTP + Shared Tensors + Qwen35 SWA Hybrid for llama.cpp — 82+ tok/s, lossless 4.25 bpv KV cache, SWA-bounded deep-context decode (w/ long-range recall) on RTX 4090

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

Curated collection of research on the limitations of next-token prediction and methods that go beyond it.

Run Hermes Agent + Claude Code locally on llama.cpp — zero API costs. A 4h / 7M-token session that would have cost $94 on Claude Opus 4.7

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Docker image. Decode-optimized to ~91–99 tok/s (min ~91 chat, max ~99 math), lossless (full GPU residency + ECC-off).

Faithful from-scratch reimplementation of DeepSeek-V3 (MLA + aux-loss-free MoE + MTP + speculative decoding), ~412M params, Chinchilla-optimal 8.4B-token training on a single A100 80GB.

PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) — written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery.

A controlled 180-run study of DeepSeek-inspired MLA, sparse MoE routing, V3-style load balancing, and multi-token prediction under constrained compute.

From-scratch PyTorch: frontier LLM techniques as of 2026-Q1 — the Muon optimizer and Multi-Token Prediction, plus a base BPE tokenizer. Self-contained, self-checking modules.

Some benchmarks on running speculative decoding with Qwen-3.6-35B-A3B with EAGLE3, DFlash and MTP.

MTP draft-head research harness for Qwen3.5-9B. Verdict: function cuts (pruning, low-rank, distillation) pay a 10-19pt off-distribution tax fidelity cuts do not — the zero-training trimmed-vocab NVFP4 recipe wins at 1.8-2.7x end-to-end in memra. Negative-result ledgers included.

Controlled benchmark of llama.cpp speculative decoding for Qwen3.8-27B on one RTX 3090. Built-in MTP at n-max 2 gives +59.8% decode and 37% less energy per request; DFlash2 (PR #27342) reaches +51.9%. Includes a verification-step cost model, memory-bandwidth vs core-clock response, and byte-level losslessness data.

Fork of DeepSeek-V3 with MTP speculative-decoding improvements, a cache-aware mask fix, CPU test suite, and docs cleanup.

Benchmarks Multi-Token Prediction in llama.cpp against standard autoregressive baselines. Measures tokens per second, perplexity delta, and speculative acceptance rate across model sizes.

Recover the native MTP predictor missing from the 8-bit MLX Qwen3.8-27B-Uncensored package, build a BF16 sidecar, and reproduce a 15.59 → 48.75 tok/s controlled M4 Max result with MTPLX.

Multi-Token Prediction benchmarks for Gemma 4 on Apple Silicon — LiteRT-LM, transformers, and llama.cpp at batch=1 on a MacBook M4 Pro. ~2× speedup reproducible in one specific runtime.

Research code for ProbeRoute, a probe-initialized sparse routing method for frozen-backbone multi-token prediction

Multi-Ensemble Memory-Elastic Token Prediction — a numpy implementation of DeepSeek's DSpark speculative decoding + the MEMTP elastic-ensemble extension, on AMD Strix Halo (gfx1151).