server: keep speculative checkpoints on device (6 lines) — free for anyone to take upstream - #1
Open
JayToltTech wants to merge 1 commit into
Open
server: keep speculative checkpoints on device (6 lines) — free for anyone to take upstream#1JayToltTech wants to merge 1 commit into
JayToltTech wants to merge 1 commit into
Conversation
qwen4exp is a recurrent hybrid, so the target context cannot partially seq_rm and the server classifies it SEQ_RM_TYPE_FULL. Every speculative round then takes a full recurrent-state checkpoint, and every rejected draft restores one. Through the host path that means serializing each GDN layer conv+state row, the 4-stream hyper-connection residual and the PLE history into a host vector with one synchronous backend read per tensor, then pushing it all back. The cost is flat in context and swamps everything else: with the MTP drafter attached, decode ran 201 ms/token against 29 ms/token on a tree that keeps the state on device, and a 120-token generation spent roughly 600 ms of each 825 ms round in checkpoint traffic. It also inflates the reported prompt eval time, which runs to first token and so absorbs the first save: 827 ms for an 11-token prompt. Request ON_DEVICE at the six speculative checkpoint sites so the state stays in device buffers. The library already implements this path; only the server never asked for it. Measured on gfx1151 Vulkan, Qwen3.8-Flash-Next UD-Q4_K_XL with a Q4_K_M MTP head: 4.77 -> 25.83 tok/s at short context, 4.33 -> 16.08 at 70k, draft acceptance unchanged at 70-80 percent. The prompt-cache checkpoints are deliberately left host-resident: they retain several historical states rather than one live round. Diagnosed by Claude Fable 5. The ON_DEVICE flag and the mechanism come from Gaetan Puleo (c8b681b6f), carried in Nathanw1014/llama.cpp as 08a3255. (cherry picked from commit 175b66c)
There was a problem hiding this comment.
🟡 Changes recommended
Unconditionally enabling on-device checkpoints can increase persistent backend memory usage per slot and can hard-abort on allocation/mismatch, so it should be gated and/or have a safe fallback to preserve server reliability.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adjusts the llama.cpp server’s speculative-decoding checkpointing to request on-device sequence-state save/load, aiming to avoid expensive host round-trips when restoring state (notably for recurrent/hybrid memories) and improve speculative throughput.
Changes:
- Add
LLAMA_STATE_SEQ_FLAGS_ON_DEVICEto speculative checkpoint save calls (update_tgt/update_dft). - Add
LLAMA_STATE_SEQ_FLAGS_ON_DEVICEto speculative checkpoint restore calls (load_tgt/load_dft).
File summaries
| File | Description |
|---|---|
| tools/server/server-context.cpp | Switch speculative checkpoint save/restore to request on-device state storage via LLAMA_STATE_SEQ_FLAGS_ON_DEVICE. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| //const int64_t t_start = ggml_time_us(); | ||
|
|
||
| ckpt.update_tgt(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY); | ||
| ckpt.update_tgt(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY | LLAMA_STATE_SEQ_FLAGS_ON_DEVICE); |
JayToltTech
changed the title
1 task
scchow
added a commit
to scchow/llama.cpp
that referenced
this pull request
Sep 2, 2026
qwen4exp is a recurrent hybrid: the server classifies it SEQ_RM_TYPE_FULL, so every speculative round checkpoints the full recurrent state (GDN conv+state, 4-stream HC residual, PLE history). Through the host path that is a synchronous per-tensor readback + push, ~600 ms of an 825 ms round on gfx1151; +61% on RTX 3090 CUDA; neutral on Metal (community-verified, JayToltTech#1). The library already implements the on-device path; the server just never asked for it. Signed-off-by: Scott Chow <scott@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Six call sites in
tools/server/server-context.cppgainLLAMA_STATE_SEQ_FLAGS_ON_DEVICE. No new code paths: the library already implements the on-device route, the server never requested it.A recurrent-hybrid target cannot partially
seq_rm, so the server takes a full recurrent-state checkpoint every speculative round and restores it on every rejection. Via the host path that is one synchronous backend read per tensor for every GDN conv+state row, the 4-stream residual and the PLE history. Flat in context, ~600 ms of an ~825 ms round.The four prompt-cache checkpoint sites are deliberately unchanged: they retain several historical states, not one live round, so the flag's invalidation contract does not hold there.
Measured
Three backends, independent testers:
Consistent reading: a win where per-round host round-trips are expensive, neutral where they are cheap. Not a regression anywhere tested.
Provenance
The
ON_DEVICEflag and mechanism are Gaetan Puleo's (c8b681b6f, carried inNathanw1014/llama.cppas08a3255). What is here is only the identification of which call sites need it. Mechanism diagnosis assisted by an AI agent (Claude).Status
Not submitted upstream, and not intended to be. It exists on this fork for anyone who wants to carry it. No authorship or attribution is claimed or required — anyone is free to take this upstream as their own without credit or consultation.
Not verified
Prompt-cache paths untested beyond being untouched. No VRAM-cost measurement per slot, or scaling with
--parallel.Requirements