Repository navigation
fix(maru_vllm): store a single-group model's KV after the drafter runs - #158
Closed
kihwan-XCENA wants to merge 1 commit into
Closed
kihwan-XCENA wants to merge 1 commit into
kihwan-XCENA wants to merge 1 commit into
Conversation
With speculative decoding, a drafter's own KV cache layers are registered beside the target's (Qwen3-4B with DSpark: 36 target and 5 draft layers), but vLLM calls save_kv_layer only from the target's. The chunkwise store of a single-group model waits until every registered layer has called it, so it never ran: nothing was stored and every request recomputed its prompt, without an error. A single-group chunkwise model with speculative decoding now stores in wait_for_save, which vLLM calls after the drafter, so each chunk's slab holds every layer, the drafter's included. Models with several KV cache groups already stored there; models without speculative decoding and the layerwise format are unchanged.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background & Motivation (Why)
With speculative decoding, a drafter's own KV cache layers are registered with the connector beside the target's (Qwen3-4B with DSpark: 36 target and 5 draft layers), but vLLM calls
save_kv_layeronly from the target's. The chunkwise store of a single-group model waits until every registered layer has called it, so it never ran: Maru stored nothing and every request recomputed its prompt, without an error.🏗️ Design Changes
wait_for_saveinstead ofsave_kv_layer. vLLM callswait_for_saveafter the drafter, so each chunk's slab holds every layer, the drafter's included.📝 Implementation Details
MaruWorkerConnectortakesstore_after_forward, set fromvllm_config.speculative_config. In that modesave_kv_layerreturns early andwait_for_savecalls_store_single_group_after_forward, which reuses the existing kernel, write-behind and per-layer fallback stores.✅ Tests
tests/unit2,549 passed, 4 skipped (vLLM 0.32.0).deepseek-ai/dspark_qwen3_4b_block7(7 speculative tokens), vLLM 0.32.0, one GPU, CXL backend, prefix caching off. A 9,066-token prompt was sent three times at temperature 0, so requests 2 and 3 load from Maru.maru_async_storeWith
--enforce-eager, DSpark off with Maru, DSpark on with Maru and DSpark on without Maru gave the same 400 output tokens in all nine requests. (Compiled engines can split at a near-tie token, here output token 18, with or without this PR: vLLM keys its torch.compile cache by the config, the KV transfer config included, so the engines run different compiled kernels.)🔗 Related Issues (optional)
📦 Release Note (for auto-generation / write in English)
NEW
CHANGED
FIXED
IMPORTANT NOTES