Skip to content

fix(maru_vllm): store a single-group model's KV after the drafter runs - #158

Closed
kihwan-XCENA wants to merge 1 commit into
mainfrom
fix/single-group-drafter-store
Closed

kihwan-XCENA wants to merge 1 commit into
mainfrom
fix/single-group-drafter-store

Conversation

@kihwan-XCENA

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

With speculative decoding, a drafter's own KV cache layers are registered with the connector beside the target's (Qwen3-4B with DSpark: 36 target and 5 draft layers), but vLLM calls save_kv_layer only from the target's. The chunkwise store of a single-group model waits until every registered layer has called it, so it never ran: Maru stored nothing and every request recomputed its prompt, without an error.

🏗️ Design Changes

  • With speculative decoding, a single-group chunkwise model stores in wait_for_save instead of save_kv_layer. vLLM calls wait_for_save after the drafter, so each chunk's slab holds every layer, the drafter's included.
  • Unchanged: models with several KV cache groups (they already store there), models without speculative decoding, and the layerwise format.

📝 Implementation Details

  • MaruWorkerConnector takes store_after_forward, set from vllm_config.speculative_config. In that mode save_kv_layer returns early and wait_for_save calls _store_single_group_after_forward, which reuses the existing kernel, write-behind and per-layer fallback stores.

✅ Tests

  • Unit tests: 5 new cases; tests/unit 2,549 passed, 4 skipped (vLLM 0.32.0).
  • Manual tests: Qwen3-4B with deepseek-ai/dspark_qwen3_4b_block7 (7 speculative tokens), vLLM 0.32.0, one GPU, CXL backend, prefix caching off. A 9,066-token prompt was sent three times at temperature 0, so requests 2 and 3 load from Maru.
Case Maru hit, requests 2 and 3 Output tokens
DSpark, before this PR 0 –
DSpark 8,960 same 400 tokens in all three requests
DSpark, maru_async_store 8,960 same 400 tokens in all three requests

With --enforce-eager, DSpark off with Maru, DSpark on with Maru and DSpark on without Maru gave the same 400 output tokens in all nine requests. (Compiled engines can split at a near-tie token, here output token 18, with or without this PR: vLLM keys its torch.compile cache by the config, the KV transfer config included, so the engines run different compiled kernels.)

🔗 Related Issues (optional)

📦 Release Note (for auto-generation / write in English)

NEW

CHANGED

FIXED

  • vLLM connector: a single-group model with a speculative decoding drafter that has its own KV cache layers, such as DSpark on Qwen3, now stores its KV in Maru.

IMPORTANT NOTES

With speculative decoding, a drafter's own KV cache layers are registered
beside the target's (Qwen3-4B with DSpark: 36 target and 5 draft layers),
but vLLM calls save_kv_layer only from the target's. The chunkwise store
of a single-group model waits until every registered layer has called it,
so it never ran: nothing was stored and every request recomputed its
prompt, without an error.

A single-group chunkwise model with speculative decoding now stores in
wait_for_save, which vLLM calls after the drafter, so each chunk's slab
holds every layer, the drafter's included. Models with several KV cache
groups already stored there; models without speculative decoding and the
layerwise format are unchanged.
@kihwan-XCENA

Copy link
Copy Markdown
Collaborator Author

Moved into #151 (commit 945ed35), since this is part of DSpark support.

@kihwan-XCENA
kihwan-XCENA deleted the fix/single-group-drafter-store branch November 19, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant