Repository navigation
feat(maru_vllm): support DSpark speculative decoding - #151
Draft
kihwan-XCENA wants to merge 51 commits into
Draft
kihwan-XCENA wants to merge 51 commits into
kihwan-XCENA wants to merge 51 commits into
Conversation
- KVGroup.slot_bytes is one layer's page on both sides; the new group_page_bytes holds an MLA group's bytes per block, known only on the worker. An MLA group whose layers keep a tensor each is refused with a message that says so. - A load whose start falls inside a block fails, instead of leaving the rest of that block unloaded. - _chunk_done_key owns the _DONE suffix, the MLA path calls a layer's position in its group "layer" throughout, and the store's duplicate checks use a set. - Error text and docstrings describe vLLM's layout, with DeepSeek-V4.1 as an example.
…eplay is off With --no-swa-bounded-replay, the V1 model runner or ROCm, vLLM prefix-caches DeepSeek-V4.1's sliding-window groups and expects the connector to restore their window after a hit. MaruKVConnector refused this; it now stores them, as LMCache does. - A SlidingWindowMLASpec group is an MLA group with a sliding window. Each chunk keeps only the blocks the window reaches back over from the chunk's end, a hit needs such a group only around its end, and a load writes those blocks and leaves alone the null blocks vLLM gives the tokens the window has left behind. - An MLA group may use a block size that divides vLLM's (the windows' 32-token blocks beside 64- or 128-token ones). - Both storage formats.
Supported Models gives DeepSeek-V4.1-Flash a row of its own and keeps GLM-5.3-Flash refused. The vLLM page gains a DeepSeek-V4.1-Flash section (tested setup, the vLLM patch SM120 needs, what is stored with SWA bounded replay on and off, CXL space) and its hybrid-model settings now allow the layerwise format and smaller block sizes for MLA groups.
With bounded replay off, vLLM sets cache_config.block_size to the smallest block size it prefix-caches, DeepSeek-V4.1's 32-token sliding windows, while its MLA group keeps 64-token blocks and vLLM schedules in their least common multiple. MaruKVConnector required an MLA group's block size to divide cache_config.block_size and refused the model at startup. With several groups the chunk is now rounded to the least common multiple of every group's block size (chunk_unit), and an MLA group may use any block size.
# Conflicts: # docs/source/integration/vllm.md # maru_vllm/connector.py # maru_vllm/kv_groups.py
- A load of an MLA group's object fails when the object's size differs from the bytes this engine's blocks hold, and vLLM recomputes the request. Keys carry no KV cache layout, so an engine with another one (DeepSeek-V4.1 writes 528-byte records on SM100 and 584-byte ones elsewhere, and other KV cache dtypes change them too) stores objects of another size under the same keys; the load read the first bytes of a larger one as its own blocks. - check_multi_group_support refuses a Mamba-type group beside an MLA group. A Mamba group pins the chunk to one block, which holds no whole block of an MLA group with larger blocks, so such a model would store empty MLA objects and load nothing for a hit. The pair has not been run with equal block sizes either (GLM-5.3-Flash).
…ayout The existing path copies KV bytes too, so "copied as bytes" did not set the MLA group apart. What differs is the unit and the layout: these groups are copied whole blocks at a time, keeping the layout vLLM gives each block, while other groups are read token by token into Maru's own format.
# Conflicts: # maru_vllm/kv_groups.py
Two docstrings still said the MLA group's blocks are kept as bytes, which the token-by-token path does too. They now say what differs: the blocks keep the layout vLLM gives them instead of being laid out token by token.
…store path - An MLA group's object key ends in _b<block size> (KVGroup.object_key, now the one place every group object key is made). Its objects keep vLLM's block layout, and engines whose blocks differ in size, such as DeepSeek-V4.1 on SM90 or SM120 (64 tokens) and SM100 (128), lay the same tokens out differently; the size check passed such an object when its bytes added up to the same total. - _gather_group_blocks gives the right reason the window blocks are still the request's: vLLM frees blocks behind the window only when it schedules a later step, so the copy must finish before the next forward. - The MLA group's store allocates in the request's pool, as the other store paths do. - The support checks run before the chunk is rounded, so a refused model logs no chunk-size warning first.
…layouts match The MLA group's keys now name the block size, so SM100 and SM90/SM120 engines no longer meet under one key, and a load of an object of another size fails and is recomputed. Say that instead of forbidding engines of different architectures from sharing a MaruServer.
…for the PR description
The model's draft layers are DSpark stages, not MTP layers, and Maru plans to support speculative decoding with them in #149.
…ding planned, for the PR description
…ts for the PR description
…eplay for the PR description
…marks or unrelated counts for the PR description
…rchitecture for the PR description
…veral KV cache groups check_multi_group_support takes the speculative decoding method instead of a boolean. It accepts DeepSeek-V4.1-Flash's DSpark, whose draft layers keep sliding windows like the target's in the target's window groups, and refuses every other method, naming it.
vLLM numbers KV cache groups by position, and another configuration of the same model can put other layers in the group of the same number with objects of the same size: DSpark's draft layers make vLLM split DeepSeek-V4.1-Flash's sliding windows into one more group. An engine could then load another engine's objects into the wrong layers without an error. Each group object key now adds _l<16-digit hex hash of the group's layer names in vLLM's order> after _g<group id>.
…he group With speculative lookahead Maru admits parked loads against its own block budget, kv_cache_config.num_blocks - 1. It counted one group's blocks per request, while vLLM allocates the blocks of every group, the skipped ones included, from that one pool. A request now counts, per group, a block per block_size tokens of its prompt and lookahead, capped in a sliding-window group at vLLM's max_admission_blocks_per_request, and one block in a circular buffer. On DeepSeek-V4.1-Flash with DSpark a 1,000-token prompt needs 273 blocks, not 16.
…1-Flash The required settings for several KV cache groups allow DSpark on DeepSeek-V4.1-Flash, the model's section says how the draft layers' windows are stored and that toggling DSpark changes the groups, and the parked-load budget counts every group's blocks.
…culative decoding note
…dsv41-dspark # Conflicts: # docs/source/integration/vllm.md
# Conflicts: # docs/source/integration/vllm.md # maru_vllm/connector.py # maru_vllm/kv_groups.py # tests/unit/test_vllm_kv_groups.py
…ba group check_multi_group_support no longer accepts only DSpark. vLLM puts a drafter's layers in KV cache groups like any other layer, so the existing checks cover them: a draft model whose KV heads differ from the target's still fails the one-KV-shape rule. A model with a Mamba-type group stays refused with speculative decoding. In align mode vLLM keeps a block per draft token at the end of a request's Mamba block list and moves one into place without telling the connector, so the block list the scheduler tracks can name a state block no kernel wrote.
kihwan-XCENA
force-pushed
the
feat/dsv41-dspark
branch
from
November 19, 2026 14:14
a718358 to
616730a
Compare
2 tasks done
With speculative decoding, a drafter's own KV cache layers are registered beside the target's (Qwen3-4B with DSpark: 36 target and 5 draft layers), but vLLM calls save_kv_layer only from the target's. The chunkwise store of a single-group model waits until every registered layer has called it, so it never ran: nothing was stored and every request recomputed its prompt, without an error. A single-group chunkwise model with speculative decoding now stores in wait_for_save, which vLLM calls after the drafter, so each chunk's slab holds every layer, the drafter's included. Models with several KV cache groups already stored there; models without speculative decoding and the layerwise format are unchanged.
With speculative decoding a model with one KV cache group stores after the drafter runs, so each chunk holds a drafter's own KV cache layers, such as DSpark's on Qwen3. The layerwise format gets no hits with such a drafter.
…that was run Models with several KV cache groups accepted any speculative decoding method without a Mamba-type group, but only DSpark was run. The connector now refuses every other method there, and keeps refusing DSpark beside a Mamba-type group. A single-group model stores after the drafter only with DSpark; other methods store as before. The comments and docs now give the cause: vLLM's V2 model runner replays DSpark's draft step as a full CUDA graph unless --enforce-eager, and a replay calls no save_kv_layer for the draft layers, so the chunkwise store that waited for every registered layer never ran. With --enforce-eager the draft layers call it and the store already ran.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background & Motivation (Why)
DSpark speculative decoding registers its draft layers' KV caches beside the target's. Maru failed with it on both kinds of model:
🏗️ Design Changes
This PR supports DSpark only, the one method that was run, and adds no speculative decoding logic to Maru. Models with several groups still refuse every other method.
"method": "dspark"unless a group is Mamba-typealignmode vLLM moves Mamba state blocks without telling the connector_l<16-hex hash of the group's layer names>wait_for_save, after the draftersave_kv_layer, so the store, waiting for all 41 layers, never ranDraft KV affects only acceptance, since the target verifies every token.
Limits:
📝 Implementation Details
kv_groups.py:DSPARK_METHOD,check_multi_group_support(speculative_method=...),layer_names_digestingroup_object_key,GroupAdmission.connector.py:_speculative_method,_group_admissionsummed by_resume_blocks,MaruWorkerConnector(store_after_forward=...)with_store_single_group_after_forward.✅ Tests
Manual runs used the CXL backend with prefix caching off, so Maru was the only cache. Each sent one prompt three times at temperature 0: request 1 stores, and requests 2 and 3 load.
Unit tests:
tests/unit2,464 passed, 4 skipped (vLLM 0.32.0, CUDA cases included).Qwen3-4B with
deepseek-ai/dspark_qwen3_4b_block7(one group): vLLM 0.32.0, a 9,066-token prompt, CUDA graphs on.Outputs match a run with speculative decoding off under
--enforce-eager. With--enforce-eager, main also stores: the bug needs CUDA graphs.DeepSeek-V4.1-Flash-1017 (ten groups): TP 4, vLLM 0.33.0 with [Bugfix] Fix DeepSeek-V4.1-Flash boot failure on SM120 (RTX PRO 6000 Blackwell): allow 64-page block size for sparse-MLA and indexer backends vllm-project/vllm#65356, DSpark with 5 tokens, a 6,688-token prompt.
On an earlier head, with the key code unchanged since, an engine without DSpark loaded none of the DSpark engine's objects.
Parked-load admission on DeepSeek-V4.1-Flash with DSpark: the GPU cache was cut to 5,671 blocks and 16 stored prompts were sent at once.
🔗 Related Issues (optional)
📦 Release Note (for auto-generation / write in English)
NEW
CHANGED
_l<hash of the group's layer names>, so objects stored by earlier versions are not found.FIXED
IMPORTANT NOTES