Skip to content

feat(maru_vllm): support DSpark speculative decoding - #151

Draft
kihwan-XCENA wants to merge 51 commits into
mainfrom
feat/dsv41-dspark
Draft

kihwan-XCENA wants to merge 51 commits into
mainfrom
feat/dsv41-dspark

Conversation

@kihwan-XCENA

@kihwan-XCENA kihwan-XCENA commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

DSpark speculative decoding registers its draft layers' KV caches beside the target's. Maru failed with it on both kinds of model:

  • Several KV cache groups (DeepSeek-V4.1-Flash): refused at startup, like every speculative decoding method.
  • One KV cache group (Qwen3 with a DSpark checkpoint): with CUDA graphs on, vLLM's default, nothing was stored and no error was raised.

🏗️ Design Changes

This PR supports DSpark only, the one method that was run, and adds no speculative decoding logic to Maru. Models with several groups still refuse every other method.

Change Why
The several-group check accepts "method": "dspark" unless a group is Mamba-type In align mode vLLM moves Mamba state blocks without telling the connector
Group object keys add _l<16-hex hash of the group's layer names> DSpark's three draft windows make vLLM split DeepSeek-V4.1-Flash's sliding windows into eight groups instead of seven, so a group id holds other layers in objects of the same size
Parked-load admission sums a request's blocks over every group vLLM allocates all groups from one pool: a 6,690-token prompt takes 1,786 blocks by this count, not 105
With DSpark, a single-group chunkwise model stores in wait_for_save, after the drafter vLLM's V2 model runner replays the draft step as a full CUDA graph, which calls no save_kv_layer, so the store, waiting for all 41 layers, never ran

Draft KV affects only acceptance, since the target verifies every token.

Limits:

  • A single-group model with DSpark in the layerwise format still gets no hits with CUDA graphs on. The docs say to use chunkwise.
  • The admission count takes vLLM's per-request cap for each sliding-window group, about 10 times the 150–190 blocks a request was seen to hold. When the cache is nearly full, loads wait sooner than they need to.

📝 Implementation Details

  • kv_groups.py: DSPARK_METHOD, check_multi_group_support(speculative_method=...), layer_names_digest in group_object_key, GroupAdmission.
  • connector.py: _speculative_method, _group_admission summed by _resume_blocks, MaruWorkerConnector(store_after_forward=...) with _store_single_group_after_forward.
  • Docs: speculative decoding rules, DSpark on DeepSeek-V4.1-Flash, key format, parked-load budget, single-group store timing.

✅ Tests

Manual runs used the CXL backend with prefix caching off, so Maru was the only cache. Each sent one prompt three times at temperature 0: request 1 stores, and requests 2 and 3 load.

  • Unit tests: tests/unit 2,464 passed, 4 skipped (vLLM 0.32.0, CUDA cases included).

  • Qwen3-4B with deepseek-ai/dspark_qwen3_4b_block7 (one group): vLLM 0.32.0, a 9,066-token prompt, CUDA graphs on.

    Code Loaded from Maru (requests 2, 3) Acceptance Output token ids
    main (ca23e6a) none 0.337 identical in all 3 requests
    This PR 8,960 tokens 0.337 identical in all 3 requests

    Outputs match a run with speculative decoding off under --enforce-eager. With --enforce-eager, main also stores: the bug needs CUDA graphs.

  • DeepSeek-V4.1-Flash-1017 (ten groups): TP 4, vLLM 0.33.0 with [Bugfix] Fix DeepSeek-V4.1-Flash boot failure on SM120 (RTX PRO 6000 Blackwell): allow 64-page block size for sparse-MLA and indexer backends vllm-project/vllm#65356, DSpark with 5 tokens, a 6,688-token prompt.

    • Every run loaded 6,656 tokens: chunkwise and layerwise, each with SWA bounded replay on and off.
    • Every request gave the same first 297 tokens. At token 297, where the answer ends or goes on, requests split in the same way whatever the prefix came from:
    Prefix from Requests Ended / went on at token 297 Acceptance
    Computed (no cache) 13 8 / 5 0.868–0.907
    Maru 15 7 / 8 0.872–0.924
    vLLM prefix cache 5 2 / 3 0.868–0.893

    On an earlier head, with the key code unchanged since, an engine without DSpark loaded none of the DSpark engine's objects.

  • Parked-load admission on DeepSeek-V4.1-Flash with DSpark: the GPU cache was cut to 5,671 blocks and 16 stored prompts were sent at once.

    • 13 loads waited for the budget. All 16 loaded from Maru and matched their stored outputs, with no preemption or hang.
    • With the one-group count none waited, and the burst also finished, in 1.2 s against 2.9 s with this PR. This load did not reach a failure that the old count allows.

🔗 Related Issues (optional)

📦 Release Note (for auto-generation / write in English)

NEW

  • vLLM connector: DSpark speculative decoding on models with several KV cache groups and no Mamba-type group, such as DeepSeek-V4.1-Flash. Other speculative decoding methods are still refused on such models.

CHANGED

  • Models with several KV cache groups: object keys add _l<hash of the group's layer names>, so objects stored by earlier versions are not found.

FIXED

  • vLLM connector: a single-group model with DSpark, such as Qwen3-4B, now stores its KV in Maru with CUDA graphs on, vLLM's default.

IMPORTANT NOTES

  • Turning DSpark on or off changes DeepSeek-V4.1-Flash's KV cache groups, so objects stored with one setting are not found with the other.

- KVGroup.slot_bytes is one layer's page on both sides; the new
  group_page_bytes holds an MLA group's bytes per block, known only on the
  worker. An MLA group whose layers keep a tensor each is refused with a
  message that says so.
- A load whose start falls inside a block fails, instead of leaving the
  rest of that block unloaded.
- _chunk_done_key owns the _DONE suffix, the MLA path calls a layer's
  position in its group "layer" throughout, and the store's duplicate
  checks use a set.
- Error text and docstrings describe vLLM's layout, with DeepSeek-V4.1 as
  an example.
…eplay is off

With --no-swa-bounded-replay, the V1 model runner or ROCm, vLLM
prefix-caches DeepSeek-V4.1's sliding-window groups and expects the
connector to restore their window after a hit. MaruKVConnector refused
this; it now stores them, as LMCache does.

- A SlidingWindowMLASpec group is an MLA group with a sliding window. Each
  chunk keeps only the blocks the window reaches back over from the
  chunk's end, a hit needs such a group only around its end, and a load
  writes those blocks and leaves alone the null blocks vLLM gives the
  tokens the window has left behind.
- An MLA group may use a block size that divides vLLM's (the windows'
  32-token blocks beside 64- or 128-token ones).
- Both storage formats.
Supported Models gives DeepSeek-V4.1-Flash a row of its own and keeps
GLM-5.3-Flash refused. The vLLM page gains a DeepSeek-V4.1-Flash section
(tested setup, the vLLM patch SM120 needs, what is stored with SWA bounded
replay on and off, CXL space) and its hybrid-model settings now allow the
layerwise format and smaller block sizes for MLA groups.
With bounded replay off, vLLM sets cache_config.block_size to the
smallest block size it prefix-caches, DeepSeek-V4.1's 32-token sliding
windows, while its MLA group keeps 64-token blocks and vLLM schedules in
their least common multiple. MaruKVConnector required an MLA group's
block size to divide cache_config.block_size and refused the model at
startup.

With several groups the chunk is now rounded to the least common
multiple of every group's block size (chunk_unit), and an MLA group may
use any block size.
# Conflicts:
#	docs/source/integration/vllm.md
#	maru_vllm/connector.py
#	maru_vllm/kv_groups.py
- A load of an MLA group's object fails when the object's size differs
  from the bytes this engine's blocks hold, and vLLM recomputes the
  request. Keys carry no KV cache layout, so an engine with another one
  (DeepSeek-V4.1 writes 528-byte records on SM100 and 584-byte ones
  elsewhere, and other KV cache dtypes change them too) stores objects of
  another size under the same keys; the load read the first bytes of a
  larger one as its own blocks.
- check_multi_group_support refuses a Mamba-type group beside an MLA
  group. A Mamba group pins the chunk to one block, which holds no whole
  block of an MLA group with larger blocks, so such a model would store
  empty MLA objects and load nothing for a hit. The pair has not been run
  with equal block sizes either (GLM-5.3-Flash).
…ayout

The existing path copies KV bytes too, so "copied as bytes" did not set
the MLA group apart. What differs is the unit and the layout: these groups
are copied whole blocks at a time, keeping the layout vLLM gives each
block, while other groups are read token by token into Maru's own format.
Two docstrings still said the MLA group's blocks are kept as bytes, which
the token-by-token path does too. They now say what differs: the blocks
keep the layout vLLM gives them instead of being laid out token by token.
…store path

- An MLA group's object key ends in _b<block size> (KVGroup.object_key,
  now the one place every group object key is made). Its objects keep
  vLLM's block layout, and engines whose blocks differ in size, such as
  DeepSeek-V4.1 on SM90 or SM120 (64 tokens) and SM100 (128), lay the
  same tokens out differently; the size check passed such an object when
  its bytes added up to the same total.
- _gather_group_blocks gives the right reason the window blocks are still
  the request's: vLLM frees blocks behind the window only when it
  schedules a later step, so the copy must finish before the next forward.
- The MLA group's store allocates in the request's pool, as the other
  store paths do.
- The support checks run before the chunk is rounded, so a refused model
  logs no chunk-size warning first.
…layouts match

The MLA group's keys now name the block size, so SM100 and SM90/SM120
engines no longer meet under one key, and a load of an object of another
size fails and is recomputed. Say that instead of forbidding engines of
different architectures from sharing a MaruServer.
The model's draft layers are DSpark stages, not MTP layers, and Maru
plans to support speculative decoding with them in #149.
…marks or unrelated counts for the PR description
…veral KV cache groups

check_multi_group_support takes the speculative decoding method instead of a
boolean. It accepts DeepSeek-V4.1-Flash's DSpark, whose draft layers keep
sliding windows like the target's in the target's window groups, and refuses
every other method, naming it.
vLLM numbers KV cache groups by position, and another configuration of the
same model can put other layers in the group of the same number with objects
of the same size: DSpark's draft layers make vLLM split DeepSeek-V4.1-Flash's
sliding windows into one more group. An engine could then load another
engine's objects into the wrong layers without an error. Each group object
key now adds _l<16-digit hex hash of the group's layer names in vLLM's order>
after _g<group id>.
…he group

With speculative lookahead Maru admits parked loads against its own block
budget, kv_cache_config.num_blocks - 1. It counted one group's blocks per
request, while vLLM allocates the blocks of every group, the skipped ones
included, from that one pool. A request now counts, per group, a block per
block_size tokens of its prompt and lookahead, capped in a sliding-window
group at vLLM's max_admission_blocks_per_request, and one block in a circular
buffer. On DeepSeek-V4.1-Flash with DSpark a 1,000-token prompt needs 273
blocks, not 16.
…1-Flash

The required settings for several KV cache groups allow DSpark on
DeepSeek-V4.1-Flash, the model's section says how the draft layers' windows
are stored and that toggling DSpark changes the groups, and the parked-load
budget counts every group's blocks.
…dsv41-dspark

# Conflicts:
#	docs/source/integration/vllm.md
@kihwan-XCENA kihwan-XCENA changed the title feat(maru_vllm): support DSpark speculative decoding on DeepSeek-V4.1-Flash feat(maru_vllm): support DSpark speculative decoding Nov 12, 2026
Base automatically changed from feat/dsv41-flash-cxl to main November 12, 2026 04:50
# Conflicts:
#	docs/source/integration/vllm.md
#	maru_vllm/connector.py
#	maru_vllm/kv_groups.py
#	tests/unit/test_vllm_kv_groups.py
…ba group

check_multi_group_support no longer accepts only DSpark. vLLM puts a
drafter's layers in KV cache groups like any other layer, so the existing
checks cover them: a draft model whose KV heads differ from the target's
still fails the one-KV-shape rule.

A model with a Mamba-type group stays refused with speculative decoding. In
align mode vLLM keeps a block per draft token at the end of a request's Mamba
block list and moves one into place without telling the connector, so the
block list the scheduler tracks can name a state block no kernel wrote.
With speculative decoding, a drafter's own KV cache layers are registered
beside the target's (Qwen3-4B with DSpark: 36 target and 5 draft layers),
but vLLM calls save_kv_layer only from the target's. The chunkwise store
of a single-group model waits until every registered layer has called it,
so it never ran: nothing was stored and every request recomputed its
prompt, without an error.

A single-group chunkwise model with speculative decoding now stores in
wait_for_save, which vLLM calls after the drafter, so each chunk's slab
holds every layer, the drafter's included. Models with several KV cache
groups already stored there; models without speculative decoding and the
layerwise format are unchanged.
With speculative decoding a model with one KV cache group stores after the
drafter runs, so each chunk holds a drafter's own KV cache layers, such as
DSpark's on Qwen3. The layerwise format gets no hits with such a drafter.
…that was run

Models with several KV cache groups accepted any speculative decoding method
without a Mamba-type group, but only DSpark was run. The connector now refuses
every other method there, and keeps refusing DSpark beside a Mamba-type group.

A single-group model stores after the drafter only with DSpark; other methods
store as before. The comments and docs now give the cause: vLLM's V2 model
runner replays DSpark's draft step as a full CUDA graph unless
--enforce-eager, and a replay calls no save_kv_layer for the draft layers, so
the chunkwise store that waited for every registered layer never ran. With
--enforce-eager the draft layers call it and the store already ran.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant