Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
6c62706
feat(maru_vllm): skip KV cache groups vLLM does not prefix-cache
kihwan-XCENA Nov 3, 2026
560f3d7
feat(maru_vllm): register an MLA-only KV cache group as packed blocks
kihwan-XCENA Nov 3, 2026
889f123
feat(maru_vllm): store an opaque KV cache group as whole packed blocks
kihwan-XCENA Nov 3, 2026
21f8cdf
feat(maru_vllm): load an opaque KV cache group into its blocks
kihwan-XCENA Nov 3, 2026
e19dd1e
feat(maru_vllm): ask only for stored groups and resume stores past vL…
kihwan-XCENA Nov 3, 2026
f243d7f
feat(maru_vllm): store an opaque KV cache group in layerwise format
kihwan-XCENA Nov 3, 2026
e3a258f
fix(maru_vllm): name the window spec when DeepSeek-V4.1 runs without …
kihwan-XCENA Nov 3, 2026
144f019
fix(maru_vllm): retry a layerwise _DONE marker that failed to register
kihwan-XCENA Nov 3, 2026
3e8af15
Merge branch 'main' into feat/dsv41-flash-cxl
kihwan-XCENA Nov 4, 2026
0ffb946
refactor(maru_vllm): rename GROUP_OPAQUE to GROUP_MLA and reword its …
kihwan-XCENA Nov 4, 2026
fd32aea
refactor(maru_vllm): tidy the MLA group path after review
kihwan-XCENA Nov 4, 2026
fb80e02
feat(maru_vllm): store DeepSeek-V4.1's sliding windows when bounded r…
kihwan-XCENA Nov 4, 2026
992f8f6
docs(maru_vllm): list DeepSeek-V4.1-Flash as supported
kihwan-XCENA Nov 4, 2026
7741f74
docs(maru_vllm): keep the DeepSeek-V4.1-Flash section to what users need
kihwan-XCENA Nov 4, 2026
f4927ac
fix(maru_vllm): round the chunk to every group's block size
kihwan-XCENA Nov 6, 2026
6070127
Merge remote-tracking branch 'origin/main' into feat/dsv41-flash-cxl
kihwan-XCENA Nov 6, 2026
8661c5a
fix(maru_vllm): refuse MLA objects of another size and Mamba beside MLA
kihwan-XCENA Nov 6, 2026
3de57d8
docs(maru_vllm): say MLA groups are copied block by block in vLLM's l…
kihwan-XCENA Nov 6, 2026
44cd05d
Merge remote-tracking branch 'origin/main' into feat/dsv41-flash-cxl
kihwan-XCENA Nov 6, 2026
2b7463f
docs(maru_vllm): describe MLA group objects as blocks in vLLM's layout
kihwan-XCENA Nov 6, 2026
2eef6a0
fix(maru_vllm): name the MLA block size in object keys, and tidy the …
kihwan-XCENA Nov 6, 2026
90cca9e
docs(maru_vllm): engines share DeepSeek-V4.1 objects only when their …
kihwan-XCENA Nov 6, 2026
98bf350
docs(maru_vllm): add the DeepSeek-V4.1-Flash KV cache figure for the …
kihwan-XCENA Nov 6, 2026
e48a9e2
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
8017966
docs(maru_vllm): add the compact DeepSeek-V4.1-Flash KV cache figure …
kihwan-XCENA Nov 6, 2026
8277511
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
1b178c3
docs(maru_vllm): add the DeepSeek-V4.1-Flash structure figure for the…
kihwan-XCENA Nov 6, 2026
d38fe27
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
fccd112
docs(maru_vllm): speculative decoding on DeepSeek-V4.1-Flash is planned
kihwan-XCENA Nov 6, 2026
ebd8d68
docs(maru_vllm): add the DeepSeek-V4.1-Flash figure, speculative deco…
kihwan-XCENA Nov 6, 2026
f0bfa5a
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
a4832bc
docs(maru_vllm): add the DeepSeek-V4.1-Flash figure without byte coun…
kihwan-XCENA Nov 6, 2026
d6d7ee4
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
0f1e8cf
docs(maru_vllm): add the DeepSeek-V4.1-Flash figure without bounded r…
kihwan-XCENA Nov 6, 2026
81ed112
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
2c2c774
docs(maru_vllm): add the DeepSeek-V4.1-Flash figure without repeated …
kihwan-XCENA Nov 6, 2026
17e0460
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 6, 2026
5a3fbc1
docs(maru_vllm): add the DeepSeek-V4.1-Flash figure without the GPU a…
kihwan-XCENA Nov 9, 2026
6bda699
docs(maru_vllm): remove the PR figure from the tree
kihwan-XCENA Nov 9, 2026
6f64488
Merge remote-tracking branch 'origin/main' into feat/dsv41-flash-cxl
kihwan-XCENA Nov 10, 2026
b2f8b49
feat(maru_vllm): accept DSpark speculative decoding on models with se…
kihwan-XCENA Nov 10, 2026
28d7691
fix(maru_vllm): name a KV cache group's layers in its object keys
kihwan-XCENA Nov 10, 2026
b4dbe0f
fix(maru_vllm): admit parked loads against the blocks of every KV cac…
kihwan-XCENA Nov 10, 2026
86888b2
docs(maru_vllm): document DSpark speculative decoding on DeepSeek-V4.…
kihwan-XCENA Nov 10, 2026
c6a2f88
docs(maru_vllm): drop the issue link from the DeepSeek-V4.1-Flash spe…
kihwan-XCENA Nov 11, 2026
f58c448
Merge remote-tracking branch 'origin/feat/dsv41-flash-cxl' into feat/…
kihwan-XCENA Nov 11, 2026
d22cb6a
Merge remote-tracking branch 'origin/main' into feat/dsv41-dspark
kihwan-XCENA Nov 13, 2026
616730a
feat(maru_vllm): accept any speculative decoding method without a Mam…
kihwan-XCENA Nov 17, 2026
945ed35
fix(maru_vllm): store a single-group model's KV after the drafter runs
kihwan-XCENA Nov 19, 2026
fceb379
docs(maru_vllm): say a single-group model stores after the drafter
kihwan-XCENA Nov 19, 2026
53ab7a3
fix(maru_vllm): support DSpark only, the speculative decoding method …
kihwan-XCENA Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 27 additions & 9 deletions docs/source/integration/vllm.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,12 @@ or EAGLE decoding), Maru retains its budget: the upstream reservation counts
prompt tails but can leave too little room for the oldest request's lookahead.
Methods without lookahead slots, such as ngram, use the ordinary version rule.
vLLM 0.31+ reserves the draft slots of every speculative method itself, so a
load Maru admits can still wait for vLLM's own reservation.
load Maru admits can still wait for vLLM's own reservation. With several KV
cache groups, the budget counts a request's blocks in every group, the groups
Maru does not store included, because vLLM allocates them all from one pool: a
block per `block_size` tokens in a full-attention or MLA group, at most vLLM's
per-request admission cap in a sliding-window group, and one block in a
circular buffer.

This budget counts GPU blocks; `maru_load_admission_window` below counts loads
in flight.
Expand All @@ -463,6 +468,16 @@ in flight.
the first token of the current step is not delayed by the GPU→CXL copy and
the metadata registration.

With DSpark speculative decoding (`"method": "dspark"`), a model with one KV
cache group, such as Qwen3-4B with `deepseek-ai/dspark_qwen3_4b_block7`,
stores after the drafter runs instead of on the last attention layer: vLLM's
V2 model runner replays DSpark's draft step as a full CUDA graph unless
`--enforce-eager`, and a replay calls no per-layer connector hook for the draft
layers. Each chunk then holds the draft layers too. The layerwise format still
stores layer by layer, so such a model gets no hits in it with CUDA graphs on:
use the chunkwise format. Other speculative decoding methods store as without
speculative decoding.

`maru_load_admission_window` bounds how many asynchronous loads may be in
flight at once. The default `0` submits every load immediately. Set a
positive value only if you need request-level backpressure.
Expand Down Expand Up @@ -622,15 +637,18 @@ Chunk keys therefore name a slice by what it holds:
|---|---|---|
| Every KV head: TP=1, or an MLA latent that each rank holds in full | `kv_<hash>` | Unchanged from earlier releases |
| KV heads `[a, b)` | `kv_<hash>_h<a>-<b>` | Llama-3.1-8B (8 KV heads) at TP=2: `_h0-4` on rank 0, `_h4-8` on rank 1 |
| Rank `r`'s part of a slice named by rank: each Mamba-type state, and the attention KV of sliding-window and full layers when TP exceeds the KV heads (below) | `kv_<hash>_tp<TP>_r<r>`, then `_c<chunk_tokens>_g<group_id>` with several KV cache groups | Qwen3.5-0.8B at TP=2: `_tp2_r0_c544_g0` on rank 0 |
| Rank `r`'s part of a slice named by rank: each Mamba-type state, and the attention KV of sliding-window and full layers when TP exceeds the KV heads (below) | `kv_<hash>_tp<TP>_r<r>`, then `_c<chunk_tokens>_g<group_id>_l<layers>` with several KV cache groups | Qwen3.5-0.8B at TP=2: `_tp2_r0_c544_g0_l<layers>` on rank 0 |

The layerwise format appends its `_L<layer>` and `_DONE` suffixes after the
slice. A model with several KV cache groups (see {ref}`vllm-hybrid-models`)
appends `_c<chunk_tokens>_g<group_id>`, followed for an MLA group by
appends `_c<chunk_tokens>_g<group_id>_l<layers>`, followed for an MLA group by
`_b<block_size>`, since its objects keep vLLM's block layout (see
{ref}`vllm-deepseek-v41`). It names the KV heads of each attention group, full
or sliding window, as above: Qwen3.5-0.8B at TP=2 stores
`kv_<hash>_h0-1_c544_g3` on rank 0.
{ref}`vllm-deepseek-v41`). `<layers>` is a 16-digit hex hash of the group's
layer names in vLLM's order: vLLM numbers groups by position, and another
configuration of the same model can put other layers in the group of the same
number, as DSpark speculative decoding does on DeepSeek-V4.1-Flash. It names
the KV heads of each attention group, full or sliding window, as above:
Qwen3.5-0.8B at TP=2 stores `kv_<hash>_h0-1_c544_g3_l<layers>` on rank 0.

Ranks that hold identical KV share one slice name: the MLA latent, and the
heads vLLM replicates when there are more TP ranks than KV heads (4 KV heads
Expand Down Expand Up @@ -726,7 +744,7 @@ backends still require TP=1.
(vllm-hybrid-models)=
## Hybrid models

vLLM puts layers that keep different kinds of KV cache into separate KV cache groups, for example sliding-window and full-attention layers, or Gated DeltaNet (Mamba-type) and full-attention layers. On vLLM 0.30+ the connector stores each group's part of a chunk as its own CXL object, keyed `<chunk_key>_c<chunk_tokens>_g<group_id>`, and reports a hit only for chunks that every group can resume from. A model with one group is stored as before.
vLLM puts layers that keep different kinds of KV cache into separate KV cache groups, for example sliding-window and full-attention layers, or Gated DeltaNet (Mamba-type) and full-attention layers. On vLLM 0.30+ the connector stores each group's part of a chunk as its own CXL object, keyed `<chunk_key>_c<chunk_tokens>_g<group_id>_l<layers>` (see {ref}`vllm-tensor-parallelism` for `<layers>`), and reports a hit only for chunks that every group can resume from. A model with one group is stored as before.

The key names the chunk length because a chunk key hashes the prompt only up to the chunk's end. With a Mamba-type group a chunk is one vLLM block, and vLLM sizes that block from the per-rank state and attention page, so engines with different TP sizes (or KV cache dtypes) can use different chunks: Qwen3.5-0.8B has 544-token chunks at TP=2 and 272-token chunks at TP=4. Their objects for the same prompt prefix then get different keys, and neither engine loads the other's.

Expand All @@ -738,7 +756,7 @@ The connector refuses to start a model with several groups unless:

- `maru_storage_backend` is `cxl` (the default) or `remote_cxl`. The CPU and mixed backends take one group only, and remote_cxl takes no MLA model, such as DeepSeek-V4.1.
- `maru_use_layerwise` is `false` (the default), unless every group the connector stores is an MLA group, as DeepSeek-V4.1's are (see {ref}`vllm-deepseek-v41`).
- Speculative decoding is off.
- Speculative decoding is off, or is DSpark (`"method": "dspark"`) on a model without a Mamba-type group, as on DeepSeek-V4.1-Flash (see {ref}`vllm-deepseek-v41`). The connector refuses every other method. It refuses DSpark too with a Mamba-type group: in `align` mode vLLM moves a Mamba state block among the request's last blocks without telling the connector.
- Mamba-type groups use vLLM's `align` Mamba cache mode. vLLM picks it when prefix caching is on, which is the default. Keep prefix caching on: with `--no-enable-prefix-caching`, vLLM sets the mode to `none` even if `--mamba-cache-mode align` is given, and the connector refuses the model. Do not pass `--mamba-cache-mode all`.
- The model does not have both Mamba-type groups and MLA groups, as GLM-5.3-Flash does.
- Every group uses vLLM's one `block_size`, except MLA groups, which may use other block sizes, as DeepSeek-V4.1's do; `maru_kv_chunk_tokens` is then rounded to a multiple of every group's block size. vLLM normally sets the groups up this way.
Expand Down Expand Up @@ -781,7 +799,7 @@ vLLM splits the model's KV cache into several groups, how many depending on the
- **The sliding-window groups.** With SWA bounded replay (`--swa-bounded-replay`, the default on CUDA with vLLM's V2 model runner), vLLM keeps them out of its prefix cache and, after a hit, recomputes the hit's last 128 tokens to rebuild them; Maru does not store them. With bounded replay off (`--no-swa-bounded-replay`, the V1 model runner, or ROCm), vLLM prefix-caches them, and Maru stores for each chunk the window blocks that a hit at the chunk's end needs.
- **The KV compressor's circular buffer** is not stored, since vLLM does not prefix-cache it.

Both formats work. The model calls no per-layer connector method, so the layerwise format also stores after the forward and loads before it, without overlapping transfers with compute. With bounded replay on, a 256-token chunk takes one 467,712-byte page on SM120 in the chunkwise format, and about 2.9 times that in the layerwise format, which gives each KV cache and the `_DONE` marker a page of their own. With bounded replay off, each chunk also keeps the sliding windows' last blocks, several times the bytes of the MLA group's. Speculative decoding, which runs the DSpark draft layers in the model's checkpoint, is not supported yet, as for any model with several groups.
Both formats work. The model calls no per-layer connector method, so the layerwise format also stores after the forward and loads before it, without overlapping transfers with compute. With bounded replay on, a 256-token chunk takes one 467,712-byte page on SM120 in the chunkwise format, and about 2.9 times that in the layerwise format, which gives each KV cache and the `_DONE` marker a page of their own. With bounded replay off, each chunk also keeps the sliding windows' last blocks, several times the bytes of the MLA group's. Speculative decoding works with the DSpark draft layers in the model's checkpoint (`--speculative-config '{"method": "dspark", "num_speculative_tokens": 5}'`; vLLM 0.33.0 does not start without `num_speculative_tokens`); the connector refuses every other speculative decoding method. Each draft layer keeps a sliding window like the target's, and vLLM deals the draft windows into the target's sliding-window groups, splitting the windows into one more group, so Maru skips or stores them with the target's windows. Because the groups then hold other layers, objects stored with DSpark off are not found with it on, and the other way around.

```bash
vllm serve deepseek-ai/DeepSeek-V4.1-Flash-1017 \
Expand Down
Loading
Loading