diff --git a/docs/source/integration/vllm.md b/docs/source/integration/vllm.md index b1094d62..0dddd3c3 100644 --- a/docs/source/integration/vllm.md +++ b/docs/source/integration/vllm.md @@ -454,7 +454,12 @@ or EAGLE decoding), Maru retains its budget: the upstream reservation counts prompt tails but can leave too little room for the oldest request's lookahead. Methods without lookahead slots, such as ngram, use the ordinary version rule. vLLM 0.31+ reserves the draft slots of every speculative method itself, so a -load Maru admits can still wait for vLLM's own reservation. +load Maru admits can still wait for vLLM's own reservation. With several KV +cache groups, the budget counts a request's blocks in every group, the groups +Maru does not store included, because vLLM allocates them all from one pool: a +block per `block_size` tokens in a full-attention or MLA group, at most vLLM's +per-request admission cap in a sliding-window group, and one block in a +circular buffer. This budget counts GPU blocks; `maru_load_admission_window` below counts loads in flight. @@ -463,6 +468,16 @@ in flight. the first token of the current step is not delayed by the GPU→CXL copy and the metadata registration. +With DSpark speculative decoding (`"method": "dspark"`), a model with one KV +cache group, such as Qwen3-4B with `deepseek-ai/dspark_qwen3_4b_block7`, +stores after the drafter runs instead of on the last attention layer: vLLM's +V2 model runner replays DSpark's draft step as a full CUDA graph unless +`--enforce-eager`, and a replay calls no per-layer connector hook for the draft +layers. Each chunk then holds the draft layers too. The layerwise format still +stores layer by layer, so such a model gets no hits in it with CUDA graphs on: +use the chunkwise format. Other speculative decoding methods store as without +speculative decoding. + `maru_load_admission_window` bounds how many asynchronous loads may be in flight at once. The default `0` submits every load immediately. Set a positive value only if you need request-level backpressure. @@ -622,15 +637,18 @@ Chunk keys therefore name a slice by what it holds: |---|---|---| | Every KV head: TP=1, or an MLA latent that each rank holds in full | `kv_` | Unchanged from earlier releases | | KV heads `[a, b)` | `kv__h-` | Llama-3.1-8B (8 KV heads) at TP=2: `_h0-4` on rank 0, `_h4-8` on rank 1 | -| Rank `r`'s part of a slice named by rank: each Mamba-type state, and the attention KV of sliding-window and full layers when TP exceeds the KV heads (below) | `kv__tp_r`, then `_c_g` with several KV cache groups | Qwen3.5-0.8B at TP=2: `_tp2_r0_c544_g0` on rank 0 | +| Rank `r`'s part of a slice named by rank: each Mamba-type state, and the attention KV of sliding-window and full layers when TP exceeds the KV heads (below) | `kv__tp_r`, then `_c_g_l` with several KV cache groups | Qwen3.5-0.8B at TP=2: `_tp2_r0_c544_g0_l` on rank 0 | The layerwise format appends its `_L` and `_DONE` suffixes after the slice. A model with several KV cache groups (see {ref}`vllm-hybrid-models`) -appends `_c_g`, followed for an MLA group by +appends `_c_g_l`, followed for an MLA group by `_b`, since its objects keep vLLM's block layout (see -{ref}`vllm-deepseek-v41`). It names the KV heads of each attention group, full -or sliding window, as above: Qwen3.5-0.8B at TP=2 stores -`kv__h0-1_c544_g3` on rank 0. +{ref}`vllm-deepseek-v41`). `` is a 16-digit hex hash of the group's +layer names in vLLM's order: vLLM numbers groups by position, and another +configuration of the same model can put other layers in the group of the same +number, as DSpark speculative decoding does on DeepSeek-V4.1-Flash. It names +the KV heads of each attention group, full or sliding window, as above: +Qwen3.5-0.8B at TP=2 stores `kv__h0-1_c544_g3_l` on rank 0. Ranks that hold identical KV share one slice name: the MLA latent, and the heads vLLM replicates when there are more TP ranks than KV heads (4 KV heads @@ -726,7 +744,7 @@ backends still require TP=1. (vllm-hybrid-models)= ## Hybrid models -vLLM puts layers that keep different kinds of KV cache into separate KV cache groups, for example sliding-window and full-attention layers, or Gated DeltaNet (Mamba-type) and full-attention layers. On vLLM 0.30+ the connector stores each group's part of a chunk as its own CXL object, keyed `_c_g`, and reports a hit only for chunks that every group can resume from. A model with one group is stored as before. +vLLM puts layers that keep different kinds of KV cache into separate KV cache groups, for example sliding-window and full-attention layers, or Gated DeltaNet (Mamba-type) and full-attention layers. On vLLM 0.30+ the connector stores each group's part of a chunk as its own CXL object, keyed `_c_g_l` (see {ref}`vllm-tensor-parallelism` for ``), and reports a hit only for chunks that every group can resume from. A model with one group is stored as before. The key names the chunk length because a chunk key hashes the prompt only up to the chunk's end. With a Mamba-type group a chunk is one vLLM block, and vLLM sizes that block from the per-rank state and attention page, so engines with different TP sizes (or KV cache dtypes) can use different chunks: Qwen3.5-0.8B has 544-token chunks at TP=2 and 272-token chunks at TP=4. Their objects for the same prompt prefix then get different keys, and neither engine loads the other's. @@ -738,7 +756,7 @@ The connector refuses to start a model with several groups unless: - `maru_storage_backend` is `cxl` (the default) or `remote_cxl`. The CPU and mixed backends take one group only, and remote_cxl takes no MLA model, such as DeepSeek-V4.1. - `maru_use_layerwise` is `false` (the default), unless every group the connector stores is an MLA group, as DeepSeek-V4.1's are (see {ref}`vllm-deepseek-v41`). -- Speculative decoding is off. +- Speculative decoding is off, or is DSpark (`"method": "dspark"`) on a model without a Mamba-type group, as on DeepSeek-V4.1-Flash (see {ref}`vllm-deepseek-v41`). The connector refuses every other method. It refuses DSpark too with a Mamba-type group: in `align` mode vLLM moves a Mamba state block among the request's last blocks without telling the connector. - Mamba-type groups use vLLM's `align` Mamba cache mode. vLLM picks it when prefix caching is on, which is the default. Keep prefix caching on: with `--no-enable-prefix-caching`, vLLM sets the mode to `none` even if `--mamba-cache-mode align` is given, and the connector refuses the model. Do not pass `--mamba-cache-mode all`. - The model does not have both Mamba-type groups and MLA groups, as GLM-5.3-Flash does. - Every group uses vLLM's one `block_size`, except MLA groups, which may use other block sizes, as DeepSeek-V4.1's do; `maru_kv_chunk_tokens` is then rounded to a multiple of every group's block size. vLLM normally sets the groups up this way. @@ -781,7 +799,7 @@ vLLM splits the model's KV cache into several groups, how many depending on the - **The sliding-window groups.** With SWA bounded replay (`--swa-bounded-replay`, the default on CUDA with vLLM's V2 model runner), vLLM keeps them out of its prefix cache and, after a hit, recomputes the hit's last 128 tokens to rebuild them; Maru does not store them. With bounded replay off (`--no-swa-bounded-replay`, the V1 model runner, or ROCm), vLLM prefix-caches them, and Maru stores for each chunk the window blocks that a hit at the chunk's end needs. - **The KV compressor's circular buffer** is not stored, since vLLM does not prefix-cache it. -Both formats work. The model calls no per-layer connector method, so the layerwise format also stores after the forward and loads before it, without overlapping transfers with compute. With bounded replay on, a 256-token chunk takes one 467,712-byte page on SM120 in the chunkwise format, and about 2.9 times that in the layerwise format, which gives each KV cache and the `_DONE` marker a page of their own. With bounded replay off, each chunk also keeps the sliding windows' last blocks, several times the bytes of the MLA group's. Speculative decoding, which runs the DSpark draft layers in the model's checkpoint, is not supported yet, as for any model with several groups. +Both formats work. The model calls no per-layer connector method, so the layerwise format also stores after the forward and loads before it, without overlapping transfers with compute. With bounded replay on, a 256-token chunk takes one 467,712-byte page on SM120 in the chunkwise format, and about 2.9 times that in the layerwise format, which gives each KV cache and the `_DONE` marker a page of their own. With bounded replay off, each chunk also keeps the sliding windows' last blocks, several times the bytes of the MLA group's. Speculative decoding works with the DSpark draft layers in the model's checkpoint (`--speculative-config '{"method": "dspark", "num_speculative_tokens": 5}'`; vLLM 0.33.0 does not start without `num_speculative_tokens`); the connector refuses every other speculative decoding method. Each draft layer keeps a sliding window like the target's, and vLLM deals the draft windows into the target's sliding-window groups, splitting the windows into one more group, so Maru skips or stores them with the target's windows. Because the groups then hold other layers, objects stored with DSpark off are not found with it on, and the other way around. ```bash vllm serve deepseek-ai/DeepSeek-V4.1-Flash-1017 \ diff --git a/maru_vllm/connector.py b/maru_vllm/connector.py index 3a4b9e9e..c195c9ef 100644 --- a/maru_vllm/connector.py +++ b/maru_vllm/connector.py @@ -54,11 +54,13 @@ ) from maru_common.storage_types import StorageError, StorageUnavailableError from maru_vllm.kv_groups import ( + DSPARK_METHOD, GROUP_FULL, GROUP_MAMBA, GROUP_MLA, GROUP_SKIP, GROUP_SLIDING_WINDOW, + GroupAdmission, GroupCache, KVGroup, check_multi_group_support, @@ -997,6 +999,19 @@ def _vllm_version_at_least(minimum: str) -> bool: return False +def _speculative_method(vllm_config: VllmConfig) -> str | None: + """The speculative decoding method vLLM runs, or None without one. + + A config that names no method reads as an unknown one, so the support + checks refuse it on a model with several groups rather than take it for + no speculative decoding. + """ + spec = vllm_config.speculative_config + if spec is None: + return None + return str(getattr(spec, "method", None) or "unknown") + + def _scheduler_lookahead_tokens(vllm_config: VllmConfig) -> int: """Read vLLM's slot margin, with the pre-property scheduler rules as fallback.""" lookahead = getattr(vllm_config, "num_lookahead_tokens", None) @@ -1030,6 +1045,38 @@ def _async_load_budget( return total - 1 if total else None # vLLM sets one null block aside. +def _group_admission( + kv_cache_config: KVCacheConfig, vllm_config: VllmConfig +) -> tuple[GroupAdmission, ...]: + """The blocks vLLM gives one request in each KV cache group. + + A full-attention or MLA group holds a block per ``block_size`` tokens; a + sliding-window group at most vLLM's per-request admission cap, the + window plus the tokens in flight; a circular buffer one block. + """ + max_model_len = vllm_config.model_config.max_model_len + in_flight = getattr(vllm_config, "max_in_flight_tokens", None) + admission = [] + for group in kv_cache_config.kv_cache_groups: + spec = group.kv_cache_spec + members = getattr(spec, "kv_cache_specs", None) + if isinstance(members, dict) and members: + spec = next(iter(members.values())) + if type(spec).__name__ == "CircularBufferSpec": + admission.append(GroupAdmission(int(spec.block_size), fixed=1)) + continue + cap = getattr(spec, "max_admission_blocks_per_request", None) + admission.append( + GroupAdmission( + int(spec.block_size), + cap=int(cap(in_flight, max_model_len)) + if cap is not None and in_flight + else None, + ) + ) + return tuple(admission) + + def _uncacheable_request(request: Any) -> str | None: """The input that makes a request's KV depend on more than its token ids. @@ -1241,9 +1288,9 @@ class _SlabTarget: A single-group model stores every layer under the chunk key. With several KV cache groups each attention group is its own target, keyed - ``_c_g`` and reading that group's - block ids. Either way the chunk key names the slice this rank holds - (``MaruWorkerConnector._slice_chunk_keys``). + ``_c_g_l`` and + reading that group's block ids. Either way the chunk key names the slice + this rank holds (``MaruWorkerConnector._slice_chunk_keys``). Attributes: num_layers: Layers in the slab. @@ -1663,7 +1710,7 @@ def __init__( self._kv_groups, block_size=self._block_size, use_layerwise=bool(extra.get("maru_use_layerwise", False)), - speculative=vllm_config.speculative_config is not None, + speculative_method=_speculative_method(vllm_config), inline_loads=not _get_knob(extra, "maru_async_load"), ) @@ -1713,6 +1760,11 @@ def __init__( num_lookahead_tokens=lookahead, max_model_len=vllm_config.model_config.max_model_len, kv_groups=self._kv_groups, + group_admission=( + _group_admission(kv_cache_config, vllm_config) + if kv_cache_config is not None and len(self._kv_groups) > 1 + else () + ), shard_suffixes=[ shard.suffix for shard in ( @@ -1811,6 +1863,7 @@ def __init__( kv_cache_config.num_blocks if kv_cache_config is not None else None ), dp_rank=_dp_index(vllm_config), + store_after_forward=_speculative_method(vllm_config) == DSPARK_METHOD, ) @property @@ -1898,13 +1951,14 @@ def save_kv_layer( def wait_for_save(self): # A single-group model's synchronous stores complete inside - # save_kv_layer. Write-behind stores intentionally outlive this - # forward; request/block lifetime is tracked through + # save_kv_layer, unless DSpark runs (MaruWorkerConnector. + # stores_after_forward). Write-behind stores intentionally outlive + # this forward; request/block lifetime is tracked through # request_finished() + get_finished(). Several groups store here, # after the forward has written every group's KV and state. - if len(self._kv_groups) <= 1: - return assert self._worker is not None + if len(self._kv_groups) <= 1 and not self._worker.stores_after_forward: + return metadata = self._get_connector_metadata() assert isinstance(metadata, MaruConnectorMetadata) self._worker.wait_for_save(metadata) @@ -2103,6 +2157,7 @@ def __init__( kv_groups: list[KVGroup] | None = None, shard_suffixes: Sequence[str] = ("",), state_suffixes: Sequence[str] = ("",), + group_admission: Sequence[GroupAdmission] = (), dp_rank: int = 0, ): self._block_size = block_size @@ -2118,6 +2173,9 @@ def __init__( self._num_gpu_blocks = num_gpu_blocks self._num_lookahead_tokens = num_lookahead_tokens self._max_model_len = max_model_len + # With several KV cache groups, the blocks one request takes in each + # (_resume_blocks); empty for one group. + self._group_admission = tuple(group_admission) self._bypass_requests: set[str] = set() # Key suffix of every distinct KV slice the engine's workers hold # (one "" at TP=1). Each worker stores only its own slice, so a chunk @@ -2765,13 +2823,19 @@ def _admit_parked(self, request: Request) -> bool: return False def _resume_blocks(self, request: Request) -> int: - """Prompt plus resume lookahead, capped as in vLLM's slot allocation.""" + """Prompt plus resume lookahead, capped as in vLLM's slot allocation. + + With several KV cache groups, the sum over every group, since vLLM + allocates them all from one block pool (``GroupAdmission``). + """ tokens = getattr(request, "num_tokens", None) if tokens is None: tokens = len(request.prompt_token_ids or ()) tokens = int(tokens) + self._num_lookahead_tokens if self._max_model_len is not None: tokens = min(tokens, self._max_model_len) + if self._group_admission: + return sum(group.blocks(tokens) for group in self._group_admission) return -(-tokens // self._block_size) def _group_block_ids( @@ -3353,6 +3417,9 @@ class MaruWorkerConnector: _state_shard: KVShard = WHOLE_MODEL _use_mla = False _kv_planes = 2 + # A single-group chunkwise model stores from save_kv_layer unless DSpark + # runs (stores_after_forward). + _store_after_forward = False def __init__( self, @@ -3368,6 +3435,7 @@ def __init__( use_mla: bool = False, num_blocks: int | None = None, dp_rank: int = 0, + store_after_forward: bool = False, ): _validate_deferred_loading(bool(_get_knob(extra_config, "maru_async_load"))) self._block_size = block_size @@ -3530,6 +3598,14 @@ def __init__( # num_chunks x num_layers. Layerwise=True keeps the per-(chunk,layer) # objects. The overlap path below works under either format. self._use_layerwise = bool(extra_config.get("maru_use_layerwise", False)) + # With DSpark a single-group chunkwise model stores in wait_for_save, + # which vLLM calls after the drafter: DSpark's draft layers are + # registered beside the target's, but vLLM's V2 model runner replays + # the draft step as a full CUDA graph, which calls no save_kv_layer, + # so a store waiting there for every registered layer would never run. + self._store_after_forward = ( + store_after_forward and not self._multi_group and not self._use_layerwise + ) async_load = bool(_get_knob(extra_config, "maru_async_load")) overlap_requested = bool( _get_knob(extra_config, "maru_overlap_load_with_compute") @@ -6928,6 +7004,9 @@ def save_kv_layer( # key. Skipping here queues nothing, so get_finished_saving still # reports this rank's finished requests at once. return + if self._store_after_forward: + # wait_for_save stores every registered layer, the draft layers too. + return if not self._use_layerwise: self._save_kv_layer_packed(layer_name, kv_layer, attn_metadata, metadata) @@ -7211,8 +7290,15 @@ def wait_for_save(self, metadata: MaruConnectorMetadata) -> None: (``_gather_packed_slabs``). On remote_cxl with ``maru_async_store`` that ``batch_store``, which writes the staged objects by RDMA, runs on the write-behind thread (``_register_objects_behind``). + + A single-group model stores here only with DSpark + (``stores_after_forward``; see ``_store_single_group_after_forward``). """ - if not self._multi_group or not self._group_caches: + if not self._multi_group: + if self._store_after_forward: + self._store_single_group_after_forward(metadata) + return + if not self._group_caches: return if not self._store_entries(metadata): return @@ -7255,6 +7341,45 @@ def wait_for_save(self, metadata: MaruConnectorMetadata) -> None: self._register_objects(keys, handles, "group") self._register_done_markers(markers) + @property + def stores_after_forward(self) -> bool: + """Whether this single-group model stores in ``wait_for_save``.""" + return self._store_after_forward + + def _store_single_group_after_forward( + self, metadata: MaruConnectorMetadata + ) -> None: + """Store a single-group model's chunks after the forward and DSpark's drafter. + + DSpark's draft layers (on Qwen3, for one) are registered beside the + target's. vLLM's V2 model runner replays the draft step as a full + CUDA graph unless ``--enforce-eager``, and a replay calls no + ``save_kv_layer``, so a store waiting for every registered layer + would never run. vLLM calls ``wait_for_save`` after the drafter, so + every layer's KV of the step is in the paged buffers here, and each + chunk is stored whole, as from the last ``save_kv_layer`` call. + """ + if not self._stores() or not self._store_entries(metadata): + return + self._ensure_handler() + if self._handler is None: + return + attn_metadata = self._last_attn_metadata + kernel = self._packed_store_kernel_ctx(attn_metadata) + if kernel is None: + # The per-layer fallback fills each chunk's slab layer by layer. + for layer_name, kv_layer in self._kv_caches.items(): + self._save_kv_layer_packed( + layer_name, kv_layer, attn_metadata, metadata + ) + elif self._write_behind: + # Launched now rather than queued for get_finished_saving, which + # vLLM may already have called this step: it reserves the keys + # before the request can be reported finished. + self._store_packed_slabs_write_behind(kernel, metadata) + else: + self._store_packed_slabs(kernel, metadata) + def _register_done_markers(self, markers: dict[str, list[str]]) -> None: """Register the ``_DONE`` marker of each chunk whose layers are stored. diff --git a/maru_vllm/kv_groups.py b/maru_vllm/kv_groups.py index 76f590b9..f40c7cf2 100644 --- a/maru_vllm/kv_groups.py +++ b/maru_vllm/kv_groups.py @@ -18,14 +18,17 @@ (DeepSeek-V4.1's compressed KV and indexer keys) keeps the chunk's blocks, copied block by block in the layout vLLM gives them. A group vLLM does not prefix-cache is not stored. A model with one group keeps the bare chunk key; -with several, each object is keyed ``_c_g`` -(see :func:`group_object_key`). +with several, each object is keyed +``_c_g_l`` (see +:func:`group_object_key`). """ from __future__ import annotations +import functools +import hashlib import math -from collections.abc import Callable +from collections.abc import Callable, Sequence from dataclasses import dataclass from typing import Any @@ -40,6 +43,11 @@ GROUP_SKIP = "skip" GROUP_UNSUPPORTED = "unsupported" +# vLLM's name for DSpark speculative decoding, the one method the connector +# supports on a model with several KV cache groups and stores a single-group +# model's draft layers for. +DSPARK_METHOD = "dspark" + @dataclass(frozen=True) class KVGroup: @@ -133,7 +141,7 @@ def object_key(self, chunk_key: str, chunk_tokens: int) -> str: chunk_key: The chunk's key, with this rank's slice suffix. chunk_tokens: Tokens per Maru chunk. """ - key = group_object_key(chunk_key, self.group_id, chunk_tokens) + key = group_object_key(chunk_key, self.group_id, chunk_tokens, self.layer_names) return f"{key}_b{self.block_size}" if self.kind == GROUP_MLA else key def object_bytes(self, chunk_tokens: int) -> int: @@ -278,7 +286,23 @@ def _group_label(group: KVGroup) -> str: return group.spec_name -def group_object_key(chunk_key: str, group_id: int, chunk_tokens: int) -> str: +@functools.lru_cache(maxsize=256) +def layer_names_digest(layer_names: tuple[str, ...]) -> str: + """Hex digest of a group's layer names, in vLLM's order. + + vLLM numbers groups by position, and another configuration of the same + model can deal other layers into the group of the same id, in objects of + the same size: DSpark's draft layers make vLLM split DeepSeek-V4.1's + sliding windows into one more group. The order also fixes where each + layer's bytes lie in a block, since vLLM lays a group's layers out from + this list and their specs. + """ + return hashlib.blake2b("\n".join(layer_names).encode(), digest_size=8).hexdigest() + + +def group_object_key( + chunk_key: str, group_id: int, chunk_tokens: int, layer_names: Sequence[str] +) -> str: """Key of the (chunk, group) object for a model with several groups. A chunk key hashes the tokens from the prompt's start to the chunk's @@ -286,9 +310,11 @@ def group_object_key(chunk_key: str, group_id: int, chunk_tokens: int) -> str: does. It varies between engines: with a Mamba group a chunk is one vLLM block, sized from the per-rank Mamba state and attention page, so Qwen3.5-0.8B has 544-token chunks at TP=2 and 272-token ones at TP=4, - where rank 0 holds the same KV head and names it alike. + where rank 0 holds the same KV head and names it alike. The key also + names the group's layers (:func:`layer_names_digest`). """ - return f"{chunk_key}_c{chunk_tokens}_g{group_id}" + layers = layer_names_digest(tuple(layer_names)) + return f"{chunk_key}_c{chunk_tokens}_g{group_id}_l{layers}" def window_chunks(group: KVGroup, end: int, chunk_tokens: int) -> range: @@ -354,6 +380,33 @@ def resumable(k: int, g: KVGroup) -> bool: return 0, prefix +@dataclass(frozen=True) +class GroupAdmission: + """The blocks vLLM gives one request in one KV cache group. + + vLLM allocates the blocks of every group, the groups Maru skips included, + from one block pool, so a parked load is admitted against their sum. + + Attributes: + block_size: Tokens per block in the group. + cap: vLLM's per-request admission cap for a sliding-window group + (``max_admission_blocks_per_request``), None for no cap. + fixed: Blocks a request holds whatever its length, as a circular + buffer's one block; None otherwise. + """ + + block_size: int + cap: int | None = None + fixed: int | None = None + + def blocks(self, tokens: int) -> int: + """Blocks a request of ``tokens`` tokens holds in this group.""" + if self.fixed is not None: + return self.fixed + blocks = -(-tokens // self.block_size) + return blocks if self.cap is None else min(blocks, self.cap) + + def chunk_unit(groups: list[KVGroup], block_size: int) -> int: """Tokens a chunk is a multiple of, so it holds whole blocks of each group. @@ -419,7 +472,7 @@ def check_multi_group_support( *, block_size: int, use_layerwise: bool, - speculative: bool, + speculative_method: str | None, inline_loads: bool, ) -> None: """Reject group combinations the connector cannot store correctly. @@ -428,8 +481,13 @@ def check_multi_group_support( Several groups are supported in chunkwise (packed) format with full attention, sliding window, Mamba (``align``) and MLA groups whose attention groups share one KV shape, but not a Mamba group beside an MLA - group, with parked loads, without speculative decoding; in layerwise - format only when every stored group is an MLA group (DeepSeek-V4.1). + group, with parked loads; in layerwise format only when every stored + group is an MLA group (DeepSeek-V4.1). The one speculative decoding + method supported is DSpark (``DSPARK_METHOD``), without a Mamba group: + vLLM puts its draft layers in groups like any other layer (DeepSeek-V4.1's + draft windows in the target's window groups), and the drafter writes a + token's draft KV in the step that computes the token, before the + connector stores it. Every group uses vLLM's block size, but an MLA group may use another (DeepSeek-V4.1's 64-token MLA blocks beside its 32-token windows); the chunk then holds whole blocks of each (``chunk_unit``). Groups vLLM does @@ -441,7 +499,8 @@ def check_multi_group_support( groups: The model's KV cache groups. block_size: ``cache_config.block_size``, which the connector uses. use_layerwise: ``maru_use_layerwise``. - speculative: Whether speculative decoding is configured. + speculative_method: The speculative decoding method configured, or + None without speculative decoding. inline_loads: Whether loads run inline (``maru_async_load`` set to false; ``MaruKVConnector`` turns it on for several groups unless set). @@ -485,12 +544,25 @@ def check_multi_group_support( "(DeepSeek-V4.1); otherwise only in chunkwise format. Set " f"maru_use_layerwise=false{one_group_hint(groups)}" ) - if speculative: + if speculative_method is not None and speculative_method != DSPARK_METHOD: raise ValueError( - "MaruKVConnector does not support speculative decoding with models " - f"that have several KV cache groups{one_group_hint(groups)}" + "MaruKVConnector supports speculative decoding on models with " + f"several KV cache groups only with DSpark ({DSPARK_METHOD!r}), " + f"not {speculative_method!r}{one_group_hint(groups)}" ) kinds = {g.kind for g in stored} + if speculative_method is not None and GROUP_MAMBA in kinds: + # In align mode vLLM keeps a block per draft token at the end of a + # request's Mamba block list and moves one into place when a state + # lands there, but a connector learns only the blocks appended. The + # block list the scheduler tracks can then name, at a chunk's end, a + # block no kernel wrote, and Maru would store it as the chunk's state. + raise ValueError( + "MaruKVConnector does not support speculative decoding " + f"({speculative_method!r}) with a Mamba-type KV cache group: vLLM " + "moves a Mamba state block within the request's last blocks " + "without telling the connector" + ) if {GROUP_MAMBA, GROUP_MLA} <= kinds: # A Mamba group pins the chunk to one block (chunk_tokens_for_groups), # which holds no whole block of an MLA group with larger blocks, and diff --git a/tests/unit/test_vllm_connector.py b/tests/unit/test_vllm_connector.py index eecc8209..214abb64 100644 --- a/tests/unit/test_vllm_connector.py +++ b/tests/unit/test_vllm_connector.py @@ -4264,11 +4264,14 @@ class TestPackedStorage: NUM_LAYERS = 3 PROMPT = 16 # 2 chunks - def _make_worker(self): + def _make_worker(self, store_after_forward: bool = False): from maru_vllm.connector import MaruWorkerConnector worker = MaruWorkerConnector( - block_size=self.BLOCK, kv_chunk_tokens=self.CHUNK, extra_config={} + block_size=self.BLOCK, + kv_chunk_tokens=self.CHUNK, + extra_config={}, + store_after_forward=store_after_forward, ) assert worker._use_layerwise is False # default worker._num_layers = self.NUM_LAYERS @@ -4538,6 +4541,103 @@ def test_write_behind_launches_only_after_forward_completion(self): ) assert worker._queued_store_batches == [] + def _store_meta(self): + from maru_vllm.connector import MaruConnectorMetadata, MaruReqMeta + + return MaruConnectorMetadata( + requests=[ + MaruReqMeta( + req_id="r1", + token_ids=list(range(self.PROMPT)), + block_ids=[0, 1, 2, 3], + is_store=True, + num_scheduled_tokens=self.PROMPT, + num_computed_tokens=0, + ) + ] + ) + + def test_draft_layers_are_stored_after_the_forward(self): + """A DSpark draft layer that never calls save_kv_layer still lands in + each chunk's slab, stored from wait_for_save.""" + from maru_vllm.connector import _chunk_keys + + worker, store = self._make_worker(store_after_forward=True) + kv_layers = self._kv_layers() + names = [f"model.layers.{i}.self_attn" for i in range(self.NUM_LAYERS)] + worker._kv_caches = dict(zip(names, kv_layers, strict=True)) + worker._packed_store_kernel_ctx = lambda attn: None # per-layer fallback + attn = self._flash_attn() + meta = self._store_meta() + + # The target's layers call save_kv_layer; the last, the drafter's, not. + for name, kv in zip(names[:-1], kv_layers[:-1], strict=True): + worker.save_kv_layer(name, kv, attn, meta) + assert store == {} + + worker.wait_for_save(meta) + + chunk_keys = _chunk_keys(list(range(self.PROMPT)), self.CHUNK) + assert set(store) == set(chunk_keys) + assert worker._pending_slabs == {} + slots = worker._build_slot_mapping([0, 1, 2, 3], self.PROMPT) + for ci, key in enumerate(chunk_keys): + slab = torch.frombuffer(bytearray(store[key]), dtype=torch.float32) + slab = slab.view(2, self.NUM_LAYERS, self.CHUNK, 1) + chunk_slots = slots[ci * self.CHUNK : (ci + 1) * self.CHUNK] + for li, kv in enumerate(kv_layers): + expected = kv.reshape(2, -1, 1)[:, chunk_slots] + assert torch.equal(slab[:, li], expected), f"chunk {ci} layer {li}" + + @pytest.mark.parametrize("write_behind", [False, True]) + def test_dspark_kernel_store_runs_once_after_the_forward(self, write_behind): + """Nothing is stored or queued from save_kv_layer; wait_for_save + stores the step once, a write-behind store launched at once.""" + worker, _ = self._make_worker(store_after_forward=True) + worker._write_behind = write_behind + kernel = ("ops",) * 6 + worker._packed_store_kernel_ctx = lambda attn: kernel + worker._store_packed_slabs = MagicMock() + worker._store_packed_slabs_write_behind = MagicMock() + attn = self._flash_attn() + meta = self._store_meta() + names = [f"model.layers.{i}.self_attn" for i in range(self.NUM_LAYERS)] + + for name, kv in zip(names, self._kv_layers(), strict=True): + worker.save_kv_layer(name, kv, attn, meta) + worker._store_packed_slabs.assert_not_called() + worker._store_packed_slabs_write_behind.assert_not_called() + assert worker._queued_store_batches == [] + + worker.wait_for_save(meta) + + used, unused = ( + worker._store_packed_slabs_write_behind, + worker._store_packed_slabs, + ) + if not write_behind: + used, unused = unused, used + used.assert_called_once_with(kernel, meta) + unused.assert_not_called() + + def test_without_dspark_wait_for_save_stores_nothing(self): + """A single-group model stores from save_kv_layer, as before.""" + worker, store = self._make_worker() + assert not worker.stores_after_forward + worker._packed_store_kernel_ctx = lambda attn: None + worker._kv_caches = dict( + zip( + [f"model.layers.{i}.self_attn" for i in range(self.NUM_LAYERS)], + self._kv_layers(), + strict=True, + ) + ) + + worker.wait_for_save(self._store_meta()) + + assert store == {} + worker._handler.alloc.assert_not_called() + def test_kernel_ctx_caching_contract(self): """_packed_store_kernel_ctx resolves once and caches the outcome: empty caches -> unresolved (re-probed later), unusable kernel -> diff --git a/tests/unit/test_vllm_kv_groups.py b/tests/unit/test_vllm_kv_groups.py index 7d970e97..1a448c60 100644 --- a/tests/unit/test_vllm_kv_groups.py +++ b/tests/unit/test_vllm_kv_groups.py @@ -9,6 +9,7 @@ import dataclasses import logging +import re import time from types import SimpleNamespace from unittest.mock import MagicMock @@ -58,6 +59,7 @@ describe_kv_groups, group_object_key, hit_chunks, + layer_names_digest, state_view, window_chunks, ) @@ -250,9 +252,17 @@ def _group(specs, side): return KVCacheGroupSpec(list(specs), spec) -def dsv41_config(block=DSV41_BLOCK, side="worker", window=None): +def dsv41_config( + block=DSV41_BLOCK, side="worker", window=None, drafts=False, windows=None +): """DSV4.1-Flash's six groups: four sliding windows, the compressed KV and - indexer keys, and the KV compressor's circular buffer.""" + indexer keys, and the KV compressor's circular buffer. + + With ``drafts``, DSpark's draft layers 40-42 each add a sliding window like + the target's, and vLLM deals the windows into one more group, as it does on + the GPU (seven groups to eight), so a window group of the same id holds + other layers. ``windows`` sets the number of window groups. + """ window = window or _window_mla(block // 4) mla = { name: MLAAttentionSpec( @@ -260,9 +270,11 @@ def dsv41_config(block=DSV41_BLOCK, side="worker", window=None): ) for name, size in DSV41_MEMBERS.items() } + layers = range(43 if drafts else 40) + windows = windows or (5 if drafts else 4) groups = [ - _group({f"layers.{i}.attn.swa": window for i in range(g, 40, 4)}, side) - for g in range(4) + _group({f"layers.{i}.attn.swa": window for i in layers[g::windows]}, side) + for g in range(windows) ] groups.append(_group(mla, side)) ring = _ring(8) @@ -275,7 +287,7 @@ def _check_dsv41(config): describe_kv_groups(config), block_size=DSV41_BLOCK, use_layerwise=False, - speculative=False, + speculative_method=None, inline_loads=False, ) @@ -296,7 +308,7 @@ def test_dsv41_passes_the_support_check_in_the_layerwise_format(): describe_kv_groups(dsv41_config()), block_size=DSV41_BLOCK, use_layerwise=True, - speculative=False, + speculative_method=None, inline_loads=False, ) @@ -337,17 +349,75 @@ def test_an_mla_object_key_names_its_block_size(): describe_kv_groups(dsv41_config(block=b))[DSV41_GROUP] for b in (DSV41_BLOCK, 2 * DSV41_BLOCK) ) - assert small.object_key("kv_x", 64) == "kv_x_c64_g4_b16" - assert large.object_key("kv_x", 64) == "kv_x_c64_g4_b32" - full = KVGroup( + layers = layer_names_digest(small.layer_names) + assert small.object_key("kv_x", 64) == f"kv_x_c64_g4_l{layers}_b16" + assert large.object_key("kv_x", 64) == f"kv_x_c64_g4_l{layers}_b32" + full = _full_group(("layers.0.attn",)) + assert full.object_key("kv_x", 64) == group_object_key( + "kv_x", 1, 64, ("layers.0.attn",) + ) + + +def _full_group(layer_names): + return KVGroup( group_id=1, kind=GROUP_FULL, spec_name="FullAttentionSpec", - layer_names=("layers.0.attn",), + layer_names=layer_names, block_size=16, slot_bytes=8, ) - assert full.object_key("kv_x", 64) == group_object_key("kv_x", 1, 64) + + +def test_a_group_object_key_names_the_groups_layers(): + """vLLM numbers groups by position, so another configuration can deal + other layers into the group of the same id. The key names the group's + layers in vLLM's order, which also fixes their order in each block.""" + key = _full_group(("layers.0.attn", "layers.2.attn")).object_key("kv_x", 64) + assert re.fullmatch(r"kv_x_c64_g1_l[0-9a-f]{16}", key) + assert _full_group(("layers.0.attn", "layers.2.attn")).object_key("kv_x", 64) == key + for other in ( + ("layers.0.attn", "layers.3.attn"), + ("layers.2.attn", "layers.0.attn"), + ("layers.0.attn",), + ): + assert _full_group(other).object_key("kv_x", 64) != key + + +def test_dspark_changes_the_keys_of_window_groups_that_hold_other_layers(): + """As on the GPU, vLLM deals DeepSeek-V4.1's 40 windows into seven groups, + and with DSpark's three draft windows into eight. Window group 0 then + holds six other layers, in objects of the same size, so only its layers + tell its objects apart. The MLA group keeps its layers under another id.""" + window = _window_mla(DSV41_BLOCK // 4, bounded_replay=False) + plain = describe_kv_groups(dsv41_config(window=window, windows=7)) + dspark = describe_kv_groups(dsv41_config(window=window, drafts=True, windows=8)) + chunk = 4 * DSV41_BLOCK + assert len(plain[0].layer_names) == len(dspark[0].layer_names) == 6 + assert plain[0].object_bytes(chunk) == dspark[0].object_bytes(chunk) + assert plain[0].object_key("kv_x", chunk) != dspark[0].object_key("kv_x", chunk) + assert plain[7].kind == dspark[8].kind == GROUP_MLA + assert plain[7].layer_names == dspark[8].layer_names + assert layer_names_digest(plain[7].layer_names) in dspark[8].object_key( + "kv_x", chunk + ) + + +@pytest.mark.parametrize("drafts", [False, True], ids=["plain", "dspark"]) +def test_scheduler_and_worker_key_dsv41_objects_alike(drafts): + """The scheduler looks objects up by the keys the worker stores them under, + though it sees each group as its first member's spec.""" + window = _window_mla(DSV41_BLOCK // 4, bounded_replay=False) + keys = [ + [ + g.object_key("kv_x", 64) + for g in describe_kv_groups( + dsv41_config(side=side, window=window, drafts=drafts) + ) + ] + for side in ("scheduler", "worker") + ] + assert keys[0] == keys[1] def test_a_model_with_only_skipped_groups_is_refused(): @@ -377,11 +447,100 @@ def test_dsv41_without_bounded_replay_passes_the_support_check(layerwise): describe_kv_groups(dsv41_config(window=window)), block_size=DSV41_BLOCK // 4, use_layerwise=layerwise, - speculative=False, + speculative_method=None, inline_loads=False, ) +@pytest.mark.parametrize("bounded_replay", [True, False], ids=["replay", "no-replay"]) +@pytest.mark.parametrize("layerwise", [False, True], ids=["chunkwise", "layerwise"]) +def test_dsv41_with_dspark_passes_the_support_check(bounded_replay, layerwise): + """DSpark's draft layers keep sliding windows like the target's, in the + target's window groups, which Maru skips or copies block by block.""" + block = DSV41_BLOCK // 4 + window = _window_mla(block, bounded_replay=bounded_replay) + check_multi_group_support( + describe_kv_groups(dsv41_config(window=window, drafts=True)), + block_size=DSV41_BLOCK if bounded_replay else block, + use_layerwise=layerwise, + speculative_method="dspark", + inline_loads=False, + ) + + +@pytest.mark.parametrize("method", ["ngram", "eagle3", "mtp", "draft_model", "unknown"]) +def test_dsv41_with_another_speculative_method_is_rejected(method): + """Only DSpark has been run with several groups.""" + with pytest.raises( + ValueError, match=f"only with DSpark \\('dspark'\\), not '{method}'" + ): + check_multi_group_support( + describe_kv_groups(dsv41_config()), + block_size=DSV41_BLOCK, + use_layerwise=False, + speculative_method=method, + inline_loads=False, + ) + + +def test_connector_starts_with_dsv41_and_dspark(): + config = _connector_config(128) + config.speculative_config = SimpleNamespace( + method="dspark", num_speculative_tokens=5 + ) + config.num_lookahead_tokens = 5 + connector = MaruKVConnector( + config, + KVConnectorRole.SCHEDULER, + dsv41_config(block=128, side="scheduler", drafts=True), + ) + assert connector._scheduler._multi_group + assert connector._scheduler._num_lookahead_tokens == 5 + + +def _dspark_scheduler(in_flight=2048): + """DeepSeek-V4.1 with DSpark as on the GPU: eight 32-token window groups, + the 64-token MLA group and the 8-token ring, 5 lookahead tokens, and a + model length of 4096.""" + config = _connector_config(64) + config.speculative_config = SimpleNamespace( + method="dspark", num_speculative_tokens=5 + ) + config.num_lookahead_tokens = 5 + config.max_in_flight_tokens = in_flight + window = _window_mla(32) + connector = MaruKVConnector( + config, + KVConnectorRole.SCHEDULER, + dsv41_config(block=64, side="scheduler", window=window, drafts=True, windows=8), + ) + return connector._scheduler, window + + +def _prompt(tokens): + return SimpleNamespace( + request_id="r", prompt_token_ids=list(range(tokens)), num_tokens=tokens + ) + + +def test_dspark_admission_counts_the_blocks_of_every_group(): + """vLLM allocates every group's blocks from one pool, the groups Maru + skips included, so a parked request needs the blocks of all of them for + its prompt and lookahead: 1,005 tokens take 16 MLA blocks, 32 blocks in + each window group, and the ring's one block.""" + sched, _ = _dspark_scheduler() + assert sched._resume_blocks(_prompt(1000)) == 16 + 8 * 32 + 1 + + +def test_dspark_admission_caps_the_window_groups_as_vllm_does(): + """A window group holds at most vLLM's per-request admission cap, the + window plus the tokens in flight, however long the prompt.""" + sched, window = _dspark_scheduler() + cap = window.max_admission_blocks_per_request(2048, 4096) + assert cap < -(-4005 // 32) + assert sched._resume_blocks(_prompt(4000)) == 63 + 8 * cap + 1 + + @pytest.mark.parametrize( ("setting", "chunk"), [(None, 256), (96, 64), (32, 64), (320, 320)] ) @@ -430,10 +589,9 @@ def test_a_mamba_group_beside_an_mla_group_is_refused(ratio): def test_a_refused_model_does_not_first_warn_about_its_chunk(caplog): """The support checks run before the chunk is rounded, so a model they refuse leaves no warning about its chunk size.""" - config = _connector_config(128, maru_kv_chunk_tokens=200) - config.speculative_config = SimpleNamespace(num_speculative_tokens=1) + config = _connector_config(128, maru_kv_chunk_tokens=200, maru_async_load=False) with caplog.at_level(logging.WARNING, logger="maru_vllm.connector"): - with pytest.raises(ValueError, match="speculative decoding"): + with pytest.raises(ValueError, match="maru_async_load=false"): MaruKVConnector( config, KVConnectorRole.SCHEDULER, @@ -695,9 +853,15 @@ def test_dsv41_store_allocates_in_the_requests_pool(): ) +def _gkey(config, chunk_key, group_id, chunk): + """The object key of ``config``'s group ``group_id`` for one chunk.""" + return describe_kv_groups(config)[group_id].object_key(chunk_key, chunk) + + def _mla_key(chunk_key, group_id, chunk, block=DSV41_BLOCK): """An MLA group's object key, which also names the group's block size.""" - return f"{group_object_key(chunk_key, group_id, chunk)}_b{block}" + names = describe_kv_groups(dsv41_config())[group_id].layer_names + return f"{group_object_key(chunk_key, group_id, chunk, names)}_b{block}" def _dsv41_keys(tokens, chunk, num_chunks, layerwise, block=DSV41_BLOCK): @@ -1340,13 +1504,17 @@ def test_dsv41_unbounded_lookup_needs_each_windows_chunk_at_the_hit_end(): def _check( - groups, block_size, use_layerwise=False, speculative=False, inline_loads=False + groups, + block_size, + use_layerwise=False, + speculative_method=None, + inline_loads=False, ): check_multi_group_support( groups, block_size=block_size, use_layerwise=use_layerwise, - speculative=speculative, + speculative_method=speculative_method, inline_loads=inline_loads, ) @@ -1358,7 +1526,7 @@ def test_supported_multi_group_models_pass(): def test_single_group_is_always_accepted(): single = describe_kv_groups(qwen35_config())[3:] - _check(single, 16, use_layerwise=True, speculative=True) + _check(single, 16, use_layerwise=True, speculative_method="eagle3") def test_subclass_of_a_supported_spec_is_rejected(): @@ -1381,9 +1549,28 @@ def test_layerwise_is_rejected(): _check(describe_kv_groups(gpt_oss_config()), 16, use_layerwise=True) -def test_speculative_decoding_is_rejected(): - with pytest.raises(ValueError, match="speculative decoding"): - _check(describe_kv_groups(gpt_oss_config()), 16, speculative=True) +def test_dspark_passes_without_a_mamba_group(): + _check(describe_kv_groups(gpt_oss_config()), 16, speculative_method="dspark") + + +@pytest.mark.parametrize("method", ["ngram", "eagle3", "mtp", "draft_model"]) +def test_another_speculative_method_is_rejected_with_several_groups(method): + with pytest.raises( + ValueError, match=f"only with DSpark \\('dspark'\\), not '{method}'" + ): + _check(describe_kv_groups(gpt_oss_config()), 16, speculative_method=method) + + +def test_dspark_is_rejected_with_a_mamba_group(): + """vLLM moves an align-mode Mamba state block among a request's last + blocks without telling the connector, so a stored state could be a block + no kernel wrote.""" + with pytest.raises(ValueError, match="speculative decoding \\('dspark'\\)"): + _check( + describe_kv_groups(qwen35_config()), + QWEN35_BLOCK, + speculative_method="dspark", + ) @pytest.mark.parametrize("mode", ["none", "all"]) @@ -1815,8 +2002,11 @@ def test_group_objects_name_their_chunk_length(): assert tp2 == tp4 heads = tp_kv_shard(0, 2, 2).suffix assert tp_kv_shard(0, 4, 2).suffix == heads == "_h0-1" - assert group_object_key(tp2 + heads, 3, QWEN35_BLOCK) == f"{tp2}_h0-1_c544_g3" - assert group_object_key(tp4 + heads, 3, QWEN35_BLOCK // 2) == f"{tp2}_h0-1_c272_g3" + layers = layer_names_digest(describe_kv_groups(qwen35_config())[3].layer_names) + tp2_key = _gkey(qwen35_config(), tp2 + heads, 3, QWEN35_BLOCK) + tp4_key = _gkey(qwen35_config(), tp4 + heads, 3, QWEN35_BLOCK // 2) + assert tp2_key == f"{tp2}_h0-1_c544_g3_l{layers}" + assert tp4_key == f"{tp2}_h0-1_c272_g3_l{layers}" class _Store: @@ -1831,12 +2021,14 @@ def batch_exists(self, keys, route_key=None): return [k in self.keys for k in keys] -def _stored_by(tokens, chunk, attention_groups, state_groups, state_chunks): - """Keys a request stores: attention objects for every chunk, states for some.""" +def _stored_by(source, tokens, chunk, attention_groups, state_groups, state_chunks): + """Keys a request stores under ``source``'s groups (a config, or the groups + a connector holds): attention objects for every chunk, states for some.""" + groups = source if isinstance(source, list) else describe_kv_groups(source) keys = _chunk_keys(tokens, chunk) - stored = {group_object_key(k, g, chunk) for k in keys for g in attention_groups} + stored = {groups[g].object_key(k, chunk) for k in keys for g in attention_groups} stored |= { - group_object_key(keys[i], g, chunk) for i in state_chunks for g in state_groups + groups[g].object_key(keys[i], chunk) for i in state_chunks for g in state_groups } return stored @@ -1867,20 +2059,24 @@ def test_qwen35_prompt_of_whole_chunks_hits_nothing(): """1,088 tokens in one prefill step: the only state is at the last chunk, which the held-back token excludes.""" a = list(range(1088)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) assert _hit(sched, a) == 0 def test_qwen35_same_prompt_hits_up_to_the_state(): a = list(range(1200)) # step 1 ends at token 1,088 - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) assert _hit(sched, a) == 1088 def test_qwen35_partial_overlap_hits_only_at_a_step_end(): """A (2,300 tokens) ended prefill steps at tokens 1,632 and 2,176.""" a = list(range(2300)) - stored = _stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) + stored = _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) sched = _qwen35_scheduler(stored) assert _hit(sched, a[:1500] + _other(300)) == 0 assert _hit(sched, a[:1700] + _other(300)) == 1632 @@ -1889,7 +2085,7 @@ def test_qwen35_partial_overlap_hits_only_at_a_step_end(): def test_gpt_oss_hits_whole_chunks(): sched = make_scheduler(16, 256, kv_groups=describe_kv_groups(gpt_oss_config())) a = list(range(1200)) - sched._handler = _Store(_stored_by(a, 256, [0, 1], [], [])) + sched._handler = _Store(_stored_by(gpt_oss_config(), a, 256, [0, 1], [], [])) assert _hit(sched, a) == 1024 # A whole-chunk prompt holds back a chunk, not a block as one group would: # a hit inside a chunk could need window tokens freed before the store. @@ -1898,19 +2094,23 @@ def test_gpt_oss_hits_whole_chunks(): def test_lookup_asks_every_group_in_one_rpc(): a = list(range(2300)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) + ) b = a[:1700] + _other(300) assert _hit(sched, b) == 1632 (query,) = sched._handler.queries keys = _chunk_keys(b, QWEN35_BLOCK)[:3] assert query == [ - group_object_key(k, g, QWEN35_BLOCK) for k in keys for g in range(4) + _gkey(qwen35_config(), k, g, QWEN35_BLOCK) for k in keys for g in range(4) ] def test_known_keys_that_give_every_chunk_skip_the_rpc(): a = list(range(2300)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) + ) b = a[:1700] + _other(300) # 3 chunks may hit, and all 3 do assert _hit(sched, b) == 1632 assert _hit(sched, b) == 1632 @@ -1919,7 +2119,7 @@ def test_known_keys_that_give_every_chunk_skip_the_rpc(): def test_a_short_hit_asks_again_for_the_missing_keys_only(): a = list(range(2300)) - stored = _stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) + stored = _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) sched = _qwen35_scheduler(stored) b = a[:1500] + _other(300) assert _hit(sched, b) == 0 @@ -1930,7 +2130,7 @@ def test_a_short_hit_asks_again_for_the_missing_keys_only(): def test_a_short_reply_leaves_the_rest_missing(): a = list(range(1200)) - stored = _stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + stored = _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) sched = _qwen35_scheduler(stored) answer = sched._handler.batch_exists # Only chunk 0's four keys are answered. @@ -1960,7 +2160,9 @@ def test_a_request_whose_load_failed_computes_without_maru(caplog): exist (a mapping error, say), so the request gets no hit until it finishes, without asking MaruServer, and the log says so once.""" a = list(range(1200)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) request = SimpleNamespace(request_id="b", prompt_token_ids=a, num_preemptions=0) assert _load_given(sched, request) == 1088 _report_failure(sched) @@ -1980,11 +2182,13 @@ def test_a_later_request_asks_maru_again_only_after_a_reported_failure(report): the same failing hit. A failure comes by its blocks, or by its request for a parked load of several groups. Without one the known keys stand.""" a = list(range(1200)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) first = SimpleNamespace(request_id="b", prompt_token_ids=a) assert _load_given(sched, first) == 1088 sched._handler.keys.discard( - group_object_key(_chunk_keys(a, QWEN35_BLOCK)[1], 1, QWEN35_BLOCK) + _gkey(qwen35_config(), _chunk_keys(a, QWEN35_BLOCK)[1], 1, QWEN35_BLOCK) ) if report == "blocks": _report_failure(sched) @@ -2008,7 +2212,9 @@ def test_a_later_request_asks_maru_again_only_after_a_reported_failure(report): def test_a_request_asked_for_once_more_trusts_the_known_keys(): """Asked again before any blocks (vLLM could not fit it): no extra RPC.""" a = list(range(1200)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) request = SimpleNamespace(request_id="b", prompt_token_ids=a) for _ in range(2): assert sched.get_num_new_matched_tokens(request, 0)[0] == 1088 @@ -2019,7 +2225,9 @@ def test_a_preempted_request_keeps_its_hit(): """A preemption is not a failure: however often the request is preempted, it loads again, from the known keys.""" a = list(range(1200)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) request = SimpleNamespace(request_id="b", prompt_token_ids=a, num_preemptions=0) assert _load_given(sched, request) == 1088 for preemptions in range(1, 4): @@ -2032,7 +2240,9 @@ def test_a_request_waiting_for_blocks_asks_maru_once(): """vLLM asks a request it cannot fit again every step: after a failure only the first of those asks goes to MaruServer.""" a = list(range(1200)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) first = SimpleNamespace(request_id="b", prompt_token_ids=a) assert _load_given(sched, first) == 1088 _report_failure(sched) @@ -2052,7 +2262,9 @@ def test_a_failed_parked_load_computes_without_maru(): {"maru_async_load": True}, kv_groups=describe_kv_groups(qwen35_config()), ) - sched._handler = _Store(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [1])) + sched._handler = _Store( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [1]) + ) request = SimpleNamespace(request_id="b", prompt_token_ids=a, num_preemptions=0) assert _load_given(sched, request) == 1088 sched.update_state_after_alloc(request, MagicMock(), 0) # the load is done @@ -2121,7 +2333,9 @@ def test_a_single_group_request_asks_maru_again_after_a_failure(): def test_hit_counts_past_the_gpu_prefix_cache(): a = list(range(2300)) - sched = _qwen35_scheduler(_stored_by(a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3])) + sched = _qwen35_scheduler( + _stored_by(qwen35_config(), a, QWEN35_BLOCK, [3], [0, 1, 2], [2, 3]) + ) b = a[:1700] + _other(300) assert _hit(sched, b, num_computed_tokens=544) == 1088 assert _hit(sched, b, num_computed_tokens=1632) == 0 @@ -2373,7 +2587,7 @@ def test_a_store_reads_split_kernel_blocks_in_token_order(version): for key, block in zip(_chunk_keys(tokens, TINY_BLOCK), (4, 5), strict=True): expected = _expected_slab(attn, [block], version, kernel_block=8) assert torch.equal( - _slab(store[group_object_key(key, 3, TINY_BLOCK)], expected), expected + _slab(store[_gkey(tiny_config(), key, 3, TINY_BLOCK)], expected), expected ) @@ -2571,19 +2785,19 @@ def test_a_step_ending_on_a_chunk_stores_every_group(version): # Attention: both chunks the step completed. Mamba: only the chunk the # step ended on; chunk 0's state was overwritten within the step. assert set(store) == { - group_object_key(k0, 3, TINY_BLOCK), - group_object_key(k1, 3, TINY_BLOCK), - *(group_object_key(k1, g, TINY_BLOCK) for g in range(3)), + _gkey(tiny_config(), k0, 3, TINY_BLOCK), + _gkey(tiny_config(), k1, 3, TINY_BLOCK), + *(_gkey(tiny_config(), k1, g, TINY_BLOCK) for g in range(3)), } attn = worker._group_caches[3].tensors for key, block in ((k0, 4), (k1, 5)): expected = _expected_slab(attn, [block], version) assert torch.equal( - _slab(store[group_object_key(key, 3, TINY_BLOCK)], expected), expected + _slab(store[_gkey(tiny_config(), key, 3, TINY_BLOCK)], expected), expected ) # A state object holds each layer's state at a stride of one block slot. for g in range(3): - obj = store[group_object_key(k1, g, TINY_BLOCK)] + obj = store[_gkey(tiny_config(), k1, g, TINY_BLOCK)] assert len(obj) == 2 * TINY_SLOT block = QWEN_BLOCKS[g][1] for p in range(2): @@ -2600,7 +2814,7 @@ def test_a_step_ending_inside_a_chunk_stores_no_state(): _store_meta(tokens, ([0, 0, 1], [0, 0, 2], [0, 0, 3], [4, 5, 6]), 0, 40) ) keys = _chunk_keys(tokens, TINY_BLOCK) - assert set(store) == {group_object_key(k, 3, TINY_BLOCK) for k in keys[:2]} + assert set(store) == {_gkey(tiny_config(), k, 3, TINY_BLOCK) for k in keys[:2]} def test_a_null_state_block_stores_no_state(): @@ -2608,8 +2822,8 @@ def test_a_null_state_block_stores_no_state(): tokens = list(range(40)) worker.wait_for_save(_store_meta(tokens, ([0, 0], [0, 2], [0, 3], [4, 5]), 0, 32)) k1 = _chunk_keys(tokens, TINY_BLOCK)[1] - assert group_object_key(k1, 0, TINY_BLOCK) not in store - assert group_object_key(k1, 1, TINY_BLOCK) in store + assert _gkey(tiny_config(), k1, 0, TINY_BLOCK) not in store + assert _gkey(tiny_config(), k1, 1, TINY_BLOCK) in store def test_shared_prefix_is_stored_once_per_key(): @@ -2637,9 +2851,13 @@ def test_sliding_window_reads_the_null_block_behind_the_window(): (k0,) = _chunk_keys(tokens, 64) swa, full = (c.tensors for c in worker._group_caches) expected = _expected_slab(swa, [0, 8, 9, 10], "0.30") - assert torch.equal(_slab(store[group_object_key(k0, 0, 64)], expected), expected) + assert torch.equal( + _slab(store[_gkey(tiny_swa_config(), k0, 0, 64)], expected), expected + ) expected = _expected_slab(full, [3, 4, 5, 6], "0.30") - assert torch.equal(_slab(store[group_object_key(k0, 1, 64)], expected), expected) + assert torch.equal( + _slab(store[_gkey(tiny_swa_config(), k0, 1, 64)], expected), expected + ) def test_without_the_kernel_a_write_behind_store_is_synchronous(): @@ -2664,7 +2882,7 @@ def test_save_kv_layer_leaves_several_groups_to_wait_for_save(): def test_connector_wait_for_save_hands_several_groups_to_the_worker(): connector = MaruKVConnector.__new__(MaruKVConnector) connector._kv_groups = describe_kv_groups(tiny_config()) - connector._worker = MagicMock() + connector._worker = MagicMock(stores_after_forward=False) meta = MaruConnectorMetadata() connector._connector_metadata = meta connector.wait_for_save() @@ -2675,6 +2893,11 @@ def test_connector_wait_for_save_hands_several_groups_to_the_worker(): connector.wait_for_save() connector._worker.wait_for_save.assert_not_called() + # One group with DSpark stores after the forward too. + connector._worker.stores_after_forward = True + connector.wait_for_save() + connector._worker.wait_for_save.assert_called_once_with(meta) + def test_a_later_step_stores_the_state_at_its_own_end(): """Tokens 32-47 of a 56-token prompt: the state is at block position 2.""" @@ -2683,9 +2906,9 @@ def test_a_later_step_stores_the_state_at_its_own_end(): blocks = ([0, 0, 1], [0, 0, 2], [0, 0, 3], [4, 5, 6]) worker.wait_for_save(_store_meta(tokens, blocks, 32, 16)) k2 = _chunk_keys(tokens, TINY_BLOCK)[2] - assert set(store) == {group_object_key(k2, g, TINY_BLOCK) for g in range(4)} + assert set(store) == {_gkey(tiny_config(), k2, g, TINY_BLOCK) for g in range(4)} shared = _shared() - obj = store[group_object_key(k2, 1, TINY_BLOCK)] + obj = store[_gkey(tiny_config(), k2, 1, TINY_BLOCK)] assert obj[:TINY_STATE] == bytes(_region(shared, 0)[2 * TINY_SLOT :][:TINY_STATE]) @@ -2712,8 +2935,12 @@ def test_a_failing_group_keeps_the_others_and_leaks_nothing(): worker.wait_for_save(meta) ka, kb = _chunk_keys(a, TINY_BLOCK), _chunk_keys(b, TINY_BLOCK) assert set(store) == { - *(group_object_key(k, 3, TINY_BLOCK) for k in ka), - *(group_object_key(k, g, TINY_BLOCK) for k in (ka[1], kb[1]) for g in range(3)), + *(_gkey(tiny_config(), k, 3, TINY_BLOCK) for k in ka), + *( + _gkey(tiny_config(), k, g, TINY_BLOCK) + for k in (ka[1], kb[1]) + for g in range(3) + ), } assert worker._handler.batch_store.call_count == 1 assert worker._handler.alloc.call_count == len(store) @@ -2784,10 +3011,10 @@ def _check_qwen_objects(worker, store, shared, tokens, version="0.23"): for key, block in ((k0, 4), (k1, 5)): expected = _expected_slab(attn, [block], version) assert torch.equal( - _slab(store[group_object_key(key, 3, TINY_BLOCK)], expected), expected + _slab(store[_gkey(tiny_config(), key, 3, TINY_BLOCK)], expected), expected ) for g in range(3): - obj = store[group_object_key(k1, g, TINY_BLOCK)] + obj = store[_gkey(tiny_config(), k1, g, TINY_BLOCK)] block = QWEN_BLOCKS[g][1] for p in range(2): state = _region(shared, p)[block * TINY_SLOT :][:TINY_STATE].cpu() @@ -2815,7 +3042,7 @@ def test_kernel_store_registers_no_group_object_whose_copy_failed(version): tokens = list(range(40)) worker.wait_for_save(_store_meta(tokens, QWEN_BLOCKS, 0, 32)) k1 = _chunk_keys(tokens, TINY_BLOCK)[1] - assert set(store) == {group_object_key(k1, g, TINY_BLOCK) for g in range(3)} + assert set(store) == {_gkey(tiny_config(), k1, g, TINY_BLOCK) for g in range(3)} assert worker._handler.free.call_count == 2 # the two attention slabs @@ -2825,7 +3052,7 @@ def test_kernel_write_behind_stores_attention_later_and_states_now(version): tokens = list(range(40)) worker.wait_for_save(_store_meta(tokens, QWEN_BLOCKS, 0, 32)) k1 = _chunk_keys(tokens, TINY_BLOCK)[1] - assert all(group_object_key(k1, g, TINY_BLOCK) in store for g in range(3)) + assert all(_gkey(tiny_config(), k1, g, TINY_BLOCK) in store for g in range(3)) worker._store_executor.shutdown(wait=True) assert len(store) == 5 _check_qwen_objects(worker, store, shared, tokens, version) @@ -2851,7 +3078,9 @@ def test_kernel_store_reads_the_null_block_behind_the_window(version): (k0,) = _chunk_keys(tokens, 64) swa = [t.cpu() for t in worker._group_caches[0].tensors] expected = _expected_slab(swa, [0, 8, 9, 10], version) - assert torch.equal(_slab(store[group_object_key(k0, 0, 64)], expected), expected) + assert torch.equal( + _slab(store[_gkey(tiny_swa_config(), k0, 0, 64)], expected), expected + ) class _HeldExecutor: @@ -3055,7 +3284,7 @@ def _reported_failures(worker): def test_a_failed_parked_load_is_reported_by_its_request(): _, _, store = _stored_qwen() k1 = _chunk_keys(list(range(40)), TINY_BLOCK)[1] - del store[group_object_key(k1, 2, TINY_BLOCK)] + del store[_gkey(tiny_config(), k1, 2, TINY_BLOCK)] worker, target = _loading_worker(store) meta = _load_meta(list(range(40)), B_BLOCKS, 2, deferred_load=True) worker.start_load_kv(_FORWARD, meta) @@ -3069,7 +3298,7 @@ def test_connector_reports_failed_parked_loads_with_their_completion(): pytest.skip("this vLLM reports load failures by block only") _, _, store = _stored_qwen() del store[ - group_object_key(_chunk_keys(list(range(40)), TINY_BLOCK)[1], 2, TINY_BLOCK) + _gkey(tiny_config(), _chunk_keys(list(range(40)), TINY_BLOCK)[1], 2, TINY_BLOCK) ] worker, _ = _loading_worker(store) connector = MaruKVConnector.__new__(MaruKVConnector) @@ -3091,13 +3320,13 @@ def test_a_load_skips_what_vllms_prefix_cache_holds(): worker.start_load_kv(_FORWARD, meta) (keys,), _ = worker._handler.batch_retrieve.call_args k0, k1 = _chunk_keys(list(range(40)), TINY_BLOCK) - assert group_object_key(k0, 3, TINY_BLOCK) not in keys + assert _gkey(tiny_config(), k0, 3, TINY_BLOCK) not in keys dst = worker._group_caches[3].tensors assert not _token_rows(dst[0], 7, "0.30").any() src = stored._group_caches[3].tensors assert torch.equal(_token_rows(dst[0], 8, "0.30"), _token_rows(src[0], 5, "0.30")) - del store[group_object_key(k1, 3, TINY_BLOCK)] + del store[_gkey(tiny_config(), k1, 3, TINY_BLOCK)] worker.start_load_kv(_FORWARD, meta) assert worker.take_failed_load_blocks() == {1, 3, 5, 8} @@ -3105,7 +3334,7 @@ def test_a_load_skips_what_vllms_prefix_cache_holds(): def test_loads_of_several_groups_carry_where_vllms_cache_ends(): a = list(range(96)) sched = _two_group_scheduler({"maru_async_load": True}) - sched._handler = _Store(_stored_by(a, 32, [0, 1], [], [])) + sched._handler = _Store(_stored_by(gpt_oss_config(), a, 32, [0, 1], [], [])) request = SimpleNamespace(request_id="r1", prompt_token_ids=a + _other(4)) # vLLM's prefix cache holds 32 tokens; Maru adds 64. assert sched.get_num_new_matched_tokens(request, 32) == (64, True) @@ -3260,7 +3489,9 @@ def test_vllm_scheduler_hands_every_group_to_the_connector(tmp_path): ) tokens = list(range(100)) connector = engine.connector - connector._scheduler._handler = _Store(_stored_by(tokens, 32, [0, 1], [], [])) + connector._scheduler._handler = _Store( + _stored_by(connector._kv_groups, tokens, 32, [0, 1], [], []) + ) finished = MagicMock(wraps=connector.request_finished_all_groups) connector.request_finished_all_groups = finished @@ -3453,7 +3684,9 @@ def _add_hit_request(engine, tokens): from vllm.v1.request import Request init_none_hash(sha256) - engine.connector._scheduler._handler = _Store(_stored_by(tokens, 16, [1], [0], [5])) + engine.connector._scheduler._handler = _Store( + _stored_by(engine.connector._kv_groups, tokens, 16, [1], [0], [5]) + ) engine.add_request( Request( "b", @@ -3580,7 +3813,7 @@ def test_an_unavailable_handler_fails_the_loads(parked): def test_an_inline_load_of_a_missing_object_is_reported(): _, _, store = _stored_qwen() k1 = _chunk_keys(list(range(40)), TINY_BLOCK)[1] - del store[group_object_key(k1, 0, TINY_BLOCK)] + del store[_gkey(tiny_config(), k1, 0, TINY_BLOCK)] worker, target = _loading_worker(store) worker.start_load_kv(_FORWARD, _load_meta(list(range(40)), B_BLOCKS, 2, 32, 8)) assert _reported_failures(worker) == FAILED_INLINE @@ -3682,12 +3915,12 @@ def test_tp_ranks_store_and_load_their_own_slices(version): k0, k1 = _chunk_keys(tokens, TINY_BLOCK) assert set(server.objects) == { *( - group_object_key(k + h, 3, TINY_BLOCK) + _gkey(tiny_config(), k + h, 3, TINY_BLOCK) for k in (k0, k1) for h in ("_h0-2", "_h2-4") ), *( - group_object_key(k1 + s, g, TINY_BLOCK) + _gkey(tiny_config(), k1 + s, g, TINY_BLOCK) for s in ("_tp2_r0", "_tp2_r1") for g in range(3) ), @@ -3752,9 +3985,11 @@ def test_tp_replica_skips_attention_but_stores_its_state(): k0, k1 = _chunk_keys(tokens, TINY_BLOCK) assert _registered_keys(workers[1]) == { - group_object_key(k1 + "_tp4_r1", g, TINY_BLOCK) for g in range(3) + _gkey(tiny_config(), k1 + "_tp4_r1", g, TINY_BLOCK) for g in range(3) } - assert group_object_key(k0 + "_h0-1", 3, TINY_BLOCK) in _registered_keys(workers[0]) + assert _gkey(tiny_config(), k0 + "_h0-1", 3, TINY_BLOCK) in _registered_keys( + workers[0] + ) target = torch.zeros_like(_shared()) worker = _tp_worker( @@ -3820,7 +4055,7 @@ def test_tp_sliding_window_ranks_load_their_own_heads(shards, names): (k0,) = _chunk_keys(tokens, 64) assert set(server.objects) == { - group_object_key(k0 + name, g, 64) for name in names for g in (0, 1) + _gkey(config, k0 + name, g, 64) for name in names for g in (0, 1) } for rank in range(2): worker = _tp_worker( @@ -3856,12 +4091,12 @@ def _qwen35_tp2_objects(keys): """What both ranks store of a 1,200-token prompt whose first step ended at chunk 1: attention for every chunk, states for chunk 1.""" attention = { - group_object_key(k + h, 3, QWEN35_BLOCK) + _gkey(qwen35_config(), k + h, 3, QWEN35_BLOCK) for k in keys for h in ("_h0-1", "_h1-2") } states = { - group_object_key(keys[1] + s, g, QWEN35_BLOCK) + _gkey(qwen35_config(), keys[1] + s, g, QWEN35_BLOCK) for s in ("_tp2_r0", "_tp2_r1") for g in range(3) } @@ -3878,9 +4113,9 @@ def test_tp_hit_needs_every_slice_of_every_group(): assert stored <= set(sched._handler.queries[0]) # One rank's part of one Mamba state is missing: no chunk can resume. - missing_state = group_object_key(keys[1] + "_tp2_r1", 0, QWEN35_BLOCK) + missing_state = _gkey(qwen35_config(), keys[1] + "_tp2_r1", 0, QWEN35_BLOCK) assert _hit(_qwen35_scheduler(stored - {missing_state}, **QWEN35_TP2), a) == 0 - missing_heads = group_object_key(keys[0] + "_h1-2", 3, QWEN35_BLOCK) + missing_heads = _gkey(qwen35_config(), keys[0] + "_h1-2", 3, QWEN35_BLOCK) assert _hit(_qwen35_scheduler(stored - {missing_heads}, **QWEN35_TP2), a) == 0 @@ -3969,7 +4204,7 @@ def test_tp_kernel_stores_each_ranks_heads(write_behind): for k in _chunk_keys(tokens, TINY_BLOCK): for rank, heads in enumerate(("_h0-2", "_h2-4")): - raw = server.objects[group_object_key(k + heads, 3, TINY_BLOCK)] + raw = server.objects[_gkey(tiny_config(), k + heads, 3, TINY_BLOCK)] slab = torch.frombuffer(bytearray(raw), dtype=torch.bfloat16) assert torch.all(slab == rank + 1) @@ -4083,7 +4318,7 @@ def test_tp_lookup_log_names_a_missing_rank_slice(caplog): a = list(range(1200)) keys = _chunk_keys(a, QWEN35_BLOCK) stored = _qwen35_tp2_objects(keys) - { - group_object_key(keys[1] + "_tp2_r1", 0, QWEN35_BLOCK) + _gkey(qwen35_config(), keys[1] + "_tp2_r1", 0, QWEN35_BLOCK) } with caplog.at_level(logging.INFO, logger="maru_vllm.connector"): assert _hit(_qwen35_scheduler(stored, **QWEN35_TP2), a) == 0 diff --git a/tests/unit/test_vllm_remote_cxl_tp_groups.py b/tests/unit/test_vllm_remote_cxl_tp_groups.py index e6e6363e..0bf45dd0 100644 --- a/tests/unit/test_vllm_remote_cxl_tp_groups.py +++ b/tests/unit/test_vllm_remote_cxl_tp_groups.py @@ -31,7 +31,7 @@ _cpu_engine_config, _create_maru_handler, ) -from maru_vllm.kv_groups import describe_kv_groups, group_object_key +from maru_vllm.kv_groups import describe_kv_groups from tests.unit.test_vllm_cpu_config import engine_config from tests.unit.test_vllm_cpu_config import extra as cpu_extra from tests.unit.test_vllm_kv_groups import ( @@ -40,6 +40,7 @@ FAILED_PARKED, QWEN_BLOCKS, TINY_BLOCK, + _gkey, _HeldExecutor, _load_meta, _reported_failures, @@ -546,7 +547,7 @@ def test_a_group_load_reads_what_staging_holds_and_releases_it(slots): def test_a_missing_group_object_fails_the_parked_load_and_frees_its_slots(): _, _, store = _stored_qwen() k1 = _chunk_keys(list(range(40)), TINY_BLOCK)[1] - del store[group_object_key(k1, 2, TINY_BLOCK)] + del store[_gkey(tiny_config(), k1, 2, TINY_BLOCK)] worker, target = session_group_worker(SessionServer(store), 64) meta = _load_meta(list(range(40)), B_BLOCKS, 2, deferred_load=True) worker.start_load_kv(_FORWARD, meta) diff --git a/tests/unit/test_vllm_tp.py b/tests/unit/test_vllm_tp.py index 71ab24c9..72752727 100644 --- a/tests/unit/test_vllm_tp.py +++ b/tests/unit/test_vllm_tp.py @@ -430,6 +430,27 @@ def test_worker_takes_mla_from_the_model_config(): assert not gqa._worker._use_mla and gqa._worker._kv_planes == 2 +def test_worker_stores_after_the_forward_with_dspark(): + """vLLM replays DSpark's draft step as a full CUDA graph, which calls no + save_kv_layer for the draft layers. Other methods store as before.""" + plain = MaruKVConnector(fake_vllm_config(1, 8), KVConnectorRole.WORKER) + assert not plain._worker.stores_after_forward + config = fake_vllm_config(1, 8) + config.speculative_config = SimpleNamespace(method="dspark") + spec = MaruKVConnector(config, KVConnectorRole.WORKER) + assert spec._worker.stores_after_forward + other = fake_vllm_config(1, 8) + other.speculative_config = SimpleNamespace(method="eagle3") + assert not MaruKVConnector( + other, KVConnectorRole.WORKER + )._worker.stores_after_forward + layerwise = fake_vllm_config(1, 8, extra={"maru_use_layerwise": True}) + layerwise.speculative_config = SimpleNamespace(method="dspark") + assert not MaruKVConnector( + layerwise, KVConnectorRole.WORKER + )._worker.stores_after_forward + + def test_worker_names_its_slice_and_scopes_its_instance_id(monkeypatch): monkeypatch.setattr(connector_module, "_tp_rank", lambda: 1) connector = MaruKVConnector(fake_vllm_config(2, 8), KVConnectorRole.WORKER)