Repository navigation
[Feature] Shard the MLA KV store across prefill-context-parallel ranks - #18
Open
ibrahimnd2000 wants to merge 1 commit into
Open
ibrahimnd2000 wants to merge 1 commit into
ibrahimnd2000 wants to merge 1 commit into
Conversation
This was referenced Nov 12, 2026
Collaborator
|
@ibrahimnd2000 Thanks for your contribution! Would you like to show your use case for the LMCache in process mode? |
Author
Collaborator
|
@ibrahimnd2000 Thanks for sharing your use case, is that a production use case? And yes we recommend to use LMCache mp mode instead. But we can keep in process mode iterate in this repository, we won't remove in process mode. |
Author
|
@maobaolong Yes, it's a production use case: long-context MLA serving (prompts up to ~200k tokens) with vLLM prefill/decode disaggregation, and these patches have been running in that deployment since October. Thanks for confirming in-process mode stays; I'll look at MP mode for new work. Two small things:
|
Ported from LMCache/LMCache#5545: the in-process engine moved to this repository in LMCache/LMCache#6254. Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
ibrahimnd2000
force-pushed
the
port/5545-pcp-shard-store
branch
from
November 19, 2026 07:07
7d8a3d5 to
1137a4e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Feature] Shard the MLA KV store across prefill-context-parallel ranks
Purpose
For MLA models LMCache stores and loads KV only on worker 0
(
save_only_first_rank, the MLA default). Under vLLM prefill context parallel(TP1 x PCP N) the MLA KV cache is replicated on every PCP rank. So on a store,
rank 0 copies the whole prompt's KV to CPU inside the forward step, and the
other N-1 ranks wait for it in the next collective. On 8 GPUs with 200k-token
prompts this stall took about 16% of prefill time in Nsight.
This PR adds an opt-in mode,
pcp_shard_store: true. Rankrstores, looks upand loads only the chunks with
chunk_index % world_size == r. Each rank thencopies 1/N of the KV to CPU, in parallel.
It also includes a small fix to the vLLM integration. When TP == 1 and PCP > 1,
the engine now uses the PCP group's broadcast functions. Upstream
devpassesthe TP group's broadcast functions to the engine. With TP1 x PCP N that group
has one rank, so for any MLA + PCP retrieve (default mode included) the other
PCP ranks never receive the KV that rank 0 loaded. Shard mode needs the PCP
group, and it checks the group at startup.
Design
Chunk
iis the chunk that starts at tokeni * chunk_size, counted fromtoken 0 of the request. Its owner is
i % world_size. Store, lookup andretrieve all work out the owner the same way. The pure logic is in
lmcache/inprocess/v1/pcp_shard.py, and the engine changes are incache_engine.py.Store. In shard mode a non-first rank does not return early. Every rank
has a
StorageManager/LocalCPUBackend. In the chunk loop a rank skips thechunks it does not own. Remote (L2) keys of MLA chunks already map to
worker_id 0 (
remote_enable_mla_worker_id_as0), so a chunk's L2 key doesnot depend on which rank wrote it.
Lookup. A lookup server runs on every rank (
lookup_server_worker_idsis forced to all ranks). Rank
rchecks only its owned chunks(
batched_contains, with prefix semantics over the owned list). It answerswith the start token of its first owned miss, or with the end of the last
chunk if it has no owned miss. The scheduler's
LMCacheLookupClientalreadytakes
min()over ranks. That minimum is exactly the longest prefix in whichevery chunk is present on its owner: if
jis the first missing chunk, itsowner answers
start(j)and every other rank answers at leaststart(j).In this mode the client no longer warns that per-rank results differ,
because they differ by design. Each rank pins its own owned hits under the
lookup id, and
lookup_unpinreleases them on every worker, as before.Retrieve (the blocking path). Every rank runs the same collective
sequence, whatever it found locally:
chunks.
first_failis its first owned chunk that it could not get.Storage exceptions, an unhealthy engine and device-staging errors count as
failures. None of them is raised before the collectives.
non_blocking, on thebroadcast stream), so all ranks do their host-to-device copies in
parallel.
world_sizebroadcast_objectcalls, with src = 0..N-1 inorder. Each rank sends
(fingerprint, first_fail, {chunk: metadata}). Thefingerprint is blake2b over the chunk bounds and chunk hashes.
agree_prefixis a pure function of the N messages, whichare identical on every rank. The prefix is 0 if the fingerprints differ.
Otherwise it is
min(first_fail), cut further at the first chunk whoseowner sent no metadata.
j < prefix, in order,broadcast(tensor, src=j % N).batched_to_gpuruns on every rank, andret_mask[:prefix tokens]isset. The owned L1 objects are released after a stream sync.
Collectives per retrieve: N object broadcasts plus K tensor broadcasts. The
default path does 1 + 2K (one metadata object per chunk).
L1 size. Each rank holds 1/N of the chunks, so its L1 size is
max_local_cpu_size / world_size. The node total stays the same.pcp_shard_max_local_cpu_size(GB per rank) overrides it. The"available memory minus reserve" clamp is divided by
world_sizetoo.Controller.
lmcache_worker_idsdefaults to all ranks in this mode, sothe controller sees every chunk. An explicit list is respected.
Invariants (deadlock safety)
minover per-rank answers equals the longest prefix ofchunks that are all present. Checked exhaustively up to 10 chunks for
W in {2, 3, 4, 8}, with a partial last chunk, and on random prompts of up to
800 chunks.
collectives (same srcs, same count) whatever it fetched. Local failures are
only reported through the exchange, never raised before it. Tests run later
retrieves after failures to show that the ranks never get out of step.
ret_mask, and every loadedchunk holds its owner's bytes.
same shorter prefix: a chunk evicted on its owner,
getreturning None,getraising, an unhealthy rank, a staging failure. If ranks are asked for
different chunk lists, the fingerprints differ and every rank loads nothing.
None of these cases hangs.
object is back to ref_count 1, and it is unpinned once
lookup_unpinruns.existing conditions gain
and not self._pcp_shard. Everything else is newcode that runs only in shard mode.
and the L1 size per rank is total / N.
As in the default path, faults after the exchange are not recoverable. For
example, an OOM while allocating a receive buffer, or an NCCL error, leaves the
other ranks blocked in the collective.
Config
pcp_shard_storeis true,the model is MLA,
save_only_first_rankis on (the MLA default), andworld_size > 1. The scheduler and the workers work it out from the sameconfig, so they cannot disagree.
use_layerwise,enable_async_loading,enable_blending,enable_scheduler_bypass_lookup,enable_pd,enable_p2porexternal_lookup_clientraisesValueErrorat startup.(TP1 x PCP N). This is checked at engine construction when the broadcast
function belongs to a vLLM
GroupCoordinator.lmcache_worker_portsneedsworld_sizeports if the controller is used.Remote backend (L2)
Under MLA,
RemoteBackendskips puts on every worker except 0 (remote_enable_mla_worker_id_as0), because worker 0 used to store everything. In shard mode every rank owns different chunks, so every rank now puts its own chunks, under the worker-0 key the read path already uses. Without this, the remote backend would hold ~1/8 of the chunks and remote prefix hits would stop after the first chunk (measured with Valkey: 2,381 puts for 18,960 stored chunks). Test:tests/inprocess/v1/pcp_shard/test_pcp_shard_remote.py(fails without the change).Measured with Valkey (
resp://) on the same node: puts = stored chunks (18,730 / 18,728); prefill throughput 57.3k tok/s vs 57.7k without a remote backend; two ~200k-token prompts evicted from GPU and the per-rank CPU cache were loaded entirely from Valkey (784 / 796 GETs) with correct answers.Test Plan
tests/inprocess/v1/pcp_shard/test_pcp_shard_logic.py): owner,per-rank lookup answer against the reference prefix (exhaustive and random),
agree_prefix, fingerprint, the mode gate and its errors, the broadcast-groupcheck, and the collective sequence with fake broadcast functions.
test_pcp_shard_config.py): lookup serverson every rank, controller worker ids, an error for unsupported combinations,
and the client's
minwith no warning in shard mode (the warning is stilllogged in the default mode).
test_pcp_shard_gloo.py):protocol_harness.py: 2, 3 and 8 ranks, 60 rounds each, with random chunkcounts, random per-rank fetch failures, raising ranks and fingerprint
mismatches.
engine_harness.py: a realLMCacheEngineper rank (realStorageManager,LocalCPUBackend, token database) with a fake GPU connector that writes andchecks deterministic KV. Scenarios: full round trip (W = 2, 4, 8, partial
last chunk); a mask prefix that vLLM already has; eviction on the owner
(W = 4, 8); get returning None and get raising; an unhealthy rank and a
staging failure; mismatched inputs; random prompts with shared prefixes and
evictions, checked against the reference prefix (W = 3, 4, 8); the L1
budget; and the default mode.
below).
Test Result
Unit and multi-process tests (this branch, macOS CPU, Python 3.12, torch 2.14.1)
Existing related suites (
tests/inprocess/v1/test_config.py,tests/inprocess/v1/lookup_client,tests/inprocess/v1/storage_backend/test_remote_mla_worker_id_as0.py,tests/inprocess/integration/vllm/test_retrieve_pin_ownership.py,tests/inprocess/v1/test_cache_engine_cleanup.py,tests/inprocess/v1/storage_backend/test_local_cpu_backend.py,tests/inprocess/v1/test_cache_engine.py): 158 passed, 69 skipped, 4 failed. The same 4test_config.pyGPU-mode NIXL tests also fail on unmodifieddevon this CPUmachine.
pre-commit run --files <changed files>: all hooks pass (isort, ruff,ruff-format, codespell, mypy, SPDX, device-usage bans).
GPU results (measured on 8x B300 with LMCache 0.5.5 + vLLM 0.31.0)
These numbers come from the same design implemented on LMCache 0.5.5 (plus the
PCP broadcast-group change in this PR), not from this branch on
dev. Setup:an MLA model in FP8, prefill context parallel 8 (TP1 x PCP8).
save_only_first_rank, rank 0 stores everything)AllGather wait fell from about 1.55 s to about 0.1 s per GPU per 9.5 s window.
to 204k tokens loaded): 3/3 needle tests correct, with the same answers as the
default path.
correct.
cache match before and after GPU-cache eviction.
Special notes for your reviewers
The mode is opt-in and the default MLA path is unchanged, apart from the
PCP broadcast-group fix in
vllm_service_factory.py, which applies to everyMLA + TP1 x PCP>1 deployment.
With
i % Nownership, prompts shorter than N chunks use only the low ranks'L1.
KV events (
enable_kv_events) are now emitted per rank for owned chunks only.move,compress,decompressand the internal API lookup run on one rankand see only that rank's chunks. They are not supported in this mode (P2P is
rejected).
this PR contains user facing changes - docs added
this PR contains unit tests