Skip to content

[Feature] Shard the MLA KV store across prefill-context-parallel ranks - #18

Open
ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5545-pcp-shard-store
Open

ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5545-pcp-shard-store

Conversation

@ibrahimnd2000

Copy link
Copy Markdown

Ported from LMCache/LMCache#5545 (opened 2026-10-07), whose in-process files moved to this repository in LMCache/LMCache#6254. Same change on current main; the new pcp_shard module now lives in lmcache/inprocess/v1/pcp_shard.py and its tests in tests/inprocess/v1/pcp_shard/.

[Feature] Shard the MLA KV store across prefill-context-parallel ranks

Purpose

For MLA models LMCache stores and loads KV only on worker 0
(save_only_first_rank, the MLA default). Under vLLM prefill context parallel
(TP1 x PCP N) the MLA KV cache is replicated on every PCP rank. So on a store,
rank 0 copies the whole prompt's KV to CPU inside the forward step, and the
other N-1 ranks wait for it in the next collective. On 8 GPUs with 200k-token
prompts this stall took about 16% of prefill time in Nsight.

This PR adds an opt-in mode, pcp_shard_store: true. Rank r stores, looks up
and loads only the chunks with chunk_index % world_size == r. Each rank then
copies 1/N of the KV to CPU, in parallel.

It also includes a small fix to the vLLM integration. When TP == 1 and PCP > 1,
the engine now uses the PCP group's broadcast functions. Upstream dev passes
the TP group's broadcast functions to the engine. With TP1 x PCP N that group
has one rank, so for any MLA + PCP retrieve (default mode included) the other
PCP ranks never receive the KV that rank 0 loaded. Shard mode needs the PCP
group, and it checks the group at startup.

Design

Chunk i is the chunk that starts at token i * chunk_size, counted from
token 0 of the request. Its owner is i % world_size. Store, lookup and
retrieve all work out the owner the same way. The pure logic is in
lmcache/inprocess/v1/pcp_shard.py, and the engine changes are in cache_engine.py.

  1. Store. In shard mode a non-first rank does not return early. Every rank
    has a StorageManager/LocalCPUBackend. In the chunk loop a rank skips the
    chunks it does not own. Remote (L2) keys of MLA chunks already map to
    worker_id 0 (remote_enable_mla_worker_id_as0), so a chunk's L2 key does
    not depend on which rank wrote it.

  2. Lookup. A lookup server runs on every rank (lookup_server_worker_ids
    is forced to all ranks). Rank r checks only its owned chunks
    (batched_contains, with prefix semantics over the owned list). It answers
    with the start token of its first owned miss, or with the end of the last
    chunk if it has no owned miss. The scheduler's LMCacheLookupClient already
    takes min() over ranks. That minimum is exactly the longest prefix in which
    every chunk is present on its owner: if j is the first missing chunk, its
    owner answers start(j) and every other rank answers at least start(j).
    In this mode the client no longer warns that per-rank results differ,
    because they differ by design. Each rank pins its own owned hits under the
    lookup id, and lookup_unpin releases them on every worker, as before.

  3. Retrieve (the blocking path). Every rank runs the same collective
    sequence, whatever it found locally:

    1. It works out the chunk list from (tokens, mask) and fetches its owned
      chunks. first_fail is its first owned chunk that it could not get.
      Storage exceptions, an unhealthy engine and device-staging errors count as
      failures. None of them is raised before the collectives.
    2. It stages its fetched chunks on its own device (non_blocking, on the
      broadcast stream), so all ranks do their host-to-device copies in
      parallel.
    3. Exchange. world_size broadcast_object calls, with src = 0..N-1 in
      order. Each rank sends (fingerprint, first_fail, {chunk: metadata}). The
      fingerprint is blake2b over the chunk bounds and chunk hashes.
    4. Agreement. agree_prefix is a pure function of the N messages, which
      are identical on every rank. The prefix is 0 if the fingerprints differ.
      Otherwise it is min(first_fail), cut further at the first chunk whose
      owner sent no metadata.
    5. For each j < prefix, in order, broadcast(tensor, src=j % N).
    6. batched_to_gpu runs on every rank, and ret_mask[:prefix tokens] is
      set. The owned L1 objects are released after a stream sync.

    Collectives per retrieve: N object broadcasts plus K tensor broadcasts. The
    default path does 1 + 2K (one metadata object per chunk).

  4. L1 size. Each rank holds 1/N of the chunks, so its L1 size is
    max_local_cpu_size / world_size. The node total stays the same.
    pcp_shard_max_local_cpu_size (GB per rank) overrides it. The
    "available memory minus reserve" clamp is divided by world_size too.

  5. Controller. lmcache_worker_ids defaults to all ranks in this mode, so
    the controller sees every chunk. An explicit list is respected.

Invariants (deadlock safety)

  • I1 Lookup. The min over per-rank answers equals the longest prefix of
    chunks that are all present. Checked exhaustively up to 10 chunks for
    W in {2, 3, 4, 8}, with a partial last chunk, and on random prompts of up to
    800 chunks.
  • I2 Same collective sequence. On every retrieve, every rank issues the same
    collectives (same srcs, same count) whatever it fetched. Local failures are
    only reported through the exchange, never raised before it. Tests run later
    retrieves after failures to show that the ranks never get out of step.
  • I3 Same result. Every rank ends with the same ret_mask, and every loaded
    chunk holds its owner's bytes.
  • I4 Failures become agreed misses. These all make every rank agree on the
    same shorter prefix: a chunk evicted on its owner, get returning None, get
    raising, an unhealthy rank, a staging failure. If ranks are asked for
    different chunk lists, the fingerprints differ and every rank loads nothing.
    None of these cases hangs.
  • I5 No leaked references. After loads, failed loads included, every L1
    object is back to ref_count 1, and it is unpinned once lookup_unpin runs.
  • I6 Mode off. With the key unset, every code path behaves as before. Three
    existing conditions gain and not self._pcp_shard. Everything else is new
    code that runs only in shard mode.
  • I7 Ownership. In shard mode each rank's L1 holds exactly its owned chunks,
    and the L1 size per rank is total / N.

As in the default path, faults after the exchange are not recoverable. For
example, an OOM while allocating a receive buffer, or an NCCL error, leaves the
other ranks blocked in the collective.

Config

extra_config:
  pcp_shard_store: true
  # optional, GB per rank; default max_local_cpu_size / world_size
  # pcp_shard_max_local_cpu_size: 30
  • The mode is active only when all of these hold: pcp_shard_store is true,
    the model is MLA, save_only_first_rank is on (the MLA default), and
    world_size > 1. The scheduler and the workers work it out from the same
    config, so they cannot disagree.
  • Requesting it together with any of use_layerwise, enable_async_loading,
    enable_blending, enable_scheduler_bypass_lookup, enable_pd,
    enable_p2p or external_lookup_client raises ValueError at startup.
  • The broadcast group must span all ranks, with group rank == worker_id
    (TP1 x PCP N). This is checked at engine construction when the broadcast
    function belongs to a vLLM GroupCoordinator.
  • lmcache_worker_ports needs world_size ports if the controller is used.

Remote backend (L2)

Under MLA, RemoteBackend skips puts on every worker except 0 (remote_enable_mla_worker_id_as0), because worker 0 used to store everything. In shard mode every rank owns different chunks, so every rank now puts its own chunks, under the worker-0 key the read path already uses. Without this, the remote backend would hold ~1/8 of the chunks and remote prefix hits would stop after the first chunk (measured with Valkey: 2,381 puts for 18,960 stored chunks). Test: tests/inprocess/v1/pcp_shard/test_pcp_shard_remote.py (fails without the change).

Measured with Valkey (resp://) on the same node: puts = stored chunks (18,730 / 18,728); prefill throughput 57.3k tok/s vs 57.7k without a remote backend; two ~200k-token prompts evicted from GPU and the per-rank CPU cache were loaded entirely from Valkey (784 / 796 GETs) with correct answers.

Test Plan

  • Unit, pure logic (tests/inprocess/v1/pcp_shard/test_pcp_shard_logic.py): owner,
    per-rank lookup answer against the reference prefix (exhaustive and random),
    agree_prefix, fingerprint, the mode gate and its errors, the broadcast-group
    check, and the collective sequence with fake broadcast functions.
  • Unit, config and lookup client (test_pcp_shard_config.py): lookup servers
    on every rank, controller worker ids, an error for unsupported combinations,
    and the client's min with no warning in shard mode (the warning is still
    logged in the default mode).
  • Multi-process on gloo (test_pcp_shard_gloo.py):
    • protocol_harness.py: 2, 3 and 8 ranks, 60 rounds each, with random chunk
      counts, random per-rank fetch failures, raising ranks and fingerprint
      mismatches.
    • engine_harness.py: a real LMCacheEngine per rank (real StorageManager,
      LocalCPUBackend, token database) with a fake GPU connector that writes and
      checks deterministic KV. Scenarios: full round trip (W = 2, 4, 8, partial
      last chunk); a mask prefix that vLLM already has; eviction on the owner
      (W = 4, 8); get returning None and get raising; an unhealthy rank and a
      staging failure; mismatched inputs; random prompts with shared prefixes and
      evictions, checked against the reference prefix (W = 3, 4, 8); the L1
      budget; and the default mode.
  • End to end on GPU (done on the original 0.5.5-based implementation; see
    below).

Test Result

Unit and multi-process tests (this branch, macOS CPU, Python 3.12, torch 2.14.1)

$ PYTHONHASHSEED=0 pytest -q tests/inprocess/v1/pcp_shard
..................................                                       [100%]
34 passed in 60.78s (0:01:00)

Existing related suites (tests/inprocess/v1/test_config.py, tests/inprocess/v1/lookup_client,
tests/inprocess/v1/storage_backend/test_remote_mla_worker_id_as0.py,
tests/inprocess/integration/vllm/test_retrieve_pin_ownership.py,
tests/inprocess/v1/test_cache_engine_cleanup.py,
tests/inprocess/v1/storage_backend/test_local_cpu_backend.py,
tests/inprocess/v1/test_cache_engine.py): 158 passed, 69 skipped, 4 failed. The same 4
test_config.py GPU-mode NIXL tests also fail on unmodified dev on this CPU
machine.

pre-commit run --files <changed files>: all hooks pass (isort, ruff,
ruff-format, codespell, mypy, SPDX, device-usage bans).

GPU results (measured on 8x B300 with LMCache 0.5.5 + vLLM 0.31.0)

These numbers come from the same design implemented on LMCache 0.5.5 (plus the
PCP broadcast-group change in this PR), not from this branch on dev. Setup:
an MLA model in FP8, prefill context parallel 8 (TP1 x PCP8).

Configuration Prefill throughput
Default (save_only_first_rank, rank 0 stores everything) 43.2k tokens/s
Store-only prototype (sharded store; lookup and load not sharded) 49.4k tokens/s
Unrelated FlashInfer fix alone 49.8k tokens/s
This mode + the unrelated FlashInfer fix 57.7k to 57.8k tokens/s
  • Nsight: rank-0 device-to-host traffic fell from 19.1 GB to 2.4 GB per rank.
    AllGather wait fell from about 1.55 s to about 0.1 s per GPU per 9.5 s window.
  • Correctness after GPU prefix-cache eviction (hits served from LMCache, 174k
    to 204k tokens loaded): 3/3 needle tests correct, with the same answers as the
    default path.
  • Short prompts, including a 5-chunk prompt (1,280 tokens loaded from LMCache):
    correct.
  • Concurrent burst, 12 documents x 3 repeats: 36/36 correct, no hangs.
  • 30-minute soak: 1,549 requests, 0 errors, KV usage back to 0% at the end.
  • The controller-side lookup sees the full prefix: the router logged a 100%
    cache match before and after GPU-cache eviction.

Special notes for your reviewers

  • The mode is opt-in and the default MLA path is unchanged, apart from the
    PCP broadcast-group fix in vllm_service_factory.py, which applies to every
    MLA + TP1 x PCP>1 deployment.

  • With i % N ownership, prompts shorter than N chunks use only the low ranks'
    L1.

  • KV events (enable_kv_events) are now emitted per rank for owned chunks only.

  • move, compress, decompress and the internal API lookup run on one rank
    and see only that rank's chunks. They are not supported in this mode (P2P is
    rejected).

  • this PR contains user facing changes - docs added

  • this PR contains unit tests

@maobaolong

Copy link
Copy Markdown
Collaborator

@ibrahimnd2000 Thanks for your contribution! Would you like to show your use case for the LMCache in process mode?

@ibrahimnd2000

Copy link
Copy Markdown
Author

@maobaolong Thanks for looking. Happy to share the use case.

We serve MLA models with vLLM in a prefill/decode split, where the prefill engine runs prefill context parallel (TP1 x PCP 8) on long prompts, up to about 200k tokens. LMCache runs in-process in that engine, with a CPU L1 and a sharded Valkey cluster as the remote tier, so a prefix computed by one prefill engine can be loaded by another.

The PRs I ported here (#18 to #34) all come from running that setup:

If in-process mode is being frozen and you would rather have this work target MP mode, please say so and I will look at what carries over. Otherwise, could a maintainer approve the CI workflow on these PRs? It is waiting for approval as I am a first-time contributor to this repository.

@maobaolong

Copy link
Copy Markdown
Collaborator

@ibrahimnd2000 Thanks for sharing your use case, is that a production use case? And yes we recommend to use LMCache mp mode instead. But we can keep in process mode iterate in this repository, we won't remove in process mode.

@ibrahimnd2000

Copy link
Copy Markdown
Author

@maobaolong Yes, it's a production use case: long-context MLA serving (prompts up to ~200k tokens) with vLLM prefill/decode disaggregation, and these patches have been running in that deployment since October. Thanks for confirming in-process mode stays; I'll look at MP mode for new work.

Two small things:

Ported from LMCache/LMCache#5545: the in-process engine moved to this
repository in LMCache/LMCache#6254.

Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
@ibrahimnd2000
ibrahimnd2000 force-pushed the port/5545-pcp-shard-store branch from 7d8a3d5 to 1137a4e Compare November 19, 2026 07:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants