Skip to content

[Feature] pcp_shard_store: support async loading (each rank prefetches its own chunks) - #19

Open
ibrahimnd2000 wants to merge 2 commits into
LMCache:mainfrom
ibrahimnd2000:port/5553-pcp-shard-async
Open

ibrahimnd2000 wants to merge 2 commits into
LMCache:mainfrom
ibrahimnd2000:port/5553-pcp-shard-async

Conversation

@ibrahimnd2000

Copy link
Copy Markdown

Ported from LMCache/LMCache#5553, moved here by LMCache/LMCache#6254. Stacked on #18: only the last commit (pcp_shard_store: support async loading) is new here.

What this PR does / why we need it:

Follow-up to #5545 (sharded PCP store). Depends on #5545: this branch is built on it, so until #5545 merges the diff includes its commits. Only the top commit (dbd78611) is new.

#5545 rejects enable_async_loading in shard mode. This adds it.

  • Lookup + prefetch. The async lookup client already sends each lookup to every rank's async lookup server and takes the minimum. In shard mode each rank now prefetches only the chunks it owns (chunk_index % world_size == rank). It passes cum_chunk_lengths = "start of my k-th owned chunk" (+ the end of the last chunk), so the number the storage manager reports is the rank's sharded lookup answer, and the minimum is the same prefix as the synchronous path. A new helper pcp_shard.async_lookup_keys builds the keys and lengths.
  • Zero hits. StorageManager.async_lookup_and_prefetch reported a hard-coded 0 when nothing hit. It now reports cum_chunk_lengths[0], which is 0 outside shard mode. In shard mode, that stops a rank whose first owned chunk misses from cutting the chunks before it.
  • Load. _pcp_shard_retrieve takes this rank's chunks from its prefetch event (helper pcp_shard.select_prefetched), then runs the same exchange + per-chunk broadcast as the sync path. The event is popped, so lookup_unpin has nothing left to release. Used objects are unpinned and released as in the default async path; unused ones are released.
  • Remote. The async get reads under the worker-0 key, like the async contains. Without it, ranks ≠ 0 ask for keys that do not exist. This is the same one-line change as #5550; whichever lands first, the other rebases.

Special notes for your reviewers:

Validated on a vLLM GLM-5.3 FP8 prefill engine (PCP8 + EP8, 8×B300) with a RESP (Valkey) remote tier, one instance per rank (#5548):

  • Correctness. 6 needle prompts of 12.8k–166.7k tokens were answered four ways: cold, from the CPU tier via async prefetch, from L2 via async prefetch after an engine restart (12,800–166,656 tokens loaded), and cold again. The code was found 24/24. Answers served from cache differ from cold ones no more than two cold runs differ from each other; the engine isn't bit-deterministic run to run.
  • Load. A hit-heavy 16k stream (450 requests, 98.5% of tokens from L2) gave async p50 0.41 s / p99 0.74 s against sync 0.41 s / 0.81 s, with 0 timeouts or errors. With a slow L2 (one instance), sync loading backed up to p99 84 s while the engine kept running other requests under async.

Tests (tests/inprocess/v1/pcp_shard):

  • New gloo scenarios with real LMCacheEngines:
    • async_full (w2/4/8): whole prompt, byte-exact load, no pins or extra refs left.
    • async_evict (w4/8): owner of chunk 5 lost it, so every rank agrees on 5 chunks.
    • async_owner_miss (w3/4): rank 1 has no hit and must answer the start of chunk 1, not 0.
  • Logic tests, including a property test over 3,000 random cases that the minimum of the ranks' async answers equals the sharded prefix.
  • The config and gate tests now assert that async loading is accepted, and that the other unsupported flags still raise.
  • Full suite: 48 passed.

If applicable:

  • this PR contains user facing changes - docs added
  • this PR contains unit tests

Ported from LMCache/LMCache#5545: the in-process engine moved to this
repository in LMCache/LMCache#6254.

Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
…s its own chunks)

Follow-up to #5545 (sharded PCP store). **Depends on #5545**: this
branch is built on it, so until #5545 merges the diff includes its
commits. Only the top commit (`dbd78611`) is new.

Ported from LMCache/LMCache#5553: the in-process engine moved to this
repository in LMCache/LMCache#6254.

Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
@ibrahimnd2000
ibrahimnd2000 force-pushed the port/5553-pcp-shard-async branch from 664f740 to 1259f1c Compare November 19, 2026 07:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant