Skip to content

[MP][Refactor] Dispatch built-in cache contexts in the server factory - #6793

Open
thegoldenflow wants to merge 2 commits into
LMCache:devfrom
thegoldenflow:mp-platform-cache-context-dispatch
Open

thegoldenflow wants to merge 2 commits into
LMCache:devfrom
thegoldenflow:mp-platform-cache-context-dispatch

Conversation

@thegoldenflow

@thegoldenflow thegoldenflow commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #6792: this branch is #6792's commit (4cc732f) plus one new commit (fc191ab). Please review only fc191ab (git diff 4cc732f0 fc191ab2). I'll rebase once #6792 merges.

What this PR does / why we need it:

Refs #6305, #6310. Follow-up to #6792.

Not a pure move. The server-side create_cache_context factory now picks the in-tree context class for the built-in backends (cpu, cuda, rocm, musa, npu) by backend_name. The four DeviceSpec.create_cache_context overrides in lmcache/v1/platform/devices/ are removed. The hook stays as the extension point for external lmcache.device_plugins and is used for any non-built-in spec. After this change, lmcache/v1/platform/ no longer imports lmcache.v1.multiprocess, gpu_connector or distributed, at module level, lazily, or under TYPE_CHECKING.

Special notes for your reviewers:

AI-assisted change; the submitter reviews every line before this leaves draft.

  • Behaviour change: a third-party spec that subclasses a built-in spec, sets its own backend_name and does not override the hook used to inherit the built-in context. It now raises NotImplementedError. No such spec exists in this repo; downstream repos were not surveyed. The alternative is isinstance matching. Which do you prefer?
  • The hook's return type is now Any, so platform does not name a server class. The alternative is a Protocol in platform.
  • Is moving built-in dispatch into the server factory acceptable, or should the hooks stay as they are in [MP][Refactor] Move cache contexts out of platform #6792 alone?

Checks: SKIP=rust-fmt,rust-clippy pre-commit run --all-files passes. pytest on the dispatch, plugin, platform/devices and seven multiprocess test files: 554 passed, 177 skipped. The new tests fail on the unchanged code, as expected. An AST scan finds 0 imports from platform into server code (9 before). Everything ran on macOS CPU-only, and every skip is for missing hardware or libraries, so the CUDA/MUSA/NPU contexts were not constructed on real devices.

AI-generated PR explainer (required when AI-generated code is included):

pr-explainer-fc191ab2.html

If applicable:

  • this PR contains user facing changes - docs added
  • this PR contains unit tests
Before marking ready for review
  • I reviewed every changed line since the latest update, including generated or
    agent-assisted changes.
  • I ran the relevant local checks and recorded the commands and results above.
  • I confirmed the description matches the final diff and documents any missing coverage
    or compatibility concerns.
  • I confirmed every commit has a DCO Signed-off-by trailer.
  • I stated whether AI assistance was used in Special notes for your reviewers.
  • If AI-generated code is included, this PR is a draft and its title, description, and
    bilingual HTML explainer all match the current head.
  • The HTML explainer is detailed enough for a reviewer without prior context, and its
    diagrams, optional motion, code walkthrough, and test rationale match the implementation.

Move the six server-only cache context modules from lmcache/v1/platform/
to a new lmcache/v1/multiprocess/cache_context/ package so that platform's
module-level import closure no longer pulls in gpu_connector, distributed
or multiprocess code.

  platform/cache_context.py                -> multiprocess/cache_context/factory.py
  platform/base/cache_context.py           -> multiprocess/cache_context/base.py
  platform/devices/<dev>/cache_context.py  -> multiprocess/cache_context/<dev>.py
                                              (cpu, cuda, musa, npu)

Pure move: no logic changes. Only import paths, docstring cross-references
and doc paths are updated. The old paths are not re-exported. The
DeviceSpec.create_cache_context hooks are kept and now lazy-import from the
new location.

Signed-off-by: Jason <31175216+thegoldenflow@users.noreply.github.com>
Follow-up to the cache context move (LMCache#6792). After that move, the
DeviceSpec.create_cache_context overrides of the cpu, cuda, musa and
npu platform backends still lazy-imported their context class from
lmcache.v1.multiprocess.cache_context, and five platform modules kept a
TYPE_CHECKING import of BaseCacheContext from there.

Move the built-in mapping into the server-side factory instead:
create_cache_context resolves the DeviceSpec as before, then maps a
built-in backend_name (cpu, cuda, rocm, musa, npu) to its in-tree
context class. Any other spec, i.e. an external lmcache.device_plugins
plugin, is still served by its create_cache_context hook, which stays
on DeviceSpec with the same default (NotImplementedError). The hook's
return annotation becomes Any so platform no longer names the server
class.

Built-in backends get the same class with the same positional
arguments as before; ROCm keeps using GPUCacheContext. Nothing under
lmcache/v1/platform/ now imports lmcache.v1.multiprocess,
lmcache.v1.gpu_connector or lmcache.v1.distributed, at module level,
lazily or under TYPE_CHECKING.

Not a pure move. Tests: the dispatch tests now patch the context class
in its server module, and cover built-in dispatch (all five backend
names, no hook call) and the plugin hook fallback. The NPU test of the
removed NpuDeviceSpec override is dropped; the parametrized npu case
covers that dispatch.

Signed-off-by: Jason <31175216+thegoldenflow@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant