Repository navigation
[MP][Refactor] Dispatch built-in cache contexts in the server factory - #6793
Open
thegoldenflow wants to merge 2 commits into
Open
thegoldenflow wants to merge 2 commits into
thegoldenflow wants to merge 2 commits into
Conversation
Move the six server-only cache context modules from lmcache/v1/platform/
to a new lmcache/v1/multiprocess/cache_context/ package so that platform's
module-level import closure no longer pulls in gpu_connector, distributed
or multiprocess code.
platform/cache_context.py -> multiprocess/cache_context/factory.py
platform/base/cache_context.py -> multiprocess/cache_context/base.py
platform/devices/<dev>/cache_context.py -> multiprocess/cache_context/<dev>.py
(cpu, cuda, musa, npu)
Pure move: no logic changes. Only import paths, docstring cross-references
and doc paths are updated. The old paths are not re-exported. The
DeviceSpec.create_cache_context hooks are kept and now lazy-import from the
new location.
Signed-off-by: Jason <31175216+thegoldenflow@users.noreply.github.com>
Follow-up to the cache context move (LMCache#6792). After that move, the DeviceSpec.create_cache_context overrides of the cpu, cuda, musa and npu platform backends still lazy-imported their context class from lmcache.v1.multiprocess.cache_context, and five platform modules kept a TYPE_CHECKING import of BaseCacheContext from there. Move the built-in mapping into the server-side factory instead: create_cache_context resolves the DeviceSpec as before, then maps a built-in backend_name (cpu, cuda, rocm, musa, npu) to its in-tree context class. Any other spec, i.e. an external lmcache.device_plugins plugin, is still served by its create_cache_context hook, which stays on DeviceSpec with the same default (NotImplementedError). The hook's return annotation becomes Any so platform no longer names the server class. Built-in backends get the same class with the same positional arguments as before; ROCm keeps using GPUCacheContext. Nothing under lmcache/v1/platform/ now imports lmcache.v1.multiprocess, lmcache.v1.gpu_connector or lmcache.v1.distributed, at module level, lazily or under TYPE_CHECKING. Not a pure move. Tests: the dispatch tests now patch the context class in its server module, and cover built-in dispatch (all five backend names, no hook call) and the plugin hook fallback. The NPU test of the removed NpuDeviceSpec override is dropped; the parametrized npu case covers that dispatch. Signed-off-by: Jason <31175216+thegoldenflow@users.noreply.github.com>
9 tasks
thegoldenflow
marked this pull request as ready for review
November 20, 2026 09:12
thegoldenflow
requested review from
ApostaC,
DongDongJu,
VincyZhang,
deng451e,
hickeyma,
hlin99,
maobaolong,
sammshen and
zhengfeihe
as code owners
November 20, 2026 09:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #6792: this branch is #6792's commit (4cc732f) plus one new commit (fc191ab). Please review only fc191ab (
git diff 4cc732f0 fc191ab2). I'll rebase once #6792 merges.What this PR does / why we need it:
Refs #6305, #6310. Follow-up to #6792.
Not a pure move. The server-side
create_cache_contextfactory now picks the in-tree context class for the built-in backends (cpu, cuda, rocm, musa, npu) bybackend_name. The fourDeviceSpec.create_cache_contextoverrides inlmcache/v1/platform/devices/are removed. The hook stays as the extension point for externallmcache.device_pluginsand is used for any non-built-in spec. After this change,lmcache/v1/platform/no longer importslmcache.v1.multiprocess,gpu_connectorordistributed, at module level, lazily, or underTYPE_CHECKING.Special notes for your reviewers:
AI-assisted change; the submitter reviews every line before this leaves draft.
backend_nameand does not override the hook used to inherit the built-in context. It now raisesNotImplementedError. No such spec exists in this repo; downstream repos were not surveyed. The alternative isisinstancematching. Which do you prefer?Any, soplatformdoes not name a server class. The alternative is aProtocolinplatform.Checks:
SKIP=rust-fmt,rust-clippy pre-commit run --all-filespasses. pytest on the dispatch, plugin,platform/devicesand seven multiprocess test files: 554 passed, 177 skipped. The new tests fail on the unchanged code, as expected. An AST scan finds 0 imports from platform into server code (9 before). Everything ran on macOS CPU-only, and every skip is for missing hardware or libraries, so the CUDA/MUSA/NPU contexts were not constructed on real devices.AI-generated PR explainer (required when AI-generated code is included):
pr-explainer-fc191ab2.html
If applicable:
Before marking ready for review
agent-assisted changes.
or compatibility concerns.
Signed-off-bytrailer.bilingual HTML explainer all match the current head.
diagrams, optional motion, code walkthrough, and test rationale match the implementation.