Repository navigation
[Bugfix] V3 GPU connector: transfer each kernel group with its own KV layout - #23
Open
ibrahimnd2000 wants to merge 1 commit into
Open
ibrahimnd2000 wants to merge 1 commit into
ibrahimnd2000 wants to merge 1 commit into
Conversation
This was referenced Nov 12, 2026
Closed
… layout `VLLMPagedMemGPUConnectorV3` (the non-MP, in-process connector) splits the KV caches into kernel groups, but it passes **one** set of layout parameters to `multi_layer_kv_transfer` for **every** group: engine KV format, page buffer size, block size, head size and block stride, all sampled from layer 0. Ported from LMCache/LMCache#5386: the in-process engine moved to this repository in LMCache/LMCache#6254. Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
ibrahimnd2000
force-pushed
the
port/5386-v3-per-group-kv-layout
branch
from
November 19, 2026 07:07
3740592 to
0f432da
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
VLLMPagedMemGPUConnectorV3(the non-MP, in-process connector) splits the KV caches into kernel groups, but it passes one set of layout parameters tomulti_layer_kv_transferfor every group: engine KV format, page buffer size, block size, head size and block stride, all sampled from layer 0.That breaks models whose layers use different KV layouts. GLM-5.x (and other DeepSeek-V3.2-style DSA models) register two layouts in the same model:
(NB, 64, 132)uint8NL_X_NB_BSV_BSS(NB, 64, 576)uint8NL_X_NB_BS_HSKVLayerGroupsManagercorrectly builds two kernel groups (hs=132, stride 8448 andhs=576, stride 36864). But V3 then transfers the MLA group with the indexer's format, head size and block stride. Two symptoms, depending on timing:to_gpu/from_gpu, butLMCacheEngine.storeallocates the chunk's memory object before calling the connector. So the first objects get the legacy single-group shape[1, L, T, 576], and the copy fails with a 576 vs 132 size mismatch.The existing warning in
_initialize_kv_cache_pointersonly fires for heterogeneous block sizes. Here both groups havebs=64, so nothing was logged.What changed:
engine_kv_formatandshape_desc(nb × bs,bs,hs,block_stride_elems), into_gpuand on bothfrom_gpupaths (direct and via the staging buffer). This matches what the MP path (GPUCacheContext) already does. The native kernel already takes all of these as per-call arguments, so no CUDA changes are needed. The old heterogeneous-block-size warning is removed, since that case is now handled.VLLMPagedMemGPUConnectorV3.build_kv_layer_groups(kvcaches)is called fromLMCacheConnectorV1Impl.register_kv_cachesbeforepost_init. SoLMCacheMetadata.get_shapes()returns per-group shapes for every memory object, including the first stored chunks.RuntimeErrorinstead of being copied into.shapesare still accepted when there is only one group. The scalar attributes (engine_kv_format,head_size, …) are kept.Tests (
tests/inprocess/v1/gpu_connector/test_v3_per_group_layout.py, CPU-runnable: CUDA streams are stubbed andmulti_layer_kv_transferis replaced by a recorder):to_gpuandfrom_gpu.get_shapes()returns per-group shapes as soon as the KV caches are registered.register_kv_cachesbuilds the layer groups beforepost_init(skipped whenvllmisn't installed).On unmodified
dev, the per-group and mismatch tests fail: the MLA group is transferred withNL_X_NB_BSV_BSS. With this PR, the new tests plus the existingtests/inprocess/v1/gpu_connector,test_kv_layer_groups_manager.pyandtest_gpu_connector.pysuites pass on a CPU build (236 passed, 27 skipped CUDA-only).pre-commitpasses on the changed files.Validation on hardware:
Validated on a GLM-5.x deployment (TP8, 8×B300) with a needle-in-a-haystack test: 30/30 exact answers served from LMCache hits after evicting the GPU prefix cache. Before the fix, the connector crashed with a 576 vs 132 layout mismatch.
Special notes for your reviewers:
test_vllm_paged_connector_v3_with_gpu_and_mlacovers multiple groups with the same engine format. A GPU round-trip test with two groups of different formats (NL_X_NB_BSV_BSS+NL_X_NB_BS_HS) would be a good addition on CI hardware. I couldn't run CUDA tests locally.VLLMPagedMemXPUConnectorV3has the same single-layout pattern. I left it unchanged here to keep the PR focused, and can follow up if wanted.If applicable: