Skip to content

[Bugfix] V3 GPU connector: transfer each kernel group with its own KV layout - #23

Open
ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5386-v3-per-group-kv-layout
Open

ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5386-v3-per-group-kv-layout

Conversation

@ibrahimnd2000

Copy link
Copy Markdown

Ported from LMCache/LMCache#5386, whose in-process files moved to this repository in LMCache/LMCache#6254. Same change on current main.

What this PR does / why we need it:

VLLMPagedMemGPUConnectorV3 (the non-MP, in-process connector) splits the KV caches into kernel groups, but it passes one set of layout parameters to multi_layer_kv_transfer for every group: engine KV format, page buffer size, block size, head size and block stride, all sampled from layer 0.

That breaks models whose layers use different KV layouts. GLM-5.x (and other DeepSeek-V3.2-style DSA models) register two layouts in the same model:

Layer type Tensor Engine KV format Bytes / token
DSA indexer k-cache (NB, 64, 132) uint8 NL_X_NB_BSV_BSS 132
MLA latent cache (NB, 64, 576) uint8 NL_X_NB_BS_HS 576

KVLayerGroupsManager correctly builds two kernel groups (hs=132, stride 8448 and hs=576, stride 36864). But V3 then transfers the MLA group with the indexer's format, head size and block stride. Two symptoms, depending on timing:

  1. Crash on the first stores. The layer groups are built lazily inside the first to_gpu/from_gpu, but LMCacheEngine.store allocates the chunk's memory object before calling the connector. So the first objects get the legacy single-group shape [1, L, T, 576], and the copy fails with a 576 vs 132 size mismatch.
  2. Silent KV corruption once shapes are right: the MLA data is read and written with the indexer's value/scale-split format and stride.

The existing warning in _initialize_kv_cache_pointers only fires for heterogeneous block sizes. Here both groups have bs=64, so nothing was logged.

What changed:

  • Per-group layout for every transfer. Each kernel group's transfer uses that group's own engine_kv_format and shape_desc (nb × bs, bs, hs, block_stride_elems), in to_gpu and on both from_gpu paths (direct and via the staging buffer). This matches what the MP path (GPUCacheContext) already does. The native kernel already takes all of these as per-call arguments, so no CUDA changes are needed. The old heterogeneous-block-size warning is removed, since that case is now handled.
  • Correct shapes from the first chunk. A new public VLLMPagedMemGPUConnectorV3.build_kv_layer_groups(kvcaches) is called from LMCacheConnectorV1Impl.register_kv_caches before post_init. So LMCacheMetadata.get_shapes() returns per-group shapes for every memory object, including the first stored chunks.
  • Fail loudly on a layout mismatch. A memory object whose per-group shapes don't match the kernel groups raises RuntimeError instead of being copied into.
  • Single-group models are unchanged. Their one group's parameters are identical to the previous layer-0 scalars, and objects without per-group shapes are still accepted when there is only one group. The scalar attributes (engine_kv_format, head_size, …) are kept.

Tests (tests/inprocess/v1/gpu_connector/test_v3_per_group_layout.py, CPU-runnable: CUDA streams are stubbed and multi_layer_kv_transfer is replaced by a recorder):

  • Each kernel group of a DSA-style model (indexer 132 + MLA 576) gets its own format, head size and block stride, for to_gpu and from_gpu.
  • get_shapes() returns per-group shapes as soon as the KV caches are registered.
  • A memory object with the legacy single-group shape is rejected for mixed-layout models.
  • Single-group (MLA-only) transfers are unchanged.
  • register_kv_caches builds the layer groups before post_init (skipped when vllm isn't installed).

On unmodified dev, the per-group and mismatch tests fail: the MLA group is transferred with NL_X_NB_BSV_BSS. With this PR, the new tests plus the existing tests/inprocess/v1/gpu_connector, test_kv_layer_groups_manager.py and test_gpu_connector.py suites pass on a CPU build (236 passed, 27 skipped CUDA-only). pre-commit passes on the changed files.

Validation on hardware:

Validated on a GLM-5.x deployment (TP8, 8×B300) with a needle-in-a-haystack test: 30/30 exact answers served from LMCache hits after evicting the GPU prefix cache. Before the fix, the connector crashed with a 576 vs 132 layout mismatch.

Special notes for your reviewers:

  • The CUDA test test_vllm_paged_connector_v3_with_gpu_and_mla covers multiple groups with the same engine format. A GPU round-trip test with two groups of different formats (NL_X_NB_BSV_BSS + NL_X_NB_BS_HS) would be a good addition on CI hardware. I couldn't run CUDA tests locally.
  • VLLMPagedMemXPUConnectorV3 has the same single-layout pattern. I left it unchanged here to keep the PR focused, and can follow up if wanted.

If applicable:

  • this PR contains user facing changes - docs added
  • this PR contains unit tests

… layout

`VLLMPagedMemGPUConnectorV3` (the non-MP, in-process connector) splits
the KV caches into kernel groups, but it passes **one** set of layout
parameters to `multi_layer_kv_transfer` for **every** group: engine KV
format, page buffer size, block size, head size and block stride, all
sampled from layer 0.

Ported from LMCache/LMCache#5386: the in-process engine moved to this
repository in LMCache/LMCache#6254.

Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
@ibrahimnd2000
ibrahimnd2000 force-pushed the port/5386-v3-per-group-kv-layout branch from 3740592 to 0f432da Compare November 19, 2026 07:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant