Skip to content

[Bugfix] Broadcast MLA chunks on the worker's current device under DP - #15

Open
nitsingh2 wants to merge 1 commit into
LMCache:mainfrom
nitsingh2:fix/mla-broadcast-current-device
Open

nitsingh2 wants to merge 1 commit into
LMCache:mainfrom
nitsingh2:fix/mla-broadcast-current-device

Conversation

@nitsingh2

@nitsingh2 nitsingh2 commented Nov 11, 2026 •

Copy link
Copy Markdown

Problem

With save_only_first_rank (the default for MLA), only the first rank fetches KV chunks on retrieve. It copies each chunk to the GPU, broadcasts it over the TP group, and since #3413 batched_to_gpu reads that GPU copy directly. The receivers allocate a GPU buffer for the incoming chunk. Both sides derived the device from worker_id.

With vLLM data parallelism, each DP engine's TP world starts at rank 0, so worker_id is 0 on every DP rank. Every rank copies its broadcast buffer to cuda:0, then the load kernel on the rank's own GPU reads it. The result is a GPU memory-access fault at the first LMCache hit on any rank other than DP0. Reproduced on CUDA (DP8, TP1, MLA model) and on ROCm with DeepSeek-V3 in 1P1D EP8/DP8 and 2P2D EP16/DP16. Same bug class as LMCache/LMCache#5501, which fixed the equivalent device binding in CreateGPUConnector.

Fix

Use torch_dev.current_device() for both the leader's broadcast copy and the receivers' buffer in
_broadcast_or_receive_memory_objs.

Testing

  • New tests/inprocess/v1/test_broadcast_device.py checks that the leader copies to the current device and the receivers allocate on it. Passes with the fix, fails against the original code.
  • test_retrieve_pin_ownership.py: the torch_dev mock now returns current_device().
  • A 2P2D EP16/DP16 DeepSeek-V3 run on ROCm with this change applied as a runtime patch completed with no faults across ~525 retrieves per role; the same setup without it faults at the first retrieve on non-DP0 ranks.

History

Same fix and test as LMCache/LMCache#5515 by @MUHAMMAD_HANDLE (Oct 6, CUDA), which could not merge after the in-process code moved here in LMCache/LMCache#6254. Supersedes LMCache/LMCache#6109. Co-authored accordingly.

@maobaolong

Copy link
Copy Markdown
Collaborator

@nitsingh2 Hi would you like to share your use case of LMCache in-process mode?

@ibrahimnd2000

Copy link
Copy Markdown

Hi @nitsingh2, this is the same fix (and test) as LMCache/LMCache#5515, which I opened on Oct 6 for the same failure with vLLM data parallelism on CUDA (DP8, TP1, MLA model: illegal memory access at the first LMCache hit on DP ranks >= 1). It couldn't merge after the in-process code moved here in LMCache/LMCache#6254.

Glad to see it reproduced on ROCm too. Rather than opening a duplicate, I'm happy for this one to go in. Could you add a reference to LMCache/LMCache#5515 in the description, and a Co-authored-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com> trailer on the commit? I'll close the old PR in favour of this one.

@nitsingh2

Copy link
Copy Markdown
Author

@maobaolong The in-process mode is used in llm-d's tiered-prefix-cache guide. Although this issue is not exposed there. We plan to add an agentic guide on llm-d in same mode where this issue would come up.

We also plan to use the in-process mode disagg recipes on ROCm MAD (ROCm/MAD#226), where I first observed the issue.

With save_only_first_rank (default for MLA), the first rank copies each hit
chunk to the GPU, broadcasts it, and batched_to_gpu reads that GPU copy.
The receivers allocate their receive buffer the same way. Both derived the
device from worker_id.

With vLLM data parallelism every DP engine's TP world starts at rank 0, so
worker_id is 0 on every DP rank: all ranks copy to cuda:0 and the load kernel
on the rank's own GPU reads it, causing a GPU memory-access fault at the
first LMCache hit on any rank other than DP0. Seen on CUDA (DP8, TP1, MLA)
and on ROCm with DeepSeek-V3 in 1P1D EP8/DP8 and 2P2D EP16/DP16.

Use the current device on both the leader and the receivers.

Same fix as LMCache/LMCache#5515, which could not merge after the in-process
code moved to this repository (LMCache/LMCache#6254).
Related: LMCache/LMCache#5501.

Co-authored-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>
Signed-off-by: Nitin Singh <nitin.singh@amd.com>
@nitsingh2
nitsingh2 force-pushed the fix/mla-broadcast-current-device branch from 7c2d18b to c6ba1e9 Compare November 12, 2026 14:09
@nitsingh2

Copy link
Copy Markdown
Author

@ibrahimnd2000 Thanks. Added the Co-authored-by trailer and the #5515 reference to the commit and description. Appreciate you letting this one carry it.

@nitsingh2
nitsingh2 marked this pull request as ready for review November 12, 2026 14:35
@nitsingh2

Copy link
Copy Markdown
Author

@maobaolong Could you please review once?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants