Repository navigation
Conversation
|
@nitsingh2 Hi would you like to share your use case of LMCache in-process mode? |
|
Hi @nitsingh2, this is the same fix (and test) as LMCache/LMCache#5515, which I opened on Oct 6 for the same failure with vLLM data parallelism on CUDA (DP8, TP1, MLA model: illegal memory access at the first LMCache hit on DP ranks >= 1). It couldn't merge after the in-process code moved here in LMCache/LMCache#6254. Glad to see it reproduced on ROCm too. Rather than opening a duplicate, I'm happy for this one to go in. Could you add a reference to LMCache/LMCache#5515 in the description, and a |
|
@maobaolong The in-process mode is used in llm-d's tiered-prefix-cache guide. Although this issue is not exposed there. We plan to add an agentic guide on llm-d in same mode where this issue would come up. We also plan to use the in-process mode disagg recipes on ROCm MAD (ROCm/MAD#226), where I first observed the issue. |
With save_only_first_rank (default for MLA), the first rank copies each hit chunk to the GPU, broadcasts it, and batched_to_gpu reads that GPU copy. The receivers allocate their receive buffer the same way. Both derived the device from worker_id. With vLLM data parallelism every DP engine's TP world starts at rank 0, so worker_id is 0 on every DP rank: all ranks copy to cuda:0 and the load kernel on the rank's own GPU reads it, causing a GPU memory-access fault at the first LMCache hit on any rank other than DP0. Seen on CUDA (DP8, TP1, MLA) and on ROCm with DeepSeek-V3 in 1P1D EP8/DP8 and 2P2D EP16/DP16. Use the current device on both the leader and the receivers. Same fix as LMCache/LMCache#5515, which could not merge after the in-process code moved to this repository (LMCache/LMCache#6254). Related: LMCache/LMCache#5501. Co-authored-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com> Signed-off-by: Nitin Singh <nitin.singh@amd.com>
7c2d18b to
c6ba1e9
Compare
|
@ibrahimnd2000 Thanks. Added the Co-authored-by trailer and the #5515 reference to the commit and description. Appreciate you letting this one carry it. |
|
@maobaolong Could you please review once? |
Problem
With
save_only_first_rank(the default for MLA), only the first rank fetches KV chunks on retrieve. It copies each chunk to the GPU, broadcasts it over the TP group, and since #3413batched_to_gpureads that GPU copy directly. The receivers allocate a GPU buffer for the incoming chunk. Both sides derived the device fromworker_id.With vLLM data parallelism, each DP engine's TP world starts at rank 0, so
worker_idis 0 on every DP rank. Every rank copies its broadcast buffer tocuda:0, then the load kernel on the rank's own GPU reads it. The result is a GPU memory-access fault at the first LMCache hit on any rank other than DP0. Reproduced on CUDA (DP8, TP1, MLA model) and on ROCm with DeepSeek-V3 in 1P1D EP8/DP8 and 2P2D EP16/DP16. Same bug class as LMCache/LMCache#5501, which fixed the equivalent device binding inCreateGPUConnector.Fix
Use
torch_dev.current_device()for both the leader's broadcast copy and the receivers' buffer in_broadcast_or_receive_memory_objs.Testing
tests/inprocess/v1/test_broadcast_device.pychecks that the leader copies to the current device and the receivers allocate on it. Passes with the fix, fails against the original code.test_retrieve_pin_ownership.py: thetorch_devmock now returnscurrent_device().History
Same fix and test as LMCache/LMCache#5515 by @MUHAMMAD_HANDLE (Oct 6, CUDA), which could not merge after the in-process code moved here in LMCache/LMCache#6254. Supersedes LMCache/LMCache#6109. Co-authored accordingly.