Skip to content

[Feature] remote backend: "{rank}" in remote_url selects a per-worker remote instance - #29

Open
ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5548-remote-url-rank
Open

ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5548-remote-url-rank

Conversation

@ibrahimnd2000

Copy link
Copy Markdown

Ported from LMCache/LMCache#5548, whose in-process files moved to this repository in LMCache/LMCache#6254. Same change on current main. Note: conflicts with #19 in remote_backend.py; whichever merges second needs a small rebase.

What this PR does / why we need it:

RemoteBackend now replaces {rank} in remote_url with the worker's rank (metadata.worker_id, 0 when unset):

remote_url: resp://valkey-{rank}.ns.svc.cluster.local:6379

When each worker only reads and writes its own chunks, for example with the sharded PCP store in #5545 (rank r owns chunk_index % world_size == r), every worker can use its own remote instance with no coordination. Rank r of every engine owns the same chunks, so cross-engine reuse still works. A URL without {rank} is unchanged.

Why: a single Valkey instance executes commands on one thread. With 12.4 MB chunks (GLM-5.3 FP8, chunk 256) it served ~0.9 GB/s, with its main thread 73–74% busy on GETs, and a 16k-token L2 hit cost 0.83–0.89 s against 0.32 s to recompute. Measured on one 8×B300 prefill engine (PCP8), replaying a production request stream where ~98% of tokens came from L2:

Aggregate GET bandwidth (8 ranks) p50 p99
1 Valkey instance 1.71 GB/s 9.71 s 83.65 s
8 instances, {rank} URL 10.76 GB/s 0.41 s 0.81 s
recompute (no cache) – 0.31 s ~1.0 s

Special notes for your reviewers:

  • The substitution happens in both places the URL is read (constructor and connection (re)initialisation), so reconnects keep the same instance.
  • Related: #5545 (sharded PCP store), #5547 (native client lost-completion fix found in the same investigation).

If applicable:

  • this PR contains user facing changes - docs added
  • this PR contains unit tests

… remote instance

`RemoteBackend` now replaces `{rank}` in `remote_url` with the worker's
rank (`metadata.worker_id`, 0 when unset):

Ported from LMCache/LMCache#5548: the in-process engine moved to this
repository in LMCache/LMCache#6254.

Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant