Skip to content

[Bugfix] remote backend: async get must read the worker-0 keys in MLA worker-id-as-0 mode - #30

Open
ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5550-remote-async-get-worker0
Open

ibrahimnd2000 wants to merge 1 commit into
LMCache:mainfrom
ibrahimnd2000:port/5550-remote-async-get-worker0

Conversation

@ibrahimnd2000

Copy link
Copy Markdown

Ported from LMCache/LMCache#5550, whose in-process files moved to this repository in LMCache/LMCache#6254. Same change on current main. Note: conflicts with #19 in remote_backend.py; whichever merges second needs a small rebase.

What this PR does / why we need it:

With remote_enable_mla_worker_id_as0 (the MLA default), chunks are stored under worker 0, and RemoteBackend.batched_async_contains and the blocking get rewrite keys to worker 0:

if self._mla_worker_id_as0_mode:
    keys = [key.with_new_worker_id(0) for key in keys]

batched_get_non_blocking (the async prefetch path) did not. So any worker other than 0 that does an async lookup + prefetch finds its chunks with EXISTS, then asks for keys that do not exist. With the RESP connector that GET also never returns: the native client blocks on the nil reply, which I'll fix in a separate PR.

This hasn't shown up upstream because only worker 0 does async lookups in MLA mode. It does once more workers prefetch. We hit it with the sharded PCP store (#5545), where every rank prefetches its own chunks: after an engine restart every L2 prefetch timed out (batched_get_non_blocking timed out) and nothing was loaded. With this fix the same run loaded 12,800–166,656 tokens per prompt from L2.

Special notes for your reviewers:

  • One-line behavioural change, mirroring batched_async_contains.
  • tests/inprocess/v1/storage_backend/test_remote_async_get_worker0_key.py: the async get now receives the same worker-0 keys as the async contains (fails on dev), and keys are unchanged outside that mode.

If applicable:

  • this PR contains user facing changes - docs added
  • this PR contains unit tests

… worker-id-as-0 mode

With `remote_enable_mla_worker_id_as0` (the MLA default), chunks are
stored under worker 0, and `RemoteBackend.batched_async_contains` and
the blocking get rewrite keys to worker 0:

Ported from LMCache/LMCache#5550: the in-process engine moved to this
repository in LMCache/LMCache#6254.

Signed-off-by: Muhammad Ibrahim <ibrahimnd2000@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant