Skip to content

[Feature] Split metadata lookup from the disk read, and release a prefetch buffer once - #35

Open
AlexNebula wants to merge 22 commits into
LMCache:mainfrom
SurData-io:split-metadata-lookup-upstream
Open

AlexNebula wants to merge 22 commits into
LMCache:mainfrom
SurData-io:split-metadata-lookup-upstream

Conversation

@AlexNebula

Copy link
Copy Markdown

What this PR does / why we need it

enable_async_loading stays an existing flag and defaults to off. This branch adds split_metadata_lookup, registered in _CONFIG_DEFINITIONS, also default off. With both on, the scheduler learns the hit length from a metadata lookup. The disk GET runs only when the preload pool has a contiguous free chunk. No free chunk is a miss, so the scheduler recomputes instead of blocking TTFT on a synchronous read.

When max_local_cpu_size is exactly 20, the hot cache splits into a store pool of 8589934592 bytes (8GiB) and a preload pool of 12884901888 bytes (12GiB). D2H uses the store pool. Disk reads use the preload pool. Store reclaim continues until one-chunk holes cover the step. Disk puts go to the existing disk_io_threads pool. No new pool-size config is added.

Release a prefetch buffer once is the fix that lets that path finish. A LOADING future is popped once. retrieve pops it at the start and releases it after a successful H2D. lookup_unpin no longer calls cleanup_memory_objs. A cleanup whose epoch does not match leaves the buffer that was already accepted. This commit stops the double free. It is not the throughput multiplier.

The branch is rebased onto current main, including #11 (calculate_local_rank_and_world_size) and #8 (daemon=True on the storage-manager event loop). Those two commits do not conflict with this branch.

Throughput

RTX PRO 6000, FP16 DeepSeek-V2-Lite-Chat, vLLM 0.33.0, LMCache 0.5.7.post1. 800 users, rate 800, online overlap, max_model_len 8192, max_num_seqs 128, max_num_batched_tokens 8192, KV HBM 12GiB, L1 20GiB. Disk is /dev/sdb2 XFS at /mnt/xfs, max_local_disk_size 100GiB. This branch has both flags on. The baseline is the same post1 wheel with inprocess left as shipped and enable_async_loading at its default of false. Every run is 800/0. Double free is 0.

Combined tok/s is generated tokens divided by pipeline wall_s. The stock LMCache+XFS run is the baseline for this protocol.

  • 800×2: this branch 967.2 = (50801+38471)/92.296, wall_s 92.296, overlap 76.898. Stock 388.7 = (50817+38740)/230.430, wall_s 230.430, overlap 216.266.
  • 800×3: this branch 827.3 = (50910+38349+34199)/149.234, wall_s 149.234, overlap 94.521 / 92.510. Stock 321.2 = (50813+38440+36915)/392.788, wall_s 392.788, overlap 265.413 / 331.169.
  • 800×4: this branch 867.1 = (50910+38894+36087+34400)/184.854, wall_s 184.854, overlap 84.442 / 95.536 / 98.268. Stock 285.2 = (50801+38906+36527+35442)/566.811, wall_s 566.811, overlap 269.466 / 396.177 / 345.859.

The gap is Turn2+ TTFT, not the latency of one disk read. The same 7962624-byte O_DIRECT read has a stock median of 62–74 ms and a branch median of 100–103 ms. Stock reads more bytes: 45.1GB vs 4.2GB (800×2), 95.8GB vs 29.8GB (800×3), 154.5GB vs 38.2GB (800×4). Stock CPU allocation failures (no memory is available, block size 7962624) are 9514, 16688, and 24813. This branch has 0. Eight hundred users of a 2048-token prefix are about 51GB, and the 20GiB pool cannot hold them. After the pool is full, stock still reads on the TTFT path.

A documented no-cache run on this machine is a different protocol (vLLM 0.15, rate 32, max_model_len 4096): 20261002-120539 at 971.2 tok/s and 20261002-161229 at 973.2 tok/s. There is no same-protocol historical SurData run on this card.

On a 3090 with AWQ, the same inprocess tree with the flags on versus off is noise: 800×2 is 395.024 vs 393.420 (+0.4%), 800×3 is 435.458 vs 427.905 (+1.8%). Double free is 0. The stock 800×4 on that machine had a disk-read median of 12.7 ms against about 2.5 ms on the other three runs, so it is not a speedup number.

Open PR #26 also releases an async-loaded object once, ported from LMCache/LMCache#5481. It is not in main. It touches cache_engine.py, event_manager.py, and storage_manager.py. This branch's release rule is the epoch-scoped pop described above.

Known, not changed here

  • cleanup_memory_objs returns before popping when abort() is true. An older DONE future can remain if a newer epoch is already open.
  • update_event_status / add_event can overwrite a slot that is already DONE.
  • retrieve pops the future without an epoch filter.

Test plan

  • CPU tests for cleanup, pre-evict, the tier pipeline, disk admission, and partial GET, run before the release commit
  • pre-commit on the files this branch changes (ruff, ruff-format, isort, codespell, whitespace). pre-commit run --all-files still rewrites legacy docs/source/legacy/**/*.rst that are unchanged from main; those rewrites are not in this PR
  • 3090 retest of the release commit, 800×2 / 800×3 / 800×4, Double free 0
  • RTX PRO 6000, this branch versus stock 0.5.7.post1, 800×2 / 800×3 / 800×4, numbers above
  • CI on this pull request
  • A stock binary with only enable_async_loading forced on. Not run. The ratios above compare the whole path with stock defaults. They are not a multiplier for the one-release commit

Made with Cursor

AlexNebula and others added 20 commits November 14, 2026 03:14
The in-tree engine left LMCache, so the waiting-queue lookup and the host-fit GET now belong in this package. The switch stays off by default.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A preemption retires the in-flight lookup and a late ACK is dropped when its epoch does not match. Abort during metadata contains does not start a GET. Only a metadata lookup that has not been acknowledged can time out.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
The completion path passes epoch into send_response_to_scheduler. The storage-manager tests rejected that keyword.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A late body ACK cannot revive an absent id. Cleanup carries the attempt epoch so it cannot abort the next lookup. Queued hits occupy the offer window without becoming misses.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A zero preloaded ACK unblocks a queued request. Worker misses and finished preemption counts are dropped so they do not grow for the life of the process.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
… on shutdown.

A lookup that arrives while the previous GET is reading unpins the new prefix and reports a miss. The scheduler wrapper is removed with the last connector.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A later request with that epoch returns before contains, so it does not pin or queue. A newer epoch is left alone.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A prefix that still does not fit used to rescan on each free and hold the GIL inside store offload. Count released bytes until the smallest waiting prefix can fit, and scan once.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A full store frees about one prefix and allocates those blocks again, so counting gross frees still walked the queue inside offload. Drain only when bytes are still free after allocate returns.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A full prefetch frees about one prefix when retrieve unpins it, and that free does not go through allocate. The net-free check never ran, so queued hits stayed deferred.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A request whose KV is not already in host was left waiting, so the GPU could idle. Load a preloaded prefix as before. Otherwise recompute when a KV block is free, and cancel the lookup so a late ACK cannot revive it.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A full 20GiB pinned pool was evicting one hot-cache chunk after every
failed allocation, and that wait was counted in offload time. The worker
now drops only the shortfall for the batch, capped at one engine step.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Budget and free space are both whole aligned chunks. Eviction frees what it can; a request that still does not fit is not copied off the GPU.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ool.

A full 20GiB local CPU cache now pins an 8GiB store pool and a 12GiB preload pool. Store admission uses the longest free span, and a request that does not fit is skipped before D2H.

Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: alex <alex@surdata.io>
Admission was the longest span, so a 7-chunk hole skipped an 8-chunk store while evictable KV stayed in the pool.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
The drain was counting free bytes in the store pool, so a full preload pool still admitted GETs that allocated nothing. Async loads also read one chunk at a time and held those pins.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
wait_for_save and a stale lookup cleanup were both dropping the same LOADING future, so Turn3 H2D asserted on an invalidated MemoryObj.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
The 11-13 runs exercised this branch on the 3090 and on the RTX PRO 6000. The README still said those images had not been built.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Ruff rejected an unused EventStatus import, and isort/ruff-format wanted the import order and line breaks these files already had in the working tree.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
@maobaolong

Copy link
Copy Markdown
Collaborator

@AlexNebula Thanks for your contribution! Would you like to show your use case for the LMCache in process mode?

@AlexNebula

Copy link
Copy Markdown
Author

The in-process engine runs inside each vLLM process. The use case is many users sharing a long prefix, including across machines: each node keeps the engine in-process, and the slow tier may be another machine's memory or a remote store, not only a local disk.

@maobaolong

Copy link
Copy Markdown
Collaborator

@AlexNebula In fact, i want to know whether you are using LMCache in-process mode in your production scenario or a research scenario?

@AlexNebula

Copy link
Copy Markdown
Author

We are currently running this in a PoC environment and are preparing to move it to production.

AlexNebula and others added 2 commits November 21, 2026 15:07
LMCache dev moved kv_format out of lmcache.v1.gpu_connector. Smoke (3.12, dev) failed while collecting tests/smoke because gpu_connectors imported the old path.

Signed-off-by: Alex Wang <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Alex Wang <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants