Repository navigation
[Feature] Split metadata lookup from the disk read, and release a prefetch buffer once - #35
Open
AlexNebula wants to merge 22 commits into
Open
AlexNebula wants to merge 22 commits into
AlexNebula wants to merge 22 commits into
Conversation
The in-tree engine left LMCache, so the waiting-queue lookup and the host-fit GET now belong in this package. The switch stays off by default. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A preemption retires the in-flight lookup and a late ACK is dropped when its epoch does not match. Abort during metadata contains does not start a GET. Only a metadata lookup that has not been acknowledged can time out. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
The completion path passes epoch into send_response_to_scheduler. The storage-manager tests rejected that keyword. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A late body ACK cannot revive an absent id. Cleanup carries the attempt epoch so it cannot abort the next lookup. Queued hits occupy the offer window without becoming misses. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A zero preloaded ACK unblocks a queued request. Worker misses and finished preemption counts are dropped so they do not grow for the life of the process. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
… on shutdown. A lookup that arrives while the previous GET is reading unpins the new prefix and reports a miss. The scheduler wrapper is removed with the last connector. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A later request with that epoch returns before contains, so it does not pin or queue. A newer epoch is left alone. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A prefix that still does not fit used to rescan on each free and hold the GIL inside store offload. Count released bytes until the smallest waiting prefix can fit, and scan once. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A full store frees about one prefix and allocates those blocks again, so counting gross frees still walked the queue inside offload. Drain only when bytes are still free after allocate returns. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A full prefetch frees about one prefix when retrieve unpins it, and that free does not go through allocate. The net-free check never ran, so queued hits stayed deferred. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A request whose KV is not already in host was left waiting, so the GPU could idle. Load a preloaded prefix as before. Otherwise recompute when a KV block is free, and cancel the lookup so a late ACK cannot revive it. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A full 20GiB pinned pool was evicting one hot-cache chunk after every failed allocation, and that wait was counted in offload time. The worker now drops only the shortfall for the batch, capped at one engine step. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Budget and free space are both whole aligned chunks. Eviction frees what it can; a request that still does not fit is not copied off the GPU. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
…ool. A full 20GiB local CPU cache now pins an 8GiB store pool and a 12GiB preload pool. Store admission uses the longest free span, and a request that does not fit is skipped before D2H. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: alex <alex@surdata.io>
Admission was the longest span, so a 7-chunk hole skipped an 8-chunk store while evictable KV stayed in the pool. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
The drain was counting free bytes in the store pool, so a full preload pool still admitted GETs that allocated nothing. Async loads also read one chunk at a time and held those pins. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
wait_for_save and a stale lookup cleanup were both dropping the same LOADING future, so Turn3 H2D asserted on an invalidated MemoryObj. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
The 11-13 runs exercised this branch on the 3090 and on the RTX PRO 6000. The README still said those images had not been built. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Ruff rejected an unused EventStatus import, and isort/ruff-format wanted the import order and line breaks these files already had in the working tree. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
3 of 5 tasks
Collaborator
|
@AlexNebula Thanks for your contribution! Would you like to show your use case for the LMCache in process mode? |
Author
|
The in-process engine runs inside each vLLM process. The use case is many users sharing a long prefix, including across machines: each node keeps the engine in-process, and the slow tier may be another machine's memory or a remote store, not only a local disk. |
Collaborator
|
@AlexNebula In fact, i want to know whether you are using LMCache in-process mode in your production scenario or a research scenario? |
Author
|
We are currently running this in a PoC environment and are preparing to move it to production. |
LMCache dev moved kv_format out of lmcache.v1.gpu_connector. Smoke (3.12, dev) failed while collecting tests/smoke because gpu_connectors imported the old path. Signed-off-by: Alex Wang <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Alex Wang <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it
enable_async_loadingstays an existing flag and defaults to off. This branch addssplit_metadata_lookup, registered in_CONFIG_DEFINITIONS, also default off. With both on, the scheduler learns the hit length from a metadata lookup. The disk GET runs only when the preload pool has a contiguous free chunk. No free chunk is a miss, so the scheduler recomputes instead of blocking TTFT on a synchronous read.When
max_local_cpu_sizeis exactly 20, the hot cache splits into a store pool of8589934592bytes (8GiB) and a preload pool of12884901888bytes (12GiB). D2H uses the store pool. Disk reads use the preload pool. Store reclaim continues until one-chunk holes cover the step. Disk puts go to the existingdisk_io_threadspool. No new pool-size config is added.Release a prefetch buffer onceis the fix that lets that path finish. ALOADINGfuture is popped once.retrievepops it at the start and releases it after a successful H2D.lookup_unpinno longer callscleanup_memory_objs. A cleanup whose epoch does not match leaves the buffer that was already accepted. This commit stops the double free. It is not the throughput multiplier.The branch is rebased onto current
main, including #11 (calculate_local_rank_and_world_size) and #8 (daemon=Trueon the storage-manager event loop). Those two commits do not conflict with this branch.Throughput
RTX PRO 6000, FP16 DeepSeek-V2-Lite-Chat, vLLM 0.33.0, LMCache 0.5.7.post1. 800 users, rate 800, online overlap,
max_model_len8192,max_num_seqs128,max_num_batched_tokens8192, KV HBM 12GiB, L1 20GiB. Disk is/dev/sdb2XFS at/mnt/xfs,max_local_disk_size100GiB. This branch has both flags on. The baseline is the same post1 wheel with inprocess left as shipped andenable_async_loadingat its default of false. Every run is 800/0. Double free is 0.Combined tok/s is generated tokens divided by pipeline
wall_s. The stock LMCache+XFS run is the baseline for this protocol.wall_s92.296, overlap 76.898. Stock 388.7 = (50817+38740)/230.430,wall_s230.430, overlap 216.266.wall_s149.234, overlap 94.521 / 92.510. Stock 321.2 = (50813+38440+36915)/392.788,wall_s392.788, overlap 265.413 / 331.169.wall_s184.854, overlap 84.442 / 95.536 / 98.268. Stock 285.2 = (50801+38906+36527+35442)/566.811,wall_s566.811, overlap 269.466 / 396.177 / 345.859.The gap is Turn2+ TTFT, not the latency of one disk read. The same 7962624-byte O_DIRECT read has a stock median of 62–74 ms and a branch median of 100–103 ms. Stock reads more bytes: 45.1GB vs 4.2GB (800×2), 95.8GB vs 29.8GB (800×3), 154.5GB vs 38.2GB (800×4). Stock CPU allocation failures (
no memory is available, block size 7962624) are 9514, 16688, and 24813. This branch has 0. Eight hundred users of a 2048-token prefix are about 51GB, and the 20GiB pool cannot hold them. After the pool is full, stock still reads on the TTFT path.A documented no-cache run on this machine is a different protocol (vLLM 0.15, rate 32,
max_model_len4096):20261002-120539at 971.2 tok/s and20261002-161229at 973.2 tok/s. There is no same-protocol historical SurData run on this card.On a 3090 with AWQ, the same inprocess tree with the flags on versus off is noise: 800×2 is 395.024 vs 393.420 (+0.4%), 800×3 is 435.458 vs 427.905 (+1.8%). Double free is 0. The stock 800×4 on that machine had a disk-read median of 12.7 ms against about 2.5 ms on the other three runs, so it is not a speedup number.
Open PR #26 also releases an async-loaded object once, ported from LMCache/LMCache#5481. It is not in
main. It touchescache_engine.py,event_manager.py, andstorage_manager.py. This branch's release rule is the epoch-scoped pop described above.Known, not changed here
cleanup_memory_objsreturns before popping whenabort()is true. An older DONE future can remain if a newer epoch is already open.update_event_status/add_eventcan overwrite a slot that is already DONE.retrievepops the future without an epoch filter.Test plan
pre-commiton the files this branch changes (ruff, ruff-format, isort, codespell, whitespace).pre-commit run --all-filesstill rewrites legacydocs/source/legacy/**/*.rstthat are unchanged frommain; those rewrites are not in this PRenable_async_loadingforced on. Not run. The ratios above compare the whole path with stock defaults. They are not a multiplier for the one-release commitMade with Cursor