Repository navigation
分拆元数据查找,预取缓冲只释放一次 - #1
Closed
AlexNebula wants to merge 18 commits into
Closed
AlexNebula wants to merge 18 commits into
AlexNebula wants to merge 18 commits into
Conversation
The in-tree engine left LMCache, so the waiting-queue lookup and the host-fit GET now belong in this package. The switch stays off by default. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A preemption retires the in-flight lookup and a late ACK is dropped when its epoch does not match. Abort during metadata contains does not start a GET. Only a metadata lookup that has not been acknowledged can time out. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
The completion path passes epoch into send_response_to_scheduler. The storage-manager tests rejected that keyword. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A late body ACK cannot revive an absent id. Cleanup carries the attempt epoch so it cannot abort the next lookup. Queued hits occupy the offer window without becoming misses. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A zero preloaded ACK unblocks a queued request. Worker misses and finished preemption counts are dropped so they do not grow for the life of the process. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
… on shutdown. A lookup that arrives while the previous GET is reading unpins the new prefix and reports a miss. The scheduler wrapper is removed with the last connector. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A later request with that epoch returns before contains, so it does not pin or queue. A newer epoch is left alone. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A prefix that still does not fit used to rescan on each free and hold the GIL inside store offload. Count released bytes until the smallest waiting prefix can fit, and scan once. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A full store frees about one prefix and allocates those blocks again, so counting gross frees still walked the queue inside offload. Drain only when bytes are still free after allocate returns. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A full prefetch frees about one prefix when retrieve unpins it, and that free does not go through allocate. The net-free check never ran, so queued hits stayed deferred. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A request whose KV is not already in host was left waiting, so the GPU could idle. Load a preloaded prefix as before. Otherwise recompute when a KV block is free, and cancel the lookup so a late ACK cannot revive it. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
A full 20GiB pinned pool was evicting one hot-cache chunk after every failed allocation, and that wait was counted in offload time. The worker now drops only the shortfall for the batch, capped at one engine step. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Budget and free space are both whole aligned chunks. Eviction frees what it can; a request that still does not fit is not copied off the GPU. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
…ool. A full 20GiB local CPU cache now pins an 8GiB store pool and a 12GiB preload pool. Store admission uses the longest free span, and a request that does not fit is skipped before D2H. Co-authored-by: Cursor <cursoragent@cursor.com>
Admission was the longest span, so a 7-chunk hole skipped an 8-chunk store while evictable KV stayed in the pool. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
The drain was counting free bytes in the store pool, so a full preload pool still admitted GETs that allocated nothing. Async loads also read one chunk at a time and held those pins. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
wait_for_save and a stale lookup cleanup were both dropping the same LOADING future, so Turn3 H2D asserted on an invalidated MemoryObj. Signed-off-by: alex <alex@surdata.io> Co-authored-by: Cursor <cursoragent@cursor.com>
Author
|
This pull request targeted the fork main. The branch is submitted upstream at LMCache#35. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
enable_async_loading与split_metadata_lookup同时打开时,调度器先拿到命中长度;盘读只在 preload 池有连续空块时放行,没有空块就报 miss,让调度器重算,不再把同步读盘算进 TTFT。enable_async_loading是原有配置,默认关闭。split_metadata_lookup是这次新增的配置,已写进_CONFIG_DEFINITIONS,默认关闭。max_local_cpu_size正好为 20 时拆成 store8589934592(8GiB)和 preload12884901888(12GiB)。这次不增加池大小的配置项。D2H 进 store 池,读盘进 preload 池。store 池按可分配块数回收。写盘进已有的disk_io_threads线程池。a934ac8让这条快路径能跑完:一条LOADINGfuture 只弹出一次。retrieve开头弹出,成功的 H2D 由 retrieve 释放;lookup_unpin不再调用cleanup_memory_objs。过期 epoch 的 cleanup 不释放已经收下的缓冲。它修的是双重释放,不是吞吐乘数。吞吐
物理机 RTX PRO 6000,FP16
DeepSeek-V2-Lite-Chat,vLLM 0.33.0,LMCache 0.5.7.post1。800 用户、rate 800、在线重叠、max_model_len8192、max_num_seqs128、max_num_batched_tokens8192、KV HBM 12GiB、L1 20GiB。盘是/dev/sdb2XFS/mnt/xfs,max_local_disk_size100GiB。PR 两个开关都开。原生是同一 post1、不覆盖 inprocess、enable_async_loading为默认 False。六场都是 800/0,Double free 为 0。综合 tok/s 按轮数分开。原生是这组协议里的 stock LMCache+XFS 基准。
wall_s92.296,重叠 76.898。原生 388.7 = (50817+38740)/230.430,wall_s230.430,重叠 216.266。wall_s149.234,重叠 94.521 / 92.510。原生 321.2 = (50813+38440+36915)/392.788,wall_s392.788,重叠 265.413 / 331.169。wall_s184.854,重叠 84.442 / 95.536 / 98.268。原生 285.2 = (50801+38906+36527+35442)/566.811,wall_s566.811,重叠 269.466 / 396.177 / 345.859。差在 Turn2 及以后的 TTFT,不在单次读盘延迟。同一次 7962624 字节的 O_DIRECT,原生中位 62–74 ms,PR 100–103 ms。原生读得更多:800×2 是 45.1GB 对 4.2GB,800×3 是 95.8GB 对 29.8GB,800×4 是 154.5GB 对 38.2GB。原生 CPU 分配失败(
no memory is available,块大小 7962624)分别是 9514、16688、24813;PR 是 0。800 用户各 2048 token 约 51GB,20GiB 池放不下。原生在池满之后仍在 TTFT 上读盘。文档里的物理机不存是另一口径,不并进上表:vLLM 0.15、rate 32、
max_model_len4096。20261002-120539综合 971.2,20261002-161229综合 973.2。这张卡上没有同协议(vLLM 0.33、rate 800、8192、在线重叠)的历史 SurData 跑次。3090、AWQ、同一份 inprocess、只比
a934ac8和开关:800×2 是 395.0 对 393.4(+0.4%),800×3 是 435.5 对 427.9(+1.8%)。Double free 为 0。那次原生 800×4 的盘读中位 12.7 ms,其余三场约 2.5 ms,不拿来当加速比。已知、这次不改
abort()返回 True 时,cleanup_memory_objs在弹出 epoch 之前就返回。更新的 epoch 已经_tier_open时,更老的 DONE future 可能留在槽里。update_event_status/add_event可以覆盖一个已经 DONE 的槽。Test plan
tests/inprocess/v1/test_cache_engine_cleanup.py、test_local_cpu_pre_evict.py、test_tier_pipeline.py以及读盘准入、部分 GET 的测试在提交前通过a934ac8的 800×2 / 800×3 / 800×4,Double free 为 0.github/workflows/ci.yml在main上,pull_request会触发;打开本 PR 之后 Actions 里一次运行都没有,ci.yml也未注册成 workflow。仓库 Actions 开关是开的,历史 run 总数为 0。要让 CI 出现,需要往main再推一次,让 GitHub 重新索引 workflow。这次没有改main。enable_async_loading的对照臂。这次没跑,所以上面的倍数是整条路径对 stock 默认,不是a934ac8单独的加速比