Skip to content

分拆元数据查找,预取缓冲只释放一次 - #1

Closed
AlexNebula wants to merge 18 commits into
mainfrom
split-metadata-lookup
Closed

AlexNebula wants to merge 18 commits into
mainfrom
split-metadata-lookup

Conversation

@AlexNebula

@AlexNebula AlexNebula commented Nov 13, 2026 •

Copy link
Copy Markdown

Summary

  • 把元数据查找和读盘拆开。enable_async_loading 与 split_metadata_lookup 同时打开时,调度器先拿到命中长度;盘读只在 preload 池有连续空块时放行,没有空块就报 miss,让调度器重算,不再把同步读盘算进 TTFT。enable_async_loading 是原有配置,默认关闭。split_metadata_lookup 是这次新增的配置,已写进 _CONFIG_DEFINITIONS,默认关闭。
  • 20GiB 热缓存在已有的 max_local_cpu_size 正好为 20 时拆成 store 8589934592(8GiB)和 preload 12884901888(12GiB)。这次不增加池大小的配置项。D2H 进 store 池,读盘进 preload 池。store 池按可分配块数回收。写盘进已有的 disk_io_threads 线程池。
  • a934ac8 让这条快路径能跑完:一条 LOADING future 只弹出一次。retrieve 开头弹出,成功的 H2D 由 retrieve 释放;lookup_unpin 不再调用 cleanup_memory_objs。过期 epoch 的 cleanup 不释放已经收下的缓冲。它修的是双重释放,不是吞吐乘数。

吞吐

物理机 RTX PRO 6000,FP16 DeepSeek-V2-Lite-Chat,vLLM 0.33.0,LMCache 0.5.7.post1。800 用户、rate 800、在线重叠、max_model_len 8192、max_num_seqs 128、max_num_batched_tokens 8192、KV HBM 12GiB、L1 20GiB。盘是 /dev/sdb2 XFS /mnt/xfs,max_local_disk_size 100GiB。PR 两个开关都开。原生是同一 post1、不覆盖 inprocess、enable_async_loading 为默认 False。六场都是 800/0,Double free 为 0。

综合 tok/s 按轮数分开。原生是这组协议里的 stock LMCache+XFS 基准。

  • 800×2:PR 967.2 = (50801+38471)/92.296,wall_s 92.296,重叠 76.898。原生 388.7 = (50817+38740)/230.430,wall_s 230.430,重叠 216.266。
  • 800×3:PR 827.3 = (50910+38349+34199)/149.234,wall_s 149.234,重叠 94.521 / 92.510。原生 321.2 = (50813+38440+36915)/392.788,wall_s 392.788,重叠 265.413 / 331.169。
  • 800×4:PR 867.1 = (50910+38894+36087+34400)/184.854,wall_s 184.854,重叠 84.442 / 95.536 / 98.268。原生 285.2 = (50801+38906+36527+35442)/566.811,wall_s 566.811,重叠 269.466 / 396.177 / 345.859。

差在 Turn2 及以后的 TTFT,不在单次读盘延迟。同一次 7962624 字节的 O_DIRECT,原生中位 62–74 ms,PR 100–103 ms。原生读得更多:800×2 是 45.1GB 对 4.2GB,800×3 是 95.8GB 对 29.8GB,800×4 是 154.5GB 对 38.2GB。原生 CPU 分配失败(no memory is available,块大小 7962624)分别是 9514、16688、24813;PR 是 0。800 用户各 2048 token 约 51GB,20GiB 池放不下。原生在池满之后仍在 TTFT 上读盘。

文档里的物理机不存是另一口径,不并进上表:vLLM 0.15、rate 32、max_model_len 4096。20261002-120539 综合 971.2,20261002-161229 综合 973.2。这张卡上没有同协议(vLLM 0.33、rate 800、8192、在线重叠)的历史 SurData 跑次。

3090、AWQ、同一份 inprocess、只比 a934ac8 和开关:800×2 是 395.0 对 393.4(+0.4%),800×3 是 435.5 对 427.9(+1.8%)。Double free 为 0。那次原生 800×4 的盘读中位 12.7 ms,其余三场约 2.5 ms,不拿来当加速比。

已知、这次不改

  • abort() 返回 True 时,cleanup_memory_objs 在弹出 epoch 之前就返回。更新的 epoch 已经 _tier_open 时,更老的 DONE future 可能留在槽里。
  • update_event_status / add_event 可以覆盖一个已经 DONE 的槽。
  • retrieve 弹出 future 时不按 epoch 过滤。

Test plan

  • tests/inprocess/v1/test_cache_engine_cleanup.py、test_local_cpu_pre_evict.py、test_tier_pipeline.py 以及读盘准入、部分 GET 的测试在提交前通过
  • 3090 上 a934ac8 的 800×2 / 800×3 / 800×4,Double free 为 0
  • 物理机 RTX PRO 6000 上 PR 与 stock 0.5.7.post1 的 800×2 / 800×3 / 800×4,数字见上
  • CI。.github/workflows/ci.yml 在 main 上,pull_request 会触发;打开本 PR 之后 Actions 里一次运行都没有,ci.yml 也未注册成 workflow。仓库 Actions 开关是开的,历史 run 总数为 0。要让 CI 出现,需要往 main 再推一次,让 GitHub 重新索引 workflow。这次没有改 main。
  • 同一份 stock 二进制只打开 enable_async_loading 的对照臂。这次没跑,所以上面的倍数是整条路径对 stock 默认,不是 a934ac8 单独的加速比

AlexNebula and others added 18 commits November 8, 2026 11:40
The in-tree engine left LMCache, so the waiting-queue lookup and the host-fit GET now belong in this package. The switch stays off by default.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A preemption retires the in-flight lookup and a late ACK is dropped when its epoch does not match. Abort during metadata contains does not start a GET. Only a metadata lookup that has not been acknowledged can time out.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
The completion path passes epoch into send_response_to_scheduler. The storage-manager tests rejected that keyword.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A late body ACK cannot revive an absent id. Cleanup carries the attempt epoch so it cannot abort the next lookup. Queued hits occupy the offer window without becoming misses.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A zero preloaded ACK unblocks a queued request. Worker misses and finished preemption counts are dropped so they do not grow for the life of the process.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
… on shutdown.

A lookup that arrives while the previous GET is reading unpins the new prefix and reports a miss. The scheduler wrapper is removed with the last connector.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A later request with that epoch returns before contains, so it does not pin or queue. A newer epoch is left alone.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A prefix that still does not fit used to rescan on each free and hold the GIL inside store offload. Count released bytes until the smallest waiting prefix can fit, and scan once.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A full store frees about one prefix and allocates those blocks again, so counting gross frees still walked the queue inside offload. Drain only when bytes are still free after allocate returns.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A full prefetch frees about one prefix when retrieve unpins it, and that free does not go through allocate. The net-free check never ran, so queued hits stayed deferred.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A request whose KV is not already in host was left waiting, so the GPU could idle. Load a preloaded prefix as before. Otherwise recompute when a KV block is free, and cancel the lookup so a late ACK cannot revive it.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
A full 20GiB pinned pool was evicting one hot-cache chunk after every
failed allocation, and that wait was counted in offload time. The worker
now drops only the shortfall for the batch, capped at one engine step.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Budget and free space are both whole aligned chunks. Eviction frees what it can; a request that still does not fit is not copied off the GPU.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ool.

A full 20GiB local CPU cache now pins an 8GiB store pool and a 12GiB preload pool. Store admission uses the longest free span, and a request that does not fit is skipped before D2H.

Co-authored-by: Cursor <cursoragent@cursor.com>
Admission was the longest span, so a 7-chunk hole skipped an 8-chunk store while evictable KV stayed in the pool.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
The drain was counting free bytes in the store pool, so a full preload pool still admitted GETs that allocated nothing. Async loads also read one chunk at a time and held those pins.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
wait_for_save and a stale lookup cleanup were both dropping the same LOADING future, so Turn3 H2D asserted on an invalidated MemoryObj.

Signed-off-by: alex <alex@surdata.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
@AlexNebula

Copy link
Copy Markdown
Author

This pull request targeted the fork main. The branch is submitted upstream at LMCache#35.

@AlexNebula AlexNebula closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant