server: separate resident sessions from active requests - #930
server: separate resident sessions from active requests#930JordiPosthumus wants to merge 2 commits into
Conversation
I think it's currently a bit more nuanced (or, rather, a bit more broken :). Even with --batched-session 10 the in-memory KV cache is only kept for 1 slot, if there is 1 request at a time (no parallel requests), due to broken logic of "which KV cache slot to use". On-disk KV cache can somewhat mask the issue, but it's unnecesary SSD trashing. See #765. |
ba7b962 to
998fc19
Compare
|
Agreed — slot selection is a separate upstream problem, and the earlier description overstated what this PR guarantees. #930 limits admission only; it does not fix the routing behavior discussed in #765. I have updated the code documentation and PR description accordingly. With current upstream routing, |
--batched-sessioncurrently controls both resident KV allocation and thenumber of requests admitted to inference. This patch adds
--max-active-requests Nso those limits can be set independently.For example:
allocates ten resident slots while allowing one assigned or running request.
Additional requests remain queued until an active request completes or is
cancelled. Omitting the option preserves current behavior by setting the
active limit equal to the resident-session count. The option requires batched
mode, must be positive, and cannot exceed the resident-session count.
This is admission control only. It does not change which slot the existing
router selects. As discussed in #765, current upstream routing may reuse one
slot for unrelated sequential conversations even when other slots exist.
Consequently, this option alone does not guarantee that ten sequential
conversations remain warm; reuse-aware slot selection is complementary work.
The scheduler regression covers the default limit, a limit of one, worker
claim, completion, cancellation, and raising the limit to two. CLI rejection
tests cover invalid combinations. Forced Metal and CPU builds,
ds4_test --server,ds4_agent_test, the DeepSeek vision image test, theserver suite under ASan+UBSan, and focused DeepSeek-V4-Flash-Vision-Exp checks
pass on Apple M3 Ultra.
Current-main recheck — 2026-09-03
Applied independently to main
b0a147a7fba6d1a104d047d5a140e9bb4bfc13cdin a cleanworktree. The default build,
ds4_test --server, andds4_agent_testpass.The admission/routing distinction above is unchanged; the ten-conversation
integration result is not being attributed to this PR alone.