Skip to content

docs: stateless chat client live KV warm recipe #956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
paride-cmd wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: stateless chat client live KV warm recipe #956

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1509,6 +1509,32 @@ re-processing if it was written to the disk KV cache. In other words, memory
cache handles the active session; disk cache is the resume mechanism for
different sessions.

### Keeping the live session warm from a stateless chat client

After a generation finishes, the live slot holds `prompt + sampled output`. A
typical chat client next sends `prompt + new user turns` and never includes that
exact assistant text, so the token-prefix check fails (`token-mismatch`) and
every request falls through to disk. On disk, cold checkpoints are trimmed to
the last user turn before the first assistant turn (`kv_cache_chat_anchor_pos`);
for a client that cannot replay, that anchor is usually the only prefix it can
hit.

To extend the live session instead:

1. Replay the previous assistant turn byte-for-byte in the next request
(plain `content`, or `tool_calls` plus the matching `tool` result). Exact
DSML tool-id replay makes tool histories match the sampled text too.
2. To create or re-warm a session after a slot steal/eviction, send the
conversation with `max_tokens: 0` (prefill only, no sampled token). Do not
use `max_tokens: 1`: the one sampled token is appended to the live session
and the next prompt misses it.
3. Witness: `usage.prompt_tokens_details.cached_tokens` should be about prior
prompt + completion on a warm follow-up (or the full prompt after a
`max_tokens: 0` prewarm).

Prefer plain-text structured output over a short tool call when no tool result
is needed: DSML scaffolding adds tokens to every warm extension.

Enable it with:

```sh
Expand Down Expand Up @@ -1646,6 +1672,9 @@ builds for this model layout.
The cache stores checkpoints at four moments:

- `cold`: after a long first prompt reaches a stable prefix, before generation.
For chat prompts this is trimmed to the last user turn before the first
assistant turn (`kv_cache_chat_anchor_pos`), so a client that cannot replay
assistant text can still resume from that shared prefix on disk.
- `continued`: when prefill or generation reaches the next absolute aligned frontier.
- `evict`: before an unrelated request replaces the live in-memory session.
- `shutdown`: when the server exits cleanly.
Expand Down