Skip to content

server: preserve live vision cache for Pi replay - #962

Open
softpudding wants to merge 2 commits into
antirez:mainfrom
softpudding:codex/ds4-pi-vision-cache
Open

server: preserve live vision cache for Pi replay#962
softpudding wants to merge 2 commits into
antirez:mainfrom
softpudding:codex/ds4-pi-vision-cache

Conversation

@softpudding

Copy link
Copy Markdown

Summary

Adds a memory-only live-KV checkpoint for multimodal (image) Chat Completions turns so a same-image follow-up request can continue from the sampled session cache instead of re-prefilling from scratch. The checkpoint stores the rendered (visible) assistant transcript as a byte key and appends only the new user suffix to the live frontier.

This tightens remember_multimodal_chat_checkpoint() in ds4_server.c:

  • Keep the raw checkpoint boundary aligned (DeepSeek no-tool thinking path): the visible key drops the pending <think>, but the raw base length still counted it. A same-image follow-up that matched the visible prefix would start tokenizing seven bytes into the next <|User|> tag (possibly mid-UTF-8), corrupting the continuation. The dropped tag length is now subtracted from the raw base length in the same branch.
  • Do not cache unfinished reasoning: the image-specific path was missing the thinking->inside guard used by should_remember_thinking_checkpoint. When a custom stop sequence fires during reasoning (before </think>), it recorded a closed visible assistant turn while the live KV remains inside thinking, so the next same-image request could append the user message inside the unfinished reasoning. The thinking state is now threaded into the checkpoint function and the checkpoint is cleared when it is still inside.

Testing

  • Command: make -B ds4_test -j2 && ./ds4_test --server
  • Machine/backend: Apple M5 Max, macOS Metal backend
  • Result: server: OK, ds4 tests: ok
  • Scope: server-side API / prompt-rendering / KV-cache bookkeeping only; no inference-backend code touched. CUDA/CPU were not exercised (change is confined to ds4_server.c HTTP/session logic).

…king

In the DeepSeek no-tool thinking path of remember_multimodal_chat_checkpoint
we drop the pending <think> from the visible key, but the raw checkpoint
boundary still counted it. A same-image follow-up matched the visible prefix
and then started tokenizing seven bytes into the next <User> tag (possibly
mid-UTF-8), corrupting the continuation. Subtract the dropped tag length from
the raw base length in the same branch.

Also, the image-specific checkpoint path no longer applied the thinking->inside
guard used by should_remember_thinking_checkpoint. When a custom stop sequence
fires during reasoning, before </think>, it recorded a closed visible assistant
turn while the live KV stays inside thinking, so the next same-image request
could append the user message inside unfinished reasoning. Pass the thinking
state through and clear the checkpoint when still inside.

Tested on macOS Metal backend (Apple M5 Max):
  make -B ds4_test -j2 && ./ds4_test --server
which passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant