server: preserve live vision cache for Pi replay - #962
Open
softpudding wants to merge 2 commits into
Open
Conversation
…king In the DeepSeek no-tool thinking path of remember_multimodal_chat_checkpoint we drop the pending <think> from the visible key, but the raw checkpoint boundary still counted it. A same-image follow-up matched the visible prefix and then started tokenizing seven bytes into the next <User> tag (possibly mid-UTF-8), corrupting the continuation. Subtract the dropped tag length from the raw base length in the same branch. Also, the image-specific checkpoint path no longer applied the thinking->inside guard used by should_remember_thinking_checkpoint. When a custom stop sequence fires during reasoning, before </think>, it recorded a closed visible assistant turn while the live KV stays inside thinking, so the next same-image request could append the user message inside unfinished reasoning. Pass the thinking state through and clear the checkpoint when still inside. Tested on macOS Metal backend (Apple M5 Max): make -B ds4_test -j2 && ./ds4_test --server which passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a memory-only live-KV checkpoint for multimodal (image) Chat Completions turns so a same-image follow-up request can continue from the sampled session cache instead of re-prefilling from scratch. The checkpoint stores the rendered (visible) assistant transcript as a byte key and appends only the new user suffix to the live frontier.
This tightens
remember_multimodal_chat_checkpoint()inds4_server.c:<think>, but the raw base length still counted it. A same-image follow-up that matched the visible prefix would start tokenizing seven bytes into the next<|User|>tag (possibly mid-UTF-8), corrupting the continuation. The dropped tag length is now subtracted from the raw base length in the same branch.thinking->insideguard used byshould_remember_thinking_checkpoint. When a custom stop sequence fires during reasoning (before</think>), it recorded a closed visible assistant turn while the live KV remains inside thinking, so the next same-image request could append the user message inside the unfinished reasoning. The thinking state is now threaded into the checkpoint function and the checkpoint is cleared when it is still inside.Testing
make -B ds4_test -j2 && ./ds4_test --serverserver: OK,ds4 tests: okds4_server.cHTTP/session logic).