ds4-server: make OpenAI-compatible image input work end to end - #950
Open
Bendr0id wants to merge 1 commit into
Open
ds4-server: make OpenAI-compatible image input work end to end#950Bendr0id wants to merge 1 commit into
Bendr0id wants to merge 1 commit into
Conversation
Analyzing an image over the OpenAI chat completions API failed even though /read in the agent handled the same file, and every failure surfaced as a generic "invalid JSON request". Three independent causes, each sufficient on its own to break it: The data URI parser matched a fixed set of literal prefixes and the base64 decoder accepted only canonical input, so payloads real clients emit were refused: varying scheme and media-type case, parameters such as ";charset=" before ";base64", MIME line breaks, the URL-safe alphabet and omitted padding. Rejections now explain themselves instead of collapsing into one opaque 400, and an untyped content block carrying an image_url is treated as an image rather than silently dropped, which had answered the prompt from the text alone as if no image had been sent. /v1/models never advertised image input, so clients that gate on modality stripped the image before it left the client. Model metadata now carries architecture.input_modalities when an encoder is loaded. Clients also commonly suppress image input for anything they classify as DeepSeek, because the upstream API is text-only, and re-enable it only when the model id carries a "vision" token, so deepseek-v4-flash-vision and deepseek-v4-pro-vision are offered too. Both are reported only when started with --vision, including the per-model endpoint. An image the server cannot decode is replayed in the transcript on every later turn, so a single WebP wedged a conversation permanently: every request failed regardless of what was actually being asked. An undecodable image now becomes a text note in the history and only the newest message rejects the request, since that is the one the caller just attached and needs to be told about. Dropping an image also removes the marker that referenced it, which rendering would otherwise emit into the prompt as literal control-character junk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Analyzing an image over the OpenAI chat completions API failed even though /read in the agent handled the same file, and every failure surfaced as a generic "invalid JSON request". Three independent causes, each sufficient on its own to break it:
The data URI parser matched a fixed set of literal prefixes and the base64 decoder accepted only canonical input, so payloads real clients emit were refused: varying scheme and media-type case, parameters such as ";charset=" before ";base64", MIME line breaks, the URL-safe alphabet and omitted padding. Rejections now explain themselves instead of collapsing into one opaque 400, and an untyped content block carrying an image_url is treated as an image rather than silently dropped, which had answered the prompt from the text alone as if no image had been sent.
/v1/models never advertised image input, so clients that gate on modality stripped the image before it left the client. Model metadata now carries architecture.input_modalities when an encoder is loaded. Clients also commonly suppress image input for anything they classify as DeepSeek, because the upstream API is text-only, and re-enable it only when the model id carries a "vision" token, so deepseek-v4-flash-vision and deepseek-v4-pro-vision are offered too. Both are reported only when started with --vision, including the per-model endpoint.
An image the server cannot decode is replayed in the transcript on every later turn, so a single WebP wedged a conversation permanently: every request failed regardless of what was actually being asked. An undecodable image now becomes a text note in the history and only the newest message rejects the request, since that is the one the caller just attached and needs to be told about. Dropping an image also removes the marker that referenced it, which rendering would otherwise emit into the prompt as literal control-character junk.