Skip to content

ds4-server: make OpenAI-compatible image input work end to end - #950

Open
Bendr0id wants to merge 1 commit into
antirez:mainfrom
Bendr0id:fix-openai-image-data-uri
Open

ds4-server: make OpenAI-compatible image input work end to end#950
Bendr0id wants to merge 1 commit into
antirez:mainfrom
Bendr0id:fix-openai-image-data-uri

Conversation

@Bendr0id

@Bendr0id Bendr0id commented Sep 2, 2026

Copy link
Copy Markdown

Analyzing an image over the OpenAI chat completions API failed even though /read in the agent handled the same file, and every failure surfaced as a generic "invalid JSON request". Three independent causes, each sufficient on its own to break it:

The data URI parser matched a fixed set of literal prefixes and the base64 decoder accepted only canonical input, so payloads real clients emit were refused: varying scheme and media-type case, parameters such as ";charset=" before ";base64", MIME line breaks, the URL-safe alphabet and omitted padding. Rejections now explain themselves instead of collapsing into one opaque 400, and an untyped content block carrying an image_url is treated as an image rather than silently dropped, which had answered the prompt from the text alone as if no image had been sent.

/v1/models never advertised image input, so clients that gate on modality stripped the image before it left the client. Model metadata now carries architecture.input_modalities when an encoder is loaded. Clients also commonly suppress image input for anything they classify as DeepSeek, because the upstream API is text-only, and re-enable it only when the model id carries a "vision" token, so deepseek-v4-flash-vision and deepseek-v4-pro-vision are offered too. Both are reported only when started with --vision, including the per-model endpoint.

An image the server cannot decode is replayed in the transcript on every later turn, so a single WebP wedged a conversation permanently: every request failed regardless of what was actually being asked. An undecodable image now becomes a text note in the history and only the newest message rejects the request, since that is the one the caller just attached and needs to be told about. Dropping an image also removes the marker that referenced it, which rendering would otherwise emit into the prompt as literal control-character junk.

Analyzing an image over the OpenAI chat completions API failed even though
/read in the agent handled the same file, and every failure surfaced as a
generic "invalid JSON request". Three independent causes, each sufficient on
its own to break it:

The data URI parser matched a fixed set of literal prefixes and the base64
decoder accepted only canonical input, so payloads real clients emit were
refused: varying scheme and media-type case, parameters such as ";charset="
before ";base64", MIME line breaks, the URL-safe alphabet and omitted
padding. Rejections now explain themselves instead of collapsing into one
opaque 400, and an untyped content block carrying an image_url is treated as
an image rather than silently dropped, which had answered the prompt from
the text alone as if no image had been sent.

/v1/models never advertised image input, so clients that gate on modality
stripped the image before it left the client. Model metadata now carries
architecture.input_modalities when an encoder is loaded. Clients also
commonly suppress image input for anything they classify as DeepSeek,
because the upstream API is text-only, and re-enable it only when the model
id carries a "vision" token, so deepseek-v4-flash-vision and
deepseek-v4-pro-vision are offered too. Both are reported only when started
with --vision, including the per-model endpoint.

An image the server cannot decode is replayed in the transcript on every
later turn, so a single WebP wedged a conversation permanently: every
request failed regardless of what was actually being asked. An undecodable
image now becomes a text note in the history and only the newest message
rejects the request, since that is the one the caller just attached and
needs to be told about. Dropping an image also removes the marker that
referenced it, which rendering would otherwise emit into the prompt as
literal control-character junk.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant