Skip to content

server: accept inline images on tool/function role messages (fixes #933) - #944

Open
Flor1an-B wants to merge 1 commit into
antirez:mainfrom
Flor1an-B:fix-server-tool-role-image-messages
Open

server: accept inline images on tool/function role messages (fixes #933)#944
Flor1an-B wants to merge 1 commit into
antirez:mainfrom
Flor1an-B:fix-server-tool-role-image-messages

Conversation

@Flor1an-B

Copy link
Copy Markdown

Fixes #933. Credit to @aj-img for the precise diagnosis and suggested fix there — this PR implements exactly what was proposed.

Problem

parse_messages rejected any OpenAI Chat Completions message carrying an inline image whose role wasn't exactly "user", always with a generic 400 "invalid JSON request". That breaks the normal shape for coding agents that attach a tool-result screenshot as an image_url content part on a role: "tool" (or role: "function") message — the same data URI on role: "user" was accepted.

Fix

As diagnosed in #933: the user-only restriction was meant to keep image tokens landing on a user-shaped turn, but DS4's own chat template already renders tool results inside a user turn (<|User|><tool_result>...), so the API-level role tag doesn't need to match that internal shape. One line in parse_messages: allow user/tool/function, keep rejecting assistant/system, file paths, and https URLs exactly as before (parse_anthropic_messages's identical-looking check is untouched — Anthropic's wire format has no separate tool role; tool_result blocks always nest inside a user-role message already, so it isn't exposed to this bug).

Testing

  • Added test_openai_tool_role_image_content, mirroring the existing test_openai_inline_image_content / test_http_image_paths_and_urls_are_rejected pair: asserts user/tool/function all parse and capture the image, and that assistant/system still fail.
  • Verified it's a real regression test, not a happy-path-only check: reverted the fix locally, confirmed the new test fails at the exact new assertions, restored it, confirmed green.
  • Verified end-to-end past the parser: built ds4-server with the fix, sent the exact three-message shape from the issue (user text → assistant with tool_callstool message with a real screenshot as image_url) over HTTP — 200, and the model's answer correctly described the screenshot's actual on-screen content (not a hallucination). Confirmed the two negative cases still 400: an image on role: assistant, and an https:// URL on role: tool.
  • Full make test green throughout, on main @ current head.

Scope

The issue's "prefer a specific error string for remaining schema failures" suggestion is explicitly marked optional there and left out, to keep this to the one behavioral fix.

)

parse_messages rejected any message carrying an inline image whose role
wasn't exactly "user", with a generic "invalid JSON request" 400. That
breaks the normal OpenAI Chat Completions shape for coding agents that
attach a tool-result screenshot as an image_url content part on a
role=tool (or role=function) message -- the same data URI on role=user
was accepted.

As diagnosed in antirez#933: the user-only restriction was meant to keep image
tokens landing on a user-shaped turn, but DS4's own chat template
already renders tool results inside a user turn
(<|User|><tool_result>...), so the role tag at the API layer doesn't
need to match that internal shape. Allow user/tool/function; keep
rejecting assistant/system roles, file paths, and https URLs exactly as
before.

Added test_openai_tool_role_image_content (mirrors the existing
test_openai_inline_image_content / test_http_image_paths_and_urls_are_rejected
pair): asserts user/tool/function all parse and capture the image, and
that assistant/system still fail. Verified as a real regression test,
not just a happy-path check: reverted the fix locally, confirmed the
new test fails at the exact assertions added, then restored it and
confirmed green.

Verified end-to-end, not just at the parser: built ds4-server with the
fix, sent the exact three-message shape from the issue (user text,
assistant tool_calls, tool message with a real screenshot as
image_url) over HTTP -- got 200, and the model's answer correctly
described the screenshot's actual content. Confirmed the two negative
cases still return 400: an image on role=assistant, and an https URL
on role=tool. Full make test suite green throughout.

Not included: the issue's "prefer a specific error string" suggestion
is explicitly marked optional there and left out to keep this change to
the one behavioral fix it's meant to be.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ds4-server 400s valid OpenAI tool-role image messages as invalid JSON

1 participant