server: accept inline images on tool/function role messages (fixes #933) - #944
Open
Flor1an-B wants to merge 1 commit into
Open
server: accept inline images on tool/function role messages (fixes #933)#944Flor1an-B wants to merge 1 commit into
Flor1an-B wants to merge 1 commit into
Conversation
) parse_messages rejected any message carrying an inline image whose role wasn't exactly "user", with a generic "invalid JSON request" 400. That breaks the normal OpenAI Chat Completions shape for coding agents that attach a tool-result screenshot as an image_url content part on a role=tool (or role=function) message -- the same data URI on role=user was accepted. As diagnosed in antirez#933: the user-only restriction was meant to keep image tokens landing on a user-shaped turn, but DS4's own chat template already renders tool results inside a user turn (<|User|><tool_result>...), so the role tag at the API layer doesn't need to match that internal shape. Allow user/tool/function; keep rejecting assistant/system roles, file paths, and https URLs exactly as before. Added test_openai_tool_role_image_content (mirrors the existing test_openai_inline_image_content / test_http_image_paths_and_urls_are_rejected pair): asserts user/tool/function all parse and capture the image, and that assistant/system still fail. Verified as a real regression test, not just a happy-path check: reverted the fix locally, confirmed the new test fails at the exact assertions added, then restored it and confirmed green. Verified end-to-end, not just at the parser: built ds4-server with the fix, sent the exact three-message shape from the issue (user text, assistant tool_calls, tool message with a real screenshot as image_url) over HTTP -- got 200, and the model's answer correctly described the screenshot's actual content. Confirmed the two negative cases still return 400: an image on role=assistant, and an https URL on role=tool. Full make test suite green throughout. Not included: the issue's "prefer a specific error string" suggestion is explicitly marked optional there and left out to keep this change to the one behavioral fix it's meant to be.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #933. Credit to @aj-img for the precise diagnosis and suggested fix there — this PR implements exactly what was proposed.
Problem
parse_messagesrejected any OpenAI Chat Completions message carrying an inline image whoserolewasn't exactly"user", always with a generic400 "invalid JSON request". That breaks the normal shape for coding agents that attach a tool-result screenshot as animage_urlcontent part on arole: "tool"(orrole: "function") message — the same data URI onrole: "user"was accepted.Fix
As diagnosed in #933: the user-only restriction was meant to keep image tokens landing on a user-shaped turn, but DS4's own chat template already renders tool results inside a user turn (
<|User|><tool_result>...), so the API-level role tag doesn't need to match that internal shape. One line inparse_messages: allowuser/tool/function, keep rejectingassistant/system, file paths, andhttpsURLs exactly as before (parse_anthropic_messages's identical-looking check is untouched — Anthropic's wire format has no separate tool role;tool_resultblocks always nest inside auser-role message already, so it isn't exposed to this bug).Testing
test_openai_tool_role_image_content, mirroring the existingtest_openai_inline_image_content/test_http_image_paths_and_urls_are_rejectedpair: assertsuser/tool/functionall parse and capture the image, and thatassistant/systemstill fail.ds4-serverwith the fix, sent the exact three-message shape from the issue (usertext →assistantwithtool_calls→toolmessage with a real screenshot asimage_url) over HTTP —200, and the model's answer correctly described the screenshot's actual on-screen content (not a hallucination). Confirmed the two negative cases still400: an image onrole: assistant, and anhttps://URL onrole: tool.make testgreen throughout, onmain@ current head.Scope
The issue's "prefer a specific error string for remaining schema failures" suggestion is explicitly marked optional there and left out, to keep this to the one behavioral fix.