server : support input_image in Responses API tool call output

[Fino-wind]

Overview

The Responses API converter rejects any tool call output whose type is not input_text, so a client that returns an image from a tool always gets:

400 {"error":{"message":"Output of tool call should be 'Input text'","type":"invalid_request_error"}}

This is inconsistent with input messages: input_image is already accepted a hundred lines above in the same file. A model with a working vision encoder cannot see an image simply because it arrived as the result of a tool call rather than in a user turn.

This is reachable with any Responses client that has a screenshot / image-returning tool — I hit it with Codex driving a local Qwen3.8-27B (vision) via --mmproj, where every screenshot step failed.

Changes

Accept input_image in function_call_output as well, mirroring the existing branch for input messages.

Chat templates cannot render media inside a tool message, so the image is emitted as a following user message rather than placed in the tool message itself. This keeps it at the same position in the conversation and lets the existing multimodal path consume it unchanged. When the tool output contains only images, a short placeholder text is kept in the tool message so the tool result is not empty.

output_type is now read with json_value(...), so a malformed entry without type produces the regular error path instead of relying on contains().

Unknown output types still throw; the message is updated to mention both accepted types.

Testing

Built with CUDA and run against Huihui-Qwen3.8-27B-abliterated-Q8_0 + mmproj-model-bf16.gguf. Same request, same model, two servers:

server	result
master (d7bd3bf)	HTTP 400 Output of tool call should be 'Input text'
this branch	HTTP 200 — model replied "这是宝可梦游戏画面" ("this is a Pokémon game screen"), 211 → 6 tokens

The request carried a real 50 KB JPEG screenshot as input_image inside function_call_output. The reply describes the actual picture content, so the image reaches the vision encoder rather than merely passing validation.

Text-only tool outputs are unaffected: the input_text path is the same conversion as before (type rewritten to text), and the string form of output is untouched.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — the patch and the test runs were produced with Claude Code under my direction, and I reviewed the change before submitting.

[ggml-gh-bot]

Hi @Fino-wind, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[inf9144]

I don't think the statement that chat templates cannot render media originating from a tool message holds in general.

We have a concrete counterexample in our Qwen3.8/Codex setup: input_image is kept as part of the tool result in llama.cpp's Responses -> Chat Completions representation, and we have tested that path end-to-end with vision.

In our Responses converter, input_image is normalized directly to image_url without changing the message role:

https://github.com/inf9144/llama.cpp/blob/responses-function-call-output/tools/server/server-chat.cpp#L46-L74

The resulting function_call_output remains role: "tool":

https://github.com/inf9144/llama.cpp/blob/responses-function-call-output/tools/server/server-chat.cpp#L224-L242

That is also what the current branch diff implements.

On the template side, the companion Qwen3.8/Codex template accepts both image and image_url in its general content renderer and emits the normal vision tokens:

<|vision_start|><|image_pad|><|vision_end|>

https://github.com/inf9144/qwen-codex-local/blob/main/templates/llama-cpp-qwen3.8-codex.jinja#L176-L223

The important part is that:

render_content(message.content, true)

is executed for every message before role-specific handling:

https://github.com/inf9144/qwen-codex-local/blob/main/templates/llama-cpp-qwen3.8-codex.jinja#L359-L362

The template then handles role == "tool" and wraps the already-rendered content in <tool_response>...</tool_response>:

https://github.com/inf9144/qwen-codex-local/blob/main/templates/llama-cpp-qwen3.8-codex.jinja#L547-L567

There is an important distinction here: Qwen's template ultimately projects tool responses into its own user-side ChatML framing. So I am not claiming that the final serialized prompt literally contains a native tool role token.

The point is that this projection happens in the model-specific template, while the Responses -> Chat Completions conversion preserves the tool association of the content.

That separation seems architecturally cleaner to me.

The image can remain semantically where it originated: in the tool result. The protocol converter does not need to turn part of that result into an additional standalone user message merely because a particular model uses user-side framing for tool responses. The template already owns the model-specific decision of how role: "tool" maps onto the model's actual prompt format.

That is also the relevant difference compared with the approach in #27958:

• the converter there changes the intermediate message list by extracting the media from the tool result and creating a separate user message;
• our converter preserves the tool association;
• only afterwards does the model-specific template decide how that tool message should be serialized for Qwen.

To be clear, this does not show that all existing llama.cpp chat templates already support multimodal tool content. Some may not.

It does show that multimodal content originating from a tool result is technically workable without requiring the generic protocol conversion layer to rewrite that content into a separate user turn.

For that reason, I would prefer keeping input_image associated with the tool result during Responses -> Chat Completions conversion and keeping the resulting function_call_output as role: "tool". Templates that support multimodal tool results can then render and project that content into whatever role/framing their model requires.

In other words: the protocol converter should preserve the semantic message structure and tool association; the model-dependent projection belongs in the template.