Skip to content

fix(chatgpt): infer image input from OpenAI catalogue - #4528

Merged
aheritier merged 1 commit into
mainfrom
fix/4521-chatgpt-image-input
Oct 6, 2026
Merged

aheritier merged 1 commit into
mainfrom
fix/4521-chatgpt-image-input

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

ChatGPT models use the OpenAI-compatible Codex backend, but the models.dev catalogue has no chatgpt provider entries. As a result, the capability lookup treated known image-capable ChatGPT models as text-only and stripped image attachments—including images returned by tools—before sending the request.

Fix

When a ChatGPT model has no direct catalogue entry, use the matching OpenAI entry to infer image input only. Direct ChatGPT entries and explicit capability overrides retain precedence; unknown models remain text-only. This does not inherit PDF, audio, video, pricing, context limits, tool support, output capabilities, or max_tokens.

Regression coverage checks lookup precedence and misses, filtering, and backend serialization for both user attachments and tool-result images. A live authenticated run of examples/chatgpt-images.yaml sent the attached PNG and returned: “A red square is centered on a bright blue square background.” The sample needs no image: true override.

The issue also reported that an explicit image: true override had no effect; that symptom was not reproduced in testing, so it remains unresolved here. Related catalogue alias gaps for fireworks, together, moonshot, and opencode-zen, and an OVH model-case mismatch, were found in a read-only audit and are intentionally out of scope for this PR; no live tests were run for those providers. They should be addressed separately.

Validation

  • Build, targeted tests, sample/docs validation, Hugo, and new-code lint passed.
  • Full tests remain blocked by a pre-existing ACP usage-cost failure, reproduced unchanged at HEAD.
  • Full lint remains blocked by three findings in untouched files.

Related to #4521.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:26
@aheritier
aheritier requested a review from a team as a code owner October 6, 2026 13:26
@aheritier aheritier changed the title ## Problem fix(chatgpt): infer image input from OpenAI catalogue Oct 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The fallback is narrowly scoped, preserves capability precedence, and has focused regression coverage with no identified blocking issues.

Review effort: Balanced
Findings: None

What changed in this PR

Restores image input for known ChatGPT models by using matching OpenAI catalogue entries when direct entries are absent.

Changes:

  • Adds an image-only fallback while preserving direct entries and explicit overrides.
  • Tests capability precedence, attachment filtering, and backend serialization.
  • Documents the behavior and adds a runnable example.
File Description
pkg/​runtime/​transforms.go Applies the fallback before filtering messages.
pkg/​runtime/​transforms_test.go Tests image retention and capability precedence.
pkg/​modelinfo/​resolve_caps_test.go Tests fallback scope, precedence, and misses.
pkg/​modelinfo/​modelinfo.go Integrates fallback into capability loading.
pkg/​modelinfo/​catalog_caps.go Implements image-only OpenAI catalogue lookup.
pkg/​model/​provider/​openai/​chatgpt_test.go Tests attachment and tool-image serialization.
pkg/​model/​provider/​capability_override_test.go Verifies ChatGPT overrides survive construction.
examples/​README.md Lists the image-input example.
examples/​chatgpt-images.yaml Adds an example without capability overrides.
docs/​providers/​chatgpt/​index.md Explains image support and fallback limits.
docs/​configuration/​models/​index.md Clarifies fallback and override behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@aheritier aheritier added area/docs Documentation changes area/models LLM model integrations and model providers area/providers/openai For features/issues/fixes related to the usage of OpenAI models area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection kind/fix PR fixes a bug (maps to fix:). Use on PRs only. labels Oct 6, 2026
@aheritier
aheritier added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit 5f7f048 Oct 6, 2026
23 checks passed
@aheritier
aheritier deleted the fix/4521-chatgpt-image-input branch October 6, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes area/models LLM model integrations and model providers area/providers/openai For features/issues/fixes related to the usage of OpenAI models area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants