Skip to content

chatgpt provider strips all image input (attachments and tool-result images); capabilities: {image: true} override has no effect #4521

Description

@cat-yyy

Versions: docker-agent v1.142.0 and v1.148.0 (linux/amd64, inside a Docker Sandbox), config version 16, provider chatgpt (ChatGPT subscription login), model gpt-6.1-sol.

What happens

Images never reach the model when the agent uses the chatgpt provider:

  • docker-agent run agent.yaml --exec --attach screenshot.png "Describe the image" → the model answers that no image is attached.
  • Images returned by MCP tools (e.g. a screenshot tool whose result carries images: [{data: <base64 png>}]) are also dropped: the next request's new input is only ~230 tokens.

The model itself accepts images: models.dev lists openai/gpt-6.1-sol with input modalities text, image, pdf, and the same account and model see images through Codex CLI (codex exec -i).

Minimal config

version: '16'
agents:
  root:
    model: sol
    instruction: Describe the attached image.
models:
  sol:
    provider: chatgpt
    model: gpt-6.1-sol
    capabilities:
      image: true

Debug log (--debug), same with and without the capabilities block:

level=DEBUG msg="Failed to resolve model capabilities for message transforms" model=chatgpt/gpt-6.1-sol error="provider \"chatgpt\" not found"
level=DEBUG msg="strip_unsupported_modalities: stripped media part" kind=image role=user reason="model does not support image input"
level=DEBUG msg="Stripped media content from message" role=user original_parts=2 remaining_parts=1

Analysis (from reading the source)

  • LocalRuntime.prepareMessagesForModel (pkg/runtime/transforms.go) looks the model up in models.dev as chatgpt/gpt-6.1-sol. models.dev has no chatgpt provider, so catalogModel is nil.
  • It then uses cfg.CapsOverride() (pkg/model/provider/contracts), which reads ModelConfig.Capabilities. Because the image is still stripped with capabilities: {image: true} set, the override appears to be nil for the chatgpt provider by the time the runtime sees it. We did not find where it is lost.
  • With no catalogue entry and no override, providerFallbackCaps returns empty capabilities, and strip_unsupported_modalities removes every image part before convertDocumentToResponseInput (which would have produced a correct input_image) is reached.

Expected

  • The chatgpt provider resolves capabilities from the openai models.dev entry (it is the same model family on the Responses API), and/or
  • an explicit capabilities: override is honoured for the chatgpt provider, as the warning text in modelinfo.warnCapsLookupMiss suggests.

Activity

  1. added
    area/providers/openaiFor features/issues/fixes related to the usage of OpenAI models
    area/runtimeRuntime engine, agent loop execution, tool dispatch, loop detection
    area/modelsLLM model integrations and model providers
    on Oct 5, 2026
  2. added theissue type on Oct 5, 2026
  3. self-assigned this
    on Oct 6, 2026
  4. aheritier commented on Oct 6, 2026

    @aheritier
    Collaborator

    I can look at this one. Last time I worked on images support I focuses more on gemini than others models

  5. aheritier commented on Oct 6, 2026

    @aheritier
    Collaborator

    The ChatGPT capability lookup had no chatgpt catalogue entry to use, so known image-capable models fell back to text-only and image attachments—including tool-result images—were stripped. PR #4528 (#4528) adds an image-input-only fallback to the matching OpenAI catalogue entry; the sample succeeded in a live authenticated run without a capabilities.image override. The reported failure with an explicit image: true override was not reproduced, so please retest and share details if it persists. Other provider catalogue aliases and the OVH model-case mismatch will be handled separately.

  6. aheritier commented on Oct 7, 2026

    @aheritier
    Collaborator

    fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/modelsLLM model integrations and model providersarea/providers/openaiFor features/issues/fixes related to the usage of OpenAI modelsarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detection

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions