Skip to content

Images without OCR text are never indexed, and ingestion ignores the vision model's AI connection #1583

Closed

Description

@paullizer

Issue

If an uploaded image has no machine-readable text, it never makes it into search. The document ends with status "Processing complete - no text found in image". It has no title, no chunks in AI Search, and nothing chat can retrieve, even when the multimodal vision model described the image successfully.

Two related defects make this worse:

  • Content Understanding only transcribes text from standalone images. It is never asked to describe them.
  • The ingestion vision call ignores the AI connection that the selected vision model belongs to.

Steps to Reproduce

Images without text:

  1. Turn on Enhanced extraction (Content Understanding), Enhanced Citations, and Multi-Modal Vision Analysis.
  2. Upload a photo with no text in it, for example an overexposed grayscale photo of an empty lab floor.
  3. Check the workspace document list and try to chat with the document.

Vision model on an AI connection:

  1. Turn on AI connections (multi-model endpoints) and pick a vision model that is deployed only on a non-default connection, for example gpt-5.6-luna on a second Azure OpenAI resource.
  2. Click Test Vision Analysis in Admin Settings. The test passes.
  3. Upload any image.

Expected Behavior

  • An image with no readable text is still indexed from its description (from the vision model, the Content Understanding image analyzer, or both). Metadata extraction gives it a title, and chat can use it.
  • The chunk embedding is built from the description, so vector search can find the image.
  • Ingestion calls the vision model through the same AI connection the admin selected and tested.

Actual Behavior

  • Content Understanding prebuilt-documentSearch returns 0 pages for a photo with no text. Ingestion falls back to Document Intelligence Layout, which also finds no words.
  • process_di_document builds image chunks only from OCR pages, so no chunks are saved. The vision analysis is stored on the Cosmos record but never reaches AI Search. Final metadata extraction is skipped because no chunks were saved.
  • A slightly less overexposed copy of the same photo "works" only by accident. DI Layout misreads a strip of floor tape as a page number (J, confidence 0.18) and emits <!-- PageNumber="J" -->. That bogus token creates a chunk, and the vision text is appended to it. Even then, the embedding comes from the OCR text alone (7 tokens), not from the appended description.
  • analyze_image_with_vision_model always calls the legacy default GPT endpoint or APIM. When the vision model lives on a different AI connection, the call returns HTTP 404 and vision analysis is skipped. The admin Test Vision Analysis button resolves the connection correctly, so the test passes anyway.
  • The configured Content Understanding image analyzer (prebuilt-imageSearch) describes both test photos well. It is only used for images embedded in Office files, though, and analyze_image_with_content_understanding ignores its Summary field and returns only the ![image](pages/1) placeholder.

Impact

Photos, unlabeled diagrams, and other images without text are invisible to search and chat. Admins who pick a vision model from an AI connection lose vision analysis on every image upload, in both workspaces and chat, and no error is shown.

Notes

Evidence from the simplechat-20260506 App Insights logs (2026-09-30, UTC):

  • 15:05:26, 2026-09-30_10-54-17.png (102,092 bytes): CU returned 0 pages, DI Layout returned nothing, and vision analysis with gpt-5.4 succeeded (200). The result was 0 chunks and "Processing complete - no text found in image".
  • 15:06:31, 2026-09-30_10-43-52.png (116,863 bytes): CU returned 0 pages and DI Layout returned <!-- PageNumber="J" -->. That produced 1 chunk with the vision text appended, a 7-token embedding, and an extracted title.
  • 15:09:11, 22222.png (102,092 bytes): the vision model was gpt-5.6-luna. The POST to simplechat-20260506-openai/.../deployments/gpt-5.6-luna returned 404 because the model is deployed on the Global connection (aoai-global-team).

Suspected areas: process_di_document, save_chunks and save_chunks_batch, and analyze_image_with_vision_model in functions_documents.py; extract_content_with_extraction_engine in functions_content.py; analyze_image_with_content_understanding in functions_content_understanding.py.

The fix lands in Development first and then in paullizer-react-v2-ui. Related: #1042 (models missing from the vision model selector).

Activity

  1. added
    bugSomething isn't working
    pythonPull requests that update python code
    P1Priority 1: urgent, address before P2 work
    on Sep 30, 2026
  2. paullizer commented on Sep 30, 2026

    @paullizer
    ContributorAuthor

    Status update

    Root cause: Neither photo has any real text. The less overexposed photo only indexed because Document Intelligence Layout read a strip of floor tape as a page number (<!-- PageNumber="J" -->, confidence 0.18), which gave the vision description a chunk to attach to. With no OCR text, the overexposed photo got no chunk. Its successful vision analysis stayed on the Cosmos record, and metadata extraction was skipped. Two other things made it worse:

    • Content Understanding sent standalone photos to the document analyzer, which returns nothing for them, and ignored the image analyzer's Summary.
    • The vision call always used the primary GPT endpoint, so gpt-5.6-luna on the Global connection returned 404.

    Fixes

    • Index images without OCR text and route vision analysis through its AI connection #1584, merged into Development (0.261.047):
      • Text-free images are indexed from their vision description.
      • Chunks are embedded from the stored text.
      • Standalone images fall back to the Content Understanding image analyzer.
      • The vision model is called through the AI connection that hosts it, including in Test Vision Analysis.
      • The image generation API version defaults to 2025-04-01-preview, and a saved old default or blank value is upgraded.
    • Show what was extracted from V2 image uploads, index text-free images, and default image generation to 2025-04-01-preview #1585, open against paullizer-react-v2-ui (0.261.210):
      • The same fixes, adjusted for V2's capability checks and embedding limits. Imported image connection profiles are upgraded too, and APIM versions are untouched.
      • V2 chat: uploads show what was extracted instead of "File content not found", and cited images show the extracted passage.
      • The uploaded image card now matches generated images: it is sized to the image, its actions appear on hover, and Show sources shows the extraction.

    Validation: new and folded functional tests pass, including real-module offline probes in normal and -O runs. The V2 Playwright tests for upload cards, the Sources panel, the file preview, processing and failed states, and cited images pass. Route policy, docs, and the BAC, XSS and swagger guardrails pass. The remaining failures also fail on the base commit.

    Follow-ups:

    • After deploying, re-upload or reprocess the overexposed image so it gets indexed.
    • The classic admin page fills its vision options from deploymentName, so Custom OpenAI-style models without one won't resolve.
    • The chat route's own vision block still includes error analyses.

    This issue will be closed when #1585 merges.

  3. paullizer commented on Sep 30, 2026

    @paullizer
    ContributorAuthor

    Fixed by #1584 (Development, 0.261.047) and #1585 (React V2, 0.261.210), both merged. After deploying, re-upload or reprocess images that were ingested before the fix so they get indexed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P1Priority 1: urgent, address before P2 workbugSomething isn't workingpythonPull requests that update python code

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions