Repository navigation
Images without OCR text are never indexed, and ingestion ignores the vision model's AI connection #1583
Description
Issue
If an uploaded image has no machine-readable text, it never makes it into search. The document ends with status "Processing complete - no text found in image". It has no title, no chunks in AI Search, and nothing chat can retrieve, even when the multimodal vision model described the image successfully.
Two related defects make this worse:
- Content Understanding only transcribes text from standalone images. It is never asked to describe them.
- The ingestion vision call ignores the AI connection that the selected vision model belongs to.
Steps to Reproduce
Images without text:
- Turn on Enhanced extraction (Content Understanding), Enhanced Citations, and Multi-Modal Vision Analysis.
- Upload a photo with no text in it, for example an overexposed grayscale photo of an empty lab floor.
- Check the workspace document list and try to chat with the document.
Vision model on an AI connection:
- Turn on AI connections (multi-model endpoints) and pick a vision model that is deployed only on a non-default connection, for example
gpt-5.6-lunaon a second Azure OpenAI resource. - Click Test Vision Analysis in Admin Settings. The test passes.
- Upload any image.
Expected Behavior
- An image with no readable text is still indexed from its description (from the vision model, the Content Understanding image analyzer, or both). Metadata extraction gives it a title, and chat can use it.
- The chunk embedding is built from the description, so vector search can find the image.
- Ingestion calls the vision model through the same AI connection the admin selected and tested.
Actual Behavior
- Content Understanding
prebuilt-documentSearchreturns 0 pages for a photo with no text. Ingestion falls back to Document Intelligence Layout, which also finds no words. process_di_documentbuilds image chunks only from OCR pages, so no chunks are saved. The vision analysis is stored on the Cosmos record but never reaches AI Search. Final metadata extraction is skipped because no chunks were saved.- A slightly less overexposed copy of the same photo "works" only by accident. DI Layout misreads a strip of floor tape as a page number (
J, confidence 0.18) and emits<!-- PageNumber="J" -->. That bogus token creates a chunk, and the vision text is appended to it. Even then, the embedding comes from the OCR text alone (7 tokens), not from the appended description. analyze_image_with_vision_modelalways calls the legacy default GPT endpoint or APIM. When the vision model lives on a different AI connection, the call returns HTTP 404 and vision analysis is skipped. The admin Test Vision Analysis button resolves the connection correctly, so the test passes anyway.- The configured Content Understanding image analyzer (
prebuilt-imageSearch) describes both test photos well. It is only used for images embedded in Office files, though, andanalyze_image_with_content_understandingignores itsSummaryfield and returns only theplaceholder.
Impact
Photos, unlabeled diagrams, and other images without text are invisible to search and chat. Admins who pick a vision model from an AI connection lose vision analysis on every image upload, in both workspaces and chat, and no error is shown.
Notes
Evidence from the simplechat-20260506 App Insights logs (2026-09-30, UTC):
- 15:05:26,
2026-09-30_10-54-17.png(102,092 bytes): CU returned 0 pages, DI Layout returned nothing, and vision analysis withgpt-5.4succeeded (200). The result was 0 chunks and "Processing complete - no text found in image". - 15:06:31,
2026-09-30_10-43-52.png(116,863 bytes): CU returned 0 pages and DI Layout returned<!-- PageNumber="J" -->. That produced 1 chunk with the vision text appended, a 7-token embedding, and an extracted title. - 15:09:11,
22222.png(102,092 bytes): the vision model wasgpt-5.6-luna. The POST tosimplechat-20260506-openai/.../deployments/gpt-5.6-lunareturned 404 because the model is deployed on the Global connection (aoai-global-team).
Suspected areas: process_di_document, save_chunks and save_chunks_batch, and analyze_image_with_vision_model in functions_documents.py; extract_content_with_extraction_engine in functions_content.py; analyze_image_with_content_understanding in functions_content_understanding.py.
The fix lands in Development first and then in paullizer-react-v2-ui. Related: #1042 (models missing from the vision model selector).
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsReleased
Activity
Address CodeQL findings on the image description indexing tests
Index images without OCR text and route vision analysis through its A…
Bump version to 0.261.047 and add release notes
paullizer commentedon Sep 30, 2026
Status update
Root cause: Neither photo has any real text. The less overexposed photo only indexed because Document Intelligence Layout read a strip of floor tape as a page number (
<!-- PageNumber="J" -->, confidence 0.18), which gave the vision description a chunk to attach to. With no OCR text, the overexposed photo got no chunk. Its successful vision analysis stayed on the Cosmos record, and metadata extraction was skipped. Two other things made it worse:Summary.gpt-5.6-lunaon the Global connection returned 404.Fixes
Development(0.261.047):2025-04-01-preview, and a saved old default or blank value is upgraded.paullizer-react-v2-ui(0.261.210):Validation: new and folded functional tests pass, including real-module offline probes in normal and
-Oruns. The V2 Playwright tests for upload cards, the Sources panel, the file preview, processing and failed states, and cited images pass. Route policy, docs, and the BAC, XSS and swagger guardrails pass. The remaining failures also fail on the base commit.Follow-ups:
deploymentName, so Custom OpenAI-style models without one won't resolve.This issue will be closed when #1585 merges.
Show what ingestion extracted for V2 chat uploads and match the uploa…
Index images without OCR text and route vision analysis through its A…
Add React V2 release notes for the image indexing, upload extraction …
Address review findings on the React V2 image fixes
Address CodeQL findings on the image description indexing tests
paullizer commentedon Sep 30, 2026
Fixed by #1584 (Development, 0.261.047) and #1585 (React V2, 0.261.210), both merged. After deploying, re-upload or reprocess images that were ingested before the fix so they get indexed.