Repository navigation
Use Azure AI Content Understanding for Enhanced document extraction #1277
Description
Summary
Enhanced extraction for PDFs and images currently means Azure Document Intelligence prebuilt-layout. It preserves tables and selection marks, but it does not produce any image or figure analysis — which is the structure that matters most for PDFs and Office documents.
Azure AI Content Understanding (prebuilt-documentSearch) returns layout-aware markdown plus AI-generated descriptions of figures, charts, and diagrams, along with structured Chart.js/Mermaid output. Enhanced extraction should use it instead.
User Value
Workspace and chat documents that rely on charts, diagrams, screenshots, and figures become searchable and citable by what the figure actually shows, not just by whatever OCR text happens to sit inside it. Documents with tables and checkbox states keep the quality they have today.
Proposed Behavior
- Standard extraction stays Document Intelligence
prebuilt-readand remains required for workspaces and chat file uploads. - Enhanced extraction uses Content Understanding, gated behind a new Enable Enhanced extraction admin toggle that reveals the Content Understanding connection settings.
- The existing Standard / Enhanced / Auto selector keeps working. Enabling Enhanced defaults the selector to Auto.
- Auto keeps sampling the first pages with Document Intelligence Layout as the cheaper detector, and additionally upgrades to Enhanced when the sample contains figures or images, not just tables and selection marks.
Graceful degradation is required
Content Understanding is only available in Azure commercial regions — it is not offered in Azure Government. Enhanced must therefore never become a hard dependency:
- In Azure Government and custom clouds, Enhanced automatically uses Document Intelligence Layout, and the admin UI should say so plainly rather than presenting configuration that cannot work.
- If Content Understanding is enabled but unconfigured or unreachable, the document falls back to Document Intelligence Layout and the reason is recorded on the document.
Images inside Office files
Content Understanding accepts DOCX/DOC/PPTX as input, but its figure analysis only applies to PDFs and images. Embedded images should be pulled out of the OOXML package and analyzed separately with whichever engine backs the selected extraction mode, then indexed as their own citable chunks. Per-document cost needs bounding via a minimum image size, duplicate detection, and a cap.
Acceptance Criteria
- Admin settings expose an Enable Enhanced extraction toggle plus Content Understanding endpoint, authentication type (key and managed identity), API version, and analyzer IDs.
- A Test Connection button validates the endpoint, credentials, and analyzer before saving, and explains the common setup failures rather than surfacing raw HTTP errors.
- An in-app guide covers Foundry resource creation, supported regions, required model deployment defaults, and the Cognitive Services User role.
- The admin UI is environment-aware: Azure Public shows the Content Understanding configuration; Azure Government and custom clouds show that Enhanced uses Document Intelligence Layout and needs no configuration.
- Enhanced falls back to Document Intelligence Layout on unsupported cloud, missing configuration, or request failure, with the reason recorded on the document and shown in workspace tooltips.
- Auto mode upgrades to Enhanced when sampled pages contain tables, selection marks, or figures/images.
- Images embedded in DOCX and PPTX are analyzed by the active engine and indexed as their own citable chunks, with size filtering, duplicate detection, and a per-document cap.
- Existing deployments already set to Enhanced or Auto keep that behavior across the upgrade rather than silently downgrading to Standard.
- Uploaded Office files are treated as untrusted input: no path traversal from archive entry names, bounded decompression, and hardened XML parsing.
- Extraction badges and tooltips name the engine that actually ran, and the Change Extraction action covers images as well as PDFs.
- Functional tests cover engine resolution and fallback, page reconstruction, figure handling, the upgrade migration, embedded image extraction, and the untrusted-archive defenses.
Notes and Constraints
- Cost:
prebuilt-documentSearchinvokes a language model per document, so Enhanced is meaningfully more expensive than Document Intelligence Layout. Auto is the best default. - Setup: Content Understanding requires default completion and embedding model deployments on the Foundry resource; requests fail until those are set.
- Compatibility: existing setting and per-document field names should be preserved so no Cosmos or search index migration is needed.
Related
- Related to [P1] Add multimodal augmentation for complex document page extraction #1137, which adds a parallel multimodal augmentation pass over flagged complex pages. That work is complementary: this issue changes which engine backs Enhanced extraction, while [P1] Add multimodal augmentation for complex document page extraction #1137 layers additional structured analysis on top of whatever extraction ran.
Activity
paullizer commentedon Aug 18, 2026
Implemented and merged into
Developmentvia #1278 (merge commitd5390c1f), shipping in v0.250.221.Leaving this open to match the repo convention —
Fixes #1277only auto-closes on merge to the default branch (main), and issues here stay open untilDevelopmentreaches it.What shipped
Enhanced extraction now uses Azure AI Content Understanding (
prebuilt-documentSearch), which returns layout-aware markdown plus AI-generated descriptions of figures, charts, and diagrams. Standard extraction is unchanged and still uses Document Intelligenceprebuilt-read.All acceptance criteria are met, including graceful degradation: Enhanced automatically falls back to Document Intelligence Layout in Azure Government and custom clouds, when Content Understanding is unconfigured, or when a request fails, with the reason recorded on the document and surfaced in workspace tooltips.
Validation
test_legacy_doc_ole_extraction.py, animportlibstub set that did not know about the new module) and was fixed in the same PR.Follow-ups worth tracking separately
enable_enhanced_extractionon for deployments already set to Enhanced or Auto, preserving their intent. Becauseprebuilt-documentSearchinvokes a language model per document, those tenants will see higher per-document extraction cost than Document Intelligence Layout. Worth calling out in release communications so it does not arrive as a surprise on a bill.Reference tracking issue microsoft#1277 in docs
Render EMF/WMF diagrams so Office figures are analyzed
paullizer commentedon Aug 18, 2026
Follow-up: EMF/WMF diagrams in Office files are now analyzed.
A real TIC 3.0 architecture document uploaded with Enhanced extraction on produced no image analysis and no log output, despite containing four diagrams. The images were EMF metafiles, and the embedded-image whitelist accepted only raster formats, so extraction matched nothing — and returned before emitting any status message, which is why the workspace log stayed silent.
Why it needed more than a whitelist entry
Neither extraction engine accepts a metafile, so EMF has to be rasterized first, and the usual routes were all closed:
hasattr(Image.core, "drawwmf")and commented "windows only" because it is GDI-backed. Relying on it would have worked in local development and silently failed in the container.azurelinux/distroless/python, with no shell and no package manager, so LibreOffice or Inkscape would mean copying a very large dependency tree into a distroless image.functions_emf_render.pyrenders EMF and WMF in-process using only Pillow,io, andstruct— no system packages, no external binaries, no platform branches, and no container change.Result
All four diagrams rasterize into recognizable Azure architecture renderings, and their text labels are recovered — Event Hub Namespace, Log Analytics Workspace, Azure Firewall Table, SysLog Table, CLAW Registered App — so labels stay searchable even when the vision engine returns no description.
Embedded image processing is also no longer silent. Extraction reports candidates found, images analyzed, and per-reason skip counts, with progress per image, and persists those counts on the document. Previously "no images in this file" and "images found but all skipped" were indistinguishable.
Tracked in #1279, all checks green. Version 0.250.223.
Note for reviewers
Rasterization is a description aid for search and citation rather than a pixel-accurate GDI reimplementation. Gradients, complex clipping, and bitmap blits inside a metafile are not reproduced, and text is drawn with a default font. The separately extracted metafile text keeps labels accurate regardless.
Support legacy Office images and optional formula extraction
paullizer commentedon Aug 18, 2026
Follow-up on coverage questions: equations, legacy binary Office formats, and PPTX.
What was already working
PPTX was already covered. Verified end-to-end by placing a real EMF diagram on slide 2 of a test deck: extracted, rasterized, slide attribution correct, 27 labels recovered.
What was not, and is now
Legacy
.docand.pptwere excluded outright. They are OLE compound documents rather than zip packages, so there is no media part to enumerate and the extractor found nothing. Their pictures and embedded equation previews are now carved out of the container by metafile signature, using the length recorded in the metafile's own header, then rasterized and analyzed like any other embedded image. Validation checks record type, signature position, and that the declared length fits the remaining bytes, so a coincidental byte sequence is not mistaken for an image.Equations were never extracted at all — no formula feature was requested anywhere in the app, so they arrived as whatever OCR guessed. Document Intelligence exposes a
FORMULASadd-on, already present in the pinned SDK, which returns equations as LaTeX. It is billed per page, so it is behind a new admin toggle that defaults to off, is guarded by the layout-mode check because the add-on applies to that model only, and the admin UI states plainly that it is a billed add-on.Boundary worth recording
Equations authored in modern Word are stored as OMML markup rather than images, so they are outside embedded image analysis entirely and the formula add-on does not reach them either. Legacy Equation Editor and MathType objects are stored with a metafile preview, and those are covered by the carving change. Both facts are now written into the feature documentation so the boundary is not misremembered later.
Tracked in #1280, all checks green. Version 0.250.224.
paullizer commentedon Aug 19, 2026
Shipped in v0.250.228 via #1278 and #1279 and #1280 and #1290.
Release note entries:
.docand.pptFiles Are Now AnalyzedFull notes: https://microsoft.github.io/simplechat/explanation/release_notes/
Closing as completed. This issue stayed open after the fix merged because pull requests target
Developmentwhile the repository default branch ismain, so GitHub never applied theFixes #keyword automatically.Keep figures in the chunk they came from