Skip to content

Commit 81c10c9

Browse files
authored
feat(mistral): extract PDF images as files and expose table format in Mistral Parser (#9856)
* feat(mistral): extract PDF images as files and expose table format in Mistral Parser * fix(mistral): type the v3 images output as optional for the shared v2 base * fix(mistral): name extracted images as storage does, drop inline table format, align image switch with tool param * docs(mistral): describe Extract Images and Tables options in the Mistral Parser page
1 parent 24aab3f commit 81c10c9

14 files changed

Lines changed: 432 additions & 131 deletions

File tree

‎apps/docs/content/docs/integrations/mistral_parse.mdx‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Mistral Parser
3-
description: Extract text from PDF documents
3+
description: Extract text, tables, and images from PDF documents
44
integration:
55
type: "mistral_parse_v3"
66
color: "#000000"
@@ -10,12 +10,12 @@ import { ReferenceDetails } from "@/components/ui/reference-details"
1010

1111

1212
{/* MANUAL-CONTENT-START:intro */}
13-
Use [Mistral's OCR API](https://mistral.ai/) to extract text and page structure from a PDF uploaded to Sim or supplied by URL. The current parser returns page Markdown, image metadata, the model, and usage information. **Specific Pages** accepts zero-based indices such as `0,1,2`; leave it empty to process all pages. The block offers Markdown, plain-text, and JSON format values, but its current result uses the structured page output documented below. Base64 image data is a hidden tool input and has no block control.
13+
Use [Mistral's OCR API](https://mistral.ai/) to extract text and page structure from a PDF uploaded to Sim or supplied by URL. The current parser returns page Markdown, image metadata, the model, and usage information. **Specific Pages** accepts zero-based indices such as `0,1,2`; leave it empty to process all pages. The block offers Markdown, plain-text, and JSON format values, but its current result uses the structured page output documented below. Turn on **Extract Images** to get the images Mistral finds in the document (figures, logos, photos) as files on the `images` output, up to 100 per parse; each page's image entry carries a `file_name` matching its file. **Max Images** and **Min Image Size** (advanced) narrow what is extracted. **Tables** keeps tables inline in the page Markdown by default, or returns them separately as Markdown or HTML in each page's `tables` list.
1414
{/* MANUAL-CONTENT-END */}
1515

1616
## Use the block [#usage-instructions]
1717

18-
Extract text from uploaded PDF documents, or from a URL.
18+
Extract text, tables, and images from uploaded PDF documents, or from a URL. Images come back as files an agent can pass on.
1919

2020
## Actions
2121

@@ -31,7 +31,11 @@ Parse PDF documents using Mistral OCR API
3131
| --------- | ---- | -------- | ----------- |
3232
| `file` | file | Yes | Normalized UserFile from file upload or file reference |
3333
| `resultType` | string | No | Legacy parser output format \(markdown, text, or json; default markdown\). Versions 2 and 3 return structured OCR pages regardless of this value. |
34+
| `includeImageBase64` | boolean | No | Extract the images in the document \(figures, logos, photos\) as files on the images output |
3435
| `pages` | array | No | Specific pages to process \(array of page numbers, starting from 0\) |
36+
| `imageLimit` | number | No | Maximum number of images to extract \(at most 100\) |
37+
| `imageMinSize` | number | No | Minimum height and width of images to extract from the PDF |
38+
| `tableFormat` | string | No | Return tables separately on each page as markdown or html instead of inline in the page markdown |
3539
| `apiKey` | string | Yes | Mistral API key \(MISTRAL_API_KEY\) |
3640

3741
#### Output
@@ -47,20 +51,21 @@ Parse PDF documents using Mistral OCR API
4751
| ↳ `top_left_y` | number | Top-left Y coordinate in pixels |
4852
| ↳ `bottom_right_x` | number | Bottom-right X coordinate in pixels |
4953
| ↳ `bottom_right_y` | number | Bottom-right Y coordinate in pixels |
50-
| ↳ `image_base64` | string | Base64-encoded image data; returned only when the hidden includeImageBase64 input is enabled |
54+
| ↳ `file_name` | string | Name of this image in the images output; present only when image extraction is on |
5155
| ↳ `dimensions` | object | Page dimensions |
5256
| ↳ `dpi` | number | Dots per inch |
5357
| ↳ `height` | number | Page height in pixels |
5458
| ↳ `width` | number | Page width in pixels |
55-
| ↳ `tables` | array | Separate table objects, referenced from the markdown via placeholders like \[tbl-0.html\]. Mistral populates these only when table_format is "markdown" or "html"; Sim never sets it, so tables stay inline in the markdown and this list is empty |
56-
| ↳ `hyperlinks` | array | Array of URL strings detected in the page \(e.g., \["https://...", "mailto:..."\]\) |
59+
| ↳ `tables` | array | Separate table objects \(\{ id, content, format \}\), referenced from the markdown via placeholders like \[tbl-0.html\]. Populated only when tableFormat is markdown or html; otherwise tables stay inline in the markdown and this list is empty |
60+
| ↳ `hyperlinks` | array | Array of URL strings detected in the page |
5761
| ↳ `header` | string | Page header content. Mistral returns it only when extract_header is true \(it defaults to false\); Sim never sets it, so this is not returned |
5862
| ↳ `footer` | string | Page footer content. Mistral returns it only when extract_footer is true \(it defaults to false\); Sim never sets it, so this is not returned |
5963
| `model` | string | Mistral OCR model identifier \(e.g., mistral-ocr-latest\) |
6064
| `usage_info` | object | Usage and processing statistics |
6165
| ↳ `pages_processed` | number | Total number of pages processed |
6266
| ↳ `doc_size_bytes` | number | Document file size in bytes |
6367
| `document_annotation` | string | Structured annotation data as JSON string \(when applicable\) |
68+
| `images` | file[] | Images extracted from the document, named page-<index>-<image id>; empty unless image extraction is on |
6469

6570
</ReferenceDetails>
6671

‎apps/sim/blocks/blocks/mistral_parse.ts‎

Lines changed: 66 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,11 @@ import {
77
IntegrationType,
88
type SubBlockType,
99
} from '@/blocks/types'
10-
import { createVersionedToolSelector, normalizeFileInput } from '@/blocks/utils'
10+
import {
11+
createVersionedToolSelector,
12+
normalizeFileInput,
13+
parseOptionalNumberInput,
14+
} from '@/blocks/utils'
1115

1216
export const MistralParseBlock: BlockConfig = {
1317
type: 'mistral_parse',
@@ -304,7 +308,9 @@ export const MistralParseV3Block: BlockConfig = {
304308
sunset: undefined,
305309
type: 'mistral_parse_v3',
306310
name: 'Mistral Parser',
307-
description: 'Extract text from PDF documents',
311+
description: 'Extract text, tables, and images from PDF documents',
312+
longDescription:
313+
'Extract text, tables, and images from uploaded PDF documents, or from a URL. Images come back as files an agent can pass on.',
308314
hideFromToolbar: false,
309315
/* v3 renamed the canonical pair's advanced member `filePath` -> `fileReference`,
310316
so the inherited sentence would reference a subblock this block lacks. */
@@ -358,6 +364,38 @@ export const MistralParseV3Block: BlockConfig = {
358364
type: 'short-input',
359365
placeholder: 'e.g. 0,1,2 (leave empty for all pages)',
360366
},
367+
{
368+
id: 'tableFormat',
369+
title: 'Tables',
370+
type: 'dropdown',
371+
options: [
372+
{ id: 'inline', label: 'Inline in page markdown' },
373+
{ id: 'markdown', label: 'Separate (Markdown)' },
374+
{ id: 'html', label: 'Separate (HTML)' },
375+
],
376+
value: () => 'inline',
377+
},
378+
{
379+
id: 'includeImageBase64',
380+
title: 'Extract Images',
381+
type: 'switch',
382+
},
383+
{
384+
id: 'imageLimit',
385+
title: 'Max Images',
386+
type: 'short-input',
387+
placeholder: 'Up to 100 (default 100)',
388+
condition: { field: 'includeImageBase64', value: true },
389+
mode: 'advanced',
390+
},
391+
{
392+
id: 'imageMinSize',
393+
title: 'Min Image Size (px)',
394+
type: 'short-input',
395+
placeholder: 'Skip images smaller than this on both sides',
396+
condition: { field: 'includeImageBase64', value: true },
397+
mode: 'advanced',
398+
},
361399
{
362400
id: 'apiKey',
363401
title: 'API Key',
@@ -416,6 +454,24 @@ export const MistralParseV3Block: BlockConfig = {
416454
parameters.pages = pagesArray
417455
}
418456

457+
parameters.tableFormat =
458+
params.tableFormat === 'markdown' || params.tableFormat === 'html'
459+
? params.tableFormat
460+
: undefined
461+
462+
if (params.includeImageBase64) {
463+
parameters.includeImageBase64 = true
464+
parameters.imageLimit = parseOptionalNumberInput(params.imageLimit, 'Max Images', {
465+
integer: true,
466+
min: 0,
467+
})
468+
parameters.imageMinSize = parseOptionalNumberInput(
469+
params.imageMinSize,
470+
'Min Image Size',
471+
{ integer: true, min: 0 }
472+
)
473+
}
474+
419475
return parameters
420476
},
421477
},
@@ -425,12 +481,20 @@ export const MistralParseV3Block: BlockConfig = {
425481
apiKey: { type: 'string', description: 'Mistral API key' },
426482
resultType: { type: 'string', description: 'Output format type' },
427483
pages: { type: 'string', description: 'Page selection' },
484+
tableFormat: { type: 'string', description: 'Table output: inline, markdown, or html' },
485+
includeImageBase64: { type: 'boolean', description: 'Extract document images as files' },
486+
imageLimit: { type: 'number', description: 'Maximum number of images to extract' },
487+
imageMinSize: { type: 'number', description: 'Minimum image size in pixels' },
428488
},
429489
outputs: {
430490
pages: { type: 'array', description: 'Array of page objects from Mistral OCR' },
431491
model: { type: 'string', description: 'Mistral OCR model identifier' },
432492
usage_info: { type: 'json', description: 'Usage statistics from the API' },
433493
document_annotation: { type: 'string', description: 'Structured annotation data' },
494+
images: {
495+
type: 'file[]',
496+
description: 'Images extracted from the document when Extract Images is on',
497+
},
434498
},
435499
}
436500

‎apps/sim/lib/internal/mistral/execute-tool.ts‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,12 @@ import {
99
mistralParseInputSchema,
1010
} from '@/lib/internal/mistral/input'
1111
import { isMistralInputWithinLimit } from '@/lib/internal/mistral/input-size'
12-
import { executeMistralParse } from '@/lib/internal/mistral/operations'
13-
import type { InternalToolOperationHandler } from '@/lib/internal/tool-operations/types'
12+
import { executeMistralToolParse } from '@/lib/internal/mistral/operations'
13+
import { isInternalToolFileResult } from '@/lib/internal/tool-operations/file-result'
14+
import type {
15+
InternalToolOperationHandler,
16+
InternalToolOperationResult,
17+
} from '@/lib/internal/tool-operations/types'
1418
import { docNotReadyMessage, isDocNotReadyError } from '@/lib/uploads/utils/doc-not-ready'
1519

1620
const logger = createLogger('MistralToolExecution')
@@ -26,7 +30,9 @@ function validationResponse(error: z.ZodError): Response {
2630
)
2731
}
2832

29-
export const executeMistralTool: InternalToolOperationHandler = async (request) => {
33+
export const executeMistralTool: InternalToolOperationHandler<InternalToolOperationResult> = async (
34+
request
35+
) => {
3036
request.signal?.throwIfAborted()
3137
if (!['mistral_parser', 'mistral_parser_v2', 'mistral_parser_v3'].includes(request.toolId)) {
3238
return Response.json(
@@ -51,14 +57,14 @@ export const executeMistralTool: InternalToolOperationHandler = async (request)
5157
if (!parsed.success) return validationResponse(parsed.error)
5258

5359
try {
54-
const result = await executeMistralParse(parsed.data, {
60+
const result = await executeMistralToolParse(request.toolId, parsed.data, {
5561
headers: request.headers,
5662
requestId: request.requestId,
5763
signal: request.signal,
5864
userId: request.context.userId,
5965
})
6066
request.signal?.throwIfAborted()
61-
return Response.json(result)
67+
return isInternalToolFileResult(result) ? result : Response.json(result)
6268
} catch (error) {
6369
request.signal?.throwIfAborted()
6470
if (error instanceof ProviderCapacityDeferredError) {
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
import { describe, expect, it } from 'vitest'
2+
import { extractOcrImageFiles, MAX_OCR_EXTRACTED_IMAGES } from '@/lib/internal/mistral/images'
3+
4+
const JPEG = Buffer.concat([Buffer.from([0xff, 0xd8, 0xff]), Buffer.from('synthetic jpeg body')])
5+
const dataUri = (bytes: Buffer) => `data:image/jpeg;base64,${bytes.toString('base64')}`
6+
7+
function image(id: string, imageBase64: string | undefined) {
8+
return {
9+
id,
10+
top_left_x: 0,
11+
top_left_y: 0,
12+
bottom_right_x: 10,
13+
bottom_right_y: 10,
14+
image_base64: imageBase64,
15+
}
16+
}
17+
18+
describe('extractOcrImageFiles', () => {
19+
it('moves image bytes into files named as storage will name them, with no base64 left in the page output', () => {
20+
const { output, files } = extractOcrImageFiles({
21+
pages: [
22+
{
23+
index: 2,
24+
markdown: '![img-0.jpeg](img-0.jpeg)',
25+
images: [image('img-0.jpeg', dataUri(JPEG))],
26+
},
27+
],
28+
})
29+
expect(files).toEqual([{ buffer: JPEG, name: 'page-2-img-0.jpg', mimeType: 'image/jpeg' }])
30+
expect(JSON.stringify(output)).not.toContain(JPEG.toString('base64'))
31+
expect(output.pages).toEqual([
32+
expect.objectContaining({
33+
images: [expect.objectContaining({ id: 'img-0.jpeg', file_name: 'page-2-img-0.jpg' })],
34+
}),
35+
])
36+
})
37+
38+
it('stores at most the image cap and keeps the bounding boxes of the rest without a file', () => {
39+
const images = Array.from({ length: MAX_OCR_EXTRACTED_IMAGES + 1 }, (_, i) =>
40+
image(`img-${i}.jpeg`, JPEG.toString('base64'))
41+
)
42+
const { output, files } = extractOcrImageFiles({ pages: [{ index: 0, images }] })
43+
expect(files).toHaveLength(MAX_OCR_EXTRACTED_IMAGES)
44+
const pageImages = (output.pages as Array<{ images: Array<Record<string, unknown>> }>)[0].images
45+
expect(pageImages).toHaveLength(MAX_OCR_EXTRACTED_IMAGES + 1)
46+
expect(pageImages.at(-1)).not.toHaveProperty('file_name')
47+
expect(pageImages.at(-1)).not.toHaveProperty('image_base64')
48+
})
49+
50+
it('skips an image over the per-image cap instead of failing the parse', () => {
51+
const huge = Buffer.alloc(21 * 1024 * 1024)
52+
const { output, files } = extractOcrImageFiles({
53+
pages: [
54+
{
55+
index: 0,
56+
images: [image('img-0.png', dataUri(huge)), image('img-1.jpg', dataUri(JPEG))],
57+
},
58+
],
59+
})
60+
expect(files.map((file) => file.name)).toEqual(['page-0-img-1.jpg'])
61+
expect(JSON.stringify(output)).not.toContain('image_base64')
62+
})
63+
})
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
import { isRecordLike } from '@sim/utils/object'
2+
import { decodeDataUriWithinLimit } from '@/lib/file-parsers/data-uri'
3+
import { isFileParserError } from '@/lib/file-parsers/errors'
4+
import type { InternalToolFile } from '@/lib/internal/tool-operations/file-result'
5+
import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types'
6+
import { resolveEffectiveMimeType } from '@/lib/uploads/utils/file'
7+
import { resolveStoredFileMetadata } from '@/lib/uploads/utils/stored-file-metadata'
8+
9+
/** Most images one parse stores; Mistral's `image_limit` is clamped to it. */
10+
export const MAX_OCR_EXTRACTED_IMAGES = 100
11+
12+
/** Largest single image kept; a bigger one is a page scan, not a figure. */
13+
const MAX_OCR_IMAGE_BYTES = 20 * 1024 * 1024
14+
15+
interface OcrImageFiles {
16+
/** The OCR output with every `image_base64` removed and each kept image named. */
17+
output: Record<string, unknown>
18+
files: InternalToolFile[]
19+
}
20+
21+
/** The decoded image, or null when it is over the per-image cap. */
22+
function decodeImage(imageBase64: string): { buffer: Buffer; mediaType: string | null } | null {
23+
const dataUri = imageBase64.startsWith('data:') ? imageBase64 : `data:;base64,${imageBase64}`
24+
try {
25+
return decodeDataUriWithinLimit(dataUri, MAX_OCR_IMAGE_BYTES)
26+
} catch (error) {
27+
if (isFileParserError(error) && error.code === 'complexity_limit') return null
28+
throw error
29+
}
30+
}
31+
32+
/**
33+
* Turns the base64 images in a Mistral OCR response into files the executor
34+
* stores, so image bytes never travel through block outputs. Each kept image
35+
* gets a `file_name` matching its stored file; an image past the count, the
36+
* per-image cap, or the total byte budget keeps its bounding box but no file.
37+
*/
38+
export function extractOcrImageFiles(output: unknown): OcrImageFiles {
39+
if (!isRecordLike(output)) return { output: {}, files: [] }
40+
if (!Array.isArray(output.pages)) return { output, files: [] }
41+
const files: InternalToolFile[] = []
42+
let totalBytes = 0
43+
const pages = output.pages.map((page: unknown) => {
44+
if (!isRecordLike(page) || !Array.isArray(page.images)) return page
45+
const pageIndex = typeof page.index === 'number' ? page.index : 0
46+
const images = page.images.map((image: unknown) => {
47+
if (!isRecordLike(image)) return image
48+
const { image_base64: imageBase64, ...rest } = image
49+
if (typeof imageBase64 !== 'string' || !imageBase64) return rest
50+
if (files.length >= MAX_OCR_EXTRACTED_IMAGES) return rest
51+
const decoded = decodeImage(imageBase64)
52+
if (!decoded || totalBytes + decoded.buffer.length > MAX_BUFFERED_TRANSFER_BYTES) return rest
53+
totalBytes += decoded.buffer.length
54+
const id = typeof image.id === 'string' && image.id ? image.id : `img-${files.length}.jpeg`
55+
/** Storage renames an image from its bytes; naming it the same way keeps `file_name` matching. */
56+
const draftName = `page-${pageIndex}-${id}`
57+
const { fileName, mimeType } = resolveStoredFileMetadata(
58+
draftName,
59+
resolveEffectiveMimeType(decoded.mediaType, draftName),
60+
decoded.buffer
61+
)
62+
files.push({ buffer: decoded.buffer, name: fileName, mimeType })
63+
return { ...rest, file_name: fileName }
64+
})
65+
return { ...page, images }
66+
})
67+
return { output: { ...output, pages }, files }
68+
}

‎apps/sim/lib/internal/mistral/input.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,9 @@ export const mistralParseInputSchema = z.object({
1212
resultType: z.string().optional(),
1313
pages: z.array(z.number()).max(MISTRAL_OCR_REQUEST_POLICY.maxPages).optional(),
1414
includeImageBase64: z.boolean().optional(),
15-
imageLimit: z.number().optional(),
16-
imageMinSize: z.number().optional(),
15+
imageLimit: z.number().int().min(0).optional(),
16+
imageMinSize: z.number().int().min(0).optional(),
17+
tableFormat: z.enum(['markdown', 'html']).optional(),
1718
[RESOLVED_SECRET_PROVENANCE_FIELD]: resolvedSecretTraceProvenanceSchema.optional(),
1819
})
1920

0 commit comments

Comments
 (0)