Skip to content

feat(parse): support task-based MinerU APIs and the online batch service #4725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
now-ing wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(parse): support task-based MinerU APIs and the online batch service #4725

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions docs/en/guides/01-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -892,11 +892,32 @@ PDF parsing configuration. Three strategies are supported: `local` (local pdfplu
| Parameter | Type | Description |
|-----------|------|-------------|
| `strategy` | str | Parsing strategy: `local` / `mineru` / `auto` (default `auto`) |
| `mineru_endpoint` | str | MinerU API **base URL** (e.g. `http://127.0.0.1:8000`) |
| `mineru_timeout` | float | Request timeout in seconds (default `300.0`) |
| `mineru_bodys` | dict | MinerU API multipart form fields |
| `mineru_endpoint` | str | MinerU API **base URL** (e.g. `http://127.0.0.1:8000`, or `https://mineru.net/api/v4` for the online API) |
| `mineru_timeout` | float | Request/polling timeout in seconds (default `300.0`) |
| `mineru_bodys` | dict | MinerU API multipart form fields (passed through) |
| `mineru_api_mode` | str | Protocol flavor: `auto` (detect from the response, default) / `sync` (legacy inline `/file_parse`) / `async` (task APIs) |
| `mineru_token` | str | Optional bearer token for the online MinerU API (sent as `Authorization: Bearer ...`) |

**MinerU protocol**: a synchronous `POST {mineru_endpoint}/file_parse` request with the PDF as the multipart `files` field; form parameters are passed through from `mineru_bodys`.
**MinerU protocols** — three flavors are supported:

- **Legacy self-hosted (`sync`)**: a single `POST {mineru_endpoint}/file_parse` answering inline with the markdown; form parameters are passed through from `mineru_bodys`.
- **Current self-hosted (`async`)**: `POST {mineru_endpoint}/tasks` answers `202` with `task_id`/`status_url`/`result_url`; the task is polled until `completed` and the markdown is downloaded from the result zip.
- **Online batch API (`async`)**: `POST {mineru_endpoint}/extract/task/batch` (Bearer-token auth via `mineru_token`), poll `{mineru_endpoint}/extract-results/batch/{batch_id}` until `state=done`, download `full_zip_url` and unpack.

Under the default `mineru_api_mode=auto` the flavor is detected automatically: the inline endpoint is tried first and a `404` (or a task-shaped response) switches to the task flow, so existing self-hosted v1 deployments keep working without config changes. The startup `/health` preflight only applies to the sync flavor (the online service has no such endpoint).

**Online example:**

```json
{
"pdf": {
"strategy": "mineru",
"mineru_endpoint": "https://mineru.net/api/v4",
"mineru_api_mode": "auto",
"mineru_token": "your-mineru-api-token"
}
}
```

### rerank

Expand Down
29 changes: 25 additions & 4 deletions docs/zh/guides/01-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -860,11 +860,32 @@ PDF 解析配置。支持三种策略：`local`（本地 pdfplumber）、`mineru
| 参数 | 类型 | 说明 |
|------|------|------|
| `strategy` | str | 解析策略：`local` / `mineru` / `auto`（默认 `auto`） |
| `mineru_endpoint` | str | MinerU API **base URL**（如 `http://127.0.0.1:8000`） |
| `mineru_timeout` | float | 请求超时秒数（默认 `300.0`） |
| `mineru_bodys` | dict | MinerU API multipart form 参数 |
| `mineru_endpoint` | str | MinerU API **base URL**（如自托管 `http://127.0.0.1:8000`，在线服务 `https://mineru.net/api/v4`） |
| `mineru_timeout` | float | 请求/轮询超时秒数（默认 `300.0`） |
| `mineru_bodys` | dict | MinerU API multipart form 参数（透传） |
| `mineru_api_mode` | str | 协议形态：`auto`（按响应自动识别，默认）/ `sync`（旧版内联 `/file_parse`）/ `async`（任务式 API） |
| `mineru_token` | str | 在线 MinerU API 的 bearer token（以 `Authorization: Bearer ...` 发送） |

**MinerU 协议**：同步调用 `POST {mineru_endpoint}/file_parse`，multipart 文件字段为 `files`，form 参数由 `mineru_bodys` 透传。
**MinerU 协议**——支持三种形态：

- **旧版自托管（`sync`）**：单次 `POST {mineru_endpoint}/file_parse` 内联返回 markdown；form 参数由 `mineru_bodys` 透传。
- **新版自托管（`async`）**：`POST {mineru_endpoint}/tasks` 返回 `202` 与 `task_id`/`status_url`/`result_url`；轮询任务至 `completed` 后从结果 zip 下载 markdown。
- **在线批量 API（`async`）**：`POST {mineru_endpoint}/extract/task/batch`（通过 `mineru_token` 做 Bearer 认证），轮询 `{mineru_endpoint}/extract-results/batch/{batch_id}` 至 `state=done`，下载 `full_zip_url` 并解包。

默认 `mineru_api_mode=auto` 时自动识别协议形态：先尝试内联端点，返回 `404`（或任务式响应）即切换到任务流，存量自托管 v1 部署无需改配置。启动时的 `/health` 预检仅适用于 sync 形态（在线服务没有该端点）。

**在线服务示例：**

```json
{
"pdf": {
"strategy": "mineru",
"mineru_endpoint": "https://mineru.net/api/v4",
"mineru_api_mode": "auto",
"mineru_token": "your-mineru-api-token"
}
}
```

### rerank

Expand Down
Loading