Skip to content

feat(pdf): prefer a configured remote parser in auto, add hosted MinerU support - #5967

Open
q2000s wants to merge 1 commit into
volcengine:mainfrom
q2000s:feat/mineru-first-strategy-standalone
Open

q2000s wants to merge 1 commit into
volcengine:mainfrom
q2000s:feat/mineru-first-strategy-standalone

Conversation

@q2000s

@q2000s q2000s commented Oct 13, 2026 •

Copy link
Copy Markdown

Summary

Two changes to PDF parsing, both aimed at making external parsers useful without making
them mandatory:

  1. auto prefers the configured remote parser (it used to try local first).
  2. mineru_token adds the hosted MinerU API, including the protocol that service
    actually speaks.

1. auto prefers the remote parser

strategy Remote parser used? On remote failure mineru_endpoint required
local no n/a no
mineru always the import fails yes
auto yes, if one is configured falls back to the local parser optional

Behavior change. auto previously tried the local parser first and only used MinerU
when local parsing failed. Since pdfplumber handles essentially every document, the
remote branch was almost never reached -- configuring mineru_endpoint had close to no
effect under auto. Set strategy: local to keep the old behaviour, or strategy: mineru
to require the remote path.

The fallback is per document, not per process.

Provider-shaped dispatch

auto iterates _remote_converters(), which lists the external converters this config
actually enables. "Is a remote parser in play" is PDFConfig.remote_enabled, expressed
over the provider list rather than any single service, and it is what both the dispatch
and the startup preflight read -- so the two cannot drift, and adding a provider does not
mean revisiting either. mineru_enabled remains the MinerU provider's own eligibility
check.

Degradation is bounded, not per-document

A batch that hits a rate limit should not pay for a doomed remote call on every
document, so auto:

  • retries 429 and 5xx once after a short backoff -- these usually clear;
  • sends 404 and other permanent errors straight to local, since retrying cannot help;
  • circuit-breaks a provider after repeated consecutive failures, so the rest of the
    batch skips it; the circuit half-opens after a cooldown, so a service that
    recovers is picked up again rather than being written off for the process lifetime;
  • counts degraded documents in pdf_remote_degraded, next to pdf_strategy_effective,
    so a batch that quietly stopped using the remote parser is visible after the fact.

2. Hosted MinerU support (mineru_token)

Config Protocol Flow
no mineru_token self-hosted (v1) POST {mineru_endpoint}/file_parse, markdown inline
mineru_token set hosted batch (v4) POST /file-urls/batch -> PUT -> poll GET /extract-results/batch/{batch_id} -> download full_zip_url

A token is treated as the signal for the hosted API rather than as a credential to attach
to the v1 call, because that service has no /file_parse route at all -- every document
would 404 and fall back. If the batch handshake is refused the parser falls back to
/file_parse once, keeping a token-gated self-hosted deployment working. Endpoints
without a token keep the existing /file_parse contract untouched.

Tests

tests/parse/test_pdf_strategy_dispatch.py, 26 cases:

  • auto prefers the remote parser; does not call local on success
  • auto falls back to local on a raised error, a timeout, and 404/405/429/500/503
  • auto uses local directly when nothing is configured; local never contacts remote
  • mineru does not fall back
  • token selects the batch protocol; no token keeps /file_parse; a refused batch
    handshake falls back
  • 429/500 are retried once, 404 is not
  • the circuit opens after repeated failures and half-opens to recover
  • batch level: 10 documents with 3 remote failures -- every document produces a
    result, pdf_strategy_effective is correct per document, and the degraded count is
    observable

47 passed across this file plus test_pdf_bookmark_extraction.py; ruff check and
ruff format --check clean.

Verified against the live hosted endpoint: strategy: auto with a token produced
api_mode=v4-batch, a real batch_id, and markdown returned by the service.

Docs

docs/{en,zh}/guides/01-configuration.md: strategy and protocol tables, the
provider-shaped predicate, the retry/circuit rules, and a behavior-change callout.
docs/{en,zh}/about/02-changelog.md: entries for all of the above under Unreleased.

Scope

Self-contained on main; depends on no other PR. tests/parse/ has 123 pre-existing
failures on main unrelated to this change (confirmed against a clean checkout).

@q2000s
q2000s force-pushed the feat/mineru-first-strategy-standalone branch 2 times, most recently from c099422 to 2c8f215 Compare October 13, 2026 00:46
@qin-ctx qin-ctx self-assigned this Oct 13, 2026
@ZaynJarvis ZaynJarvis added kernel Core server, runtime, storage, retrieval, SDK, and CLI. labels Oct 13, 2026
@q2000s
q2000s force-pushed the feat/mineru-first-strategy-standalone branch 3 times, most recently from 91c3786 to 044f416 Compare October 14, 2026 04:27
@q2000s q2000s changed the title feat(pdf): add mineru_first strategy with local fallback feat(pdf): prefer a configured remote parser in auto, add hosted MinerU support Oct 14, 2026
Adds a PDF parsing strategy that prefers MinerU and degrades to the local
parser instead of failing the import. It sits between 'mineru' (hard
dependency on a reachable endpoint) and 'auto' (local first):

  local        never calls MinerU
  mineru       always calls MinerU, failure fails the import
  mineru_first calls MinerU first, failure falls back to local
  auto         calls MinerU only after local fails

Because the strategy is defined as recoverable, MinerU problems never
become startup problems: should_preflight_mineru now covers mineru_first
so the preflight runs, but its failure stays advisory and the fallback
happens per document.

The result metadata records the converter that actually ran as
pdf_strategy_effective ('mineru' or 'local'), so operators can audit
conversion quality after the fact rather than inferring it from logs.

Spelled mineru_first rather than mineru-first to match the snake_case
convention used by the other enumerations in ov.conf. This spelling is
used in configuration, ov.conf, logs and metadata.

Tests cover the fallback path, which is the contract this strategy
exists to provide, plus config validation.
@q2000s
q2000s force-pushed the feat/mineru-first-strategy-standalone branch from 044f416 to d4eaccd Compare October 14, 2026 10:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Core server, runtime, storage, retrieval, SDK, and CLI.

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

4 participants