Repository navigation
Conversation
q2000s
force-pushed
the
feat/mineru-first-strategy-standalone
branch
2 times, most recently
from
October 13, 2026 00:46
c099422 to
2c8f215
Compare
q2000s
force-pushed
the
feat/mineru-first-strategy-standalone
branch
3 times, most recently
from
October 14, 2026 04:27
91c3786 to
044f416
Compare
Adds a PDF parsing strategy that prefers MinerU and degrades to the local
parser instead of failing the import. It sits between 'mineru' (hard
dependency on a reachable endpoint) and 'auto' (local first):
local never calls MinerU
mineru always calls MinerU, failure fails the import
mineru_first calls MinerU first, failure falls back to local
auto calls MinerU only after local fails
Because the strategy is defined as recoverable, MinerU problems never
become startup problems: should_preflight_mineru now covers mineru_first
so the preflight runs, but its failure stays advisory and the fallback
happens per document.
The result metadata records the converter that actually ran as
pdf_strategy_effective ('mineru' or 'local'), so operators can audit
conversion quality after the fact rather than inferring it from logs.
Spelled mineru_first rather than mineru-first to match the snake_case
convention used by the other enumerations in ov.conf. This spelling is
used in configuration, ov.conf, logs and metadata.
Tests cover the fallback path, which is the contract this strategy
exists to provide, plus config validation.
q2000s
force-pushed
the
feat/mineru-first-strategy-standalone
branch
from
October 14, 2026 10:35
044f416 to
d4eaccd
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to PDF parsing, both aimed at making external parsers useful without making
them mandatory:
autoprefers the configured remote parser (it used to try local first).mineru_tokenadds the hosted MinerU API, including the protocol that serviceactually speaks.
1.
autoprefers the remote parserstrategymineru_endpointrequiredlocalmineruautoBehavior change.
autopreviously tried the local parser first and only used MinerUwhen local parsing failed. Since
pdfplumberhandles essentially every document, theremote branch was almost never reached -- configuring
mineru_endpointhad close to noeffect under
auto. Setstrategy: localto keep the old behaviour, orstrategy: mineruto require the remote path.
The fallback is per document, not per process.
Provider-shaped dispatch
autoiterates_remote_converters(), which lists the external converters this configactually enables. "Is a remote parser in play" is
PDFConfig.remote_enabled, expressedover the provider list rather than any single service, and it is what both the dispatch
and the startup preflight read -- so the two cannot drift, and adding a provider does not
mean revisiting either.
mineru_enabledremains the MinerU provider's own eligibilitycheck.
Degradation is bounded, not per-document
A batch that hits a rate limit should not pay for a doomed remote call on every
document, so
auto:429and5xxonce after a short backoff -- these usually clear;404and other permanent errors straight to local, since retrying cannot help;batch skips it; the circuit half-opens after a cooldown, so a service that
recovers is picked up again rather than being written off for the process lifetime;
pdf_remote_degraded, next topdf_strategy_effective,so a batch that quietly stopped using the remote parser is visible after the fact.
2. Hosted MinerU support (
mineru_token)mineru_tokenPOST {mineru_endpoint}/file_parse, markdown inlinemineru_tokensetPOST /file-urls/batch->PUT-> pollGET /extract-results/batch/{batch_id}-> downloadfull_zip_urlA token is treated as the signal for the hosted API rather than as a credential to attach
to the v1 call, because that service has no
/file_parseroute at all -- every documentwould 404 and fall back. If the batch handshake is refused the parser falls back to
/file_parseonce, keeping a token-gated self-hosted deployment working. Endpointswithout a token keep the existing
/file_parsecontract untouched.Tests
tests/parse/test_pdf_strategy_dispatch.py, 26 cases:autoprefers the remote parser; does not call local on successautofalls back to local on a raised error, a timeout, and404/405/429/500/503autouses local directly when nothing is configured;localnever contacts remoteminerudoes not fall back/file_parse; a refused batchhandshake falls back
429/500are retried once,404is notresult,
pdf_strategy_effectiveis correct per document, and the degraded count isobservable
47 passedacross this file plustest_pdf_bookmark_extraction.py;ruff checkandruff format --checkclean.Verified against the live hosted endpoint:
strategy: autowith a token producedapi_mode=v4-batch, a realbatch_id, and markdown returned by the service.Docs
docs/{en,zh}/guides/01-configuration.md: strategy and protocol tables, theprovider-shaped predicate, the retry/circuit rules, and a behavior-change callout.
docs/{en,zh}/about/02-changelog.md: entries for all of the above under Unreleased.Scope
Self-contained on
main; depends on no other PR.tests/parse/has 123 pre-existingfailures on
mainunrelated to this change (confirmed against a clean checkout).