Repository navigation
Conversation
The PDF parser assumed the legacy self-hosted contract: a single
POST /file_parse answering inline with md_content. The current
self-hosted /tasks API and the online batch API are task-based
(create -> poll -> zip download), so both made OpenViking fail:
the online service has no /file_parse endpoint at all (404) and no
/health endpoint for the startup preflight.
- add pdf.mineru_api_mode ("auto" | "sync" | "async", default "auto"):
auto detects the flavor from the first response, so existing v1
deployments keep working without config changes
- add pdf.mineru_token, sent as Authorization: Bearer on every call
- v2-tasks flow: POST /tasks -> poll status_url (pending/processing/
completed) -> download the result zip
- online-batch flow: POST /extract/task/batch -> poll
extract-results/batch/{batch_id} until state=done -> download
full_zip_url
- unpack the zip via safe_extract_zip, persist images/ into the media
store and rewrite markdown references (mirrors the v1 image path)
- skip the /health startup preflight for the async flavors (the
online service has no such endpoint)
- docs (en/zh): protocol table and online example
Fixes volcengine#4708
q2000s
force-pushed
the
feat/iss-4708-mineru-async-api
branch
from
October 2, 2026 03:50
73c4f40 to
9d9f672
Compare
ZaynJarvis
added
the
kernel
Author
|
Now, this PR has been compatible with v0.5 of OV. Please take a look. |
Contributor
|
At current head The fallback/async behavior also needs repository tests. This head changes 13 files, including deployment scripts and copied owner modules, while the PR description still presents the small strategy-only change. Splitting that deployment material and updating the description would make the protocol change easier to review. AI-assisted review. |
Adds a PDF parsing strategy that prefers MinerU and degrades to the local parser instead of failing the import. It sits between 'mineru' (hard dependency) and 'auto' (local first): local never calls MinerU mineru always calls MinerU, failure fails the import mineru_first calls MinerU first, failure falls back to local auto calls MinerU only after local fails Named miner_first rather than miner-first to match the snake_case convention used by the other enumerations in ov.conf. Because the strategy is defined as recoverable, MinerU problems never become startup problems: the preflight result is advisory and the fallback happens per document. The parser records the path actually taken in the result metadata as pdf_strategy_effective so operators can audit which converter handled a document after the fact. The strategy name is spelled mineru_first in configuration, ov.conf and all logs and metadata. Stacks on volcengine#4725, which already provides the task-based protocols, mineru_api_mode and mineru_token this change builds on.
q2000s
force-pushed
the
feat/iss-4708-mineru-async-api
branch
from
October 12, 2026 23:38
a206099 to
832fc43
Compare
main advanced 510 commits while this branch was stacked on volcengine#4725. Resolved a single import-order conflict in pdf.py (main added \ hreading\ for the PDFium render lock in volcengine#5952; this branch added \ empfile\), keeping both. No behavioural change from the resolution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
mineru_firstPDF parsing strategy: MinerU is preferred, and any failuredegrades to the local parser instead of failing the import.
strategymineru_endpointrequiredlocalminerumineru_firstautoBecause the strategy is defined as recoverable, MinerU problems never become
startup problems: the preflight result is advisory and the fallback happens per
document. The parser records the path actually taken as
pdf_strategy_effective(
mineruorlocal) so operators can audit which converter handled a document.Naming
Spelled
mineru_firstrather thanmineru-firstto match the snake_caseconvention used by the other enumerations in
ov.conf(e.g.semantic_and_vectors).This spelling is used in configuration,
ov.conf, logs and metadata.Stacked on #4725
This branch is based on
feat/iss-4708-mineru-async-apifrom #4725, which alreadyprovides the task-based protocols,
mineru_api_modeandmineru_token. This PRonly adds the
mineru_firststrategy on top of them and does not reimplement anyof that machinery.
Tests
Added to the existing
tests/parse/test_pdf_mineru.pyrather than a new file:test_mineru_first_uses_mineru_when_availabletest_mineru_first_falls_back_to_local_on_mineru_failure— the fallback pathtest_mineru_first_records_effective_strategytest_mineru_first_is_a_valid_strategytest_mineru_first_requires_endpoint19 passed;ruff checkandruff format --checkare clean.Docs
docs/{en,zh}/guides/01-configuration.md: addedmineru_firstto the strategytable plus a comparison table covering when to use it and how it differs from
mineruandauto.