feat(rag): add ViDoRe V3 agentic RAG evaluation with reasoning-based page selectors - #43
Merged
Conversation
Contributor
|
✅ DCO Check Passed Thanks @ceberam, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
ceberam
force-pushed
the
dev/vidore-eval
branch
from
August 12, 2026 10:50
41fc568 to
c45f75d
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
ceberam
force-pushed
the
dev/vidore-eval
branch
from
August 14, 2026 17:44
a0fdb6d to
6ca47ca
Compare
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
numpy 2.5.0 ships stubs that require Python 3.12 syntax,it can be installed into a 3.12 environment that is being type-checked against a 3.10 language target. Constraint numpy to '<2.5.0' for mypy compatibility checks. Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
ceberam
force-pushed
the
dev/vidore-eval
branch
from
August 17, 2026 08:58
6ca47ca to
4e4cae2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces an end-to-end evaluation pipeline for a novel chunkless, reasoning-driven RAG approach on the
ViDoRe V3 benchmark. Rather than splitting documents into fixed-size chunks and relying on vector similarity, the approach preserves the
DoclingDocumentstructure, enriches it with AI-generated summaries, and uses an LLM to navigate the document hierarchy and select the most relevant pages per query.What's new
Enricher — dual summarisation styles
DoclingEnrichingAgent._summarize_pagesgains astyleparameter ("sentences"/"keyphrases"). When"keyphrases"is chosen, page-level enrichment stores keyphrase lists inmeta.keywordsinstead of prose summaries inmeta.summary. Resume logic and the skip-if-already-enriched guard both respect the chosen style.Two page-selector strategies
ReasoningBasedPageSelector— evaluates pages in iterative, sliding-window batches; works with both page-level and element-level enrichment. Includes multi-document support viaselect_relevant_documents, per-documentselect_pages, and cross-documentrerank_across_documents.TreeGuidedPageSelector— traverses the heading hierarchy top-down, drilling into the most promising sections at each step; requires element-level enrichment.Both selectors are exported from
docling_agent.agents.perfs/evaluation harnessA four-step pipeline script and YAML config template cover:
DoclingEditingAgentranxThe pipeline is fully resumable — Step 3 checkpoints after every page and Step 4 flushes results after every query. The
perfs/README is updated to document both the existing extraction-quality evaluator and the new RAG evaluator side by side.