Refactor imports and enhance RAG pipeline

[nedeadinside]

This pull request introduces several significant enhancements and refactors to the RAG system, focusing on improved document chunking, retrieval, and verification workflows. The main changes include a complete overhaul of the DoclingClient to support asynchronous chunking and polling, updates to the Qdrant vector store integration for hybrid dense/sparse retrieval, and the introduction of a verification prompt and structured output support for LLM completions. Configuration files and data models have also been updated to support these new features and provide greater flexibility.

Docling and Chunking Improvements:

• Refactored DoclingClient to use asynchronous document submission, status polling, and result fetching, with support for timeouts, polling intervals, and conversion/chunking options. The client now handles task submission, polling, and result retrieval in separate methods, providing robust error handling and deadline management. (rag/src/clients/docling.py)
• Added new configuration options for docling conversion and chunking, such as OCR, heading hierarchy, image inclusion, max tokens, and peer merging. These are reflected in both the config file and the new data models. (config.yaml, rag/src/models/config.py, rag/src/models/__init__.py) [1] [2] [3]

Hybrid Dense/Sparse Retrieval with Qdrant:

• Updated Qdrant client and configuration to support hybrid dense (embedding) and sparse (BM25) vector retrieval, including reciprocal rank fusion (RRF) of results. The collection schema and search logic were updated to handle named vectors and prefetching for both modes. (rag/src/clients/qdrant.py, config.yaml) [1] [2] [3]

LLM Client Enhancements:

• Extended the LLM client to support structured output via Pydantic models, enabling schema-constrained LLM completions. This is used for downstream verification and other structured tasks. (rag/src/clients/llm.py) [1] [2] [3]

Verification Workflow:

• Introduced a new verification prompt for answer validation, instructing the LLM to determine if provided fragments fully answer the question. This is reflected in the prompts configuration and new model definitions. (prompts.yaml, rag/src/models/__init__.py) [1] [2]

Configuration and Import Refactoring:

• Updated imports throughout the codebase to use absolute paths from src, improving module clarity and maintainability. (rag/src/clients/embedder.py, rag/src/clients/reranker.py, rag/src/clients/webhook.py, rag/src/config.py, rag/src/clients/qdrant.py, etc.) [1] [2] [3] [4] [5] [6]

Configuration Enhancements:

• Increased job timeout and added new retrieval and webhook options (e.g., prefetch multiplier, verification flag) to support the new retrieval and verification workflows. (config.yaml) [1] [2]

These changes collectively modernize the RAG pipeline, enabling more accurate and robust document processing, retrieval, and answer validation.

---

Docling and Chunking Enhancements

• Refactored DoclingClient for async document submission, polling, and result fetching with deadline and error management.
• Added config and models for docling conversion/chunking options (OCR, heading hierarchy, max tokens, peer merging, etc.). [1] [2] [3]

Hybrid Retrieval and Qdrant Integration

• Enabled hybrid dense/sparse retrieval in Qdrant, including named vectors, BM25 config, and reciprocal rank fusion. [1] [2] [3]

LLM and Verification

• Added structured output support to LLM client for schema-constrained completions. [1] [2] [3]
• Introduced verification prompt and model for answer validation. [1] [2]

Configuration and Imports

• Standardized imports to use absolute src paths for clarity and maintainability. [1] [2] [3] [4] [5] [6]
• Increased job timeout and added retrieval/webhook options for new workflows. [1] [2]