Multi-Agent System with RAG

Multi-agent system with RAG capabilities, fine-tuning pipeline, and monitoring.

Features

• Multi-agent orchestration (RAG, simple, multi-agent workflows)
• RAG pipeline with document processing (PDF/HTML/Markdown), embeddings, vector search
• LLM integration: Ollama, HuggingFace, OpenAI
• PEFT/LoRA fine-tuning pipeline
• Automated evaluation metrics + human evaluation UI
• Prometheus + Grafana monitoring
• Data connectors: REST API, SQL, CSV, Google Drive, Confluence

Quick Start

Prerequisites: Docker, Docker Compose, Python 3.11+, Node.js 18+

git clone <repo>
cd multi-agent-system
cp .env.example .env
docker compose up -d

Access:

• Frontend: http://localhost:3000
• API: http://localhost:8000/docs
• Grafana: http://localhost:3001

API Endpoints

Agents: POST /v1/agents, GET /v1/agents, GET /v1/agents/{id}, DELETE /v1/agents/{id}

Queries: POST /v1/queries/{agent_id}, GET /v1/queries/history/{agent_id}

Documents: POST /v1/documents/upload, POST /v1/documents/{id}/index, GET /v1/documents, DELETE /v1/documents/{id}

Training: POST /v1/training/examples/generate, POST /v1/training/finetune, POST /v1/training/evaluate

Metrics: GET /v1/metrics/overview, GET /v1/metrics/latency-distribution

Usage Example

import requests

agent = {
    "name": "support-agent",
    "agent_type": "rag",
    "model_name": "microsoft/phi-2",
    "config": {"collection": "docs"}
}
requests.post("http://localhost:8000/v1/agents", json=agent)

with open("doc.pdf", "rb") as f:
    upload = requests.post("http://localhost:8000/v1/documents/upload", files={"file": f})
    doc_id = upload.json()["id"]

requests.post(f"http://localhost:8000/v1/documents/{doc_id}/index?collection_name=docs")

query = {"query": "How do I reset my password?", "stream": False}
response = requests.post(f"http://localhost:8000/v1/queries/{agent_id}", json=query)
print(response.json()["response"])

Tech Stack

Backend: FastAPI, SQLAlchemy, Postgres, LangChain, HuggingFace, PEFT/LoRA, Qdrant, FAISS, Redis

Frontend: React 18, Vite, TailwindCSS

Infrastructure: Docker, Prometheus, Grafana, MinIO

Development

cd backend && python -m venv venv && source venv/bin/activate && pip install -r requirements.txt
uvicorn app.main:app --reload

cd frontend && npm install && npm run dev

License

MIT

Features

• Multi-Agent Orchestration: RAG, simple, and multi-agent workflows
• RAG Pipeline: Document parsing (PDF/HTML/Markdown), chunking, embeddings, vector search
• LLM Integration: Ollama (local models), HuggingFace models + OpenAI API support
• Fine-tuning: PEFT/LoRA training pipeline
• Evaluation: Automated metrics (precision, recall, MRR, NDCG) + human evaluation UI
• Monitoring: Prometheus + Grafana dashboards
• Data Connectors: REST API, SQL, CSV, Google Drive, Confluence

Quick Start

Prerequisites

• Docker & Docker Compose
• Python 3.11+
• Node.js 18+

Installation

1. Clone and setup:

git clone <repo>
cd multi-agent-system
cp .env.example .env

2. Start services:

docker compose up -d

3. Setup Ollama with DeepSeek R1 1.5B (recommended):

chmod +x scripts/setup_ollama.sh
./scripts/setup_ollama.sh

Or use HuggingFace/OpenAI - see OLLAMA_SETUP.md for configuration.

4. Access:

• Frontend: http://localhost:3000
• API: http://localhost:8000
• API Docs: http://localhost:8000/docs
• Grafana: http://localhost:3001 (admin/admin)
• Prometheus: http://localhost:9090
• Ollama API: http://localhost:11434

Development

Backend

cd backend
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload

Frontend

cd frontend
npm install
npm run dev

API Endpoints

Agents

• POST /v1/agents - Create agent
• GET /v1/agents - List agents
• GET /v1/agents/{id} - Get agent
• DELETE /v1/agents/{id} - Delete agent

Queries

• POST /v1/queries/{agent_id} - Execute query
• GET /v1/queries/history/{agent_id} - Query history

Documents

• POST /v1/documents/upload - Upload document
• POST /v1/documents/{id}/index - Index document
• GET /v1/documents - List documents
• DELETE /v1/documents/{id} - Delete document

Training

• POST /v1/training/examples/generate - Generate training examples
• POST /v1/training/finetune - Start fine-tuning
• POST /v1/training/evaluate - Evaluate response
• GET /v1/training/examples - List training examples

Metrics

• GET /v1/metrics/overview - Metrics overview
• GET /v1/metrics/latency-distribution - Latency stats

Usage Examples

Create RAG Agent

import requests

agent = {
    "name": "support-agent",
    "description": "Technical support assistant",
    "agent_type": "rag",
    "model_name": "microsoft/phi-2",
    "config": {"collection": "docs"}
}

response = requests.post("http://localhost:8000/v1/agents", json=agent)
print(response.json())

Upload and Index Document

with open("manual.pdf", "rb") as f:
    files = {"file": f}
    upload = requests.post("http://localhost:8000/v1/documents/upload", files=files)
    doc_id = upload.json()["id"]

requests.post(f"http://localhost:8000/v1/documents/{doc_id}/index?collection_name=docs")

Query Agent

query = {
    "query": "How do I reset my password?",
    "stream": False
}

response = requests.post(f"http://localhost:8000/v1/queries/{agent_id}", json=query)
print(response.json()["response"])

Training Pipeline

1. Collect evaluations through UI
2. Generate training examples:

curl -X POST "http://localhost:8000/v1/training/examples/generate?min_rating=4"

3. Start fine-tuning:

curl -X POST "http://localhost:8000/v1/training/finetune?model_name=custom&num_examples=200"

Data Connectors

REST API

from app.services.connectors import ConnectorFactory

connector = ConnectorFactory.create(
    "rest",
    url="https://api.example.com/data",
    headers={"Authorization": "Bearer token"}
)
data = await connector.fetch()

SQL Database

connector = ConnectorFactory.create(
    "sql",
    connection_string="postgresql://user:pass@host/db",
    query="SELECT * FROM support_tickets"
)
data = await connector.fetch()

Monitoring

Access Grafana at http://localhost:3001:

• Query latency (p50, p95, p99)
• Request volume
• Error rates
• Evaluation metrics
• Active agents

Tech Stack

Backend:

• FastAPI
• SQLAlchemy + Postgres
• LangChain
• HuggingFace Transformers
• PEFT/LoRA
• Qdrant + FAISS
• Redis

Frontend:

• React 18
• Vite
• TailwindCSS
• Recharts

Infrastructure:

• Docker Compose
• Prometheus
• Grafana
• MinIO

Project Structure

multi-agent-system/
├── backend/
│   ├── app/
│   │   ├── api/v1/          # API endpoints
│   │   ├── core/            # Config, database
│   │   ├── models/          # SQLAlchemy models
│   │   ├── schemas/         # Pydantic schemas
│   │   └── services/        # Business logic
│   ├── requirements.txt
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── api/             # API client
│   │   ├── pages/           # React pages
│   │   └── App.jsx
│   ├── package.json
│   └── Dockerfile
├── monitoring/
│   ├── prometheus.yml
│   └── grafana/
├── docker-compose.yml
└── README.md

Production Deployment

Kubernetes

kubectl apply -f k8s/

Environment Variables

See .env.example for configuration options.

Testing

cd backend
pytest