Reliable RAG Agent Tutorial

A production-ready, traceable RAG (Retrieval-Augmented Generation) agent built on DigitalOcean Gradient for serverless inference and Briefcase AI for decision-state tracking. Every LLM call is automatically captured, enabling drift detection, deterministic replay, cost tracking, and reliability scoring — all deployed on DigitalOcean App Platform.

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Client / API Consumer                 │
└────────────────────────┬────────────────────────────────┘
                         │  HTTP (POST /ask, POST /drift, ...)
                         ▼
┌─────────────────────────────────────────────────────────┐
│          DigitalOcean App Platform (basic-xxs)          │
│  ┌───────────────────────────────────────────────────┐  │
│  │              FastAPI Service (port 8080)           │  │
│  │                                                   │  │
│  │  /ask ──► app.agent.ask_agent()                   │  │
│  │              │                                    │  │
│  │              ├── @capture (Briefcase AI)           │  │
│  │              │     └── Decision record ──► DB      │  │
│  │              │                                    │  │
│  │              └── Gradient Serverless Inference     │  │
│  │                    (OpenAI-compatible API)         │  │
│  │                                                   │  │
│  │  /replay ──► ReplayEngine (strict mode)           │  │
│  │  /drift  ──► DriftCalculator                      │  │
│  │  /cost   ──► CostCalculator                       │  │
│  │  /reliability ──► Aggregate scoring               │  │
│  └───────────────────────────────────────────────────┘  │
│                           │                             │
│                           ▼                             │
│              ┌─────────────────────┐                    │
│              │   PostgreSQL (dev)  │                    │
│              │   decisions-db      │                    │
│              └─────────────────────┘                    │
└─────────────────────────────────────────────────────────┘

Prerequisites

• DigitalOcean account with Gradient AI Platform access
• Python 3.9+
• doctl installed and authenticated (install guide)
• Gradient Model Access Key (from DO Control Panel → Agent Platform → Models → API Keys)

Quick Start (Local)

# Clone the repository
git clone https://github.com/briefcasebrain/reliable-agent-tutorial.git
cd reliable-agent-tutorial

# Set up environment
cp .env.example .env
# Edit .env and fill in your Gradient keys

# Install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Run the server
uvicorn app.main:app --host 0.0.0.0 --port 8080 --reload

# Test the health endpoint
curl http://localhost:8080/health
# → {"status": "ok"}

# Ask a question
curl -X POST http://localhost:8080/ask -H "Content-Type: application/json" -d '{"question": "What is compliance?", "context": "Compliance means adhering to rules."}'

Deploy to DigitalOcean

# Authenticate doctl
doctl auth init

# Validate the app spec
doctl apps spec validate .do/app.yaml

# Create the app (first deploy)
# This provisions the App Platform service + dev PostgreSQL database.
doctl apps create --spec .do/app.yaml

# Set secret environment variables
# After first deploy, set Gradient secrets via the control panel:
# Apps → reliable-rag-agent → Settings → Environment Variables
# Or via CLI:
doctl apps list --format ID,Spec.Name
doctl apps update <APP_ID> --spec .do/app.yaml
# Then set SECRET values through the control panel (CLI cannot set SECRET values directly).

# Trigger a new deployment (deploy_on_push: true means pushing to main auto-deploys)
doctl apps create-deployment <APP_ID>

# Check deployment status
doctl apps list-deployments <APP_ID> --format ID,Phase,Progress,Created

# View logs
doctl apps logs <APP_ID> --type run

# Get the app URL
doctl apps list --format DefaultIngress

Set Up Gradient Agent

The easiest way to create a Gradient agent is through the DO Control Panel → Agent Platform → Agents → Create Agent. Alternatively, use the DigitalOcean API directly:

# List available models
curl -s -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/models" | python3 -m json.tool

# Check available GenAI regions (currently only tor1)
curl -s -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/regions" | python3 -m json.tool

# Create an agent (region must be tor1)
curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/agents" \
  -d '{
    "name": "compliance-rag-agent",
    "model_uuid": "<MODEL_UUID>",
    "instruction": "Answer questions based on the provided compliance documents.",
    "description": "RAG agent for compliance Q&A",
    "project_id": "<YOUR_PROJECT_ID>",
    "region": "tor1"
  }'

# Create a model API key (for serverless inference → GRADIENT_MODEL_ACCESS_KEY)
curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/models/api_keys" \
  -d '{"name": "tutorial-serverless-key"}'

# Create an agent API key (for agent endpoint → GRADIENT_AGENT_ACCESS_KEY)
curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/gen-ai/agents/<AGENT_UUID>/api_keys" \
  -d '{"name": "tutorial-agent-key"}'

After creating the agent and keys, copy the values into your .env file:

• GRADIENT_MODEL_ACCESS_KEY → the secret_key from the model API key response
• GRADIENT_AGENT_ACCESS_KEY → the secret_key from the agent API key response
• GRADIENT_AGENT_ENDPOINT → the agent's endpoint URL (e.g. https://<id>.agents.do-ai.run)

API Reference

Method	Path	Request Body	Response
GET	/health	—	{"status": "ok"}
POST	/ask	{"question": str, "context": str, "model"?: str}	{"answer": str, "latency_ms": float, "tokens": int, "finish_reason": str}
GET	/replay/{snapshot_fingerprint}	—	{"snapshot_fingerprint": str, "result": object}
POST	/drift	{"outputs": [str, str, ...]}	{"consistency_score": float, "drift_score": float, "total_samples": int, "is_drifted": bool}
POST	/cost	{"model": str, "input_tokens": int, "output_tokens": int}	{"model": str, "input_tokens": int, "output_tokens": int, "total_cost": float, "currency": str}
GET	/reliability?limit=100	—	{"total_snapshots": int, "reliability_pct": float, "avg_consistency": float, "total_cost": float}

Default model for /ask and /cost is llama3.3-70b-instruct.

Configuration

Variable	Default	Description
GRADIENT_MODEL_ACCESS_KEY	—	Gradient serverless inference key (DO Control Panel → Agent Platform → Models → API Keys)
GRADIENT_AGENT_ACCESS_KEY	—	Gradient agent endpoint key (DO Control Panel → Agent Platform → Agents → Endpoint Keys)
GRADIENT_AGENT_ENDPOINT	—	Gradient agent endpoint URL
BRIEFCASE_STORAGE_BACKEND	sqlite	Storage backend: memory, sqlite, or postgresql
DATABASE_URL	—	PostgreSQL connection string (auto-injected by App Platform in production)
BRIEFCASE_ENABLE_OTEL	false	Enable OpenTelemetry tracing
LOG_LEVEL	info	Logging level: debug, info, warning, error

Cleanup

# Delete the app and associated resources
doctl apps delete <APP_ID> --force

Tutorial Article

This repository accompanies the DigitalOcean tutorial: Building a Reliable RAG Agent with Gradient and Briefcase AI.

License

MIT