Internal Developer Assistant - AgentGPA Demo

AI Agent evaluation using the Goal / Plan / Act framework with Snowflake Cortex LLM-as-Judge.

Demo Story

An LLM-based "Internal Developer Assistant" with 3 tools is evaluated in two versions:

• v1 — Weak prompts + incomplete knowledge base → hallucination on Case 3
• v2 — Grounded prompts + expanded docs → correct answers

Case	Query	Expected	v1 Issue
1	"What is 25% of 800?"	200	None (baseline)
2	"API handles 50 req/s, how many in 30 min?"	90,000	None (baseline)
3	"What is the Python indentation rule in our codebase?"	2-space indent	Hallucination: PEP8 (4 spaces)

v1 → v2 Improvement (Feedback Loop)

Problem: v1 Case 3 gets Goal=0.0, Act=0.0 because:

1. Internal coding standards doc is missing from knowledge base
2. Prompt says "enrich with industry best practices" → LLM answers PEP8 (4 spaces)

Fix (two axes):

1. Data: Added doc-006: Python Coding Standards (2-space indent rule)
2. Prompt: Changed from "enrich with best practices" → "answer ONLY from context"

Result: Case 3 GPA improved from 0.33 → 0.83.

AgentGPA Framework

Dimension	Measures	Evaluation Method	AgentGPA Formal Metrics
Goal	User's intent achieved?	LLM judge: answer vs expected	1A Answer Correctness, 1B Answer Relevance
Plan	Right tool selected?	LLM judge: tool appropriateness	4B Tool Selection
Act	Faithful to source data?	LLM judge: groundedness	1C Groundedness

Note: This demo uses a simplified 3-dimension evaluation (Goal/Plan/Act). The full AgentGPA framework defines 8 metrics: 1A Answer Correctness, 1B Answer Relevance, 1C Groundedness, 2 Logical Consistency, 3 Execution Efficiency, 4A Plan Quality, 4B Tool Selection, 5A Plan Adherence, 5B Tool Calling.

All metrics scored 0.0–1.0 by Snowflake Cortex (llama3.1-70b) as LLM-as-Judge.

Architecture

Developer Query → LLM Router (Tool Selection Prompt)
                     ├─ technical → documentation_search → LLM Answer Generator → Response
                     ├─ policy    → hr_policy_search    → LLM Answer Generator → Response
                     └─ math      → calculator          → Response (bypasses LLM)

• LLM Router and Answer Generator use an OpenAI-compatible REST API via src/llm_client.py
• Default provider: Snowflake Cortex REST API (llama3.1-70b) with Programmatic Access Token (PAT)
• v1/v2 behavior controlled by different prompts (src/prompts_v1.py, src/prompts_v2.py)
• v1 uses data/documentation.json, v2 uses data/documentation_v2.json (with coding standards)

Prerequisites

• Python 3.11+
• uv package manager
• Snowflake account with Cortex access (or OpenAI / Anthropic API key)

Setup

git clone <repo-url>
cd demo-development-tool-agent

uv sync

cp .env.example .env
# Edit .env with your credentials (see LLM Provider section below)

LLM Provider Configuration

Set LLM_PROVIDER in .env to select a provider. Default is cortex.

Snowflake Cortex (default)

Requires a Programmatic Access Token (PAT).

LLM_PROVIDER=cortex
LLM_MODEL=llama3.1-70b
SNOWFLAKE_ACCOUNT=<account-identifier>
SNOWFLAKE_TOKEN=<your-pat>

Alternatively, set SNOWFLAKE_CONNECTION_NAME to use a profile from ~/.snowflake/connections.toml.

OpenAI

LLM_PROVIDER=openai
LLM_MODEL=gpt-4o
OPENAI_API_KEY=sk-...

Also works with any OpenAI-compatible endpoint (Azure OpenAI, etc.) by setting OPENAI_BASE_URL.

Anthropic

LLM_PROVIDER=anthropic
LLM_MODEL=claude-sonnet-4-20250514
ANTHROPIC_API_KEY=sk-ant-...

See .env.example for all supported providers (AWS Bedrock, Google Vertex AI, HuggingFace).

1. Run the AI Agent

uv run python main.py

Programmatic usage:

from src.agent import InternalDeveloperAssistant

agent = InternalDeveloperAssistant(version="v2")
response = agent.run("What is the Python indentation rule in our codebase?")
# response.answer → "2-space indentation..."
# response.tool_used → "documentation_search"

2. Run Evaluation

Runs all 3 test cases for v1 and v2, scores with Cortex LLM-as-Judge:

uv run python evaluation/trulens_eval.py

Results saved to evaluation/trulens_results.json.

3. Launch Streamlit Dashboard

uv run streamlit run app.py

Opens at http://localhost:8501 and displays:

• Agent Architecture — Mermaid diagram of tool routing
• Section 1 — v1 prompts + scores
• Section 2 — v2 improvements (diff view) + scores
• Section 3 — v1 vs v2 comparison chart (Goal/Plan/Act color-coded)

Project Structure

├── src/
│   ├── agent.py           # LLM-based agent with tool selection + answer gen
│   ├── llm_client.py      # Unified LLM client (Cortex REST / OpenAI / Anthropic)
│   ├── prompts_v1.py      # v1 prompts (weak: "enrich with best practices")
│   ├── prompts_v2.py      # v2 prompts (grounded: "ONLY from context")
│   └── tools.py           # Tool implementations (doc search, HR search, calculator)
├── data/
│   ├── documentation.json      # v1 docs (no coding standards)
│   ├── documentation_v2.json   # v2 docs (includes Python coding standards)
│   └── hr_policies.json        # HR policies corpus
├── evaluation/
│   ├── trulens_eval.py         # AgentGPA evaluation (Cortex LLM-as-Judge)
│   └── trulens_results.json    # Pre-computed evaluation results
├── app.py                 # Streamlit dashboard
├── main.py                # Quick interactive demo
├── .env.example           # Environment variable template
├── pyproject.toml         # Dependencies + ruff config (Google style)
└── .python-version        # Python 3.11

Linting

uv run ruff check .
uv run ruff format .

Key Takeaway

With 3 test cases and an internal developer assistant, AgentGPA evaluates agent quality on Goal (intent achieved), Plan (right tool), and Act (faithful to source). The v1→v2 feedback loop demonstrates how prompt engineering + knowledge base expansion eliminates hallucination.