Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions build/agents/build-your-agent/evals.mdx
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
---
title: 'Evals'
sidebarTitle: 'Evals'

Check warning on line 3 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L3

Did you really mean 'Evals'?
description: 'Test and evaluate your AI Agents with scenario-based evaluations and reusable Checks'
---

<Info>
**Rollout Status**: Evals is rolling out progressively, starting with Enterprise customers. If you don't see this feature in your account yet, reach out to your account manager to discuss access.

Check warning on line 8 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L8

Did you really mean 'Evals'?
</Info>

The Evals section is your command center for testing and evaluating AI Agent performance. Located in the **Evaluate** tab (next to the Build and Use tabs) in the Agent builder, Evals lets you create test sets, define reusable Checks, run automated evaluations, and monitor live Agent quality — all without manual testing. Evals apply to both individual Agents and Workforces, including the sub-agents and tools inside a Workforce.

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

![Evaluate tab showing the Evals sidebar (Test, Runs, Checks, Publish, Monitor) and a Monitor dashboard with overall score, total runs, and Checks breakdown](/images/agent/agent-evals.png)

Check warning on line 13 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L13

Did you really mean 'Evals'?

## What you can do with Evals

Check warning on line 15 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L15

Did you really mean 'Evals'?

<CardGroup cols={3}>
<Card title="Run tests" icon="flask-vial">
Expand All @@ -28,11 +28,11 @@

---

## Evals sections

Check warning on line 31 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L31

Did you really mean 'Evals'?

The Evals area has five sections, shown in the left sidebar of the Evaluate tab:

Check warning on line 33 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L33

Did you really mean 'Evals'?

- **Test** — Create and manage test sets. Each test set holds scenarios that simulate users; running a scenario produces a conversation with your Agent that gets scored by attached Checks.

Check warning on line 35 in build/agents/build-your-agent/evals.mdx

View workflow job for this annotation

GitHub Actions / Documentation Lint Checks

5 settings listed as bullet points — consider using a table instead so they're easier to scan. [technical: 5 consecutive bullet items matching **Key**: value or **Key** — value pattern]
- **Runs** — Past evaluation run results. Browse average scores, tasks evaluated, progress status, cost (Credits and Actions), and creation date for every run.
- **Checks** — The reusable set of evaluation criteria. Create a Check once, then attach it to scenarios, to Monitor dashboards, or to one-off evaluations of completed tasks.
- **Publish** — Choose which test sets must pass before your Agent can be published. Set a minimum pass rate and optionally block publishing on failure.
Expand Down Expand Up @@ -216,8 +216,23 @@
You can select specific scenarios within a test set to run a subset at once, or run all scenarios in the test set together. Note that you cannot bulk-select and run multiple test sets at the same time.

1. Enter a name for the run (e.g., "Scenario run - Jan 14, 12:14 PM"). A default name with timestamp is provided.
2. Checks already attached to the scenarios are always included. To add Checks from the Checks tab, select the ones you want under **Additional global checks**.
3. Click **Run** to begin. The simulator generates conversations with your Agent based on your scenario prompts and the selected Checks score each conversation.
2. Select which version to evaluate from the **Version** dropdown. It defaults to the current active version. Select any historical or draft version to test against instead. This applies to both individual agents and workforces.
3. Checks already attached to the scenarios are always included. To add Checks from the Checks tab, select the ones you want under **Additional global checks**.
4. Click **Run** to begin. The simulator generates conversations with your Agent based on your scenario prompts and the selected Checks score each conversation.

### Running evaluations via MCP

You can trigger evaluation runs programmatically using the `relevance_run_evaluation` MCP tool available through the [Relevance AI MCP server](/integrations/mcp/programmatic-gtm/mcp-server). This mirrors the version picker available in the run modals.

| Parameter | Required | Description |
|-----------|----------|-------------|
| `agent_id` | Yes | The ID of the agent or workforce to evaluate |
| `test_set_id` | Yes | The ID of the test set to run |
| `version_id` | No | A specific agent version ID to evaluate against. Omit to use the current active version |

<Note>
Workforce evaluations now correctly run against the version specified — either via the Version dropdown in the run modal or the `version_id` parameter via MCP. Previously, workforce evaluations always used the active published version regardless of the selection.
</Note>

---

Expand All @@ -233,7 +248,7 @@
|--------|-------------|
| **Average Score** | Overall pass rate across all scenarios and Checks |
| **Tasks** | How many Agent tasks were evaluated |
| **Agent Version** | The version of the Agent that was tested |
| **Agent Version** | The version of the agent or workforce that was evaluated, as selected in the run modal |

### Scenario results

Expand Down Expand Up @@ -290,7 +305,7 @@

You also get:

- **Overall score timeseries** to spot regressions or improvements over time.

Check warning on line 308 in build/agents/build-your-agent/evals.mdx

View workflow job for this annotation

GitHub Actions / Documentation Lint Checks

4 features listed as bullet points — consider using cards instead so they stand out visually. [technical: 4 consecutive bullet items matching **Feature** pattern, use <CardGroup> with <Card> components]
- **Per-Check charts** so you can see which criteria are slipping.
- **Version markers** that line up score changes with Agent publishes.
- **A list of evaluation runs** with score, name, and a drill-in to the full conversation.
Expand Down Expand Up @@ -389,7 +404,7 @@
</Accordion>

<Accordion title="Can I rerun a previous evaluation?">
Yes, you can run the same scenarios again at any time. Each run is saved in your Runs history, allowing you to compare results across different Agent versions.
Yes, you can run the same scenarios again at any time. Each run is saved in your Runs history. You can also target a specific historical or draft version using the **Version** dropdown in the run modal, letting you compare results across different agent versions directly.
</Accordion>

<Accordion title="Where do my Checks live?">
Expand Down
13 changes: 13 additions & 0 deletions integrations/mcp/programmatic-gtm/mcp-server.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
</Accordion>

<Accordion title="ChatGPT" icon="comment">
ChatGPT supports MCP servers through Developer Mode, available on Pro, Team, Enterprise, and Edu plans.

Check warning on line 42 in integrations/mcp/programmatic-gtm/mcp-server.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

integrations/mcp/programmatic-gtm/mcp-server.mdx#L42

Did you really mean 'Edu'?

1. Open ChatGPT **Settings**
2. Go to **Connectors** → **Advanced** → **Developer Mode**
Expand Down Expand Up @@ -242,6 +242,19 @@

---

## Available tools

The Relevance AI MCP server exposes the following tools to connected AI clients:

| Tool | Description |
|------|-------------|
| `relevance_trigger_agent` | Triggers an agent synchronously and waits up to 120 seconds for the result. |
| `relevance_trigger_agent_async` | Triggers an agent and returns immediately with a `conversation_id` for polling. Use for long-running agents. |
| `relevance_poll_agent_result` | Polls for the result of an async agent run using the `conversation_id`. |
| `relevance_run_evaluation` | Runs an evaluation test set against an agent or workforce. Accepts `agent_id` (required), `test_set_id` (required), and an optional `version_id` to evaluate a specific historical or draft version instead of the current active version. |

---

## Handling long-running agent executions

When triggering agents via MCP, you have two execution modes available depending on how long your agent takes to complete.
Expand Down
Loading