Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions build/agents/build-your-agent/evals.mdx
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
---
title: 'Evals'
sidebarTitle: 'Evals'

Check warning on line 3 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L3

Did you really mean 'Evals'?
description: 'Test and evaluate your AI Agents with scenario-based evaluations and reusable Checks'
---

<Info>
**Rollout Status**: Evals is rolling out progressively, starting with Enterprise customers. If you don't see this feature in your account yet, reach out to your account manager to discuss access.

Check warning on line 8 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L8

Did you really mean 'Evals'?
</Info>

The Evals section is your command center for testing and evaluating AI Agent performance. Located in the **Evaluate** tab (next to the Build and Use tabs) in the Agent builder, Evals lets you create test sets, define reusable Checks, run automated evaluations, and monitor live Agent quality — all without manual testing.

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

![Evaluate tab showing the Evals sidebar (Test, Runs, Checks, Publish, Monitor) and a Monitor dashboard with overall score, total runs, and Checks breakdown](/images/agent/agent-evals.png)

Check warning on line 13 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L13

Did you really mean 'Evals'?

## What you can do with Evals

Check warning on line 15 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L15

Did you really mean 'Evals'?

<CardGroup cols={3}>
<Card title="Run tests" icon="flask-vial">
Expand All @@ -28,11 +28,11 @@

---

## Evals sections

Check warning on line 31 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L31

Did you really mean 'Evals'?

The Evals area has five sections, shown in the left sidebar of the Evaluate tab:

Check warning on line 33 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L33

Did you really mean 'Evals'?

- **Test** — Create and manage test sets. Each test set holds scenarios that simulate users; running a scenario produces a conversation with your Agent that gets scored by attached Checks.

Check warning on line 35 in build/agents/build-your-agent/evals.mdx

View workflow job for this annotation

GitHub Actions / Documentation Lint Checks

5 settings listed as bullet points — consider using a table instead so they're easier to scan. [technical: 5 consecutive bullet items matching **Key**: value or **Key** — value pattern]
- **Runs** — Past evaluation run results. Browse average scores, tasks evaluated, progress status, credit spend, and creation date for every run.
- **Checks** — The reusable set of evaluation criteria. Create a Check once, then attach it to scenarios, to Monitor dashboards, or to one-off evaluations of completed tasks.
- **Publish** — Choose which test sets must pass before your Agent can be published. Set a minimum pass rate and optionally block publishing on failure.
Expand Down Expand Up @@ -89,6 +89,23 @@
| **Tool** | Select the tool to check for |
| **Position** | Whether the tool was used anywhere, used first, or used last |
| **Comparison** | Check if the tool was used at least, exactly, or at most X times |

#### Argument conditions

You can add conditions to verify the specific arguments passed to tool calls, not just whether the tool was called. Each condition specifies an argument path and a match type, and all conditions must be satisfied for the check to pass.

| Match type | Description |
|------------|-------------|
| **Exact match** | The argument value must equal the specified value. Optionally enable case-insensitive matching to ignore capitalization. |
| **Contains** | The argument value must include the specified substring. |
| **Regex** | The argument value must match the specified regular expression pattern. |
| **Not empty** | The argument must have any non-empty value. |

To add conditions, click **+ Add condition** in the **Conditions** section of the eval rule form. For each condition, specify the argument path and the value or pattern to match against. A plain-English preview of your conditions appears in the form.

<Info>
Argument conditions work for both single-agent evals and workforce (node-scoped) evals. This feature is currently behind the `evals-toolusage-argument-matcher` feature flag.
</Info>
</Accordion>
</AccordionGroup>

Expand Down Expand Up @@ -284,7 +301,7 @@

You also get:

- **Overall score timeseries** to spot regressions or improvements over time.

Check warning on line 304 in build/agents/build-your-agent/evals.mdx

View workflow job for this annotation

GitHub Actions / Documentation Lint Checks

4 features listed as bullet points — consider using cards instead so they stand out visually. [technical: 4 consecutive bullet items matching **Feature** pattern, use <CardGroup> with <Card> components]
- **Per-Check charts** so you can see which criteria are slipping.
- **Version markers** that line up score changes with Agent publishes.
- **A list of evaluation runs** with score, name, and a drill-in to the full conversation.
Expand Down
Loading