Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
f051d10
Merge pull request #689 from future-agi/docs/agentcc-video
khushalsonawat Jun 25, 2026
00a4f2b
Merge pull request #691 from future-agi/docs/agentcc-video
khushalsonawat Jun 25, 2026
f9e2a0a
Merge pull request #692 from future-agi/dev
khushalsonawat Jun 26, 2026
1b2f18e
Merge pull request #699 from future-agi/dev
khushalsonawat Jul 1, 2026
5399406
Merge pull request #708 from future-agi/dev
khushalsonawat Jul 6, 2026
889c8b2
Merge pull request #718 from future-agi/dev
nik13 Jul 7, 2026
6185080
Merge pull request #738 from future-agi/dev
khushalsonawat Jul 10, 2026
9662993
Merge pull request #756 from future-agi/dev
khushalsonawat Jul 16, 2026
15a2586
Merge pull request #761 from future-agi/dev
khushalsonawat Jul 16, 2026
07a43fe
Merge pull request #766 from future-agi/dev
khushalsonawat Jul 16, 2026
03d9fed
Merge pull request #768 from future-agi/dev
khushalsonawat Jul 16, 2026
99f71e7
docs(changelog): add automation insertion marker
Jul 23, 2026
f591791
ci(changelog): add release transform script with tests
Jul 23, 2026
2bdd45a
ci(changelog): open changelog PR on platform-release dispatch
Jul 23, 2026
4fd5ce2
docs(changelog): remove placeholder version entries
Jul 23, 2026
9f0d647
fix(changelog): escape MDX-significant chars in release body
Jul 23, 2026
d418859
fix(release-notes): target docs/release-notes page, not the orphan ch…
atharva-bhange Jul 24, 2026
0558d1f
docs(changelog): restore changelog.mdx untouched
atharva-bhange Jul 24, 2026
7d9368a
Merge pull request #772 from future-agi/chore/changelog-sync
atharva-bhange Jul 28, 2026
1a11d8d
ci(changelog): open the release-notes PR with the App token, not GITH…
atharva-bhange Jul 28, 2026
bbddf1b
Merge pull request #774 from future-agi/chore/changelog-sync-app-token
atharva-bhange Jul 28, 2026
19aea5f
Merge pull request #800 from future-agi/dev
khushalsonawat Aug 10, 2026
55f0537
Merge pull request #808 from future-agi/dev
khushalsonawat Aug 13, 2026
99765bc
Merge pull request #811 from future-agi/dev
khushalsonawat Aug 14, 2026
d994467
Merge pull request #813 from future-agi/dev
khushalsonawat Aug 14, 2026
782e936
Merge pull request #815 from future-agi/dev
khushalsonawat Aug 14, 2026
42c1791
Merge pull request #818 from future-agi/dev
khushalsonawat Aug 17, 2026
8614c25
Merge pull request #820 from future-agi/dev
khushalsonawat Aug 17, 2026
19d4a87
Merge pull request #823 from future-agi/dev
khushalsonawat Aug 18, 2026
83f8d95
Merge pull request #831 from future-agi/dev
khushalsonawat Aug 19, 2026
f468f3b
docs(release-notes): add weeks of Jul 22 to Aug 10, 2026
Aug 25, 2026
81c1a5f
Merge pull request #833 from future-agi/docs/release-notes-2026-07-to-08
khushalsonawat Aug 25, 2026
aaacc14
Merge pull request #836 from future-agi/dev
khushalsonawat Aug 25, 2026
120d9af
Merge pull request #838 from future-agi/dev
khushalsonawat Aug 27, 2026
d3beece
Merge pull request #848 from future-agi/dev
khushalsonawat Sep 7, 2026
8abe575
docs: add SRE root cause accuracy evaluator page
NivedhN160 Sep 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/lib/navigation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,7 @@ export const tabNavigation: NavTab[] = [
{ title: 'Conversation Hallucination', href: '/docs/evaluation/builtin/conversation-hallucination' },
{ title: 'Tool Call Accuracy', href: '/docs/evaluation/builtin/tool-call-accuracy' },
{ title: 'Trajectory Match', href: '/docs/evaluation/builtin/trajectory-match' },
{ title: 'SRE Root Cause Accuracy', href: '/docs/evaluation/builtin/sre-root-cause-accuracy' },
{ title: 'Step Count', href: '/docs/evaluation/builtin/step-count' },
]
},
Expand Down
1 change: 1 addition & 0 deletions src/pages/docs/evaluation/builtin/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ Multi-turn behavior: coherence, goal completion, and how a customer-facing agent
| [**Customer Agent: Task Completion**](/docs/evaluation/builtin/customer-agent-task-completion) | Checks whether an agent fully resolved the customer's request, including valid policy-based refusals. | `agent_prompt`, `conversation` | Conversation, Chat | LLM as Judge |
| [**Tool Call Accuracy**](/docs/evaluation/builtin/tool-call-accuracy) | Compares an agent's actual tool calls against expected calls, scoring matches on function name and arguments. | `output`, `expected` | Agents, Tool Use | Code |
| [**Trajectory Match**](/docs/evaluation/builtin/trajectory-match) | Compares an agent's actual action sequence against an expected trajectory using configurable matching modes. | `output`, `expected` | Agents, Tool Use | Code |
| [**SRE Root Cause Accuracy**](/docs/evaluation/builtin/sre-root-cause-accuracy) | Scores an AI SRE or investigation agent's final diagnosis against evidence such as logs, metrics, traces, and runbooks. | `diagnosis`, `context` | Agents, Root Cause Analysis | LLM as Judge |
| [**Step Count**](/docs/evaluation/builtin/step-count) | Validates the number of steps in an agent trajectory against an exact count or a min/max range. | `output` | Agents | Code |

## Output quality & format
Expand Down
72 changes: 72 additions & 0 deletions src/pages/docs/evaluation/builtin/sre-root-cause-accuracy.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: "SRE Root Cause Accuracy"
description: "Scores an AI SRE or investigation agent's final diagnosis against evidence such as logs, metrics, traces, and runbooks."
---

SRE Root Cause Accuracy checks whether an AI SRE or investigation agent correctly identifies the root cause of an incident based on the provided evidence.

## What it does

This evaluator uses an LLM-as-a-judge to score the agent's diagnosis on a scale from 0.0 to 1.0. It penalizes hallucinated causes, claims that lack evidence, and destructive remediation suggestions.

### Input

| Required Input | Type | Description |
| --- | --- | --- |
| `diagnosis` | `string` | The agent's final root-cause diagnosis. |
| `context` | `string` \| `array` | The evidence gathered during the incident (logs, metrics, traces, runbooks). |
| `trajectory` | `string` \| `array` | (Optional) The agent's tool-use trajectory. |

### Output

| Field | Type | Description |
| --- | --- | --- |
| Result | `score` | A numeric score between 0.0 and 1.0. Higher scores indicate the agent's diagnosis correctly identified the root cause and is grounded in the provided context. |
| Reason | `string` | A plain-language explanation of the score. |

### Run it from code

Call `evaluate()` with the template name and the eval's required inputs. It returns the score and the reason.

<Note>
Before running: [install the SDK and set `FI_API_KEY` / `FI_SECRET_KEY`](/docs/evaluation/reference/sdk-api).
</Note>

<CodeGroup>

```python Python
from fi.evals import evaluate

result = evaluate(
"sre_root_cause_accuracy",
diagnosis="The database went down due to an OutOfMemory error.",
context="Logs show OOMKilled at 10:05 PM for the db-service container.",
)

print(result.score)
print(result.reason)
```

```typescript TypeScript
import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
"sre_root_cause_accuracy",
{
diagnosis: "The database went down due to an OutOfMemory error.",
context: "Logs show OOMKilled at 10:05 PM for the db-service container.",
}
);

console.log(result);
```

</CodeGroup>

## When to use

Run SRE Root Cause Accuracy when you are building AI agents that investigate systems, diagnose outages, or suggest remediations based on telemetry and logs.

## What to do when SRE Root Cause Accuracy is low

Read the reason string to identify whether the agent is hallucinating causes or failing to ground its claims in the provided logs and metrics. If the agent frequently invents causes, consider providing stricter system prompts requiring citations for every claim.
Loading