Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/pages/docs/observe/guides/setup-evals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,20 @@ The name autofills. For a built-in eval the instructions and output type are pre
- **Error localization** (optional): tick it to pinpoint which part of the input caused a failure
- **Variable mapping** (required): map each of the eval's inputs to your data. The choices are the attributes your project actually carries at the level you picked: span attribute keys for **Spans**, trace fields (`input`, `output`, `metadata`, and more) plus paths into each span for **Traces**, and session fields plus paths into their traces and spans for **Sessions**. Here, map the eval's `output` to the trace's `output`

### Map nested JSON attributes

Use dot notation to map a nested object field. For arrays, use the numeric item position as a path segment, such as `.0` or `.1`; bracket notation such as `[0]` is not supported.

A mapping can also start from a top-level span attribute whose value is a JSON object or array encoded as a string. The resolver parses that attribute, then continues walking the path.

```json
{
"attr_1": "{\"x\": {\"y\": [{\"z\": \"Your refund has been approved and will appear in your account within five business days.\"}]}}"
}
```

Map the eval variable to `attr_1.x.y.0.z` to pass `Your refund has been approved and will appear in your account within five business days.` to the evaluator.

Click **Add Evaluation** to attach it to the task.

<img src="/images/docs/observe/evals-configure.png" alt="Toxicity eval configuration showing the read-only instructions, Pass/fail output, an error-localization checkbox, and a variable-mapping panel" style={{ borderRadius: '5px' }} />
Expand Down
Loading