feat(harbor): add native Harbor evaluation plugin (#631)

Resolves
[SDK-64](https://linear.app/braintrustdata/issue/SDK-64/first-class-harbor-integration).

Adds a native Braintrust plugin for
[Harbor](https://github.com/harbor-framework/harbor).

## Setup

Requires Python 3.12+ and a minimum Harbor version of `0.16.0`. The
plugin ships in the normal `braintrust` distribution and Harbor
discovers it through the `harbor.plugins` entry point, so there is
nothing to import or register.

```bash
pip install harbor braintrust

export BRAINTRUST_API_KEY=...
```

Enable it by selecting the `braintrust` plugin:

```bash
harbor run \
  -d terminal-bench/terminal-bench-2@latest \
  -a claude-code \
  -m anthropic/claude-sonnet-4-6 \
  -n 32 \
  --plugin braintrust \
  --plugin-kwarg project_name=agent-benchmarks
```

That is the whole setup. The API key stays in the host process and is
never written into Harbor config, results, manifests, or the task
container.

## Configuration

Options are passed as Harbor kwargs, and every option also has a
`HARBOR_BRAINTRUST_*` environment fallback:

```bash
harbor run ... \
  --plugin braintrust \
  --plugin-kwarg project_name=agent-benchmarks \
  --plugin-kwarg 'score_keys=["correctness","pass_*"]' \
  --plugin-kwarg 'reward_rules={"error_rate":{"type":"score","direction":"minimize","min":0,"max":1}}'
```

```bash
# Equivalent, so CI can configure it once:
export HARBOR_BRAINTRUST_PROJECT=agent-benchmarks
export HARBOR_BRAINTRUST_SCORE_KEYS='["correctness","pass_*"]'
```

Precedence is `--plugin-kwarg` > `HARBOR_BRAINTRUST_*` > default.
Complex values are JSON because Harbor only accepts flat kwargs.

| Option | Values | Purpose |
|---|---|---|
| `project_name` / `project_id` | string | Target project. Mutually
exclusive. Env: `HARBOR_BRAINTRUST_PROJECT`. |
| `dataset_mode` | `sync` (default), `none` | Upsert a managed dataset
and associate it with the experiment. |
| `trajectory_mode` | `atif` (default), `summary`, `native` | Full ATIF
import, aggregates only, or skip (agent instrumented elsewhere). |
| `content_mode` | `metadata`, `messages` (default), `full` | How much
payload to capture. `full` currently captures the same as `messages`. |
| `score_keys` / `metric_keys` | JSON glob arrays | Route reward keys to
scores vs metrics. Overlap is rejected before any network I/O. |
| `reward_rules` | JSON object | Per-reward type, direction, and min/max
normalization. |
| `classifier_rules` | JSON object | Map a classifier name to a
documented result path. |
| `invalid_score_policy` | `metric` (default), `drop`, `error` | What to
do when a configured score is out of range. |
| `attachments` | `none`, `verifier-details` (default), `all` | `all`
also enables `artifact_include` globs. |
| `max_content_bytes`, `max_attachment_bytes`,
`max_total_attachment_bytes` | ints | Size bounds. |
| `strict` | bool (default `false`) | Raise instead of isolating sync
failures, where Harbor permits it. |

## What lands in Braintrust

```
Harbor job
├── resolved task selection ──────► Dataset (one per Harbor source)
├── dataset × semantic agent ─────► Experiment
└── final TrialResult ────────────► eval [root]
    ├── task
    │   ├── environment_setup
    │   ├── agent_setup
    │   ├── agent_execution
    │   │   ├── chat.completions.create  [llm]
    │   │   ├── read_file                [tool]
    │   │   └── chat.completions.create  [llm]
    │   └── verification
    ├── correctness  [score, purpose=scorer]
    └── category     [classifier, purpose=scorer]
```

**Datasets.** One managed dataset per Harbor source. Each resolved task
is one record with a deterministic UUIDv5 ID, so reruns and backfill
upsert instead of duplicating. Input holds task-authored semantics
(identity, instruction, step instructions); `expected` stays null rather
than treating solution or verifier code as expected output.

**Experiments.** Partitioned by `(dataset identity, semantic agent
config, skill digests)`. Agent, model, kwargs, MCP config, and skills
split experiments; concurrency, retry policy, and output paths do not.
Each retained final `TrialResult` is one root row — execution retries do
not inflate counts, while intentional `n_attempts` stay separate rows.

**Rewards.** Harbor rewards stay authoritative and the raw dict is
preserved at `metadata.harbor.raw_rewards`. Classification is semantic,
not range-guessing: an exact `reward_rules` entry, then a
`score_keys`/`metric_keys` glob, then conventional `reward` when it is
in `[0, 1]`; every other numeric reward becomes a metric even if it
happens to land in `[0, 1]`. A missing reward with no exception is
flagged as unevaluated rather than scored zero.

**ATIF traces are conformance-gated.** A step becomes an `llm` span only
with exactly one model call plus provider/model identity, canonical
messages, token usage, and valid timing; a tool span needs arguments
plus a correlated result or error. Anything deterministic, aggregated,
truncated, or incomplete is downgraded to a `task` summary with a
warning on the eval root instead of being mislabeled. Missing timestamps
are interpolated inside the agent phase, outliers clamped, and every
repair recorded. Subagents become nested task trees.

**Privacy.** Secret-like keys, credentials, and oversized payloads are
redacted or bounded, and absolute paths are stripped from host-side
metadata. Paths inside trajectory content are kept, because they name
container files the agent actually operated on — redacting them would
empty out filesystem tool calls.

## Backfill

Backfill reuses the same identity, normalization, partitioning, reward,
and ATIF core as live sync, so it reconciles an existing experiment
rather than creating a second one. It never reruns trials.

```python
import asyncio
from braintrust.integrations.harbor import backfill_job

asyncio.run(backfill_job("jobs/my-harbor-run", project_name="agent-benchmarks"))
```

Every job directory also gets a credential-free `braintrust-sync.json`
with the dataset and experiment IDs, per-trial terminal state, retry
counts, warnings, and completion state.

Author: AbhiPrasad

examples/README.md

@@ -69,6 +69,7 @@ Unless noted otherwise, every example below uses `braintrust.auto_instrument()`.
 | `dspy/` | DSPy `ReAct` agent with two tools (LiteLLM token metrics propagate) |
 | `evals/` | The `Eval` framework — does **not** use `auto_instrument()` |
 | `google_genai/` | Google GenAI `generate_content` against Gemini |
+| `harbor/` | Native Harbor evaluation plugin — does **not** use `auto_instrument()` |
 | `langchain/` | LangChain `prompt | model` chain — global handler installed by `auto_instrument()` |
 | `langsmith/` | Migration helper for projects coming from LangSmith — uses `setup_langsmith()` |
 | `litellm/` | LiteLLM `completion` |

examples/harbor/.env.example

@@ -0,0 +1,3 @@
+BRAINTRUST_API_KEY=
+OPENAI_API_KEY=
+HARBOR_BRAINTRUST_PROJECT=example-harbor

examples/harbor/.gitignore

@@ -0,0 +1 @@
+jobs/

examples/harbor/README.md

@@ -0,0 +1,60 @@
+# Harbor + Braintrust
+
+Runs a small, self-contained [Harbor](https://harborframework.com/) evaluation and uses Harbor's native Braintrust job plugin to sync the result. Braintrust receives a managed dataset, an experiment row for the final trial, verifier rewards, and the Harbor lifecycle and ATIF trace.
+
+The plugin is discovered automatically through Harbor's `braintrust` entry point. The Braintrust API key remains in the host process; it is not passed into the task container.
+
+## Setup
+
+Install the example's dependencies:
+
+```bash
+uv sync
+```
+
+The command below reads credentials from the repository's root `.env`. It requires:
+
+```dotenv
+BRAINTRUST_API_KEY=...
+OPENAI_API_KEY=...
+```
+
+Alternatively, copy `.env.example` to `.env` in this directory and change `--env-file ../../.env` below to `--env-file .env`.
+
+## Run
+
+Docker must be running. From this directory, run:
+
+```bash
+uv run harbor run \
+  --path task \
+  --agent terminus-2 \
+  --model openai/gpt-4.1-mini \
+  --job-name braintrust-harbor-example \
+  --jobs-dir jobs \
+  --env-file ../../.env \
+  --plugin braintrust \
+  --plugin-kwarg project_name=example-harbor \
+  --yes
+```
+
+The agent solves the task in `task/`, and Harbor's verifier emits a normalized `reward` plus an `answer_length` metric. The plugin creates `jobs/braintrust-harbor-example/braintrust-sync.json` after synchronization.
+
+Harbor also accepts plugin options through `HARBOR_BRAINTRUST_*` variables. For example, setting this in `.env` removes the need for the `project_name` plugin argument:
+
+```dotenv
+HARBOR_BRAINTRUST_PROJECT=example-harbor
+```
+
+Then omit `--plugin-kwarg project_name=example-harbor` from the command.
+
+## Backfill an existing job
+
+To synchronize the persisted job again without rerunning the agent or verifier:
+
+```bash
+uv run --env-file ../../.env python backfill.py jobs/braintrust-harbor-example \
+  --project example-harbor
+```
+
+Backfill uses the same deterministic dataset, experiment, and span identities, so it reconciles the existing Braintrust data instead of creating duplicate rows.

examples/harbor/backfill.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+"""Backfill a persisted Harbor job into Braintrust."""
+
+import argparse
+import asyncio
+from pathlib import Path
+
+from braintrust.integrations.harbor import backfill_job
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("job_dir", type=Path, help="Persisted Harbor job directory")
+    parser.add_argument("--project", help="Braintrust project name (otherwise read from the environment)")
+    args = parser.parse_args()
+
+    options = {"project_name": args.project} if args.project else {}
+    asyncio.run(backfill_job(args.job_dir, **options))
+
+
+if __name__ == "__main__":
+    main()

examples/harbor/pyproject.toml

@@ -0,0 +1,12 @@
+[project]
+name = "braintrust-harbor-example"
+version = "0.1.0"
+description = "Run a Harbor evaluation and sync it to Braintrust"
+requires-python = ">=3.12"
+dependencies = [
+    "braintrust",
+    "harbor==0.20.0",
+]
+
+[tool.uv.sources]
+braintrust = { path = "../../py", editable = true }

examples/harbor/task/environment/Dockerfile

@@ -0,0 +1,3 @@
+FROM ubuntu:24.04
+
+WORKDIR /app

examples/harbor/task/instruction.md

@@ -0,0 +1 @@
+Calculate 17 × 6. Create `/app/answer.txt` containing only the decimal result and a trailing newline.

examples/harbor/task/task.toml

@@ -0,0 +1,24 @@
+schema_version = "1.3"
+artifacts = []
+
+[metadata]
+category = "arithmetic"
+
+[verifier]
+timeout_sec = 60.0
+collect = []
+
+[verifier.env]
+
+[agent]
+timeout_sec = 300.0
+
+[environment]
+network_mode = "public"
+build_timeout_sec = 300.0
+os = "linux"
+mcp_servers = []
+
+[environment.env]
+
+[solution.env]

examples/harbor/task/tests/test.sh

@@ -0,0 +1,10 @@
+#!/bin/sh
+
+answer="$(tr -d '[:space:]' < /app/answer.txt 2>/dev/null || true)"
+if [ "$answer" = "102" ]; then
+  reward=1
+else
+  reward=0
+fi
+
+printf '{"reward":%s,"answer_length":%s}\n' "$reward" "${#answer}" > /logs/verifier/reward.json