Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .agents/skills/e2e-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ Try not to use specific test narrowing commands unless hunting down a very nasty

Cassettes mock provider HTTP responses (OpenAI, Anthropic, ...) so external-provider scenarios replay in CI without provider keys. The harness starts a local `@braintrust/seinfeld` cassette server and points provider SDK base URL env vars at that server, so subprocesses and SDK-launched binaries are covered too.

- Default to cheap provider models for real API calls and cassette recordings, unless the user explicitly requests otherwise. Choose a model that supports the behavior under test, and keep prompts, output limits, and media duration small while preserving meaningful coverage.

- External-provider tests should thread `runContext: { variantKey: "...", originalScenarioDir }` into the scenario runner. In normal replay mode, missing cassette entries fail loudly instead of skipping or falling back to live providers. In `record` / `record-missing` mode, they run so new cassettes can be authored.
- Thread `runContext: { variantKey: "...", originalScenarioDir }` into `runScenarioDir`/`runNodeScenarioDir`. Cassettes live at `e2e/scenarios/<name>/__cassettes__/<variantKey>.cassette.json` (parallel to `__snapshots__/`). Only set `runContext.cassette` explicitly for unusual cases, such as `cassette: false` on a non-provider mode inside an otherwise provider-backed scenario.
- To re-record after changing a scenario:
Expand Down
2 changes: 2 additions & 0 deletions .agents/skills/instrumentation/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,5 @@ Map the change before editing:
## Process

Before implementing or changing instrumentation it is advisable to add or adjust the e2e tests for the desired change, make it fail, then implement the new instrumentation until the test passes.

When validating instrumentation with real provider APIs or recording e2e cassettes, default to cheap models that support the behavior under test, unless the user explicitly requests otherwise. Keep prompts, output limits, and media duration small while preserving meaningful coverage.
5 changes: 5 additions & 0 deletions .changeset/elevenlabs-synchronous-instrumentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"braintrust": minor
---

feat: Add instrumentation for synchronous elevenlabs SDK APIs
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ pnpm run test:checks # Hermetic tests (core + vitest wrapper)

Each scenario runs the SDK in a subprocess against a mock Braintrust server and snapshots the results. No API keys required for replay; recording needs provider keys.

For real-provider instrumentation tests and e2e recordings, default to cheap models that support the behavior under test, unless the user explicitly requests otherwise. Keep prompts, output limits, and media duration small while preserving meaningful coverage.

```bash
pnpm run test:e2e # Run all e2e scenarios (from repo root)
pnpm run test:e2e:update # Update e2e snapshots without re-recording cassettes
Expand Down
15 changes: 15 additions & 0 deletions e2e/config/pr-comment-scenarios.json
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,21 @@
}
]
},
{
"scenarioDirName": "elevenlabs-instrumentation",
"label": "ElevenLabs Instrumentation",
"metadataScenario": "elevenlabs-instrumentation",
"variants": [
{
"variantKey": "elevenlabs-v2",
"label": "v2 pinned"
},
{
"variantKey": "elevenlabs-v2-latest",
"label": "v2 latest"
}
]
},
{
"scenarioDirName": "google-generative-ai-instrumentation",
"label": "Google Generative AI Instrumentation",
Expand Down
3 changes: 3 additions & 0 deletions e2e/helpers/scenario-harness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,7 @@ function getCassetteServerRoutes(): CassetteServerRoute[] {
prefix: "/aws-bedrock-runtime",
upstreamOrigin: `https://bedrock-runtime.${getBedrockRegion()}.amazonaws.com`,
},
{ prefix: "/elevenlabs", upstreamOrigin: "https://api.elevenlabs.io" },
{ prefix: "/cohere", upstreamOrigin: "https://api.cohere.com" },
{ prefix: "/cursor/v1", upstreamOrigin: "https://api.cursor.com/v1" },
{ prefix: "/cursor", upstreamOrigin: "https://api2.cursor.sh" },
Expand Down Expand Up @@ -336,6 +337,7 @@ function getCassetteEnv(wiring: ActiveCassetteWiring): Record<string, string> {
ANTHROPIC_BASE_URL: `${serverUrl}/anthropic`,
ANTHROPIC_BEDROCK_BASE_URL: `${serverUrl}/anthropic-bedrock`,
AWS_BEDROCK_RUNTIME_BASE_URL: `${serverUrl}/aws-bedrock-runtime`,
ELEVENLABS_BASE_URL: `${serverUrl}/elevenlabs`,
COHERE_BASE_URL: `${serverUrl}/cohere`,
COHERE_API_URL: `${serverUrl}/cohere`,
CURSOR_BACKEND_URL: `${serverUrl}/cursor`,
Expand Down Expand Up @@ -392,6 +394,7 @@ const CASSETTE_PROVIDER_KEYS: Array<{
envVars: ["COHERE_API_KEY", "CO_API_KEY"],
placeholder: "cassette-placeholder",
},
{ envVars: ["ELEVENLABS_API_KEY"], placeholder: "cassette-placeholder" },
{ envVars: ["CURSOR_API_KEY"], placeholder: "key_cassette-placeholder" },
{
envVars: ["GOOGLE_API_KEY", "GOOGLE_GENAI_API_KEY", "GEMINI_API_KEY"],
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Loading
Loading