-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
docs: Fix stale AI integration paths and patterns in AGENTS.md and add-ai-integration skill #24257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
RulaKhaled
wants to merge
3
commits into
develop
Choose a base branch
from
docs/fix-stale-ai-integration-paths
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+84
−64
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
f0aba94
docs: Fix stale AI integration paths in AGENTS.md and add-ai-integrat…
RulaKhaled 694550a
docs: Rewrite AI integration patterns to match orchestrion/diagnostic…
RulaKhaled 3ad69e8
docs: Correct truncation, span op, and streaming guidance in AI skill
RulaKhaled File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,113 +9,132 @@ argument-hint: <provider-name> | |
| ## Decision Tree | ||
|
|
||
| ``` | ||
| Does the AI SDK have native OpenTelemetry support? | ||
| |- YES -> Does it emit OTel spans automatically? | ||
| | |- YES (like Vercel AI) -> Pattern 1: OTel Span Processors | ||
| | +- NO -> Pattern 2: OTel Instrumentation (wrap client) | ||
| +- NO -> Does the SDK provide hooks/callbacks? | ||
| |- YES (like LangChain) -> Pattern 3: Callback/Hook Based | ||
| +- NO -> Pattern 4: Client Wrapping | ||
| Does the SDK publish its own `diagnostics_channel` telemetry? | ||
| |- YES (ai >= 7) -> Pattern 1: Native tracing channel | ||
| +- NO -> Does the SDK expose callback/exporter hooks? | ||
| |- YES (LangChain, Mastra) -> Pattern 3: Callback/Exporter | ||
| +- NO (OpenAI, Anthropic, Google GenAI, ai < 7) -> Pattern 2: Orchestrion-injected channels | ||
| ``` | ||
|
|
||
| ## Runtime-Specific Placement | ||
| ## Placement | ||
|
|
||
| If an AI SDK only works in one runtime, code lives exclusively in that runtime's package. Do NOT add it to `packages/core/`. | ||
| AI instrumentation lives in `packages/server-utils/`, not `packages/core/` and not the runtime packages: | ||
|
|
||
| - **Node.js-only** -> `packages/node/src/integrations/tracing/{provider}/` | ||
| - **Cloudflare-only** -> `packages/cloudflare/src/integrations/tracing/{provider}.ts` | ||
| - **Browser-only** -> `packages/browser/src/integrations/tracing/{provider}/` | ||
| - **Multi-runtime** -> shared core in `packages/core/src/tracing/{provider}/` with runtime-specific wrappers | ||
| - **Instrumentation logic** -> `packages/server-utils/src/ai/{provider}/` | ||
| - **Integration** (wires it up, registered in `getTracingIntegrations()`) -> `packages/server-utils/src/integrations/{provider}.ts` | ||
| - **Runtime packages** (`node`, `cloudflare`, `bun`, ...) re-export the integration from `@sentry/server-utils` -- they do not define their own | ||
|
|
||
| Cloudflare-only client wrapping (Workers AI) is the exception: it is applied in `packages/cloudflare/src/instrumentations/worker/instrumentEnv.ts`, wrapping the binding from `env`. | ||
|
|
||
| ## Span Hierarchy | ||
|
|
||
| - `gen_ai.invoke_agent` — parent/pipeline spans (chains, agents, orchestration) | ||
| - `gen_ai.chat`, `gen_ai.generate_text`, etc. — child spans (actual LLM calls) | ||
| - `gen_ai.chat`, `gen_ai.generate_content`, `gen_ai.embeddings`, `gen_ai.execute_tool` — child spans (actual LLM/tool calls) | ||
|
|
||
| Do not hand-write the op string. Derive it with `getGenAiSpanOp(operationName)` from `ai/core/utils.ts`, and take the constants from `@sentry/conventions/op` (`GEN_AI_CHAT`, `GEN_AI_GENERATE_CONTENT`, `GEN_AI_EMBEDDINGS`, `GEN_AI_EXECUTE_TOOL`, `GEN_AI_HANDOFF`, `GEN_AI_INVOKE_AGENT`, `GEN_AI_RERANK` — that is the full set). An operation with no convention op (currently only `unknown`) falls back to the generic `function` op; the raw name is still preserved on `gen_ai.operation.name`. | ||
|
|
||
| ## Shared Utilities (`packages/core/src/tracing/ai/`) | ||
| ## Shared Utilities (`packages/server-utils/src/ai/core/`) | ||
|
|
||
| - `gen-ai-attributes.ts` — OTel Semantic Convention attribute constants. **Always use these, never hardcode.** | ||
| - `utils.ts` — `setTokenUsageAttributes()`, `getTruncatedJsonString()`, `truncateGenAiMessages()`, `buildMethodPath()` | ||
| - Attribute keys come from `@sentry/conventions/attributes` — import them there directly at the call site. **Never hardcode attribute strings.** | ||
| - `gen-ai-attributes.ts` — only the gap-fillers: attributes with no `@sentry/conventions` equivalent, Sentry-internal meta attributes, and keys we intentionally emit differently. Check conventions first; add here only if it genuinely has no equivalent. | ||
| - `utils.ts` — `setTokenUsageAttributes()`, `buildMethodPath()`, `resolveAIRecordingOptions()`, `getGenAiSpanOp()`, `endStreamSpan()`, `extractSystemInstructions()` | ||
| - Only use attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/). | ||
|
|
||
| ## Streaming | ||
|
|
||
| - **Non-streaming:** `startSpan()`, set attributes from response | ||
| - **Streaming:** `startSpanManual()`, accumulate state via async generator or event listeners, set `GEN_AI_RESPONSE_STREAMING_ATTRIBUTE: true`, call `span.end()` in finally block | ||
| - Detect via `params.stream === true` | ||
| - References: `openai/streaming.ts` (async generator), `anthropic-ai/streaming.ts` (event listeners) | ||
| How the span is opened depends on the path: | ||
|
|
||
| - **Channel path** (Patterns 1 & 2 — how auto-instrumentation actually runs): build the span with `startInactiveSpan()` inside the `getSpan` callback of `bindTracingChannelToSpan()` and let the binding own its lifecycle. For a streamed call, return `true` from the `deferSpanEnd` option to hand span-ending ownership to the stream wrapper; non-streaming results end through the normal `beforeSpanEnd` path. Detect the stream from the **result shape** (async-iterable, or the SDK's stream object), not from `params.stream` — see `wrapStreamResult()` in `integrations/openai.ts` and `integrations/anthropic.ts`. | ||
| - **Manual client wrapping** (`instrumentOpenAiClient()`, `instrumentAnthropicAiClient()`, ... in `ai/{provider}/index.ts`, the public manual-instrumentation API): non-streaming uses `startSpan()`; streaming uses `startSpanManual()` and detects via `params.stream === true` (or a method that always streams). | ||
|
|
||
| Either way, do not set streaming response attributes by hand. Accumulate into a `StreamResponseState` and call `endStreamSpan(span, state, recordOutputs)` from `ai/core/utils.ts` — in a `finally` for an async generator, or from the stream's terminal event for a listener-based stream. It sets `GEN_AI_RESPONSE_STREAMING`, response id/model, token usage, finish reasons, output text and tool calls, and ends the span. | ||
|
|
||
| References: `ai/openai/streaming.ts` (`instrumentStream`, async generator), `ai/anthropic-ai/streaming.ts` (`instrumentMessageStream`, event listeners) | ||
|
|
||
| ## Token Accumulation | ||
|
|
||
| - **Child spans:** Set tokens directly from API response via `setTokenUsageAttributes()` | ||
| - **Parent spans (`invoke_agent`):** Accumulate from children using event processor (see `vercel-ai/`) | ||
| - **Parent spans (`invoke_agent`):** Accumulate inside the channel subscriber as usage/finish chunks arrive, then set on the open parent span before ending it (see `integrations/vercel-ai/vercel-ai-dc-subscriber.ts`). There is no event processor doing this rollup. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we need to do accumulation anymore since this doesn't really work in the span streaming world, iirc this will be done product side where they have the full span tree |
||
|
|
||
| ## Pattern 1: Native Tracing Channel | ||
|
|
||
| **Use when:** the SDK publishes to `diagnostics_channel` itself (`ai` >= 7 publishes `ai:telemetry`) | ||
|
|
||
| 1. Write the subscriber in `packages/server-utils/src/integrations/{provider}/{provider}-dc-subscriber.ts` — read the channel payloads, open spans, set gen_ai attributes | ||
| 2. Subscribe from the integration's `setupOnce()`, wrapped in `waitForTracingChannelBinding()` so it waits for the async-context binding: | ||
|
|
||
| ```ts | ||
| setupOnce() { | ||
| if (!dc.tracingChannel) return; | ||
| waitForTracingChannelBinding(() => { | ||
| subscribe{Provider}TracingChannel(dc.tracingChannel, options); | ||
| }); | ||
| } | ||
| ``` | ||
|
|
||
| ## Pattern 1: OTel Span Processors | ||
| Subscribing is a no-op on SDK versions that never publish, so it is always safe to call. | ||
|
|
||
| **Use when:** SDK emits OTel spans automatically (Vercel AI) | ||
| Reference: `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` | ||
|
|
||
| 1. **Core:** Create `add{Provider}Processors()` in `packages/core/src/tracing/{provider}/index.ts` — registers `spanStart` listener + event processor | ||
| 2. **Node.js:** Add `callWhenPatched()` optimization in `packages/node/src/integrations/tracing/{provider}/index.ts` — defers registration until package is imported | ||
| 3. **Edge:** Direct registration in `packages/cloudflare/src/integrations/tracing/{provider}.ts` — no OTel, call processors immediately | ||
| ## Pattern 2: Orchestrion-Injected Channels | ||
|
|
||
| Reference: `packages/node/src/integrations/tracing/vercelai/` | ||
| **Use when:** the SDK has no telemetry of its own (OpenAI, Anthropic, Google GenAI, `ai` < 7) | ||
|
|
||
| ## Pattern 2: OTel Instrumentation (Client Wrapping) | ||
| Orchestrion injects `diagnostics_channel` tracing channels into the target module's functions at load time; we then subscribe to those injected channels. This replaced the old OTel instrumentation packages — there is no `@opentelemetry/instrumentation-*` dependency in this path. | ||
|
|
||
| **Use when:** SDK has no native OTel support (OpenAI, Anthropic, Google GenAI) | ||
| 1. Create the span-building/attribute logic in `packages/server-utils/src/ai/{provider}/` | ||
| 2. Declare the module, version range, and methods to inject in `packages/server-utils/src/orchestrion/config/{provider}.ts` | ||
| 3. In `packages/server-utils/src/integrations/{provider}.ts`, call `invokeOrchestrionInstrumentation(client, {provider}ModuleNames, fn, [options])` from `setup(client)`, and bind each injected channel to a span with `bindTracingChannelToSpan()`. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. | ||
|
|
||
| 1. **Core:** Create `instrument{Provider}Client()` in `packages/core/src/tracing/{provider}/index.ts` — Proxy to wrap client methods, create spans manually | ||
| 2. **Node.js `instrumentation.ts`:** Patch module exports, wrap client constructor. Check `_INTERNAL_shouldSkipAiProviderWrapping()` for LangChain compatibility. | ||
| 3. **Node.js `index.ts`:** Export integration function using `generateInstrumentOnce()` helper | ||
| Reference: `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` | ||
|
|
||
| Reference: `packages/node/src/integrations/tracing/openai/` | ||
| **A provider can need both patterns.** `vercelAIIntegration` subscribes to the native `ai:telemetry` channel for `ai` >= 7 _and_ runs orchestrion injection for `ai` v4-v6, in the same integration. | ||
|
|
||
| ## Pattern 3: Callback/Hook Based | ||
| ## Pattern 3: Callback/Exporter | ||
|
|
||
| **Use when:** SDK provides lifecycle hooks (LangChain, LangGraph) | ||
| **Use when:** SDK provides lifecycle hooks or an exporter interface (LangChain, LangGraph, Mastra) | ||
|
|
||
| 1. **Core:** Create `create{Provider}CallbackHandler()` — implement SDK's callback interface, create spans in callbacks | ||
| 2. **Node.js `instrumentation.ts`:** Auto-inject callbacks by patching runnable methods. Disable underlying AI provider wrapping. | ||
| 1. Create `create{Provider}CallbackHandler()` in `packages/server-utils/src/ai/{provider}/index.ts` — implement the SDK's callback/exporter interface, create spans in the callbacks | ||
| 2. In `packages/server-utils/src/integrations/{provider}.ts`, auto-inject the handler by patching the relevant methods, and call `_INTERNAL_skipAiProviderWrapping()` to disable the underlying AI provider wrapping | ||
|
|
||
| Reference: `packages/node/src/integrations/tracing/langchain/` | ||
| Reference: `packages/server-utils/src/ai/langchain/`, and `packages/server-utils/src/ai/mastra/` for an exporter-shaped agent framework | ||
|
|
||
| ## Auto-Instrumentation (Node.js) | ||
| ## Registration | ||
|
|
||
| **Mandatory** for Node.js AI integrations. OTel only patches when the package is imported (zero cost if unused). | ||
| **Mandatory.** Patching only happens once the target package is imported (zero cost if unused). | ||
|
|
||
| ### Steps | ||
|
|
||
| 1. **Add to `getAutoPerformanceIntegrations()`** in `packages/node/src/integrations/tracing/index.ts` — LangChain MUST come first | ||
| 2. **Add to `getOpenTelemetryInstrumentationToPreload()`** for OTel-based integrations | ||
| 3. **Export from `packages/node/src/index.ts`**: integration function + options type | ||
| 1. **Add to `getTracingIntegrations()`** in `packages/server-utils/src/integrations/index.ts` — LangChain MUST come first, so it can disable the AI provider integrations before they instrument | ||
| 2. **Export from `packages/server-utils/src/index.ts`**: integration function + options type | ||
| 3. **Re-export from the runtime packages** that support it (e.g. `packages/node/src/index.ts`, `packages/cloudflare/src/index.ts`) | ||
| 4. **Add E2E tests:** | ||
| - Node.js: `dev-packages/node-integration-tests/suites/tracing/{provider}/` | ||
| - Cloudflare: `dev-packages/cloudflare-integration-tests/suites/tracing/{provider}/` | ||
| - Browser: `dev-packages/browser-integration-tests/suites/tracing/ai-providers/{provider}/` | ||
|
|
||
| ## Key Rules | ||
|
|
||
| 1. Respect `dataCollection.genAI` for recording input and output messages | ||
| 1. Gate input/output message recording behind `resolveAIRecordingOptions()`, which resolves the integration's `recordInputs`/`recordOutputs` against the client's `dataCollection.genAI` settings. Never read `dataCollection.genAI` directly. | ||
| 2. Set `SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN = 'auto.ai.{provider}'` (alphanumerics, `_`, `.` only) | ||
| 3. Truncate large data with helper functions from `utils.ts` | ||
| 3. **Do not truncate message payloads.** The `enableTruncation` flag and all AI truncation/media-stripping logic were removed in v11 (#23045); recorded messages are serialized with `stringify()` and set on the span as-is. Nothing downstream caps them either — `maxValueLength` only applies to `request.url` and exception values, and event normalization limits depth/breadth, not string length. Size limiting is handled server-side, so it is not a contributor concern. | ||
| 4. `gen_ai.invoke_agent` for parent ops, `gen_ai.chat` for child ops | ||
|
|
||
| ## Checklist | ||
|
|
||
| - [ ] Runtime-specific code placed only in that runtime's package | ||
| - [ ] Added to `getAutoPerformanceIntegrations()` in correct order (Node.js) | ||
| - [ ] Added to `getOpenTelemetryInstrumentationToPreload()` (Node.js with OTel) | ||
| - [ ] Exported from appropriate package index | ||
| - [ ] Instrumentation in `packages/server-utils/src/ai/`, integration in `packages/server-utils/src/integrations/` | ||
| - [ ] Added to `getTracingIntegrations()` in correct order (LangChain first) | ||
| - [ ] Exported from `packages/server-utils/src/index.ts` and re-exported from the supported runtime packages | ||
| - [ ] E2E tests added and verifying auto-instrumentation | ||
| - [ ] Only used attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) | ||
| - [ ] JSDoc says "enabled by default" or "not enabled by default" | ||
| - [ ] Documented how to disable (if auto-enabled) | ||
| - [ ] Verified OTel only patches when package imported (Node.js) | ||
| - [ ] Only used attributes from [Sentry Gen AI Conventions](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/), with span ops derived via `getGenAiSpanOp()` | ||
| - [ ] Input/output recording gated on `resolveAIRecordingOptions()`; no truncation logic added | ||
| - [ ] JSDoc on the exported integration names the channels it subscribes to, the supported SDK versions, and the prerequisite (orchestrion-injected channels "require the Sentry runtime hook or bundler plugin") | ||
| - [ ] Verified patching only happens when the target package is imported | ||
|
|
||
| ## Reference Implementations | ||
|
|
||
| - **Pattern 1 (Span Processors):** `packages/node/src/integrations/tracing/vercelai/` | ||
| - **Pattern 2 (Client Wrapping):** `packages/node/src/integrations/tracing/openai/` | ||
| - **Pattern 3 (Callback/Hooks):** `packages/node/src/integrations/tracing/langchain/` | ||
| - **Pattern 1 (Native channel):** `packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts` | ||
| - **Pattern 2 (Orchestrion channels):** `packages/server-utils/src/integrations/openai.ts` + `packages/server-utils/src/orchestrion/config/openai.ts` | ||
| - **Pattern 3 (Callback/Exporter):** `packages/server-utils/src/ai/langchain/`, `packages/server-utils/src/ai/mastra/` | ||
| - **Both patterns at once:** `packages/server-utils/src/integrations/vercel-ai/index.ts` | ||
|
|
||
| **When in doubt, follow the pattern of the most similar existing integration.** | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could just point at https://getsentry.github.io/sentry-conventions/ instead of enumerating ops in here etc. I think this will go out of date pretty soon