feat(telemetry): full OTel GenAI semantic conventions + in-process PII stripping by theomonnom · Pull Request #7104 · livekit/agents

theomonnom · 2026-09-03T05:48:20Z

No description provided.

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 6 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-09-03T05:55:54Z

+            # callers pass user data through `extra={"lk.pii.<name>": ...}` precisely
+            # because a log body cannot be redacted; drop those before export
+            log_record.attributes = pii.redact_attributes(attributes)


🟥 Redacted logs expose exception details

Redaction filters only structured attributes, leaving exception text in log bodies. Realtime failures can export customer content to every configured log destination.

Was this helpful? React with 👍 or 👎 to provide feedback.

…I stripping Emit the complete `gen_ai.*` attribute set from the OpenTelemetry GenAI semantic conventions (open-telemetry/semantic-conventions-genai) so a LiveKit trace is understood by Datadog Agent Observability, Langfuse and any other GenAI-aware backend without a LiveKit-specific mapping. Span mapping: agent_session -> invoke_workflow, agent_turn -> invoke_agent, llm_request/llm_node -> chat, function_tool -> execute_tool, start_agent_activity -> create_agent, realtime turns -> generate_content with output.type=speech. Existing lk.* attributes and span names are unchanged. Adds the convention's metrics (gen_ai.client.token.usage split by token type, client.operation.duration, client.operation.time_to_first_chunk, execute_tool.duration, invoke_agent.duration) alongside the lk.agents.* instruments, and sets error.type on every recorded exception. Message content (gen_ai.input.messages, output.messages, system_instructions, tool.definitions, tool.call.arguments/result) is captured by default, matching the lk.pii.* content already recorded. Turn it off process-wide with telemetry.gen_ai.set_capture_content(False) or OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false. PII stripping now happens in-process. telemetry/pii.py registers a span processor ahead of every exporter that removes each lk.pii.* attribute and each GenAI content attribute - whose names the convention fixes, so the marker cannot be applied - from span attributes, span events and log records whenever the session enabled redaction. Previously this only happened at the LiveKit Cloud collector, so a third-party exporter sharing the tracer provider received unredacted content. Also fixes JobContext._otel_metadata, which stamped the redaction flag for the per-session option but not for project-wide redaction.

…ndings PII stripping is no longer all-or-nothing. Conversational content, tool payloads and exception details are stripped in-process before any exporter that is not LiveKit Cloud's; the pre-redaction payload is stashed and restored on that one export path, since what Cloud may keep is the project's setting in the dashboard rather than the SDK's call. `set_tracer_provider(..., allow_pii=True)` grants a provider's exporters the content (LIVEKIT_TELEMETRY_ALLOW_PII for setups that adopt the ambient provider and have no call site), and the project's redaction flag overrides that grant, withholding PII from every destination. This replaces the process-wide set_redaction switch, which could weaken a project-mandated redaction and implied the SDK could grant Cloud something the dashboard had not. Review fixes: - realtime turns get a nested `realtime_inference` span, so the inference attributes and the provider metrics no longer overwrite the agent_turn span's `invoke_agent` identity - invoke_agent duration uses a monotonic clock, matching the tool duration - both _otel_metadata call sites stamp the same resolved redaction flag - gen_ai.output.type follows the session's configured modality instead of assuming speech, since a realtime model can be text-only - the span processor strips exception.message/stacktrace too: record_exception resolves redaction from the ambient context, which can disagree with the span's own stamp - adopted logger providers get a redaction processor, so log exporters the integrator attached are covered like span exporters - gen_ai.tool.definitions omits `parameters`, which the convention marks NOT RECOMMENDED by default, and which cost a schema build per request

- contains_pii and conversation_id are internal, not public API - remove the section banners in trace_types: the constant names already carry the grouping (gen_ai.request.*, gen_ai.usage.*, ...) - drop a comment that restated the constant names below it - merge two tests that covered the same scenario from two angles

… stripping

chenghao-mou · 2026-09-03T18:19:15Z

+    "grok": "x_ai",
+    "mistral": "mistral_ai",
+    "moonshot": "moonshot_ai",
+}


I don't think the key values match our plugin values. e.g. Mistral LLM reports MistralAI:

This is part of the spec:

For the 16 enumerated providers (openai, anthropic, mistral_ai, aws.bedrock, gcp.vertex_ai, gcp.gemini, gcp.gen_ai, x_ai, perplexity, groq, cohere, deepseek, azure.ai.*, ibm.watsonx.ai, moonshot_ai) the registry spelling is MUST

Yes, I was referring to "key" values. I think we missed Amazon.

chenghao-mou · 2026-09-03T18:36:36Z

    @abstractmethod
    async def _run(self) -> None: ...

+    def _record_genai_request(self, span: trace.Span) -> None:


Codex informed met that the best practice for these data:

Full (buffered) content
Model instructions, user messages, and model outputs are considered sensitive and are often large in size.

Recording large or sensitive content in telemetry may be problematic due to high storage costs, regulatory requirements, or the need to enforce different access models for operational and user data.

OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in.

so it is an opt-in vs opt-out case, wdyt?

chenghao-mou · 2026-09-03T18:37:46Z

@@ -252,6 +258,7 @@ async def _traceable_main_task() -> None:
            ) as span:
                for name, attributes in _chat_ctx_to_otel_events(self._chat_ctx):
                    span.add_event(name, attributes)


do we still need the for loop here if we record the request in the following line?

@chenghao-mou

…orters by default Review from @chenghao-mou. The provider alias table matched almost nothing the plugins actually report. Of 15 keys exactly one hit: plugins expose `provider` either as a display name ("MistralAI", "AWS Bedrock", "Vertex AI", "Gemini", "xAI") or, for the OpenAI-compatible clients, as the base URL's host ("api.openai.com", "api.anthropic.com", "api.mistral.ai"), while the table was keyed on short ids that were never used. So OpenAI and Anthropic reported a hostname as the provider, which the convention makes a MUST violation — the registry spelling is mandatory for an enumerated provider, since backends use the attribute as the discriminator for provider-specific parsing. Both shapes are now mapped: by host (with suffix rules for Azure, Bedrock and Vertex endpoints), then by the display name reduced to lowercase alphanumerics. A provider outside the registry keeps its own id, which the convention allows. A test walks the values the plugins report and asserts each resolves, plus that every mapping targets a real registry value. Content stays on by default and PII now reaches every exporter unless withheld (`allow_pii=False`, or LIVEKIT_TELEMETRY_ALLOW_PII=0): a GenAI backend can only render inputs/outputs and the chat view if it receives them, which is the point of the conventions. The project's redaction setting still overrides the grant. The legacy per-message span events (gen_ai.user.message and friends) duplicate gen_ai.input.messages / output.messages, and are no longer part of the convention — the current event model is gen_ai.client.inference.operation.details. They are kept for the backends that still read them, but now honour the content switch, which previously left them shipping the full chat context after set_capture_content(False).

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 1 new potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-09-03T19:30:52Z

🟥 Sensitive sync errors bypass redaction

Realtime sync failures store str(e) under unmarked error attributes. Provider exceptions can expose customer content in exported logs despite redaction.

(Refers to this code)

Was this helpful? React with 👍 or 👎 to provide feedback.

@chenghao-mou

…ep them internal Review from @chenghao-mou: the SDK removes whole fields, it does not mask identifiable entities within them, and "PIIRedactingSpanProcessor" read like the latter. Renamed to _PIIFilteringSpanProcessor / _PIIFilteringLogProcessor and made private — nothing outside the framework installs them. redact_attributes is now filter_attributes. "Redaction" is left to mean the project setting (the LiveKit Cloud dashboard, or record={"redaction": True}); this module is what the client does about it, which the module header now states. Also pins the behaviour that decision rests on: with redaction enabled the filtering applies to every destination, LiveKit Cloud included, so the client never depends on a collector knowing a newly added gen_ai key. That already held for spans; the log path was untested, and now is.

The registry was carried over wholesale, which left 35 constants no code references: inference request knobs no plugin exposes, agent id/description/ version, the embeddings/retrieval/memory/evaluation/prompt families for operations LiveKit has no spans for, server.address/port, and three metrics we do not record. A constant we never set adds nothing a backend can read, so this costs no convention coverage — it only removes inventory, and 35 bookkeeping entries from the guard test's safe list. The content attributes of those families stay: they are registered as PII, where the entry is a cheap safety net should a plugin ever set one.

@chenghao-mou

… exception status Review from @chenghao-mou: the AWS realtime model reports "Amazon", which the provider table missed. It runs on Bedrock, so it maps to aws.bedrock. This was the only remaining miss across every provider value the LLM and realtime plugins report. Review from Devin: - finish_reason_for checked function_calls before interrupted, so a generation that emitted a tool call and then failed reported tool_call rather than error. Interruption now wins. - record_exception writes the message onto the span status as well as the attributes and the `exception` event. The status was left intact, so a third-party exporter could still read it; it is now replaced with the same marker. A test asserts the message reaches none of the three.

Regression from the previous commit, caught by Devin: replacing the span status for third-party exporters left LiveKit Cloud with the redacted description too, since restore_pii only put back attributes and events. The status is now stashed alongside them, and the test asserts Cloud still receives the message on all three carriers while third parties receive it on none.

theomonnom requested a review from a team as a code owner September 3, 2026 05:48

theomonnom mentioned this pull request Sep 3, 2026

feat(telemetry): full OTel GenAI semantic conventions + in-process PII stripping livekit/agents-js#2411

Open

theomonnom force-pushed the theo/genai-semconv-pii-stripping branch from 45033c8 to 45d4e4c Compare September 3, 2026 05:53

devin-ai-integration Bot reviewed Sep 3, 2026

View reviewed changes

theomonnom force-pushed the theo/genai-semconv-pii-stripping branch from 45d4e4c to 5f8c130 Compare September 3, 2026 05:59

This comment was marked as resolved.

Sign in to view

theomonnom added 2 commits September 3, 2026 00:09

docs(telemetry): correct comments that still described all-or-nothing…

0edd3f4

… stripping

chenghao-mou reviewed Sep 3, 2026

View reviewed changes

devin-ai-integration Bot reviewed Sep 3, 2026

View reviewed changes

theomonnom added 3 commits September 3, 2026 15:21

This comment was marked as resolved.

Sign in to view

Uh oh!

Conversation

theomonnom commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

chenghao-mou Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

theomonnom Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

chenghao-mou Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

chenghao-mou Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenghao-mou Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

theomonnom commented Sep 3, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

devin-ai-integration Bot Sep 3, 2026 •

edited

Loading

chenghao-mou Sep 3, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading