Skip to content

feat(telemetry): interruption detail, handoff span, fallback events, text input - #7137

Open
davidzhao wants to merge 8 commits into
dz/telemetry-rpcfrom
dz/telemetry-coverage
Open

feat(telemetry): interruption detail, handoff span, fallback events, text input#7137
davidzhao wants to merge 8 commits into
dz/telemetry-rpcfrom
dz/telemetry-coverage

Conversation

@davidzhao

@davidzhao davidzhao commented Sep 5, 2026

Copy link
Copy Markdown
Member

What

Smaller coverage gaps on existing spans, each of which came up when reading a trace and not being able to answer a question from it.

Interruptions

agent_turn carried a single lk.interrupted boolean. It now records an interrupted event with lk.interruption.source, set by the caller that knows the cause:

source meaning
audio_activity barge-in from VAD / STT activity
user_turn a committed user turn preempting the reply
programmatic session.interrupt(), a tool, teardown

The pipeline and say paths also stamp lk.playout.position, how many seconds had actually played when the user cut in. The false-interruption path records a playout_paused event when a barge-in pauses playout and a false_interruption event with lk.false_interruption.resumed when the timer resolves it.

Agent handoff

update_agent() spans a new update_agent bar (parent agent_session) with lk.previous_agent_label and lk.agent_label. The old agent's drain_agent_activity (with on_exit inside) and the new agent's start_agent_activity nest under it. The initial start stays under session_start.

Fallback adapters (LLM and TTS)

The request span gets a fallback_provider_failed event per provider that failed (lk.fallback.label, lk.fallback.index) and, on success, lk.fallback.label / lk.fallback.index for the provider that served the request. STT fallback is left for the deferred STT-events work since STT has no request span to hang these on yet.

Text input

A text_input event on agent_session (participant identity under PII, byte size) when a lk.chat text stream arrives.

  • keyterm_detection span around the keyterm-detection LLM pass (stt_context_options.keyterm_detection), nested under the agent_turn that answers the user message (the conversation event fires from that reply: it is the agent's work on the turn), falling back to agent_session when there is no reply. Its llm_request used to sit directly under agent_turn, looking like a second inference step. Attributes are counts only (lk.keyterms.count/added/removed) plus model and provider; the terms themselves stay in the session report as lk.pii.keyterms.

  • Fallback adapters report who serves. FallbackAdapter.model / .provider (LLM, TTS, STT) follow the active instance: the primary before any traffic, then whichever last served, like metrics_metadata already did. llm_node, tts_node, start_agent_activity and the adapter's own request span therefore name a real model instead of FallbackAdapter/livekit. Because request attributes are stamped when a request starts, a failover mid-request is recorded on the response side: gen_ai.response.model and gen_ai.provider.name of the instance that answered are set on the adapter request span and on llm_node/tts_node when the node finishes. fallback_provider_failed events and the serving run carry the instance's gen_ai.request.model and gen_ai.provider.name next to lk.fallback.label/index. Usage metrics needed no change: the inner streams already emit them with their own model and the adapters suppress their own.

Tests

tests/test_coverage_spans.py: barge-in source and playout position through a fake session, the update_agent span with drain/on_exit/start nested, LLM fallback events and the serving provider, and interrupt-source precedence.

Not in this layer

STT events on the turn (deferred behind eou_wait feedback), realtime session lifecycle, and IPC high-ping / CPU pressure events.

Stacked on #7134.

🤖 Generated with Claude Code

@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from d274cf0 to f35af04 Compare September 5, 2026 20:42
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from f35af04 to dd9b4df Compare September 5, 2026 20:48
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from dd9b4df to 2d2ae2a Compare September 5, 2026 21:00
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 2d2ae2a to 40ce794 Compare September 5, 2026 21:07
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 40ce794 to 8f34d94 Compare September 5, 2026 21:10
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from d491a27 to 9e40051 Compare September 6, 2026 00:31
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from 2bdd416 to def301b Compare September 6, 2026 01:09
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from def301b to 87874d3 Compare September 6, 2026 01:28
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 87874d3 to 14c2b7c Compare September 6, 2026 01:37
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from 3731e28 to 9289b47 Compare September 6, 2026 05:23
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 6113f9f to 895ab3d Compare September 6, 2026 05:42
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch 2 times, most recently from 0aff7da to fbc6f67 Compare September 6, 2026 21:10
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from fbc6f67 to 2b84cc5 Compare September 6, 2026 22:13
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 2b84cc5 to b8b5f3a Compare September 6, 2026 22:18
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 4bedb0b to 02b24cd Compare September 6, 2026 23:04
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from a2849a9 to b00f8eb Compare September 7, 2026 00:10
@davidzhao
davidzhao marked this pull request as ready for review September 7, 2026 03:40
@davidzhao
davidzhao requested a review from a team as a code owner September 7, 2026 03:40
devin-ai-integration[bot]

This comment was marked as resolved.

@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from b00f8eb to 4801016 Compare September 7, 2026 03:54
devin-ai-integration[bot]

This comment was marked as resolved.

@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from 1f2a1e1 to fc68466 Compare September 7, 2026 05:05
@davidzhao
davidzhao force-pushed the dz/telemetry-coverage branch from fc68466 to 28c70f8 Compare September 7, 2026 05:48
davidzhao and others added 8 commits September 6, 2026 23:47
…text input

Smaller coverage gaps on existing spans, each of which came up when reading a
trace and not being able to answer a question from it.

Interruptions: agent_turn carried a single lk.interrupted boolean. It now
records an `interrupted` event with lk.interruption.source, set by the caller
that knows the cause: `audio_activity` (barge-in), `user_turn` (a committed
turn preempting the reply), or `programmatic` (session.interrupt(), a tool,
teardown). The pipeline and say paths also stamp lk.playout.position, how many
seconds had actually played when the user cut in. The false-interruption path
records `playout_paused` when a barge-in pauses playout and a
`false_interruption` event with lk.false_interruption.resumed when the timer
resolves it.

Agent handoff: update_agent() spans a new `update_agent` bar (parent:
agent_session) with lk.previous_agent_label and lk.agent_label; the old agent's
drain_agent_activity (with on_exit inside) and the new agent's
start_agent_activity nest under it. The initial start stays under session_start.

Fallback adapters (LLM and TTS): the request span gets a
`fallback_provider_failed` event per provider that failed (lk.fallback.label,
lk.fallback.index) and, on success, lk.fallback.label / lk.fallback.index for
the provider that served the request. Previously the adapter only renamed the
request span, so a silent failover was invisible.

Text input: a `text_input` event on agent_session (participant, byte size) when
a lk.chat text stream arrives, so text-mode conversations show their input edge.

Tests: tests/test_coverage_spans.py covers barge-in source and playout position
through a fake session, the update_agent span and its children, LLM fallback
events, and interrupt-source precedence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The keyterm-detection LLM pass ran in the reply task's context, so its
llm_request appeared directly under agent_turn as if it were a second
inference step. It is STT context for later turns, not part of any reply: a
keyterm_detection span under agent_session, with model, provider and keyterm
counts (the terms themselves stay in the session report as lk.pii.keyterms).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ecord the model that answered

FallbackAdapter.model/provider (LLM, TTS, STT) now follow the active instance,
the primary before any traffic and then whichever last served, as
metrics_metadata already did. llm_node, tts_node, start_agent_activity and the
adapter's own request span name a real model instead of the adapter. Since
request attributes are stamped at request start, a failover mid-request is
recorded on the response side: gen_ai.response.model and gen_ai.provider.name
of the instance that answered, on the adapter request span and on the node
when it finishes. Fallback events carry the instance's model and provider next
to its label. Usage metrics needed no change: inner streams emit them with
their own model and the adapters suppress their own.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the message

The conversation event that starts a detection pass fires from the reply
answering the user message, so the pass is that agent's work on the turn.
Parent it to the current span at trigger time (the agent_turn), falling back to
the session root when nothing is current (a skipped reply, user code editing
the history).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ource before the pause cancel, handoff start under update_agent, text input size in bytes

- fallback adapters stamp the span the request was made under (llm_node/tts_node)
  with the instance that served, at the moment it serves, instead of the node
  reading the adapter's shared active instance when it finishes: concurrent
  requests on one adapter can be served by different instances
- a paused reply resolved by a user turn is interrupted by the pause cancel
  before its cause was named; name it first, and in the pause cancel itself
- a handoff's start_agent_activity takes the update_agent span as its parent
  explicitly rather than relying on the session start context being cleared
- lk.text_input.size counts UTF-8 bytes like the RPC payload sizes

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…alize the keyterm span's provider

model/provider read the first instance marked available (the primary once all
are down), so a recovered primary is reported again before it has served; the
instance that actually serves is stamped per request by the stream.
keyterm_detection now runs the provider through gen_ai_provider_name like every
other span, omitting it when unknown.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
self._authorized_at: float | None = None
# telemetry: who interrupted this speech, recorded by the caller that knows
self._interrupt_source: str | None = None
self._interruption_recorded = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't think this is used?

trace_types.ATTR_AGENT_LABEL: self._next_activity.agent.label,
},
)
handoff_token = otel_context.attach(trace.set_span_in_context(handoff_span))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found by Codex:

AgentActivity.start() creates long-lived background tasks in _start_session(). Because this span is current here, asyncio.create_task() copies it into those tasks. Detaching in finally only restores this task, so later STT or scheduling spans can still use the ended update_agent span as their parent. Could we keep the root context current while creating those tasks and pass the handoff context only to the lifecycle spans?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants