feat(telemetry): interruption detail, handoff span, fallback events, text input - #7137
Open
davidzhao wants to merge 8 commits into
Open
feat(telemetry): interruption detail, handoff span, fallback events, text input#7137davidzhao wants to merge 8 commits into
davidzhao wants to merge 8 commits into
Conversation
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
2 times, most recently
from
September 5, 2026 20:42
d274cf0 to
f35af04
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 5, 2026 20:48
f35af04 to
dd9b4df
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 5, 2026 21:00
dd9b4df to
2d2ae2a
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 5, 2026 21:07
2d2ae2a to
40ce794
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 5, 2026 21:10
40ce794 to
8f34d94
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
2 times, most recently
from
September 6, 2026 00:31
d491a27 to
9e40051
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
2 times, most recently
from
September 6, 2026 01:09
2bdd416 to
def301b
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 6, 2026 01:28
def301b to
87874d3
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 6, 2026 01:37
87874d3 to
14c2b7c
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
2 times, most recently
from
September 6, 2026 05:23
3731e28 to
9289b47
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 6, 2026 05:42
6113f9f to
895ab3d
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
2 times, most recently
from
September 6, 2026 21:10
0aff7da to
fbc6f67
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 6, 2026 22:13
fbc6f67 to
2b84cc5
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 6, 2026 22:18
2b84cc5 to
b8b5f3a
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 6, 2026 23:04
4bedb0b to
02b24cd
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 7, 2026 00:10
a2849a9 to
b00f8eb
Compare
davidzhao
marked this pull request as ready for review
September 7, 2026 03:40
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 7, 2026 03:54
b00f8eb to
4801016
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 7, 2026 05:05
1f2a1e1 to
fc68466
Compare
davidzhao
force-pushed
the
dz/telemetry-coverage
branch
from
September 7, 2026 05:48
fc68466 to
28c70f8
Compare
…text input Smaller coverage gaps on existing spans, each of which came up when reading a trace and not being able to answer a question from it. Interruptions: agent_turn carried a single lk.interrupted boolean. It now records an `interrupted` event with lk.interruption.source, set by the caller that knows the cause: `audio_activity` (barge-in), `user_turn` (a committed turn preempting the reply), or `programmatic` (session.interrupt(), a tool, teardown). The pipeline and say paths also stamp lk.playout.position, how many seconds had actually played when the user cut in. The false-interruption path records `playout_paused` when a barge-in pauses playout and a `false_interruption` event with lk.false_interruption.resumed when the timer resolves it. Agent handoff: update_agent() spans a new `update_agent` bar (parent: agent_session) with lk.previous_agent_label and lk.agent_label; the old agent's drain_agent_activity (with on_exit inside) and the new agent's start_agent_activity nest under it. The initial start stays under session_start. Fallback adapters (LLM and TTS): the request span gets a `fallback_provider_failed` event per provider that failed (lk.fallback.label, lk.fallback.index) and, on success, lk.fallback.label / lk.fallback.index for the provider that served the request. Previously the adapter only renamed the request span, so a silent failover was invisible. Text input: a `text_input` event on agent_session (participant, byte size) when a lk.chat text stream arrives, so text-mode conversations show their input edge. Tests: tests/test_coverage_spans.py covers barge-in source and playout position through a fake session, the update_agent span and its children, LLM fallback events, and interrupt-source precedence. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The keyterm-detection LLM pass ran in the reply task's context, so its llm_request appeared directly under agent_turn as if it were a second inference step. It is STT context for later turns, not part of any reply: a keyterm_detection span under agent_session, with model, provider and keyterm counts (the terms themselves stay in the session report as lk.pii.keyterms). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ecord the model that answered FallbackAdapter.model/provider (LLM, TTS, STT) now follow the active instance, the primary before any traffic and then whichever last served, as metrics_metadata already did. llm_node, tts_node, start_agent_activity and the adapter's own request span name a real model instead of the adapter. Since request attributes are stamped at request start, a failover mid-request is recorded on the response side: gen_ai.response.model and gen_ai.provider.name of the instance that answered, on the adapter request span and on the node when it finishes. Fallback events carry the instance's model and provider next to its label. Usage metrics needed no change: inner streams emit them with their own model and the adapters suppress their own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the message The conversation event that starts a detection pass fires from the reply answering the user message, so the pass is that agent's work on the turn. Parent it to the current span at trigger time (the agent_turn), falling back to the session root when nothing is current (a skipped reply, user code editing the history). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ource before the pause cancel, handoff start under update_agent, text input size in bytes - fallback adapters stamp the span the request was made under (llm_node/tts_node) with the instance that served, at the moment it serves, instead of the node reading the adapter's shared active instance when it finishes: concurrent requests on one adapter can be served by different instances - a paused reply resolved by a user turn is interrupted by the pause cancel before its cause was named; name it first, and in the pause cancel itself - a handoff's start_agent_activity takes the update_agent span as its parent explicitly rather than relying on the session start context being cleared - lk.text_input.size counts UTF-8 bytes like the RPC payload sizes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…alize the keyterm span's provider model/provider read the first instance marked available (the primary once all are down), so a recovered primary is reported again before it has served; the instance that actually serves is stamped per request by the stream. keyterm_detection now runs the provider through gen_ai_provider_name like every other span, omitting it when unknown. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
chenghao-mou
reviewed
Sep 7, 2026
| self._authorized_at: float | None = None | ||
| # telemetry: who interrupted this speech, recorded by the caller that knows | ||
| self._interrupt_source: str | None = None | ||
| self._interruption_recorded = False |
Member
There was a problem hiding this comment.
nit: I don't think this is used?
| trace_types.ATTR_AGENT_LABEL: self._next_activity.agent.label, | ||
| }, | ||
| ) | ||
| handoff_token = otel_context.attach(trace.set_span_in_context(handoff_span)) |
Member
There was a problem hiding this comment.
Found by Codex:
AgentActivity.start()creates long-lived background tasks in_start_session(). Because this span is current here,asyncio.create_task()copies it into those tasks. Detaching infinallyonly restores this task, so later STT or scheduling spans can still use the endedupdate_agentspan as their parent. Could we keep the root context current while creating those tasks and pass the handoff context only to the lifecycle spans?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Smaller coverage gaps on existing spans, each of which came up when reading a trace and not being able to answer a question from it.
Interruptions
agent_turncarried a singlelk.interruptedboolean. It now records aninterruptedevent withlk.interruption.source, set by the caller that knows the cause:audio_activityuser_turnprogrammaticsession.interrupt(), a tool, teardownThe pipeline and
saypaths also stamplk.playout.position, how many seconds had actually played when the user cut in. The false-interruption path records aplayout_pausedevent when a barge-in pauses playout and afalse_interruptionevent withlk.false_interruption.resumedwhen the timer resolves it.Agent handoff
update_agent()spans a newupdate_agentbar (parentagent_session) withlk.previous_agent_labelandlk.agent_label. The old agent'sdrain_agent_activity(withon_exitinside) and the new agent'sstart_agent_activitynest under it. The initial start stays undersession_start.Fallback adapters (LLM and TTS)
The request span gets a
fallback_provider_failedevent per provider that failed (lk.fallback.label,lk.fallback.index) and, on success,lk.fallback.label/lk.fallback.indexfor the provider that served the request. STT fallback is left for the deferred STT-events work since STT has no request span to hang these on yet.Text input
A
text_inputevent onagent_session(participant identity under PII, byte size) when alk.chattext stream arrives.keyterm_detectionspan around the keyterm-detection LLM pass (stt_context_options.keyterm_detection), nested under theagent_turnthat answers the user message (the conversation event fires from that reply: it is the agent's work on the turn), falling back toagent_sessionwhen there is no reply. Itsllm_requestused to sit directly underagent_turn, looking like a second inference step. Attributes are counts only (lk.keyterms.count/added/removed) plus model and provider; the terms themselves stay in the session report aslk.pii.keyterms.Fallback adapters report who serves.
FallbackAdapter.model/.provider(LLM, TTS, STT) follow the active instance: the primary before any traffic, then whichever last served, likemetrics_metadataalready did.llm_node,tts_node,start_agent_activityand the adapter's own request span therefore name a real model instead ofFallbackAdapter/livekit. Because request attributes are stamped when a request starts, a failover mid-request is recorded on the response side:gen_ai.response.modelandgen_ai.provider.nameof the instance that answered are set on the adapter request span and onllm_node/tts_nodewhen the node finishes.fallback_provider_failedevents and the serving run carry the instance'sgen_ai.request.modelandgen_ai.provider.namenext tolk.fallback.label/index. Usage metrics needed no change: the inner streams already emit them with their own model and the adapters suppress their own.Tests
tests/test_coverage_spans.py: barge-in source and playout position through a fake session, theupdate_agentspan with drain/on_exit/start nested, LLM fallback events and the serving provider, and interrupt-source precedence.Not in this layer
STT events on the turn (deferred behind
eou_waitfeedback), realtime session lifecycle, and IPC high-ping / CPU pressure events.Stacked on #7134.
🤖 Generated with Claude Code