feat(telemetry): nest event loop stalls under the blocked span and sample their stacks

[davidzhao]

Stacked on #2501. Not a port: the Python monitor gets these from sys._current_frames() and the blocked task's context, which Node has no equivalent for; this layer reaches the same observable result with Node mechanisms.

Description

A cloud export from the JS build showed every event_loop_blocked span parented to agent_session with no stack, which made the stalls hard to act on. Two changes, both matching what the Python monitor reports:

A stall nests under the span that was blocked. Every framework span is created and ended on the main thread, so a span created before a stall and still open (or ended when the blocking call returned) was current during it. BlockedSpanTracker, a span processor installed wherever the framework installs its own processors, keeps creation and end times; the monitor parents a stall under the innermost such span, then the session root, then the job root. A slow tool lands under function_tool, a slow RPC under rpc_handler, a slow hook under on_user_turn_completed. Creation time is used rather than the span's start time, so a back-dated eou_wait cannot claim a stall that predates it.

lk.blocking.stack is populated. The watchdog worker thread attaches an inspector session to the main thread and, when the heartbeat is a warn threshold late, posts Debugger.pause: V8 honours it inside a spinning script, hands back full symbolized frames (optimized code included), and resumes about a millisecond later. A second sample at 10x the threshold, like Python's late sample. The format matches the Python monitor: # loop thread sampled Nms into the stall then the innermost 20 frames; Node internals dropped except innermost; frames outer than the framework's job runner cut. The warn log names the innermost frame of the agent's own code.

Sampling is adaptive by default: off until the process reports its first code-caused stall, then on for that process, so a healthy process never enables the debugger domain. LIVEKIT_AGENTS_LOOP_BLOCK_STACKS=always turns it on from the start, never off. Enabling the debugger domain costs one loop gap of about 90 ms with the framework loaded (script enumeration), once per process; in adaptive mode that gap is itself reported as a stall with the note that the sampler was starting, and in always mode it happens at monitor start, before any job work. Not enabled while an inspector is attached to the process. The worker process and an idle child sample too (third commit): their stalls only reach the log, and a log with a stack names the blocking code just as a span does, so no stall is reported with a note saying its process does not sample.

Rejected alternative, not what ships: V8's sampling profiler was tried first. On Node 24 only the first profile of a process names code optimized before it started, reading its samples requires restarting the profile, and each restart blocked the loop for 25 to 40 ms with the framework loaded. Those figures describe the profiler, not the Debugger.pause mechanism above.

Measured overhead of the shipped mechanism with the framework loaded (ten interleaved rounds, medians): with the debugger domain enabled a tight compute loop is 1.1% slower and a mixed JSON/regex/promise workload 0.6% slower; idle CPU with the 20 ms heartbeat is unchanged (0.33% vs 0.31% of a core); a pause sample costs about 1 ms and only while a stall is under way. In adaptive mode none of this is paid until the process's first code-caused stall.

Limits. A native call that does not check for interrupts (sync file or child-process calls, native addons) keeps the pause waiting until it returns; the sample then shows the caller still in the frame that made the call, which is what the Python watchdog sees once a native call releases the GIL. lk.blocking.task stays unset: Node has no task names.

Verified live

Hotel receptionist example with a test RPC that spins for 600 ms, called twice from a client: both rpc_handler spans under agent_session, both stalls under their rpc_handler, the first without a stack (adaptive), the second sampled 121 ms into the stall naming the handler, the SDK's invokeRpcHandler, and the interceptor's interceptIncoming.

Testing

• telemetry/loop_monitor_stacks.test.ts (13 tests): parenting under an open span, fallbacks, tracker retention, adaptive / always / never, the formatted stack from a real blocked call, restart bounds, env parsing. The existing 28 monitor tests unchanged.
• Full agents suite green; build, typecheck, lint, API report updated.

API report noise (second commit). Exports in @livekit/agents are not release-tagged, so API Extractor wrote an ae-missing-release-tag warning for every one of them into agents.api.md (854 by this layer), burying the real changes in each review. agents/api-extractor.json now silences that one message for this package and the report is regenerated: the diff is only the removed warning lines. Other warning kinds (ae-forgotten-export, ae-unresolved-link, ...) are untouched.

Review follow-ups (fourth commit)

• Parenting: when two in-flight operations of the same kind qualify for a stall (two function_tools of one turn, two rpc_handlers), the tracker no longer picks the newer one, which timing cannot justify: the stall lands on their nearest common ancestor among the open spans, or on the session/job root when there is none. Operations of different kinds (a user_turn open while an rpc_handler runs) still resolve to the newest. An exact answer for sampled stalls is possible as a follow-up: evaluate the active span on the paused main thread during the sample, which reads the blocked operation's async context the way Python reads the blocked task's context variables.
• Sample offsets are measured from the stall's start as the report dates it; the header used to overstate by one heartbeat interval.
• stacks: always|adaptive with watchdog: false is normalized to never (the sampler runs on the watchdog thread), documented, so such a configuration does not note on every stall that sampling starts after the first one.
• An inspector attached after sampling started stops it: the heartbeat checks the inspector URL each tick and has the watchdog disable the debugger domain and disconnect, so a debugger's breakpoints are never resumed by the sampler. A debugger; statement in user code is still resumed while no inspector is attached: it only pauses at all because the sampler enabled the domain. Later stalls note that an inspector is attached. Tested against a real inspector.open().

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 776c050

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-meta	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 1 new potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟡 Unrelated spans capture stall parenting

With another provider span open during a job stall, blockedSpan can select it as the parent. Candidates are never restricted to the fallback job trace. The stall enters an unrelated trace instead of the job timeline.

Learn more

A span processor receives every recording span created by its provider, not only LiveKit framework spans. Custom providers can therefore contribute application or auto-instrumentation spans from unrelated traces. blockedContext receives a base context carrying the current session or job root, but blockedSpan does not use that trace identity when building candidates. A newer unrelated leaf can become the parent, and trace.setSpan then replaces the base span with that unrelated span.

Example: A background HTTP request opens span http.request in trace A while job trace B runs a blocking tool. Since http.request was created later and overlaps the stall window, the emitted event_loop_blocked span becomes its child in trace A. It was expected under the blocked tool or the session root in trace B.

Recommended fix: Pass the fallback span context's trace ID into blockedSpan and discard candidates whose spanContext().traceId differs. Apply the filter before leaf and ambiguity selection so unrelated spans cannot suppress a valid same-trace candidate.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Devin Review found 1 new potential issue.

4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)