feat(telemetry): trace RPC calls made and handled - #7134
Open
davidzhao wants to merge 4 commits into
Open
Conversation
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 20:07
2c854c3 to
fcf2513
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 20:42
fcf2513 to
6c9ac55
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 20:48
6c9ac55 to
9aa16ed
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 5, 2026 21:00
9aa16ed to
13e4d84
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 5, 2026 21:10
1306080 to
aa343e0
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 00:16
aa343e0 to
26a2704
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 00:31
26a2704 to
6632f30
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 00:56
6632f30 to
93134c5
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 01:09
93134c5 to
8c372ed
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 01:28
8c372ed to
0840636
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 6, 2026 05:23
3371ae3 to
c690ca5
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 6, 2026 05:56
3b4d43e to
78b2948
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 08:24
dc1ab37 to
cb73c69
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 16:48
cb73c69 to
ab2d489
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 16:58
ab2d489 to
9fa37ca
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 17:13
9fa37ca to
22ff30c
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 21:10
22ff30c to
e36dc93
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 6, 2026 22:13
e36dc93 to
0c833cb
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
3 times, most recently
from
September 7, 2026 00:10
fe5b4a4 to
9fae149
Compare
davidzhao
marked this pull request as ready for review
September 7, 2026 03:38
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 7, 2026 05:05
d2fae93 to
6c714bf
Compare
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
2 times, most recently
from
September 7, 2026 06:26
0fea92f to
8dc4304
Compare
Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. livekit-rtc is gaining an RpcInterceptor hook (add_rpc_interceptor on LocalParticipant) that wraps every perform_rpc call and every dispatched handler invocation; this installs a tracing interceptor on the job's local participant and turns each call into a span following the OpenTelemetry RPC semantic conventions: - rpc_call (SpanKind.CLIENT) for outgoing calls, parented to the current span so an RPC issued from a tool nests under function_tool; - rpc_handler (SpanKind.SERVER) for incoming invocations, parented to the primary session's root span so it lands on the session timeline. Attributes: rpc.system=livekit, rpc.method, lk.rpc.request_id, lk.pii.rpc.caller_identity / lk.pii.rpc.destination_identity, lk.rpc.payload_size, lk.pii.rpc.payload (truncated to 1 KiB), lk.rpc.response_size, lk.rpc.response_timeout, lk.rpc.error_code, and lk.rpc.handler_registered (false when a client called a method the agent never registered). RpcError and handler exceptions set error status. Installed from JobContext.connect() after the room connects, and from RoomIO.start() for a room connected elsewhere. On an SDK without the hook (the current livekit==1.1.17 pin) install is a no-op with one debug log, so this can land ahead of the SDK release; tracing activates once the pin moves to the release carrying RpcInterceptor. Tests exercise the interceptor with fake continuations (attributes, parenting, truncation, error codes, unregistered method, handler exceptions) and the install/degrade behavior; they pass against both the pinned SDK and the interceptor-capable SDK source tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rpc.system is always livekit here, so it carried no information. Responses are now recorded like requests: lk.pii.rpc.response, truncated to 1 KiB, alongside the existing size, so a client/agent disagreement about a message format is readable from both sides of the span. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Participant identities are application identifiers, not end-user data. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
RoomIO.start only instrumented a room that was already connected; a room that connected later (ctx.connect() after session.start(), or one the user connects) got no RPC spans unless JobContext.connect() installed it. The install now lives in the connection-state handler, so every ordering and reconnects are covered; it is idempotent, one interceptor instance deduped by identity in the SDK. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
davidzhao
force-pushed
the
dz/telemetry-rpc
branch
from
September 7, 2026 06:50
8dc4304 to
a3701b2
Compare
chenghao-mou
reviewed
Sep 7, 2026
|
|
||
|
|
||
| def _truncate(payload: str) -> str: | ||
| return payload if len(payload) <= MAX_PAYLOAD_ATTR_LEN else payload[:MAX_PAYLOAD_ATTR_LEN] |
Member
There was a problem hiding this comment.
payload[:MAX_PAYLOAD_ATTR_LEN] alone should be enough.
Member
There was a problem hiding this comment.
also, should we use byte length check here?
| return attrs | ||
|
|
||
|
|
||
| def _handler_parent_context() -> otel_context.Context | None: |
Member
There was a problem hiding this comment.
nit: we should reuse session_context.session_root_context
| ) as span: | ||
| try: | ||
| response = await next(invocation) | ||
| except rtc.RpcError as e: |
Member
There was a problem hiding this comment.
found by Codex:
The SDK enforces the response deadline by cancelling the incoming interceptor chain. This produces
CancelledErrorhere; the SDK converts it toRpcError(RESPONSE_TIMEOUT)only after this interceptor has unwound. SinceCancelledErroralso bypasses OpenTelemetry’sexcept Exception, the handler span ends withUNSETstatus and no error code even though the caller receivesRESPONSE_TIMEOUT.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Neither user-registered RPC methods nor the avatar datastream RPCs appeared in a trace. This installs a tracing interceptor on the job's local participant and turns each RPC into a span.
Depends on the
RpcInterceptorhook inlivekit-rtc(livekit/python-sdks#806). On the currentlivekit==1.1.17pininstallis a no-op with one debug log, so this can land first; tracing activates once the pin moves to the SDK release carrying the hook.Spans
rpc_callfunction_toolrpc_handlerAttributes:
rpc.method,lk.rpc.request_id,lk.rpc.caller_identity,lk.rpc.destination_identity,lk.rpc.payload_size,lk.rpc.response_size,lk.rpc.response_timeout,lk.rpc.error_code,lk.rpc.handler_registered(false when a client called a method the agent never registered). Request and response bodies are under PII keys,lk.pii.rpc.payloadandlk.pii.rpc.response, truncated to 1 KiB.RpcErrorand handler exceptions set error status.Where
telemetry/rpc.py:TracingRpcInterceptor(subclassesrtc.RpcInterceptorwhen present) andinstall(local_participant).JobContext.connect()after the room connects, and fromRoomIO.start()for a room connected elsewhere. One singleton interceptor; the SDK dedups registrations by identity, so repeated installs are no-ops.Tests
tests/test_rpc_tracing.pydrives the interceptor with fake continuations: attributes and parenting for both directions, request and response truncation, error codes and status, the unregistered-method flag, handler exceptions, andinstallregistering once or degrading. Passes against both the pinned SDK and the interceptor-capable SDK source tree.Stacked on #7131.
🤖 Generated with Claude Code