Skip to content

feat(server): publish [:grpc, :server, :rpc, :abort] telemetry - #588

Open
ziomecka wants to merge 1 commit into
elixir-grpc:masterfrom
ziomecka:feat/server-rpc-abort-telemetry
Open

feat(server): publish [:grpc, :server, :rpc, :abort] telemetry#588
ziomecka wants to merge 1 commit into
elixir-grpc:masterfrom
ziomecka:feat/server-rpc-abort-telemetry

Conversation

@ziomecka

@ziomecka ziomecka commented Sep 2, 2026

Copy link
Copy Markdown

An RPC that the adapter stops before it returns publishes no server telemetry. The exit signal that stops the RPC process does not unwind it, so the :telemetry.span/3 around the call (telemetry.ex:95) publishes neither :stop nor :exception. Expired deadlines, client cancellations and dropped connections all leave no record.

This adds [:grpc, :server, :rpc, :abort], published from the cowboy handler process, which outlives the RPC process.

  • Publishes the event just before the exit signal, at both abort sites: send_error/4 (handler.ex:466) and terminate/3 (handler.ex:528). Both now go through one abort_rpc/2.
  • Metadata is :stream, :server, :endpoint, :path, :pid, :reason, with :duration measured from the arrival of the request. There is no :function_name: it is resolved in the RPC process.
  • The loop state gains stream, endpoint, route and started_at.
  • The event is guarded to one per call. Both sites can run for the same call, and Process.exit/2 is asynchronous, so Process.alive?/1 can still be true at the second one.
  • grpc_core is documentation only. The event name is built from the public GRPC.Telemetry.server_rpc_prefix/0, so grpc_server still compiles against the released grpc_core.

Measured on a service behind a proxy enforcing a 2400 ms deadline, over 6 hours at ~800 req/s: the longest server span recorded for the hot method was 2386 ms and none was above 2400 ms, against ~40 ms normally. Every call that crossed the deadline left no server-side record.

Tests in handler_test.exs:

  • An expired deadline publishes :abort, and neither :stop nor :exception.
  • A dropped connection publishes :abort, covering the terminate/3 site.
  • One :abort per call when both sites run, using a handler that traps exits so the RPC process is still alive at terminate/3.
  • A call that returns publishes :stop and no :abort, since terminate/3 runs for every request.

Three of the four fail with the publish call removed. Full suite green, mix format --check-formatted and MIX_ENV=test mix compile --warnings-as-errors clean, CHANGELOG entry under Unreleased. Verified on Elixir 1.20.3 / OTP 28 only; CI will be the first run of the older matrix rows.

Two choices I am glad to change: the one-per-call guard uses the process dictionary, and threading it through the loop state instead means changing send_error/4 to return {req, state} and updating its seven call sites; and :abort is a guess at your preferred name.

An RPC the adapter stops before it returns — expired deadline, client
cancellation, a dropped connection — is stopped with an exit signal that
does not unwind its process. `after` blocks never run, so the
`:telemetry.span/3` around the call publishes neither `:stop` nor
`:exception`, and such calls leave no telemetry at all.

Publish `:abort` from the cowboy handler process, which outlives the RPC
process, carrying the stream, request path, RPC pid and exit reason. Both
abort paths can run for one call and the exit signal is asynchronous, so
the event is guarded to one per call.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant