feat(server): publish [:grpc, :server, :rpc, :abort] telemetry - #588
Open
ziomecka wants to merge 1 commit into
Open
feat(server): publish [:grpc, :server, :rpc, :abort] telemetry#588ziomecka wants to merge 1 commit into
ziomecka wants to merge 1 commit into
Conversation
An RPC the adapter stops before it returns — expired deadline, client cancellation, a dropped connection — is stopped with an exit signal that does not unwind its process. `after` blocks never run, so the `:telemetry.span/3` around the call publishes neither `:stop` nor `:exception`, and such calls leave no telemetry at all. Publish `:abort` from the cowboy handler process, which outlives the RPC process, carrying the stream, request path, RPC pid and exit reason. Both abort paths can run for one call and the exit signal is asynchronous, so the event is guarded to one per call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An RPC that the adapter stops before it returns publishes no server telemetry. The exit signal that stops the RPC process does not unwind it, so the
:telemetry.span/3around the call (telemetry.ex:95) publishes neither:stopnor:exception. Expired deadlines, client cancellations and dropped connections all leave no record.This adds
[:grpc, :server, :rpc, :abort], published from the cowboy handler process, which outlives the RPC process.send_error/4(handler.ex:466) andterminate/3(handler.ex:528). Both now go through oneabort_rpc/2.:stream,:server,:endpoint,:path,:pid,:reason, with:durationmeasured from the arrival of the request. There is no:function_name: it is resolved in the RPC process.stream,endpoint,routeandstarted_at.Process.exit/2is asynchronous, soProcess.alive?/1can still be true at the second one.grpc_coreis documentation only. The event name is built from the publicGRPC.Telemetry.server_rpc_prefix/0, sogrpc_serverstill compiles against the releasedgrpc_core.Measured on a service behind a proxy enforcing a 2400 ms deadline, over 6 hours at ~800 req/s: the longest server span recorded for the hot method was 2386 ms and none was above 2400 ms, against ~40 ms normally. Every call that crossed the deadline left no server-side record.
Tests in
handler_test.exs::abort, and neither:stopnor:exception.:abort, covering theterminate/3site.:abortper call when both sites run, using a handler that traps exits so the RPC process is still alive atterminate/3.:stopand no:abort, sinceterminate/3runs for every request.Three of the four fail with the publish call removed. Full suite green,
mix format --check-formattedandMIX_ENV=test mix compile --warnings-as-errorsclean, CHANGELOG entry under Unreleased. Verified on Elixir 1.20.3 / OTP 28 only; CI will be the first run of the older matrix rows.Two choices I am glad to change: the one-per-call guard uses the process dictionary, and threading it through the loop state instead means changing
send_error/4to return{req, state}and updating its seven call sites; and:abortis a guess at your preferred name.