Add account and container metrics for client aborts - #3296
Merged
jitheshtr merged 4 commits intoAug 25, 2026
Merged
Conversation
A request the client abandons mid-flight is reported to the client as
400, and that 400 is what ContainerMetrics records. Per container, an
abort is therefore indistinguishable from a genuinely malformed request:
both land in BadRequestCount and ClientErrorCount. getErrorResponse's own
comment already notes this ("This would then be recorded as client error
in ContainerMetrics").
NettyMetrics has always kept the two apart - clientEarlyTerminationTest
asserts that aborts do not count toward NettyResponseChannel's
BadRequestCount - so this brings ContainerMetrics in line with that.
Add ClientAbortCount to EntityOperationMetrics, so it is emitted at both
container and account level, and set it from the existing client
termination branch of NettyResponseChannel#getErrorResponse via a flag on
RestRequestMetricsTracker.
The change is deliberately additive: aborts keep counting toward
BadRequestCount and ClientErrorCount, and the new counter is what lets a
dashboard subtract them back out. Reclassifying them instead would change
the value of existing series, putting a step at rollout and breaking
comparison across it. Tests assert the additive property so it cannot be
changed by accident.
Note this only covers aborts that happen after AccountAndContainerInjector
has resolved the account and container; earlier aborts have no container
metrics to record against.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record additive per-account and per-container counters for safely proven remote terminations and the subset retaining a 5xx status. Preserve every existing response and status metric classification so the subset can discount server-error telemetry without changing client behavior. Mark outbound server close intent on the channel, classify remote inactivity and failed response writes at the network boundary, and share an atomic request-scoped guard across both paths. Exclude idle/server closes, lifecycle-only RequestChannelClosed errors, synthetic cleanup callbacks, and failures while writing an existing server-error response. Cover real remote, server, and idle closure ordering; response-write and 100-Continue failures; genuine server errors; retained status metrics; and exactly-once recording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use one request-scoped termination outcome across idle timeout, channel inactivity, and response-write failure. Preserve existing status-metric ordering, retain general abort telemetry for failed writes of genuine 5xx responses, and exclude service-down Netty shutdown closes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover streamed response failures, synthetic cleanup callbacks, and non-client write failures with focused mutation-killing tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3296 +/- ##
=============================================
- Coverage 64.24% 50.70% -13.54%
+ Complexity 10398 8689 -1709
=============================================
Files 840 938 +98
Lines 71755 80652 +8897
Branches 8611 9722 +1111
=============================================
- Hits 46099 40898 -5201
- Misses 23004 36350 +13346
- Partials 2652 3404 +752 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jitheshtr
marked this pull request as ready for review
August 24, 2026 23:03
beijxu
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ambry needs a per-account/container signal for requests that the remote client abandons. This change adds additive abort counters for remote inactivity and failed response writes without changing response status codes or normalizing existing status-metric ordering.
What changed
ClientAbortCountfor all safely attributed client terminations andServerErrorClientAbortCountonly when the abort caused the request's recorded 5xx classification. Both counters are emitted at container and account level, allowing the subset to discount client-caused server errors without hiding a genuine 5xx whose response write later failed.RestServerState; service-down shutdown closes are excluded even when Netty bypasses outbound handlers.RequestChannelClosedclassification. A failed write of an existing 5xx records the general abort and duration but stays out of the subtractable subset.Suggested reading order:
PublicAccessLogHandler,NettyMessageProcessor,NettyResponseChannel,RestRequestMetricsTracker, then metric registration and tests.Known gaps
Attribution starts only after
AccountAndContainerInjectorsupplies container metrics. Errors such asRequestChannelClosedor routerChannelClosedremain excluded unless a Netty boundary independently proves remote termination. The change adds two series to each existing account/container-operation and account-operation tuple, increasing the per-tuple entity-operation series from 11 to 13.For bodiless non-GET responses, metadata completion can precede any observable disconnect, so the causal 5xx subset can legitimately remain sparse even when the general abort counter captures later response-write terminations.
Risk Assessment
This is metrics and termination-attribution logic only. It does not modify storage, named-blob operations, TTL, deletion, retries, response codes, response bodies, or client-visible timing. The channel evidence is terminal and channel-local; the termination outcome is request-local and atomic. Negative tests cover idle timeout during streaming, outbound server close, unsafe service shutdown, genuine 5xx response-write failure, lifecycle-only exceptions, inline and deferred inactive-write ordering, and duplicate termination paths. Durability risk: none.
Testing Done
./gradlew :ambry-api:test :ambry-rest:test./gradlew :ambry-api:javadoc :ambry-rest:javadocTests generated with unit-tests plugin
🤖 Generated with GitHub Copilot CLI