Skip to content

Add account and container metrics for client aborts - #3296

Merged
jitheshtr merged 4 commits into
linkedin:masterfrom
jitheshtr:jirajan/g3-client-abort-container-metric
Aug 25, 2026
Merged

Add account and container metrics for client aborts#3296
jitheshtr merged 4 commits into
linkedin:masterfrom
jitheshtr:jirajan/g3-client-abort-container-metric

Conversation

@jitheshtr

Copy link
Copy Markdown
Contributor

Summary

Ambry needs a per-account/container signal for requests that the remote client abandons. This change adds additive abort counters for remote inactivity and failed response writes without changing response status codes or normalizing existing status-metric ordering.

What changed

  1. Added ClientAbortCount for all safely attributed client terminations and ServerErrorClientAbortCount only when the abort caused the request's recorded 5xx classification. Both counters are emitted at container and account level, allowing the subset to discount client-caused server errors without hiding a genuine 5xx whose response write later failed.
  2. Replaced separate termination guards with one request-scoped atomic outcome shared by idle timeout, channel inactivity, and response-write failure. The first cause wins, so one request contributes to at most one termination-duration histogram and idle cleanup cannot become a client abort.
  3. Captured server provenance at the Netty boundary. Outbound server closure sets a channel marker, while the channel also retains the existing RestServerState; service-down shutdown closes are excluded even when Netty bypasses outbound handlers.
  4. Covered actual response-write failures, inactive writes, and 100-Continue failures while excluding synthetic cleanup callbacks and lifecycle-only RequestChannelClosed classification. A failed write of an existing 5xx records the general abort and duration but stays out of the subtractable subset.

Suggested reading order: PublicAccessLogHandler, NettyMessageProcessor, NettyResponseChannel, RestRequestMetricsTracker, then metric registration and tests.

Known gaps

Attribution starts only after AccountAndContainerInjector supplies container metrics. Errors such as RequestChannelClosed or router ChannelClosed remain excluded unless a Netty boundary independently proves remote termination. The change adds two series to each existing account/container-operation and account-operation tuple, increasing the per-tuple entity-operation series from 11 to 13.

For bodiless non-GET responses, metadata completion can precede any observable disconnect, so the causal 5xx subset can legitimately remain sparse even when the general abort counter captures later response-write terminations.

Risk Assessment

This is metrics and termination-attribution logic only. It does not modify storage, named-blob operations, TTL, deletion, retries, response codes, response bodies, or client-visible timing. The channel evidence is terminal and channel-local; the termination outcome is request-local and atomic. Negative tests cover idle timeout during streaming, outbound server close, unsafe service shutdown, genuine 5xx response-write failure, lifecycle-only exceptions, inline and deferred inactive-write ordering, and duplicate termination paths. Durability risk: none.

Testing Done

  • Local code review completed
Command Result
Focused API and REST test classes 71 passed
./gradlew :ambry-api:test :ambry-rest:test API: 235 passed, 6 skipped; REST: 115 passed
./gradlew :ambry-api:javadoc :ambry-rest:javadoc Passed
Targeted regression mutations Streamed-content attribution, synthetic-cleanup exclusion, and non-client cause filtering each failed under its matching mutation

Tests generated with unit-tests plugin

🤖 Generated with GitHub Copilot CLI

Jithesh Rajan and others added 4 commits August 21, 2026 21:10
A request the client abandons mid-flight is reported to the client as
400, and that 400 is what ContainerMetrics records. Per container, an
abort is therefore indistinguishable from a genuinely malformed request:
both land in BadRequestCount and ClientErrorCount. getErrorResponse's own
comment already notes this ("This would then be recorded as client error
in ContainerMetrics").

NettyMetrics has always kept the two apart - clientEarlyTerminationTest
asserts that aborts do not count toward NettyResponseChannel's
BadRequestCount - so this brings ContainerMetrics in line with that.

Add ClientAbortCount to EntityOperationMetrics, so it is emitted at both
container and account level, and set it from the existing client
termination branch of NettyResponseChannel#getErrorResponse via a flag on
RestRequestMetricsTracker.

The change is deliberately additive: aborts keep counting toward
BadRequestCount and ClientErrorCount, and the new counter is what lets a
dashboard subtract them back out. Reclassifying them instead would change
the value of existing series, putting a step at rollout and breaking
comparison across it. Tests assert the additive property so it cannot be
changed by accident.

Note this only covers aborts that happen after AccountAndContainerInjector
has resolved the account and container; earlier aborts have no container
metrics to record against.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record additive per-account and per-container counters for safely proven remote terminations and the subset retaining a 5xx status. Preserve every existing response and status metric classification so the subset can discount server-error telemetry without changing client behavior.

Mark outbound server close intent on the channel, classify remote inactivity and failed response writes at the network boundary, and share an atomic request-scoped guard across both paths. Exclude idle/server closes, lifecycle-only RequestChannelClosed errors, synthetic cleanup callbacks, and failures while writing an existing server-error response.

Cover real remote, server, and idle closure ordering; response-write and 100-Continue failures; genuine server errors; retained status metrics; and exactly-once recording.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use one request-scoped termination outcome across idle timeout, channel inactivity, and response-write failure. Preserve existing status-metric ordering, retain general abort telemetry for failed writes of genuine 5xx responses, and exclude service-down Netty shutdown closes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover streamed response failures, synthetic cleanup callbacks, and non-client write failures with focused mutation-killing tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov-commenter

codecov-commenter commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 27.71084% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.70%. Comparing base (52ba813) to head (3ca9d3e).
⚠️ Report is 416 commits behind head on master.

Files with missing lines Patch % Lines
...va/com/github/ambry/rest/NettyResponseChannel.java 0.00% 21 Missing ⚠️
...a/com/github/ambry/rest/NettyMessageProcessor.java 0.00% 18 Missing ⚠️
.../com/github/ambry/rest/PublicAccessLogHandler.java 63.15% 6 Missing and 1 partial ⚠️
...m/github/ambry/rest/RestRequestMetricsTracker.java 53.84% 5 Missing and 1 partial ⚠️
...va/com/github/ambry/frontend/ContainerMetrics.java 0.00% 4 Missing ⚠️
.../github/ambry/frontend/EntityOperationMetrics.java 42.85% 4 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3296       +/-   ##
=============================================
- Coverage     64.24%   50.70%   -13.54%     
+ Complexity    10398     8689     -1709     
=============================================
  Files           840      938       +98     
  Lines         71755    80652     +8897     
  Branches       8611     9722     +1111     
=============================================
- Hits          46099    40898     -5201     
- Misses        23004    36350    +13346     
- Partials       2652     3404      +752     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jitheshtr
jitheshtr marked this pull request as ready for review August 24, 2026 23:03
@jitheshtr
jitheshtr merged commit 376f0b7 into linkedin:master Aug 25, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants