Skip to content

Tolerate servlet responses without Servlet 3.0 header accessors (quick fix) - #12691

Draft
dougqh wants to merge 4 commits into
masterfrom
dougqh/servlet3-response-headers-unsupported
Draft

dougqh wants to merge 4 commits into
masterfrom
dougqh/servlet3-response-headers-unsupported

Conversation

@dougqh

@dougqh dougqh commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Makes the servlet 3.0 response-header reader tolerate response classes that don't implement the Servlet 3.0 header accessors.

HttpServletExtractAdapter.Response called HttpServletResponse.getHeaderNames() / getHeader(...) on every response. A class compiled against Servlet 2.5 that implements HttpServletResponse itself doesn't implement them, so the call throws AbstractMethodError. That escaped through HttpServerDecorator.callIGCallbackResponseAndHeaders, was reported as "Failed to decorate span on response", and stopped the response-header callbacks for every response of that class.

Now the accessors are guarded by a ClassLatch keyed on the response class:

  • Healthy path: unchanged apart from one plain flag read inside ClassLatch.
  • First failure of a class: handleAbstractMethod(response, "getHeaderNames", ...) latches the class; a getHeader failure latches it the same way, through latchIfNamed. Either way the visit stops without re-emitting keys.
  • Latched class: visits no headers.

Revised from the first version: this used to hand-roll the same pieces (a static tripwire, a per-class ClassValue, a reflection probe) plus a fallback that unwrapped ServletResponseWrapper to find a usable delegate. ClassLatch provides the first three. The unwrap is dropped because it can almost never apply: HttpServletResponseWrapper comes from the container's Servlet 3.0+ jar, so a wrapper subclass compiled against 2.5 inherits working, delegating accessors and never throws. The classes that do throw implement HttpServletResponse directly and aren't wrappers. A test pins that inheritance.

Motivation

Instrumentation telemetry shows this at HttpServletExtractAdapter$Response.getHeaderNames:48: about 10.5k events a day, steady at 400-500 an hour, on tracers 1.58 to 1.66, mostly Java 8 services.

Muzzle can't catch this. It verifies that HttpServletResponse.getHeaderNames() exists in the API, and it does. The failure is a particular runtime implementation not honouring it.

Additional Notes

Depends on #12799 (stacked on it). On Java 8, AbstractMethodError has no message once the call site has seen a working class, and without #12799 ClassLatch never latches there. The two latch tests warm the call sites with a working response first; they fail on Java 8 against master's ClassLatch and pass with #12799.

For the AppSec/IAST reviewers, please look at this:

  • responseStarted(status) fires before the header visit, so it was never affected. What was skipped is the response-header callbacks and igKeyClassifier.done() (responseHeaderDone).
  • For a latched class the visitor returns without visiting, so done() still runs with no headers. The servlet 2.x decorator instead returns a null responseGetter(), which skips the header callbacks and done() entirely. I went with "empty headers" so responseHeaderDone still fires. Happy to switch to the 2.x behaviour if AppSec prefers it.
  • Only the accessor calls are guarded. An AbstractMethodError from the classifier.accept callback says nothing about the response class and doesn't latch it (there's a test for this).

Tests: HttpServletExtractAdapterTest, 7 cases, using ASM-generated response classes so a real AbstractMethodError is thrown. They cover a class missing both accessors, a class missing only getHeader (not producible by javac, but possible from bytecode generators), a 2.5-style wrapper subclass, healthy responses after a failure, and a callback failure. They pass on the default JDK and on Java 8. The full javax-servlet-3.0 suite passes (1,360 tests).

🤖 Generated with Claude Code

@dougqh dougqh added type: bug fix Bug fix tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM inst:servlet Servlet instrumentation labels Sep 29, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 58.93% (-0.54%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bea559a | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.97 s 13.87 s [+0.0%; +1.4%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.91 s 12.97 s [-1.2%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 17.19 s 17.18 s [-0.9%; +1.1%] (no difference)
startup:petclinic:iast:Agent 17.00 s 17.10 s [-1.3%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 16.69 s 16.80 s [-1.5%; +0.3%] (no difference)
startup:petclinic:sca:Agent 17.14 s 17.10 s [-0.5%; +1.0%] (no difference)
startup:petclinic:tracing:Agent 16.14 s 16.14 s [-1.0%; +1.0%] (no difference)

Commit: bea559a1 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh changed the title Tolerate servlet responses without Servlet 3.0 header accessors Tolerate servlet responses without Servlet 3.0 header accessors (quick fix) Sep 29, 2026
ClassLatch attributes an AbstractMethodError to a class by parsing HotSpot's
message. JDK 8 throws it with a null message once the call site has
dispatched to a class that does implement the method, which in production is
nearly always, so handleAbstractMethod never latched on Java 8 and every call
for a deficient class kept throwing.

When there is no message, attribute the error by checking whether the key
class still has a public abstract method of that name, as a concrete class
that never implemented an interface method does. A delegating wrapper has a
concrete implementation, so an error from its delegate is still not blamed
on it. The reflection runs only on the failure path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/servlet3-response-headers-unsupported branch from 5a1c21e to bac191e Compare October 8, 2026 18:43
@dougqh
dougqh changed the base branch from master to dougqh/classlatch-jdk8-null-message October 8, 2026 18:44
public static final class Response extends HttpServletExtractAdapter<HttpServletResponse> {
public static final Response GETTER = new Response();

static final HeaderAccessLatch HEADER_ACCESS = new HeaderAccessLatch();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To Claude, let's just call this HEADER_LATCH


@Override
public void forEachKey(HttpServletResponse carrier, AgentPropagation.KeyClassifier classifier) {
final Collection<String> headerNames = HEADER_ACCESS.tryApply(carrier);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This latching situation isn't as straight-forward as others, but I think this application makes sense. I'm curious to hear what others think.

The latch is really driven by calls to getHeader in the loop below, but once the latch is triggered, we cut off the flow of calls to getHeaders to short-circuit the entire process.

dougqh and others added 3 commits October 8, 2026 16:07
Class.getMethods() resolves every public method's signature, so a class with
a method that references a type missing from the class path raises
NoClassDefFoundError. That escaped from the AbstractMethodError handler and
out of tryApply, where before the null-message attribution the fallback was
returned. Catch LinkageError alongside SecurityException and cache "cannot
tell", so the class is not latched and the fallback is returned as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HttpServletExtractAdapter.Response called HttpServletResponse.getHeaderNames()
and getHeader(...) on every response. A class compiled against Servlet 2.5
that implements HttpServletResponse itself does not implement them, so the
calls threw AbstractMethodError, which escaped as "Failed to decorate span on
response" and skipped the response-header callbacks for every response of
that class.

Guard the accessors with a ClassLatch keyed on the response class. A latched
class visits no headers, so the header-done callback still fires. Subclasses
of HttpServletResponseWrapper need no special handling: they inherit the
container's delegating accessors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Response overrides forEachKey, so the base class's getHeaderNames/getHeader
contract only served Request; move it there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/servlet3-response-headers-unsupported branch from bac191e to bea559a Compare October 8, 2026 20:07
Base automatically changed from dougqh/classlatch-jdk8-null-message to master October 9, 2026 22:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst:servlet Servlet instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant