Repository navigation
Conversation
When the StatsD connection ran out of connect retries it logged at debug and stopped for the life of the JVM, so default customer logs showed no sign that tracer metrics were never sent. Warn instead, with the StatsD address, the number of attempts, and the last error.
When the DogStatsD pipe or port was missing at startup, the StatsD connection retried 20 times, 10 seconds apart, then gave up for the life of the JVM, so a DogStatsD that came up later was never used. In Azure App Service this happens when the tracer starts before DogStatsD listens. - Retry forever, doubling the delay from 10 seconds to a cap of 60 seconds. Drop the jitter on retries so the delay stays within the cap. - Warn once on the first failure, log later attempts at debug, and log once at info when a connection succeeds after failures.
The StatsD client opens its named pipe once and never reopens it. After the DogStatsD it was writing to went away, every write failed for the life of the JVM, even once a new DogStatsD served the same pipe name. In an Azure App Service overlapping restart, tracer metrics were still missing 50 minutes after the old worker's DogStatsD exited. - Track write errors in handle(). Errors more than 60 seconds apart start a new streak; a streak of at least 3 errors spanning at least 30 seconds triggers a recreate. The 60 second gap is longer than the 30 second tracer health metrics flush, so a dead pipe keeps one streak going, while brief bursts and isolated timeouts under load do not trigger it. - Recreate on the task scheduler, not the client's own thread: close the client and reconnect through the existing connect path, which retries with backoff if the pipe is still missing. - Warn on recreate, rate limited to once every 5 minutes.
NamedPipeSocketFactory kept one NamedPipeSocket per pipe and returned it to every OkHttp connection (#3493). NamedPipeSocket ignores read timeouts, so when OkHttp checks a pooled connection's health it blocks on an unbounded pipe read, and every other agent request (traces, stats, remote config, telemetry) waits behind it on the shared handle. On Azure Functions with DD_TRACE_PIPE_NAME set, the tracer queued spans but never sent or failed a trace upload until the JVM restarted. Open a new pipe handle for each socket, as the .NET tracer does for each request. #3493 shared the handle because a named pipe was thought to allow only one client at a time; that change was untested on Windows, and the .NET tracer's per-request pipe connections show the agent's pipe server accepts several clients.
NamedPipeSocket can't time out a read, so when OkHttp checks a pooled connection's health before reusing it, the check can block forever on the pipe. Give named-pipe clients a connection pool that keeps no idle connections, so each request opens and closes its own pipe connection, as the .NET tracer does. For named pipes this replaces the maxRunningRequests connection pool; the dispatcher limits are unchanged.
When a DogStatsD named pipe was set and no port was configured, DDAgentStatsDConnection left the port null, and building the client threw NullPointerException on .port(port). Tracer health metrics and JMXFetch JVM metrics were never sent over a pipe. Use port 0 when a named pipe is set and no port is configured. The client connects to the pipe and ignores the port; port 0 also selects the same packet size and socket settings the tracer uses for Unix domain sockets.
Lewis-E
added
type: bug fix
This was referenced Oct 15, 2026
Contributor
Contributor
🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
This fixes the tracer's agent connections over Windows named pipes, and makes the StatsD connection recover when DogStatsD starts late or goes away. Each commit is one change, and its message has the details.
NamedPipeSocketFactoryopens a new pipe handle for every socket instead of returning one sharedNamedPipeSocketper pipe. This reverses Reuse named pipe sockets #3493.maxRunningRequestsconnection pool, while the dispatcher limits stay as they are.DDAgentStatsDConnectionuses port 0 when the config sets a named pipe and no port. Before, it threwNullPointerException.Motivation
Customers on two Azure products hit these bugs.
Azure Functions with named pipes (SVLS-8247): Java tracing over a named pipe has never worked. Functions on a shared Windows plan need per-function pipes: the sandbox blocks loopback connections to other apps, while TCP ports still conflict across apps on one instance.
RealConnection.isHealthy) blocked in a pipe read on the sharedNamedPipeSocket.NamedPipeSocket.setSoTimeoutdoes nothing, so that read never times out. Traces, stats, remote config, and telemetry all queued behind that one handle, and the tracer never logged a trace as sent or failed until the JVM restarted. Removing onlyDD_TRACE_PIPE_NAMEmade the same app send traces over TCP right away.DDAgentStatsDConnectionleft the port null when a pipe was set, and.port(port)threwNullPointerException, so tracer health metrics and JMXFetch JVM metrics never went out over a pipe. The StatsD bug is in every version we checked, 1.30.0 through 1.65.1, and the shared socket dates from Reuse named pipe sockets #3493 in 2022.Azure App Service overlapping restarts (SVLS-9057, escalation SLES-2826): during an overlapped IIS recycle, the old and new worker share the DogStatsD pipe. In our repro, the new JVM's StatsD client kept failing for 50 minutes after the old worker's DogStatsD exited, even though the new worker's own DogStatsD was serving the pipe. The java-dogstatsd-client never reopens a named pipe. With commits 2 and 3 in a repro build, the first connection failed once and then connected, and after the overlap the client recreated itself. Metrics came back about 34 s after the old DogStatsD exited.
Removing #3493's shared socket is safe for the agents the tracer connects to. #3493 shared one socket per pipe because "named pipes can be opened at most twice (1 client and 1 server)", and noted that it was untested on Windows. A Windows pipe server can create several instances of one pipe name. The .NET tracer opens a new pipe stream for every request to the same trace agent and serverless-compat pipes, and it doesn't hang. The StatsD client opens its own pipe, with a different name, so the shared HTTP socket never protected it.
Additional Notes
Design choices and known gaps:
NamedPipeSocketstill can't time out. With one connection per request, a hung agent blocks only the request waiting on it, not every agent request. HonoringsetSoTimeoutwould take overlapped IO, which this PR doesn't attempt. Nothing logs a hung request either, because it never fails. The only default-level sign isRemoteWriter's rate-limited "Dropped a kept trace due to a buffer overflow" warning once the queue fills, and that reads as too much traffic, not a stuck agent. In App Service, Recover Azure App Service agents after overlapping restarts #12899's health check detects a hung trace agent on its own pipe handle and logs it.DDAgentStatsDConnection. The alternative is reconnect-on-write-failure in java-dogstatsd-client, which would also help customer StatsD clients. If that's the better home, commit 3 can leave this PR and move there. The other commits don't depend on it.none.NamedPipeSocketFactoryhas a package-privateFileconstructor so that its test can use a regular file as the pipe on any OS.Tests: the new
NamedPipeSocketFactoryTestfailed on master with the shared socket.DDAgentStatsDConnectionTestis new and covers the retry delay, the write-failure streak, and the named-pipe port, which failed before commit 6. Commits 1 to 3 ran in the App Service repro on Windows. Commits 4 to 6 haven't run on Windows yet. Next is an Azure Functions end-to-end run with a dev build of this branch, on shared Elastic Premium and Dedicated plans.#12899 (App Service agent supervision and logging) is stacked on this branch. datadog-serverless-compat-java#43 gives each Windows Azure Function its own named pipes, and it requires the first dd-trace-java release with this PR.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: SVLS-8247 [SVLS-9057]