Skip to content

Disable service discovery in CI Visibility agentless mode - #12908

Open
daniel-mohedano wants to merge 1 commit into
masterfrom
daniel.mohedano/gradle-launcher-cme
Open

daniel-mohedano wants to merge 1 commit into
masterfrom
daniel.mohedano/gradle-launcher-cme

Conversation

@daniel-mohedano

@daniel-mohedano daniel-mohedano commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Disables service discovery by default when CI Visibility runs in agentless mode. Setting DD_TRACE_SERVICE_DISCOVERY_ENABLED=true still turns it back on.

Motivation

On Linux, 1 second after startup, service discovery loads JNA on a background thread to write the tracer metadata file. JNA adds new system properties (jnidispatch.path, jna.loaded, jna.platform.library.path).

On Java 8, system properties are a Hashtable with fail-fast iterators. Any application thread iterating them at that moment throws ConcurrentModificationException. The Gradle launcher does exactly that at startup, so Gradle builds instrumented with the tracer can fail with:

java.util.ConcurrentModificationException
  at java.util.Hashtable$Enumerator.next
  at org.gradle.launcher.cli.converter.LayoutToPropertiesConverter.configureFromSystemPropertiesOfThisJvm

This caused 13 failures of GradleLauncherSmokeTest on Java 8 on master over the last 3 weeks.

The metadata file is only read by an Agent running on the same host. In CI Visibility agentless mode there is usually no Agent, and the instrumented processes are short-lived build tools and test JVMs. So the file brings little value there.

Additional Notes

The race still exists for other Java 8 applications on Linux that iterate system properties around startup. That would need a separate fix in service discovery.

test-environment-trigger: skip

@daniel-mohedano daniel-mohedano added comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM type: bug labels Oct 16, 2026
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 54.44 57.69 $\color{green}{\blacktriangledown}$ -3.25 56.55 $\color{green}{\blacktriangledown}$ -2.11 64/312
agentEvpProxy 54.07 n/a n/a n/a n/a -

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 38.82 37.15 $\color{red}{\blacktriangle}$ +1.67 37.15 $\color{red}{\blacktriangle}$ +1.67 33/170
agentless 37.98 37.15 $\color{red}{\blacktriangle}$ +0.83 36.42 $\color{red}{\blacktriangle}$ +1.56 33/143
agentlessCodeCoverage 46.72 45.38 $\color{red}{\blacktriangle}$ +1.34 45.38 $\color{red}{\blacktriangle}$ +1.34 33/142
agentlessLineCoverage 58.01 56.55 $\color{red}{\blacktriangle}$ +1.46 56.55 $\color{red}{\blacktriangle}$ +1.46 32/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@datadog-prod-us1-5

This comment has been minimized.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 87.16 91.39 $\color{green}{\blacktriangledown}$ -4.23 89.58 $\color{green}{\blacktriangledown}$ -2.42 33/171
agentless 81.04 82.69 $\color{green}{\blacktriangledown}$ -1.65 82.69 $\color{green}{\blacktriangledown}$ -1.65 33/141
agentlessCodeCoverage 95.49 99.00 $\color{green}{\blacktriangledown}$ -3.51 97.04 $\color{green}{\blacktriangledown}$ -1.55 33/141
agentlessLineCoverage 110.83 113.88 $\color{green}{\blacktriangledown}$ -3.05 113.88 $\color{green}{\blacktriangledown}$ -3.05 33/142

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 6.96 10.75 $\color{green}{\blacktriangledown}$ -3.79 10.13 $\color{green}{\blacktriangledown}$ -3.17 35/166
agentless 6.70 10.33 $\color{green}{\blacktriangledown}$ -3.63 10.13 $\color{green}{\blacktriangledown}$ -3.43 32/134
agentlessCodeCoverage 14.29 18.08 $\color{green}{\blacktriangledown}$ -3.79 17.73 $\color{green}{\blacktriangledown}$ -3.44 28/122

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 22.99 22.54 $\color{red}{\blacktriangle}$ +0.45 22.09 $\color{red}{\blacktriangle}$ +0.90 35/171
agentless 18.96 19.59 $\color{green}{\blacktriangledown}$ -0.63 19.59 $\color{green}{\blacktriangledown}$ -0.63 33/137
agentlessCodeCoverage 20.44 20.39 $\color{red}{\blacktriangle}$ +0.05 19.99 $\color{red}{\blacktriangle}$ +0.45 33/137
agentlessLineCoverage 27.03 27.52 $\color{green}{\blacktriangledown}$ -0.49 27.52 $\color{green}{\blacktriangledown}$ -0.49 20/93

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 9.05 9.92 $\color{green}{\blacktriangledown}$ -0.87 9.92 $\color{green}{\blacktriangledown}$ -0.87 32/139

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@dd-octo-sts

dd-octo-sts Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.76 s 14.65 s [+0.1%; +1.3%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.64 s 13.65 s [-0.9%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 17.05 s 16.41 s [-0.6%; +8.4%] (no difference)
startup:petclinic:iast:Agent 16.87 s 16.88 s [-1.0%; +0.8%] (no difference)
startup:petclinic:profiling:Agent 16.81 s 16.60 s [+0.3%; +2.3%] (maybe worse)
startup:petclinic:sca:Agent 16.98 s 16.88 s [-0.4%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 15.59 s 16.08 s [-7.1%; +1.0%] (no difference)

Commit: baefd31a · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 11.65 13.67 $\color{green}{\blacktriangledown}$ -2.02 13.40 $\color{green}{\blacktriangledown}$ -1.75 35/171
agentless 10.71 12.12 $\color{green}{\blacktriangledown}$ -1.41 12.12 $\color{green}{\blacktriangledown}$ -1.41 35/141
agentlessCodeCoverage 14.85 15.72 $\color{green}{\blacktriangledown}$ -0.87 15.41 $\color{green}{\blacktriangledown}$ -0.56 34/140
agentlessLineCoverage 16.56 17.38 $\color{green}{\blacktriangledown}$ -0.82 17.38 $\color{green}{\blacktriangledown}$ -0.82 33/139

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 94.39 95.12 $\color{green}{\blacktriangledown}$ -0.73 95.12 $\color{green}{\blacktriangledown}$ -0.73 36/172
agentless 90.22 91.39 $\color{green}{\blacktriangledown}$ -1.17 89.58 $\color{red}{\blacktriangle}$ +0.64 35/143
agentlessCodeCoverage 97.59 101.00 $\color{green}{\blacktriangledown}$ -3.41 99.00 $\color{green}{\blacktriangledown}$ -1.41 34/142
agentlessLineCoverage 100.28 103.04 $\color{green}{\blacktriangledown}$ -2.76 101.00 $\color{green}{\blacktriangledown}$ -0.72 39/150

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 19.15 19.20 $\color{green}{\blacktriangledown}$ -0.05 19.99 $\color{green}{\blacktriangledown}$ -0.84 35/169
agentless 18.15 18.45 $\color{green}{\blacktriangledown}$ -0.30 19.20 $\color{green}{\blacktriangledown}$ -1.05 36/145
agentlessCodeCoverage 20.45 21.65 $\color{green}{\blacktriangledown}$ -1.20 22.99 $\color{green}{\blacktriangledown}$ -2.54 37/147
agentlessLineCoverage 37.37 37.91 $\color{green}{\blacktriangledown}$ -0.54 39.45 $\color{green}{\blacktriangledown}$ -2.08 33/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 15.96 16.36 $\color{green}{\blacktriangledown}$ -0.40 16.36 $\color{green}{\blacktriangledown}$ -0.40 35/167
agentless 10.49 9.92 $\color{red}{\blacktriangle}$ +0.57 9.92 $\color{red}{\blacktriangle}$ +0.57 35/140
agentlessCodeCoverage 14.36 13.67 $\color{red}{\blacktriangle}$ +0.69 13.67 $\color{red}{\blacktriangle}$ +0.69 35/142
agentlessLineCoverage 23.58 22.54 $\color{red}{\blacktriangle}$ +1.04 22.54 $\color{red}{\blacktriangle}$ +1.04 32/138

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

@daniel-mohedano
daniel-mohedano marked this pull request as ready for review October 16, 2026 10:51
@daniel-mohedano
daniel-mohedano requested a review from a team as a code owner October 16, 2026 10:51
@daniel-mohedano
daniel-mohedano requested review from bric3 and removed request for a team October 16, 2026 10:51
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T10:54:27.827872Z baefd31 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-16T10:54:26.489135Z baefd31 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The new default disables service discovery only when CI Visibility and agentless mode are both enabled, while preserving explicit overrides. The completed static review found no actionable regressions.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit baefd31

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 44.96 17.38 $\color{red}{\blacktriangle}$ +27.58 15.41 $\color{red}{\blacktriangle}$ +29.55 35/169
agentless 41.46 13.94 $\color{red}{\blacktriangle}$ +27.52 13.13 $\color{red}{\blacktriangle}$ +28.33 35/141
agentlessCodeCoverage 71.91 79.45 $\color{green}{\blacktriangledown}$ -7.54 79.45 $\color{green}{\blacktriangledown}$ -7.54 35/142
agentlessLineCoverage 109.61 123.36 $\color{green}{\blacktriangledown}$ -13.75 123.36 $\color{green}{\blacktriangledown}$ -13.75 31/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM type: bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant