Report JVM crash logs from CI Visibility smoke tests

[daniel-mohedano]

What Does This Do

CI Visibility smoke tests now report JVM crash logs (hs_err_pid*.log) written by the processes they start: the Gradle launcher, the Gradle test workers, and Maven.

• After each test, the smoke test looks for crash logs in the test project folder.
• It prints the first lines of each log to the test output.
• It copies each log to build/reports/jvm-crash-logs, which CI keeps as a job artifact.

Motivation

Forked JVMs write their crash log to their working directory. In these tests that is a temporary folder that is deleted after each test. The tests also do not print the process output. So when a forked JVM crashed, the failure only showed a non-zero exit code (134), with nothing to tell why.

On master, about 109 smoke test failures over the last 3 weeks were JVM crashes on Linux arm64. Only 2 of them had any crash details in the job log.

Additional Notes

Checked locally by making the forked JVM crash on purpose (an agent that writes to address 0). The crash logs were printed and copied for the Gradle launcher, the Gradle test workers and Maven.

test-environment-trigger: skip

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	56.23	57.69	$\color{green}{\blacktriangledown}$ -1.46	56.55	$\color{green}{\blacktriangledown}$ -0.32	68/316
agentEvpProxy	55.30	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.41	10.75	$\color{red}{\blacktriangle}$ +1.66	10.13	$\color{red}{\blacktriangle}$ +2.28	35/166
agentless	12.49	10.33	$\color{red}{\blacktriangle}$ +2.16	10.13	$\color{red}{\blacktriangle}$ +2.36	33/135
agentlessCodeCoverage	20.40	18.08	$\color{red}{\blacktriangle}$ +2.32	17.73	$\color{red}{\blacktriangle}$ +2.67	29/123

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.89	22.54	$\color{green}{\blacktriangledown}$ -0.65	22.09	$\color{green}{\blacktriangledown}$ -0.20	36/172
agentless	18.99	19.59	$\color{green}{\blacktriangledown}$ -0.60	19.59	$\color{green}{\blacktriangledown}$ -0.60	33/137
agentlessCodeCoverage	19.79	20.39	$\color{green}{\blacktriangledown}$ -0.60	19.99	$\color{green}{\blacktriangledown}$ -0.20	34/138
agentlessLineCoverage	26.99	27.52	$\color{green}{\blacktriangledown}$ -0.53	27.52	$\color{green}{\blacktriangledown}$ -0.53	20/93

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.64	37.15	$\color{green}{\blacktriangledown}$ -0.51	37.15	$\color{green}{\blacktriangledown}$ -0.51	35/172
agentless	38.30	37.15	$\color{red}{\blacktriangle}$ +1.15	36.42	$\color{red}{\blacktriangle}$ +1.88	35/145
agentlessCodeCoverage	46.87	45.38	$\color{red}{\blacktriangle}$ +1.49	45.38	$\color{red}{\blacktriangle}$ +1.49	35/144
agentlessLineCoverage	56.35	56.55	$\color{green}{\blacktriangledown}$ -0.20	56.55	$\color{green}{\blacktriangledown}$ -0.20	33/142

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	90.72	91.39	$\color{green}{\blacktriangledown}$ -0.67	89.58	$\color{red}{\blacktriangle}$ +1.14	35/173
agentless	83.54	82.69	$\color{red}{\blacktriangle}$ +0.85	82.69	$\color{red}{\blacktriangle}$ +0.85	35/143
agentlessCodeCoverage	97.77	97.04	$\color{red}{\blacktriangle}$ +0.73	97.04	$\color{red}{\blacktriangle}$ +0.73	34/142
agentlessLineCoverage	112.41	113.88	$\color{green}{\blacktriangledown}$ -1.47	113.88	$\color{green}{\blacktriangledown}$ -1.47	34/143

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.86	9.92	$\color{green}{\blacktriangledown}$ -0.06	9.92	$\color{green}{\blacktriangledown}$ -0.06	34/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.98 s	13.96 s	[-0.6%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent	12.92 s	12.93 s	[-0.8%; +0.7%] (no difference)
startup:petclinic:appsec:Agent	16.55 s	16.96 s	[-6.8%; +2.0%] (no difference)
startup:petclinic:iast:Agent	16.95 s	16.95 s	[-0.9%; +0.9%] (no difference)
startup:petclinic:profiling:Agent	16.54 s	16.77 s	[-2.5%; -0.3%] (maybe better)
startup:petclinic:sca:Agent	17.02 s	16.71 s	[+0.9%; +2.8%] (maybe worse)
startup:petclinic:tracing:Agent	16.09 s	15.70 s	[-1.7%; +6.8%] (no difference)

Commit: eb5c5311 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.67	13.67	$\color{gray}{\bullet}$ 0.00	13.40	$\color{red}{\blacktriangle}$ +0.27	35/171
agentless	10.96	12.12	$\color{green}{\blacktriangledown}$ -1.16	12.12	$\color{green}{\blacktriangledown}$ -1.16	35/141
agentlessCodeCoverage	13.98	15.72	$\color{green}{\blacktriangledown}$ -1.74	15.41	$\color{green}{\blacktriangledown}$ -1.43	34/140
agentlessLineCoverage	16.97	17.38	$\color{green}{\blacktriangledown}$ -0.41	17.38	$\color{green}{\blacktriangledown}$ -0.41	32/138

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	94.22	95.12	$\color{green}{\blacktriangledown}$ -0.90	95.12	$\color{green}{\blacktriangledown}$ -0.90	36/172
agentless	91.33	91.39	$\color{green}{\blacktriangledown}$ -0.06	89.58	$\color{red}{\blacktriangle}$ +1.75	35/143
agentlessCodeCoverage	99.14	101.00	$\color{green}{\blacktriangledown}$ -1.86	99.00	$\color{red}{\blacktriangle}$ +0.14	35/143
agentlessLineCoverage	101.54	103.04	$\color{green}{\blacktriangledown}$ -1.50	101.00	$\color{red}{\blacktriangle}$ +0.54	41/152

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.38	19.20	$\color{green}{\blacktriangledown}$ -2.82	19.99	$\color{green}{\blacktriangledown}$ -3.61	35/169
agentless	17.90	18.45	$\color{green}{\blacktriangledown}$ -0.55	19.20	$\color{green}{\blacktriangledown}$ -1.30	36/145
agentlessCodeCoverage	19.44	21.65	$\color{green}{\blacktriangledown}$ -2.21	22.99	$\color{green}{\blacktriangledown}$ -3.55	37/147
agentlessLineCoverage	34.92	37.91	$\color{green}{\blacktriangledown}$ -2.99	39.45	$\color{green}{\blacktriangledown}$ -4.53	34/142

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.57	16.36	$\color{red}{\blacktriangle}$ +0.21	16.36	$\color{red}{\blacktriangle}$ +0.21	35/167
agentless	9.80	9.92	$\color{green}{\blacktriangledown}$ -0.12	9.92	$\color{green}{\blacktriangledown}$ -0.12	35/140
agentlessCodeCoverage	14.24	13.67	$\color{red}{\blacktriangle}$ +0.57	13.67	$\color{red}{\blacktriangle}$ +0.57	35/142
agentlessLineCoverage	23.20	22.54	$\color{red}{\blacktriangle}$ +0.66	22.54	$\color{red}{\blacktriangle}$ +0.66	33/139

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T10:54:46.791152Z	eb5c531	Draft marked ready
🔒 Security Review	⚠️ Failed 2026-10-16T10:57:03.224906Z	eb5c531	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-prod-us1-4]

More details

Crash-log collection runs before temporary-project cleanup, and its destination is covered by CI artifact collection.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit eb5c531}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	8.10	17.38	$\color{green}{\blacktriangledown}$ -9.28	15.41	$\color{green}{\blacktriangledown}$ -7.31	35/169
agentless	8.75	13.94	$\color{green}{\blacktriangledown}$ -5.19	13.13	$\color{green}{\blacktriangledown}$ -4.38	35/141
agentlessCodeCoverage	65.74	79.45	$\color{green}{\blacktriangledown}$ -13.71	79.45	$\color{green}{\blacktriangledown}$ -13.71	35/141
agentlessLineCoverage	95.09	120.92	$\color{green}{\blacktriangledown}$ -25.83	123.36	$\color{green}{\blacktriangledown}$ -28.27	34/144

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.