Skip to content

Collect diagnostics when OSGi smoke test application hangs - #12938

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
masterfrom
alexeyk/osgi-smoke-test-hang-diagnostics
Oct 19, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
masterfrom
alexeyk/osgi-smoke-test-hang-diagnostics

Conversation

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor

What Does This Do

Adds diagnostics to the OSGi smoke tests (EquinoxSmokeTest, FelixSmokeTest, KnopflerfishSmokeTest) for the case where the test application never exits:

  • Replaces the unbounded testedProcess.waitFor() with a 2-minute wait. Normally the application exits within seconds.
  • On timeout, collectDiagnostics() writes into build/reports/<Class>.<timestamp>-dumps/ and then fails the test:
    • HotSpot: two thread dumps taken 10 seconds apart with jcmd <pid> Thread.print -l.
    • IBM/OpenJ9: two javacores (kill -3), then a system dump (kill -USR2). The test waits for the JVM to log Processed dump event "user2" before cleanup kills the process. To support this, the application now starts with -Xdump:directory=<dumps dir> and -Xdump:system:events=user2,request=exclusive+prepwalk.
    • collection.log: the PID and the outcome of each command.
  • The dumps folder uses the same timestamp as the matching testProcess.<Class>.<timestamp>.0.log, so dumps from a retried test don't overwrite earlier ones. To allow this, ProcessManager.logFileTimestamp changes from private to protected.

No CI configuration changes are needed. .gitlab/collect_reports.sh already copies workspace/**/build/reports/* into reports.tar, which is uploaded as a job artifact (when: always). Because the test now fails instead of hanging until the job times out, after_script and artifact collection run normally.

Motivation

Recently the OSGi smoke tests have been hanging on GitLab more often than usual, and those runs leave no diagnostics. The job just times out, and there is nothing that shows where the application is stuck. This change collects thread dumps (and a system dump on IBM JVMs) so the next hang can be investigated.

Additional Notes

  • Tested locally by temporarily making the application hang. Thread dumps were collected on HotSpot 8 and HotSpot 17, and javacores on Semeru 8. All three OSGi smoke tests still pass on the default JVM and on Semeru 8.
  • On macOS the OpenJ9 system dump reports JVMDUMP012E when the JVM runs under Gradle. Whether it works on CI depends on the container's core_pattern and core size limit. The javacores record both, so the first hang on CI will show whether it works there.

🤖 Generated with Claude Code

Fail the OSGi smoke test after 2 minutes instead of waiting forever, and
collect two thread dumps of the application (plus a system dump on IBM
JVMs) into the build reports so they are included in the GitLab job
artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added comp: testing Testing tag: no release notes Changes to exclude from release notes type: refactoring tag: diagnostics Diagnostics related changes tag: ai generated Largely based on code generated by an AI or LLM labels Oct 19, 2026
Use a single constant for the exit timeout, resolve jcmd only for
HotSpot, and create the dumps folder only when diagnostics are collected
so passing IBM runs no longer leave an empty folder in reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD self-assigned this Oct 19, 2026
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD marked this pull request as ready for review October 19, 2026 18:35
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD requested review from a team as code owners October 19, 2026 18:35
@AlexeyKuznetsov-DD
AlexeyKuznetsov-DD requested review from bric3 and jordan-wong and removed request for a team October 19, 2026 18:35
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T18:38:24.732863Z 7b2b647 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-19T18:40:04.730898Z 7b2b647 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

PID retrieval was verified on HotSpot 8 and 21, supporting diagnostic collection across Java versions. OpenJ9 dump completion and the full OSGi smoke-test flow remain unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 7b2b647

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.46% (-0.38%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7b2b647 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.75 s 14.54 s [+0.9%; +2.1%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.57 s 13.60 s [-0.8%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.02 s 16.98 s [-0.6%; +1.2%] (no difference)
startup:petclinic:iast:Agent 16.44 s 16.97 s [-7.3%; +1.0%] (no difference)
startup:petclinic:profiling:Agent 16.14 s 16.62 s [-7.1%; +1.4%] (no difference)
startup:petclinic:sca:Agent 17.01 s 16.79 s [+0.5%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent 15.97 s 16.02 s [-1.2%; +0.5%] (no difference)

Commit: 7b2b647b · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@AlexeyKuznetsov-DD

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-19 21:31:55 UTC ℹ️ Start processing command /merge


2026-10-19 21:31:59 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-10-19 22:49:12 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 60a99e2 into master Oct 19, 2026
612 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the alexeyk/osgi-smoke-test-hang-diagnostics branch October 19, 2026 22:49
@github-actions github-actions Bot added this to the 1.68.0 milestone Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: testing Testing tag: ai generated Largely based on code generated by an AI or LLM tag: diagnostics Diagnostics related changes tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants