Skip to content

Fix Kotlin coroutine scope propagation before suspension - #12755

Draft
amarziali wants to merge 5 commits into
masterfrom
andrea.marziali/kotlin
Draft

amarziali wants to merge 5 commits into
masterfrom
andrea.marziali/kotlin

Conversation

@amarziali

Copy link
Copy Markdown
Contributor

What Does This Do

A Kotlin coroutine can resume on another worker before the previous worker restores its thread context, losing a manually opened scope and leaving it unclosed.

Capture the scope stack before standard suspension and yield decisions, and prevent an older restoration from overwriting newer context. Enable instrumentation of SafeContinuation with a narrow allow-list exception.

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@amarziali amarziali changed the title Fix Kotlin coroutine scope propagation before suspensio Fix Kotlin coroutine scope propagation before suspension Oct 6, 2026
@amarziali amarziali added the tag: override groovy enforcement Override the "Enforce Groovy Migration" check label Oct 6, 2026
@amarziali

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.29% (-0.04%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2c6da9f | Docs | Give us feedback!

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

On kotlinx-coroutines 1.8, SelectBuilderImpl no longer implements Continuation, so Byte Buddy rejects the typed @Advice.This binding at transformation time before any runtime suppression can apply. The fix is to bind the receiver as Object and add a Continuation instanceof guard, letting modern select builders delegate to the already-instrumented CancellableContinuationImpl.

Open Bits AI session

🤖 Bits Code Review · Commit a9bcd5c · @DataDog review to ask questions

@dd-octo-sts

dd-octo-sts Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.72 s 14.75 s [-1.1%; +0.6%] (no difference)
startup:insecure-bank:tracing:Agent 13.61 s 13.76 s [-1.7%; -0.4%] (maybe better)
startup:petclinic:appsec:Agent 17.20 s 16.99 s [+0.2%; +2.3%] (maybe worse)
startup:petclinic:iast:Agent 17.07 s 16.40 s [-0.2%; +8.4%] (no difference)
startup:petclinic:profiling:Agent 16.50 s 16.87 s [-3.5%; -0.9%] (maybe better)
startup:petclinic:sca:Agent 17.06 s 16.96 s [-0.5%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.14 s 16.19 s [-1.2%; +0.6%] (no difference)

Commit: 2c6da9fa · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

mcculls
mcculls previously approved these changes Oct 6, 2026

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch

@amarziali
amarziali marked this pull request as ready for review October 7, 2026 12:32
@amarziali
amarziali requested review from a team as code owners October 7, 2026 12:32
@amarziali
amarziali requested review from ValentinZakharov and bric3 and removed request for a team October 7, 2026 12:32
@dd-octo-sts

dd-octo-sts Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@amarziali amarziali added type: bug fix Bug fix inst: kotlin Kotlin instrumentation labels Oct 7, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T12:38:26.729132Z c46d067 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-07T12:39:16.008499Z c46d067 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Two distinct issues are flagged: the CAS restoration guard in DatadogThreadContextElement cannot distinguish legitimate nested restorations from stale concurrent ones, risking lost manual scopes and mis-parented spans; and the method-name matcher in SuspensionInstrumentation does not cover the Kotlin-mangled getResult$kotlinx_coroutines_core variant introduced in coroutines 1.8, causing the wrong scope to be restored when a dispatcher switch follows an inline suspension.

Open Bits AI session

🤖 Bits Code Review · Commit c46d067 · @DataDog review to ask questions

@amarziali

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Two distinct production bugs are flagged in DatadogThreadContextElement.java: clearing originalContext after the first restoration causes a NullPointerException when Kotlin 1.8's UndispatchedCoroutine attempts a second restoration via the same token; and a dispatch-ordering race allows the destination coroutine to claim element.exchange before the source records its manual scope, silently losing span parentage and producing stale context on resumption.

Open Bits AI session

🤖 Bits Code Review · Commit c365c5d · @DataDog review to ask questions

@amarziali

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

A synchronous suspendCoroutine call can leave a snapshot of a scope that subsequently closes. If a later coroutineScope child completes before source restoration, the resumed coroutine reactivates that closed scope and subsequent spans inherit the wrong parent.

Open Bits AI session

🤖 Bits Code Review · Commit a40328e · @DataDog review to ask questions

@amarziali

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Static analysis of exchange ownership, nested restoration, and suspension hooks supports the scope-stack handoff change. Runtime behavior remains unverified because tests were not executed.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 2c6da9f · @DataDog review to ask questions

@mcculls
mcculls dismissed their stale review October 10, 2026 16:33

Dismissing stale review - reminder to myself to re-review the latest changes

@amarziali
amarziali requested a review from mcculls October 13, 2026 08:48
@mcculls

mcculls commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

#12864 describes an alternative approach, which requires more changes to the legacy scope manager but avoids having to continually take snapshots at suspension points.

@amarziali

Copy link
Copy Markdown
Contributor Author

#12864 describes an alternative approach, which requires more changes to the legacy scope manager but avoids having to continually take snapshots at suspension points.

@mcculls yes your approach looks very interesting and I think can superseed this proposal. I'm drafting this PR while we're assessing the other approach so. thanks for having tested this alternative

@amarziali
amarziali marked this pull request as draft October 15, 2026 07:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: kotlin Kotlin instrumentation tag: override groovy enforcement Override the "Enforce Groovy Migration" check type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants