Skip to content

Fix Kotlin coroutines losing scopes when resumed on another thread #12864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mcculls wants to merge 26 commits into
base: master
Choose a base branch
from
+2,420 −73
Open

Fix Kotlin coroutines losing scopes when resumed on another thread #12864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
bbba15c
Port Kotlin suspension handoff tests and run latestDepTest on kotlinx…
mcculls Oct 13, 2026
4c70b8f
Share each Kotlin coroutine's scope stack between its worker threads
mcculls Oct 13, 2026
7b922f0
Add unit tests for shared coroutine scope stacks
mcculls Oct 13, 2026
c3b2ca5
Avoid needless locking when swapping shared scope stacks
mcculls Oct 13, 2026
fa7a117
Skip trace continuations for structured Kotlin child coroutines
mcculls Oct 13, 2026
cf9a891
Treat children of scoped coroutines as structured Kotlin children
mcculls Oct 13, 2026
a8bfcdf
Give saved and shared scope stack contexts a common base
mcculls Oct 13, 2026
1504a0b
Read the shared scope stack's top and depth without locking
mcculls Oct 13, 2026
63ce94a
Reuse the token that swaps a thread's own scope stack back after a co…
mcculls Oct 13, 2026
bb47eee
Don't keep a Kotlin coroutine's parent context alive after capturing it
mcculls Oct 13, 2026
c4babe1
Name the reusable scope stack token after its same-thread guarantee
mcculls Oct 13, 2026
d3d8c9d
Publish the shared scope stack's depth through its volatile top
mcculls Oct 13, 2026
a5df070
Tidy the Kotlin element's context capture and activation
mcculls Oct 13, 2026
38607b2
Publish a shared scope stack's new top before activation listeners run
mcculls Oct 13, 2026
f13ae79
Update shared scope reference counts under the stack's lock
mcculls Oct 13, 2026
513b673
Guard a shared scope's reference count on the scope itself
mcculls Oct 13, 2026
43a69b3
Clear a shared scope's reference count under its lock
mcculls Oct 13, 2026
2da5c66
Clarify shared scope cleanup visibility
mcculls Oct 13, 2026
8ceb2e6
Bound shared scope stress test shutdown and clarify comments
mcculls Oct 13, 2026
c0cdb74
Avoid allocating a lambda when replacing coroutine context elements
mcculls Oct 14, 2026
5657855
Simplify coroutine regression tests and bound worker shutdown
mcculls Oct 14, 2026
b20c584
Document scope concurrency invariants in narrow SpotBugs suppressions
mcculls Oct 14, 2026
0c7bb96
Cover scope context views and cross-thread restoration
mcculls Oct 14, 2026
f3f8dc8
Fix scoped coroutines releasing their owner's trace continuation
mcculls Oct 15, 2026
e4ba85d
Fix Kotlin continuation leak when launching with a cancelled parent
mcculls Oct 16, 2026
3efdc69
Fix scope listener ordering when restoring coroutine stacks
mcculls Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...main/java/datadog/trace/instrumentation/java/lang/jdk21/VirtualThreadInstrumentation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ public final class VirtualThreadInstrumentation extends InstrumenterModule.Conte
*/
private static final String[] PRELOAD_CLASS_NAMES = {
VIRTUAL_THREAD_STATE_CLASS_NAME,
"datadog.trace.core.scopemanager.ScopeContext",
"datadog.trace.core.scopemanager.SavedScopeContext",
"datadog.trace.core.scopemanager.ScopeStack"
};

Expand Down
4 changes: 3 additions & 1 deletion dd-java-agent/instrumentation/kotlin-coroutines-1.3/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,11 @@ dependencies {
testImplementation testFixtures(project(':dd-java-agent:instrumentation:kotlin-coroutines-1.3'))

latestDepTestImplementation libs.kotlin
latestDepTestImplementation "org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.6.+"
latestDepTestImplementation "org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.8.+"
latestDepTestImplementation testFixtures(project(':dd-java-agent:instrumentation:kotlin-coroutines-1.3'))

testFixturesCompileOnly(libs.bundles.groovy)
testFixturesCompileOnly(libs.bundles.spock)
testFixturesCompileOnly libs.junit.jupiter
testFixturesCompileOnly libs.tabletest
}
29 changes: 15 additions & 14 deletions dd-java-agent/instrumentation/kotlin-coroutines-1.3/gradle.lockfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,23 +99,24 @@ org.jetbrains.kotlin:kotlin-scripting-common:2.1.20=kotlinBuildToolsApiClasspath
org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:2.1.20=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathLatestDepTest,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinCompilerPluginClasspathTestFixtures
org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:2.1.20=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathLatestDepTest,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinCompilerPluginClasspathTestFixtures
org.jetbrains.kotlin:kotlin-scripting-jvm:2.1.20=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathLatestDepTest,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinCompilerPluginClasspathTestFixtures
org.jetbrains.kotlin:kotlin-stdlib-common:1.6.21=compileClasspath,compileOnlyDependenciesMetadata,latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.6.21=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.21=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib:1.6.21=compileClasspath,compileOnlyDependenciesMetadata,latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib-common:1.6.21=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib-common:1.9.21=latestDepTestImplementationDependenciesMetadata
org.jetbrains.kotlin:kotlin-stdlib:1.6.21=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib:1.9.21=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib:2.1.20=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathLatestDepTest,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinCompilerPluginClasspathTestFixtures,kotlinKlibCommonizerClasspath
org.jetbrains.kotlinx:atomicfu:0.17.3=latestDepTestImplementationDependenciesMetadata
org.jetbrains.kotlinx:kotlinx-coroutines-bom:1.6.4=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.6.4=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlinx:atomicfu:0.23.1=latestDepTestImplementationDependenciesMetadata
org.jetbrains.kotlinx:kotlinx-coroutines-bom:1.8.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.8.0=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath
org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.8.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.0=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains:annotations:13.0=compileClasspath,compileOnlyDependenciesMetadata,kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathLatestDepTest,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinCompilerPluginClasspathTestFixtures,kotlinKlibCommonizerClasspath,latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains.kotlinx:kotlinx-coroutines-core:1.8.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jetbrains:annotations:13.0=compileClasspath,compileOnlyDependenciesMetadata,kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathLatestDepTest,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinCompilerPluginClasspathTestFixtures,kotlinKlibCommonizerClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.jetbrains:annotations:23.0.0=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath
org.jspecify:jspecify:1.0.0=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesApiDependenciesMetadata,testFixturesCompileClasspath,testFixturesImplementationDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:5.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:5.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-engine:5.14.1=latestDepTestRuntimeClasspath,testFixturesRuntimeClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-params:5.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.jupiter:junit-jupiter:5.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-params:5.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.jupiter:junit-jupiter:5.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.platform:junit-platform-commons:1.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesCompileOnlyDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.platform:junit-platform-engine:1.14.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesCompileOnlyDependenciesMetadata,testFixturesRuntimeClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.junit.platform:junit-platform-launcher:1.14.1=latestDepTestRuntimeClasspath,testFixturesRuntimeClasspath,testRuntimeClasspath
Expand Down Expand Up @@ -144,7 +145,7 @@ org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j
org.snakeyaml:snakeyaml-engine:2.9=buildTimeInstrumentationPlugin,latestDepTestRuntimeClasspath,muzzleTooling,runtimeClasspath,testFixturesRuntimeClasspath,testRuntimeClasspath
org.spockframework:spock-bom:2.4-groovy-3.0=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesCompileOnlyDependenciesMetadata,testImplementationDependenciesMetadata,testRuntimeClasspath
org.spockframework:spock-core:2.4-groovy-3.0=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testFixturesCompileOnlyDependenciesMetadata,testImplementationDependenciesMetadata,testRuntimeClasspath
org.tabletest:tabletest-junit:1.2.2=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.tabletest:tabletest-parser:1.2.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.tabletest:tabletest-junit:1.2.2=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.tabletest:tabletest-parser:1.2.1=latestDepTestCompileClasspath,latestDepTestImplementationDependenciesMetadata,latestDepTestRuntimeClasspath,testCompileClasspath,testFixturesCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath
org.xmlresolver:xmlresolver:5.3.3=spotbugs
empty=apiDependenciesMetadata,intransitiveDependenciesMetadata,kotlinCompilerPluginClasspath,kotlinNativeCompilerPluginClasspath,kotlinScriptDefExtensions,latestDepTestApiDependenciesMetadata,latestDepTestIntransitiveDependenciesMetadata,latestDepTestKotlinScriptDefExtensions,spotbugsPlugins,testApiDependenciesMetadata,testFixturesAnnotationProcessor,testFixturesIntransitiveDependenciesMetadata,testFixturesKotlinScriptDefExtensions,testIntransitiveDependenciesMetadata,testKotlinScriptDefExtensions
29 changes: 29 additions & 0 deletions ...ation/kotlin-coroutines-1.3/src/latestDepTest/groovy/StandardSuspensionHandoffTest.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import datadog.trace.instrumentation.kotlin.coroutines.AbstractStandardSuspensionHandoffTest
import datadog.trace.instrumentation.kotlin.coroutines.StandardSuspensionHandoffTests

class StandardSuspensionHandoffTest extends AbstractStandardSuspensionHandoffTest {
def "undispatched coroutine can replay its restoration token"() {
when:
new StandardSuspensionHandoffTests().runReplayedRestoration()

then:
noExceptionThrown()
}

def "scope-touching thread context elements do not corrupt the handed-off stack"() {
when:
def violations = new StandardSuspensionHandoffTests().runScopeTouchingElements(1000)

then:
// Overlapping workers can temporarily see each other's element scopes. The coroutine must
// retain its base context, but restore callbacks can observe a different top during overlap.
def expectedDuringOverlap = [
'element scope leaked from previous worker',
'before scope not on top at restore',
'after scope not on top at restore',
'before restore did not return to previous context',
'after restore did not return to previous context'
]
(violations.keySet() - expectedDuringOverlap).isEmpty()
}
}
3 changes: 3 additions & 0 deletions ...java/datadog/trace/instrumentation/kotlin/coroutines/DatadogThreadContextElementTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package datadog.trace.instrumentation.kotlin.coroutines;

class DatadogThreadContextElementTest extends AbstractDatadogThreadContextElementTest {}
12 changes: 12 additions & 0 deletions ...nt/instrumentation/kotlin-coroutines-1.3/src/latestDepTest/kotlin/KotlinCoroutineTests.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,18 @@ class KotlinCoroutineTests(dispatcher: CoroutineDispatcher) : CoreKotlinCoroutin
3
}

@Trace
override fun structuredChildrenOutliveSeedSpan(): Int = super.structuredChildrenOutliveSeedSpan()

@Trace
override fun structuredChildrenOfScopedCoroutines(): Int = super.structuredChildrenOfScopedCoroutines()

@Trace
override fun childWithReplacedJob(supervisor: Boolean): Int = super.childWithReplacedJob(supervisor)

@Trace
override fun structuredChildOfCancelledCoroutine(): Int = super.structuredChildOfCancelledCoroutine()

@Trace
override fun tracePreventedByCancellation(): Int = super.tracePreventedByCancellation()

Expand Down
Loading
Loading