Skip to content

fix(otel): end all recording spans on non-terminal invocations - #633

Draft
ayushiahjolia wants to merge 1 commit into
mainfrom
fix/otel-end-spans-on-non-terminal
Draft

fix(otel): end all recording spans on non-terminal invocations#633
ayushiahjolia wants to merge 1 commit into
mainfrom
fix/otel-end-spans-on-non-terminal

Conversation

@ayushiahjolia

@ayushiahjolia ayushiahjolia commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

#624

Description

  • Deferred the Workflow span in both plugins so it's started and ended in a single call on the terminal invocation, instead of being started every invocation and dropped un-ended on PENDING/RETRYING.
  • Deferred operation spans in ExecutionOtelPlugin to onOperationEnd (retaining only a deterministic context between start and end), so a suspended operation no longer leaves a recording span abandoned.
  • Force-end any still-open attempt spans at invocation end in ExecutionOtelPlugin so no recording span is ever abandoned.
  • Carried the trace's sampling flags onto the synthetic Workflow/operation parent contexts in both plugins, so parent-based samplers drop children of an unsampled trace instead of forcing them to record as orphans.
  • Retained operation start timestamps in ExecutionOtelPlugin as a fallback for virtual FLAT map/parallel ops, whose onOperationEnd reports null timestamps, so their spans start at the real start time.
  • Added executionStartTime to InvocationEndInfo and populated it in onInvocationEnd, so plugins read the Workflow span start time from the end hook instead of caching it across hooks.

Demo/Screenshots

N/A

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Unit Tests

Have unit tests been written for these changes? Yes, updated

Integration Tests

Have integration tests been written for these changes? N/A

Examples

Has a new example been added for the change? (if applicable) N/A

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the fix/otel-end-spans-on-non-terminal branch from 895352f to 4c2aec6 Compare August 19, 2026 19:51
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 19, 2026 22:30 — with GitHub Actions Inactive
Comment thread sdk/src/main/java/software/amazon/lambda/durable/plugin/InvocationEndInfo.java Outdated
@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the fix/otel-end-spans-on-non-terminal branch from 4c2aec6 to fcd7317 Compare August 20, 2026 16:55
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 20, 2026 17:41 — with GitHub Actions Inactive
Comment thread sdk/src/main/java/software/amazon/lambda/durable/plugin/InvocationEndInfo.java Outdated
@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the fix/otel-end-spans-on-non-terminal branch from fcd7317 to b865c79 Compare August 20, 2026 20:55
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 20, 2026 21:03 — with GitHub Actions Inactive
Comment on lines +95 to +96
if (sdkTracerProvider == null) {
return TraceFlags.getSampled();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review

[P1] Do not assume an opaque agent provider is sampled

In the recommended no-arg Java-agent path, the provider is commonly hidden behind a classloader wrapper, making sdkTracerProvider null regardless of its sampler. Marking the synthetic Workflow parent sampled causes ExecutionOtelPlugin operations to export under a parent-based sampler even when the terminal Workflow root is dropped, producing orphan spans. Bridge the actual sampling result through the auto-configuration/classloader boundary and add a wrapped-provider test with a root-dropping sampler.

Comment on lines +98 to +108
var decision = sdkTracerProvider
.getSampler()
.shouldSample(
Context.root(),
workflowTraceId,
workflowSpanName,
SpanKind.INTERNAL,
Attributes.empty(),
Collections.emptyList())
.getDecision();
return decision == SamplingDecision.RECORD_AND_SAMPLE ? TraceFlags.getSampled() : TraceFlags.getDefault();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review

[P2] Reuse the Workflow root's exact sampling result

This probe uses empty attributes, while the real Workflow span is sampled with durable.execution.arn; the terminal span also invokes the sampler a second time. Attribute-based or stateful/rate-limiting samplers can therefore give the synthetic context and real root different decisions, and sampler-updated trace state is discarded. Query with the real attributes and preserve the complete SamplingResult through a sampler wrapper/cache so the terminal root reuses it. Add attribute-dependent and stateful sampler tests.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Two sampling-consistency regressions need correction before merge. Current tests cover only visible, deterministic samplers, missing the Java-agent and stateful-sampler failure modes.

Reviewed commit b865c79b2287d93efb7d317355685a578a33a458. Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant