feat(plugin): propagate chained invoke trace context in START checkpoints

.github/workflows/otel-conformance-tests.yml

@@ -68,7 +68,7 @@ jobs:
       actions: write
       contents: read
       id-token: write
-    uses: aws/aws-durable-execution-conformance-tests/.github/workflows/opentelemetry-orchestrator.yml@a66037abbbfa55fde97f714e30f0bc262edefd63
+    uses: aws/aws-durable-execution-conformance-tests/.github/workflows/opentelemetry-orchestrator.yml@f18bd0b5f28c5c90e288d0fb8bca08a849b51863
     with:
       runs_on: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'ubuntu-latest' || format('codebuild-github-actions-runner-{0}-{1}', github.run_id, github.run_attempt) }}
       language: java

docs/advanced/propagation-metadata.md

@@ -0,0 +1,66 @@
+# Chained-invoke trace propagation
+
+For a new `CHAINED_INVOKE` operation, `InvokeOperation.startInvocation()` collects optional metadata after
+`onOperationStart` has established the operation's span and before checkpoint transport serialization. The customer
+payload first uses its existing serializer, so a payload failure does not trigger optional metadata collection. It writes
+that header through the generated `ChainedInvokeOptions.builder().xAmznTraceId(...)` method. The wire member is the
+flat optional `XAmznTraceId`; there is no nested `PropagationMetadata` structure or W3C transport field.
+
+Function name, tenant ID, payload, operation name, ID, parent ID and checkpoint behavior retain their existing
+paths. Each operation carries its own header, including when several invokes share a checkpoint request. No
+execution-global or HTTP-request trace header can substitute for those distinct calling-operation parents.
+
+## Plugin contract and fallback
+
+`PropagationInput` is an immutable SDK-owned snapshot with `executionArn`, `operationId`, optional `parentOperationId`,
+and `targetFunctionName`. `PropagationMetadata` is an immutable SDK-owned contribution with optional `xAmznTraceId`.
+Both types use only Java strings; generated Lambda and OTel model types do not enter the plugin interface.
+The typed default hook adapts a String-only producer overload, so bundled plugin layers do not reference newly added
+core classes. Older cores continue to load those layers and retain their original behavior. New collection requires
+an updated core; registration/provider API versions and existing hooks are unchanged.
+
+The optional synchronous `providePropagationMetadata` default method returns null. The dispatcher supplies the same
+immutable snapshot in configured order. The first non-null supported member wins; equal values do not conflict.
+Different later values warn with the winning and conflicting plugin identities and a conflict count, without logging
+the header. Null contributions abstain; blank headers and ordinary hook failures are logged and skipped. A missing
+contribution omits the transport member so the backend can retain its existing inherited-header fallback.
+The collector preserves Java's current event dispatch policy: `Exception` (including `CancellationException`) is
+contained, while `Error` propagates. A failing logging backend does not interrupt collection.
+
+Both OTel views encode the resolved canonical trace ID, actual calling-operation span ID and sampling decision as
+`Root=1-<8 hex>-<24 hex>;Parent=<16 hex>;Sampled=0|1`. They require active invocation ownership. The producer does not
+create a span or sample again, and retains the configured provider/resource and upstream-parent behavior. Explicit
+upstream `Sampled=0` is preserved. The producer can derive an initial operation ID before a start hook, but the real
+invoke path collects after that hook and therefore uses the operation context it already established.
+
+## Replay and retry
+
+Replaying a stored START polls for the existing operation; replaying a terminal operation returns or raises its
+stored outcome. Neither path collects metadata or sends another START. If a START checkpoint failed or was lost before
+commit, a later invocation can create it again and collect again. Plugins must be deterministic and side-effect free;
+this hook is not exactly-once delivery. Checkpoint batching and retries preserve each submitted operation's options.
+
+## Model and backend dependencies
+
+The SDK path and tests are implemented assuming the reviewed generated model member exists. The pinned public Lambda
+model `2.55.11` currently lacks `ChainedInvokeOptions.XAmznTraceId` (both its builder setter and getter), so compilation
+against that model is expected to fail. The implementation does not hide the missing member with reflection, runtime
+capability checks, serializer bypasses or raw HTTP fields. Rebase onto the published model and rerun normal tests when
+it is available.
+
+The design also adds `XAmznTraceId` to `DistributedMapOptions`. This Java SDK currently exposes no distributed-map
+operation or START dispatch; its existing `map` and `parallel` APIs use CONTEXT operations. There is no new distributed
+map/fanout/HTTP API here. The future generated model and high-level operation path must integrate their corresponding
+field when introduced.
+
+Before release, the generated model must be published and the backend must persist/forward the per-operation header.
+Then validate deployed parent/child traces for durable and ordinary Lambda targets. Keep this PR draft until those
+dependencies are ready; this remains related to issue #764. Runtime inbound headers and plugin-instance lifetime are
+separate changes.
+
+Tests exercise real public invoke and checkpoint paths, pending/terminal replay, failed uncommitted START recovery,
+ordinary plugin fallbacks and Error propagation, sampled/unsampled OTel views, batched invokes with distinct parents,
+custom payload and tenant preservation, generated-model copies, and the normal Lambda client's JSON marshaller with
+only HTTP transport replaced. Batch-boundary tests include the encoded options and trace header in the 750 KiB size
+budget, including JSON escaping and UTF-8. An isolated local model-preview fixture can test the assumed member shape,
+but cannot establish that the public model or backend supports it.

otel-plugin/README.md

@@ -40,6 +40,13 @@ If you configure your own `SdkTracerProviderBuilder`, add the OpenTelemetry SDK
 </dependency>
 ```
 
+## Chained-invoke propagation
+
+Both views provide the calling operation's X-Ray context to the SDK's synchronous collector. New invoke START
+checkpoints carry that context in the flat optional `ChainedInvokeOptions.XAmznTraceId` field. This draft requires
+the corresponding generated model and backend support; the current public model does not yet compile the new
+typed setter. See [the implemented path and remaining dependencies](../docs/advanced/propagation-metadata.md).
+
 ## Choose one durable OTel view
 
 Configure exactly one of `InvocationOtelPlugin` or `ExecutionOtelPlugin` when enabling durable tracing.
@@ -62,6 +69,13 @@ on it replaces the complete plugin list without reading `DURABLE_EXECUTION_PLUGI
 plugins from the copy. Use `DurableConfig.builder()` when creating a fresh configuration that should honor the current
 environment selection.
 
+View exclusivity is declared with inherited `@ExclusivePluginGroup("durable-otel-view")` metadata.
+Configuration reads this explicit opt-in annotation from the entire superclass chain; it does not call application methods
+that happen to be named `getExclusiveGroup`. Existing subclasses retain their own methods while inheriting the bundled
+view restriction. A subclass may add another group, but cannot replace a superclass's group; repeated group names in one
+class hierarchy are checked once.
+Older cores ignore the optional annotation and retain their prior behavior; no provider-version floor is raised.
+
 ## Quick Start using X-Ray/CloudWatch Tracing (ADOT Java Agent)
 
 1. Add the ADOT Lambda Layer to your function
@@ -407,10 +421,3 @@ var otelPlugin = new InvocationOtelPlugin(
 ## License
 
 Apache-2.0
-
-View exclusivity is declared with inherited `@ExclusivePluginGroup("durable-otel-view")` metadata.
-Configuration reads this explicit opt-in annotation from the entire superclass chain; it does not call application methods
-that happen to be named `getExclusiveGroup`. Existing subclasses retain their own methods while inheriting the bundled
-view restriction. A subclass may add another group, but cannot replace a superclass's group; repeated group names in one
-class hierarchy are checked once.
-Older cores ignore the optional annotation and retain their prior behavior; no provider-version floor is raised.

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPlugin.java

@@ -348,6 +348,26 @@ public void onInvocationEnd(InvocationEndInfo info) {
         }
     }
 
+    /** Produces metadata without creating spans or changing the configured provider/resource. */
+    @Override
+    public String providePropagationMetadata(
+            String executionArn, String operationId, String parentOperationId, String targetFunctionName) {
+        if (!tracingEnabled || !executionArn.equals(durableExecutionArn)) return null;
+        var trace = executionTrace;
+        if (trace == null) return null;
+        // An observed operation's actual context is authoritative, including invocation-view continuation segments.
+        var context = operationContexts.get(operationId);
+        if (context == null) {
+            // Before a new operation starts, use the same deterministic ID its initial span will receive.
+            context = SpanContext.create(
+                    trace.traceId(),
+                    idGenerator.generateSpanIdForOperation(durableExecutionArn, operationId),
+                    effectiveTraceFlags(),
+                    effectiveTraceState());
+        }
+        return OtelPropagationMetadata.fromContext(context);
+    }
+
     // ─── Operation hooks ─────────────────────────────────────────────────
 
     @Override

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPlugin.java

@@ -354,6 +354,26 @@ public void onInvocationEnd(InvocationEndInfo info) {
         }
     }
 
+    /** Produces metadata without creating spans or changing the configured provider/resource. */
+    @Override
+    public String providePropagationMetadata(
+            String executionArn, String operationId, String parentOperationId, String targetFunctionName) {
+        if (!tracingEnabled || !executionArn.equals(durableExecutionArn)) return null;
+        var trace = executionTrace;
+        if (trace == null) return null;
+        // An observed operation's actual context is authoritative, including invocation-view continuation segments.
+        var context = operationContexts.get(operationId);
+        if (context == null) {
+            // Before a new operation starts, use the same deterministic ID its initial span will receive.
+            context = SpanContext.create(
+                    trace.traceId(),
+                    idGenerator.generateSpanIdForOperation(durableExecutionArn, operationId),
+                    effectiveTraceFlags(),
+                    effectiveTraceState());
+        }
+        return OtelPropagationMetadata.fromContext(context);
+    }
+
     // ─── Operation hooks ─────────────────────────────────────────────────
 
     @Override

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPropagationMetadata.java

@@ -0,0 +1,18 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+package software.amazon.lambda.durable.otel;
+
+import io.opentelemetry.api.trace.SpanContext;
+
+/** Pure encoding of an existing operation context into the reviewed X-Ray carrier shape. */
+final class OtelPropagationMetadata {
+    private OtelPropagationMetadata() {}
+
+    static String fromContext(SpanContext context) {
+        if (context == null || !context.isValid()) return null;
+        var trace = context.getTraceId();
+        var header = "Root=1-" + trace.substring(0, 8) + "-" + trace.substring(8) + ";Parent=" + context.getSpanId()
+                + ";Sampled=" + (context.isSampled() ? "1" : "0");
+        return header;
+    }
+}