[Feature]: Propagate per-operation trace context for chained invokes #751

What would you like?

Implement the SDK/plugin portion of per-operation X-Ray propagation so a chain-invoked Lambda execution can descend from the caller's CHAINED_INVOKE operation span. Support durable and non-durable targets; a non-durable target must not need the Durable Execution SDK merely to receive the tracing header.

This is a tracing-model and plugin-capability enhancement. The existing inherited-parent behavior remains the fallback when operation metadata is absent. The new capability lets an optional plugin contribute metadata to a persisted operation request; it must not change customer payloads or make durable execution depend on instrumentation.

Possible Implementation

Shared SDK/plugin contract

1. Add an optional synchronous propagation hook. The proposed contract is providePropagationMetadata(input) -> PropagationMetadata | absent, with idiomatic names in each SDK. The initial read-only input contains executionArn, operationId, parentOperationId, and targetFunctionName. Use SDK-owned input/result types; do not expose generated Lambda model objects, OTel types, or the mutable checkpoint request to plugins.
2. Collect results before the operation's START checkpoint is serialized. The hook is deterministic for the same execution/operation identity and side-effect free. Replaying an already checkpointed START must not invoke it again. A failed/uncommitted checkpoint can cause a later attempt to recompute metadata, so do not promise exactly-once hook invocation; recomputation must remain stable.
3. Merge supported metadata members in configured plugin order. For each member, the first non-null value wins. Different plugins may fill different members. If a later plugin supplies a different value for an already populated member, keep the first value, warn with both plugin identities, and increment a conflict counter. Omit empty metadata. Missing hooks, absent results and ordinary plugin failures must preserve the existing execution/fallback behavior; retain the SDK's fatal-error and cancellation rules.
4. Map the result to optional, flat wire fields in the core SDK: ChainedInvokeOptions.XAmznTraceId and DistributedMapOptions.XAmznTraceId. Metadata belongs to each operation update, because one checkpoint request can contain multiple invokes with different desired parents. Do not use a single tracing header on the checkpoint HTTP request as a substitute. Publish/consume the required service model and client dependency versions so serialization actually retains the new field.
5. Implement the producer in both OTel views. Use the resolved execution trace ID, the actual deterministic span ID for the calling operation, and the resolved sampling decision. Emit X-Ray header encoding, for example Root=1-<8 hex>-<24 hex>;Parent=<operation span ID>;Sampled=0|1. The plugin owns tracing/encoding; the core transports the resulting string. Do not send a W3C 00-... value in this field, adopt a transient checkpoint HTTP span as the operation parent, or change cross-language ID algorithms as part of this work.
6. Keep the contract extensible without adding unrelated operation APIs. The model proposal also includes distributed-map propagation and names fanout/HTTP START paths. Preserve the optional field in applicable models/serializers, and confirm the operation-specific inputs and supported dispatch paths before wiring those paths. New operation APIs or end-to-end W3C/baggage propagation need their own scope. Customer payload and ClientContext propagation remain customer-owned.

Repository-specific work

• Add an optional/no-op synchronous hook using SDK-owned frozen input/result types, and ordered metadata collection in PluginExecutor. Preserve plugin ordering and failure isolation instead of treating the returned data as a lifecycle notification.
• Call it in InvokeOperationExecutor.check_result_status() only when creating a new invoke START, before OperationUpdate.create_invoke_start. Account for checkpoint failures and the method's pre/post-checkpoint calls without invoking it for existing replay state.
• Extend the SDK's ChainedInvokeOptions wrapper with an optional tracing member, mapping it to XAmznTraceId in both from_dict/to_dict paths and through operation-update serialization. Preserve omission for absent values. Align any distributed-map wrappers and the boto3/botocore service model dependency when that model is available.
• Implement generation in both OTel views from the resolved execution trace and operation span identity; keep the data invocation-local and independent of ambient worker-thread context. Add serializer, dispatcher, public-API and replay tests.

Acceptance criteria

• Existing plugins with no propagation hook and executions without an OTel plugin keep working with the new fields omitted.
• A new invoke START carries the expected operation-specific header; no propagation callback is run when replay consumes an existing checkpoint. Cover failed checkpoint/retry recomputation without asserting exactly-once hook delivery.
• Two invokes batched in one checkpoint carry distinct operation parent IDs; tenant, target and payload fields are preserved.
• Merge tests cover absent values, different members, equal repeated values, conflicting values, ordering, warning/counter behavior, and throwing plugins without changing the execution outcome.
• Both OTel views preserve canonical trace identity and sampled/unsampled decisions, with inherited context and supported fallback-context cases.
• Serialization/round-trip tests prove the generated/custom client models retain optional XAmznTraceId fields, including distributed-map models where supported; older inputs without the fields still work.
• Once backend support is available, deployed tests cover durable and non-durable children, mixed-language parent/child combinations, suspension/resume and both Active and PassThrough tracing. Under Active tracing, Lambda may insert its own segment: assert the correct ancestry rather than requiring every child Workflow to have the operation as its immediate parent.
• Coordinate updated topology requirements and language handlers with the conformance repository. Scope the new assertions to operation-level propagation while retaining the old fallback contract.
• Document the hook, ordering/failure/replay semantics, supported versions, and rollout dependencies before declaring the end-to-end capability complete.

Is this a breaking change?

The new wire members and hook should be optional and additive; existing plugin implementations should not be forced to implement a new method. Enabling operation-level propagation intentionally changes the child trace hierarchy, so the telemetry contract, affected tests and release/migration guidance need review. Coordinate with any ongoing plugin-factory API migration rather than introducing avoidable successive registration changes.

Does this require an RFC?

Yes: this adds a plugin capability that contributes to persisted operation requests and changes trace topology. Finalize the SDK-owned metadata type/member names and generic input shape during shared design review. Keep the reviewed flat XAmznTraceId wire shape and X-Ray encoding consistent across implementations; older nested/W3C examples are not a separate wire contract.

Additional Context

Current source inspected at 631b78d1b505b157eee758662291fc5e2353bbb6. Proposed language-specific hook name: provide_propagation_metadata.

• Invoke START construction
• Plugin contract and executor
• SDK Lambda model wrappers/serialization
• Execution-view OTel producer
• Invocation-view OTel producer

The worker-context issue #428 is separate from supplying outbound operation metadata. The completed tracing work in #674 does not add the new checkpoint field or return-value hook.

Service-model publication and backend forwarding/rollout are external dependencies. This issue tracks the SDK and OTel-plugin implementation, not backend deployment. The backend must accept the operation-level header and retain its normalized result for child invocations; SDK-only changes cannot establish the new hierarchy by themselves. No new end-to-end implementation or runtime validation is claimed by this tracking issue.

Related SDK implementation tracking: JavaScript #952, Java #764.