Skip to content

Otel improvements - #7798

Draft
ramonsmits wants to merge 40 commits into
masterfrom
otel
Draft

ramonsmits wants to merge 40 commits into
masterfrom
otel

Conversation

@ramonsmits

Copy link
Copy Markdown
Member

No description provided.

Comment thread src/NServiceBus.Core/OpenTelemetry/InstrumentationOptions.cs Outdated
Comment thread src/NServiceBus.Core/OpenTelemetry/InstrumentationOptions.cs Outdated
Comment thread src/NServiceBus.Core/OpenTelemetry/InstrumentationOptions.cs Outdated
Comment thread src/NServiceBus.Core/OpenTelemetry/MessagePayloadToTagsBehaviors.cs Outdated
{
var json = JsonSerializer.Serialize(instance, instance.GetType());
var base64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
activity.SetTag("nservicebus.message.body", base64);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it should always use base64. If the payload if text, then it should just be utf8.

Second, we should NOT do extra (de)serialization. We should do this where we already have access to byte[] and then write this either as utf8 string or base64 string. To decide that we have probe for \0, force if via config (log payload as base64 or utf8), or assume utf8, catch exception on GetBytes and fallback to base64 encoding, maybe even another option.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI As part of the assembly scanning efforts we have tried to sort as much as possible the paths that require dynamically referenced code. The JSonSerializer usage here would regress on that because it introduces another method usage that is not trimming friendly

[System.Diagnostics.CodeAnalysis.RequiresDynamicCode("JSON serialization and deserialization might require types that cannot be statically analyzed and might need runtime code generation. Use System.Text.Json source generation for native AOT applications.")]
[System.Diagnostics.CodeAnalysis.RequiresUnreferencedCode("JSON serialization and deserialization might require types that cannot be statically analyzed. Use the overload that takes a JsonTypeInfo or JsonSerializerContext, or make sure all of the required types are preserved.")]
public static string Serialize(object? value, Type inputType, System.Text.Json.JsonSerializerOptions? options = default);

we are planning to add some kind of "regression" approval soon that should capture things like that in the future

While I'm here, I would like to add something else. I’m concerned about adding the full message body as an activity tag.

The OTel messaging semantic conventions define messaging.message.body.size, but that is only the size of the body in bytes, not the body content itself. I don’t see a standard semantic convention for attaching the full message payload to a span attribute.

Adding the full body as nservicebus.message.body, even base64 encoded, seems risky:

  • high-cardinality span attributes
  • potentially large telemetry payloads
  • possible PII/secrets leakage
  • increased indexing/storage cost in observability backends
  • base64 does not provide sanitization

But I'm not in the weeds as you are so I might miss important context.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI #7800

Comment thread src/NServiceBus.Core/OpenTelemetry/MessagePayloadToTagsBehaviors.cs Outdated
&& operations.Length > 0
&& operations[0].AddressTag is UnicastAddressTag unicastTag
&& outgoingMessage.Headers.TryGetValue(Headers.MessageIntent, out var intentStr)
&& intentStr is "Send" or "Reply")

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis should not use magic strings

MergeDispatchProperties(publishContext, options.DispatchProperties);

using var activity = activityFactory.StartOutgoingPipelineActivity(ActivityNames.OutgoingEventActivityName, ActivityDisplayNames.PublishEvent, publishContext);
var publishDisplayName = activityFactory.Options.UseMessageDestinationInSpanNames

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This likely should be in ActivityDecorator


activity.DisplayName = ActivityDisplayNames.ProcessMessage;
activity.DisplayName = Options.UseMessageDestinationInSpanNames
? $"{ActivityDisplayNames.ProcessOperation} {context.ReceiveAddress}"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we append the incoming message type? Like the 1st type from EnclodedMessageTypes?

irinascurtu and others added 5 commits June 10, 2026 14:15
…opagator (#7820)

* initial migration to the DistributedContextPropagator for W3C compatibility

* new propagator with backwards compatiblity tests

* fixing the startnewtrace header progagation

* refactor ContextPropagation.cs

* small tweak

* warning fixes

* att tests reflect supported baggage and tracestate formats
Activity baggage with a null value used to throw ArgumentNullException
during context propagation (Uri.EscapeDataString(null)). The switch to
DistributedContextPropagator on this branch fixes the root cause; this
test pins the behavior so it cannot regress.
ramonsmits and others added 7 commits June 19, 2026 14:10
…ards compatible until v11) (#7825)

* ✨ Make DistributedContextPropagator opt-in via AppContext switch

The switch to System.Diagnostics.DistributedContextPropagator changes the
OpenTelemetry baggage wire format (W3C OWS encoding + whitespace trimming),
which is breaking on rolling upgrades. Keep the legacy percent-encoded
propagator as the default and gate the new propagator behind the
NServiceBus.Core.OpenTelemetry.UseDistributedContextPropagator AppContext
switch (default in v11).

All temporary code (switch plumbing + legacy propagator) lives in
obsolete_v11.cs so v11 cleanup is a single file deletion plus removing the two
delegation blocks in ContextPropagation.cs. Follows the existing
AppContextSwitches.UseV2DeterministicGuid / PreObsolete pattern.

- ContextPropagation: delegate to the legacy propagator unless the switch is on
- obsolete_v11.cs: switch + byte-for-byte revert of the pre-10.3 propagator
- Tests asserting the new W3C format enable the switch per-fixture
- New ContextPropagationDefaultBehaviorTests locks in legacy default behavior
- Acceptance baggage assertion reverted to the legacy default wire format

Span naming (UseMessageDestinationInSpanNames) is already opt-in and unchanged.

* Refactor: Replace `ObsoleteV11` with `LegacyContextPropagation`

* Add test to verify null baggage value does not throw in legacy propagator

* Added comments about preserve legacy baggage handling behavior when escaping and trimming values

* Add test to verify that we are preserving whitespace in legacy propagator baggage values.

* add comments to clarify intent behind legacy propagator handling

* update ContextPropagationCompatibilityTests to use correct LegacyContextPropagation delegates

* rename ContextPropagationTests to LegacyContextPropagationTests and remove outdated baggage handling tests

* allow changing the propagator implementation at runtime

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
…Source (#7844)

Handler (InvokeHandler) spans can now be emitted from a dedicated
"NServiceBus.Core.Handler" ActivitySource so they can be filtered or
sampled independently of the pipeline spans (#7284). Opting in without
subscribing to the new source suppresses handler spans entirely, making
Activity.Current inside handlers/behaviors the "process message" span —
the flattened trace requested in the issue.

Existing OpenTelemetry configurations only subscribe to "NServiceBus.Core"
and would silently lose handler spans, so the new source is opt-in via the
NServiceBus.Core.OpenTelemetry.UseHandlerActivitySource AppContext switch
until v11, following the DistributedContextPropagator pattern:

- ActivitySources: add permanent Handler source
- ActivityFactory.StartHandlerActivity: pick source based on the switch
- obsolete_v11.cs: cached switch with PreObsolete markers; in v11 delete
  the class and make ActivitySources.Handler unconditional
- Unit tests cover default source, opt-in source, preserved span data,
  and no-listener suppression; acceptance test locks in the default
* Optout on dispatching events

* Aproval tests

* Update src/NServiceBus.Core.Tests/ApprovalFiles/APIApprovals.ApproveNServiceBus.approved.txt

Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>

* Renamed Suppress to Emmit

* Wrap `AddEvent` call in null-check for `TryGetRecordingIncomingPipelineActivity`.

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>
Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
* gauge meter counting active handler invocations

* update MeterAPI approval file to reflect version 0.3.0 changes and new active handlers metric

* Improvements

* switching to active messages gauge

* remove obsolete MessageMetadataRegistry dependency from IncomingPipelineMetrics and capture enclosed message types on the gauge meter

* remove unused Unicast.Messages imports from test files

* Apply suggestion from @tmasternak

* move `ActiveMessageScope` definition to correct location in `IncomingPipelineMetrics` file

---------

Co-authored-by: Irina Dominte <scurtuirina@gmail.com>
* ✨ Add TraceConnector enum and endpoint-level trace connector defaults to InstrumentationOptions

* ✨ Honor endpoint-level trace connector defaults in send/publish behaviors with per-message overrides

* ✨ Add acceptance tests for endpoint-level trace connector defaults and per-message overrides

* ✨ Approve public API additions for trace connector configuration

* Replace `TraceConnector` with `TraceMode` for clearer terminology and updated tracing behavior.

* Update approval file after replacing `TraceConnector` with `TraceMode` in public API

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
* first stab at metric counter

* Text key fix

* Apply suggestion from @tmasternak

* update metric name for deduplication tracking

* rename meterFactory to fakeMeterFactory in unit tests

* subscribe test meter listern to all available instruments to handle instruments with duplicated names caused by multiple creations of IncomingPipelineMetrics objects

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>
Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
* Add tracing for recoverability pipeline using ActivityFactory

* Refactor recoverability pipeline to use `Activity` fallback for tagging recoverability actions

* Replace `new NoOpActivityFactory()` with `NoOpActivityFactory.Instance` in tests and implementation to enforce singleton usage.

* Normalize recoverability action tags and update Activity display names for better trace clarity in recoverability pipeline.

* Normalize recoverability action tags and update Activity display names to improve trace clarity in the recoverability pipeline.

* Fixed approval file

* Update ActivityTagsTests.Verify_ActivityTags.approved.txt

* Ensure that tracemode operations for delayed messages (user operations, sagas, and recoverability) are happening before RoutingContext stage, and that StartNewTrace header is properly set before reaching that state.

* Replace RecoverabilityTraceMode with the existing TraceMode enum; only set recoverability trace metadata for DelayedRetry

* Remove custom recoverability action spans; keep this PR focused on delayed-message trace mode

The dedicated "Recoverability" span/activity work (ActivityFactory.StartRecoverabilityActivity
and its wiring/tagging) has been split out to the recoverability-action-spans branch. This PR
now only concerns how trace mode is decided for delayed sends, saga timeouts, and delayed
retries via InstrumentationOptions.

* Simplify `InstrumentationOptions` by removing `MoveToErrorTraceMode`; refactor `ActivityFactory` constructor for improved initialization consistency

* Add acceptance tests for delayed-message trace mode; fix MoveToError regression

Covers the three delayed-message origins this PR configures: explicit delayed
sends (SendOperationTraceMode), saga timeouts (SagaTimeoutTraceMode), and
recoverability delayed retries (Recoverability.DelayedRetryTraceMode). Each
area verifies the backward-compatible default (start a new linked trace) and
that the new option flips it to continuing the existing trace. The delayed-send
tests also cover per-message override precedence: an explicit StartNewTraceOnReceive
always wins, and a ContinueExistingTraceOnReceive request cannot defeat the
delayed-send backward-compatible default.

Also fixes When_retrying_messages.Should_correlate_delayed_retry_with_send, which
asserted the opposite (same-trace) behavior and never actually ran due to
Requires.DelayedDelivery() gating - switched to the in-memory transport's native
delayed delivery support so it exercises the real code path.

Restores PopulateRecoverabilityTraceMetadataBehavior's MoveToError handling
(dropped when DelayedRetry-only gating was introduced), which had silently
broken When_incoming_message_moved_to_error_queue.Should_add_start_new_trace_header.
MoveToError itself stays hardcoded to always start a new trace, matching
pre-existing behavior; it was not part of this PR's configurable scope after
MoveToErrorTraceMode was removed from InstrumentationOptions.

* Fix trailing whitespace flagged by IDE0055

* test tweaks

* Apply suggestion from @tmasternak

* Update delayed message trace mode configuration in acceptance tests

Replaces `TestEndpointContinuingTrace` with a unified `TestEndpoint` that dynamically configures `TraceMode` for delayed sends and saga timeouts. Simplifies `RetryingEndpoint` initialization. Removes redundant handler setup in trace tests.

* Apply suggestions from code review

Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>

* fixing att: fix endpoint setups for delayed retries and fix access modifiers in otel tests saga

* Simplify delayed message trace tests by consolidating scenarios and removing redundant per-message overrides

* better test names

---------

Co-authored-by: Irina Dominte <scurtuirina@gmail.com>
Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>
irinascurtu and others added 8 commits July 29, 2026 12:57
* Add tracing for recoverability pipeline using ActivityFactory

* Refactor recoverability pipeline to use `Activity` fallback for tagging recoverability actions

* Replace `new NoOpActivityFactory()` with `NoOpActivityFactory.Instance` in tests and implementation to enforce singleton usage.

* Normalize recoverability action tags and update Activity display names for better trace clarity in recoverability pipeline.

* Normalize recoverability action tags and update Activity display names to improve trace clarity in the recoverability pipeline.

* Fixed approval file

* Update ActivityTagsTests.Verify_ActivityTags.approved.txt

* recoverabilty span should not be parenting the follow-up processing attempts. Span tags and display names update moved directly after recoverability aciton resolution

* refactor: centralize recoverability span updates and add display name adjustments

* fix tests

* fix: correct typo in DelayedRetryOperation constant

* Enable the Recoverability ActivitySource in acceptance tests; add dedicated recoverability span tests

TestingActivityListener now accepts multiple source names so
OpenTelemetryAcceptanceTest can subscribe to both "NServiceBus.Core" and
"NServiceBus.Core.Recoverability" - previously only Main was subscribed, so
recoverability spans were silently never created in any acceptance test.

Adds When_recoverability_action_occurs.cs covering all four recoverability
outcomes (immediate retry, delayed retry, move to error, discard): span
creation, display name, and the nservicebus.recoverability_action tag, plus
the UseMessageDestinationInSpanNames toggle for immediate retry.

* Simplify When_recoverability_action_occurs: one endpoint, no ErrorSpy

Collapsed all five scenarios onto a single RecoverabilityEndpoint, moving
the per-test differences (Recoverability policy, UseMessageDestinationInSpanNames)
into each test's CustomConfig callback instead of separate endpoint classes.

Replaced the ErrorSpy endpoint (used only to signal test completion for the
move-to-error case) with a Done() predicate that waits directly on the
recoverability span being captured - the actual signal the test cares about,
rather than an indirect one requiring a second endpoint and message hop.

* Combine immediate/delayed/move-to-error/discard into one recoverability test

One endpoint config now drives a single message through the default policy's
full immediate-retry -> delayed-retry -> move-to-error cascade, plus a second
message routed straight to Discard via a CustomPolicy branch that otherwise
delegates to DefaultRecoverabilityPolicy.Invoke for the first message. Asserts
by matching on the nservicebus.recoverability_action tag rather than activity
position/count, since the exact number of immediate-retry attempts before
falling through isn't a contract worth pinning down.

Kept the destination-naming test separate since it needs a different
Tracing() configuration that can't share the same CustomConfig block.

* bug fixes and att for recoverability spans

* Fix ActivitySource versioning and make sure the versiosn are checked in tests

* small ActivityFactory refactor

* fix: add missing newline at end of ActivityTagsTests.cs

---------

Co-authored-by: Ramon Smits <ramon.smits@gmail.com>
Co-authored-by: Irina Dominte <scurtuirina@gmail.com>
* Introduce log based exception recording mode

* in the dup exception capturing mode the exception deatils are captured only on the inner most span.

* Rename ExceptionRecordingMode.Dup to SpanAndLogs for clarity

"Dup" didn't convey what the mode actually does; SpanAndLogs states it directly.

* Relocate legacy exception tags into obsolete_v11 with removal notes

otel.status_code/otel.status_description predate Activity.SetStatus and
exception.escaped is Deprecated by the OTel semantic conventions. Neither
belongs long-term; group them in LegacyExceptionTags alongside the file's
other v11-scheduled removals and document exactly what to delete then.

* Default ExceptionRecordingMode from OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN unless explicitly configured

Lets users opt into the exceptions-as-logs model ahead of NServiceBus adopting
it as the default, per the (unratified) OTel migration convention. Tracks
whether the user explicitly touched ExceptionRecordingMode via a field-backed
setter so an explicit setting always wins over the environment variable.
Scaffolding lives in obsolete_v12.cs since the whole mechanism is meant to be
removed once the migration settles.

* Remove RecoverabilityAction.LogMessage, build log messages via switch in RecoverabilityPipelineExecutor

Keeps the message-formatting logic out of the public RecoverabilityAction
API surface; RecoverabilityPipelineExecutor is the only consumer, so it can
own the mapping from action type to log message directly.

* Migrate recoverability logging from static ILog to DI-resolved MEL ILogger

Starts the move away from NServiceBus.Logging's static LogManager pattern in
favor of Microsoft.Extensions.Logging, resolved from the DI container that's
already flowing through the pipeline. Extracts all of it into a dedicated
RecoverabilityActionLogger: each recoverability action type gets its own
ILogger<T> (ImmediateRetry, DelayedRetry, MoveToError, Discard, falling back
to ILogger<RecoverabilityAction> for anything else), so hosts can filter/level
recoverability logs per action type. Message templates are literal per
LoggerMessage-generated method to satisfy CA2254 rather than building a
variable string, which also gives cheap structured logging for free.

RecoverabilityPipelineExecutor now just delegates to
RecoverabilityActionLogger.LogRecoverabilityAction in one line instead of
owning the switch/DI-resolution logic itself.

* Give OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN precedence over explicit ExceptionRecordingMode configuration

Lets operators force the exception-signal behavior via environment variable
without a code change/redeploy, overriding whatever the endpoint explicitly
configured. Drops ExceptionRecordingModeSetByUser, which is no longer needed
now that precedence doesn't depend on whether the user touched the setting.

* fixing att test after minor changes to the recoverability action log messages
* Started performance metrics

* Removed the Performance metrics from public API

* Renamed a metric

* nullable enable

* remove nullable

* added a line

* test fix

* trailing whitespace

* Removed execution result

* add enclosed message type to the deserialization insturmentation

* Renamed classes and made the tags backwards compatible

* Apply suggestions from code review

Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>

* fix: correct tag formatting and indentation in OpenTelemetry tests

* undo the type renamings for now

* fix: rename IncomingPipelineMetricsTags to IncomingPipelineMetricTags

* add MetersOptions to API approvals

* fix: correct variable name from messagingMetricsMetricses to messagingMetricsMeters

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>
* adjust log levels for recoverability actions

* split exception handling in recoverability action log messages to ensure backwards compatibility

* updated exception log message formats in recoverability tests for consistency
…ion Data (#7911)

* removed RecordedExceptions tracking in favor of directly using exception.Data

* moved ExceptionRecordedFlag where it's used

* Update src/NServiceBus.Core/OpenTelemetry/Tracing/ActivityFactory.cs

Co-authored-by: Daniel Marbach <daniel.marbach@openplace.net>

---------

Co-authored-by: Daniel Marbach <daniel.marbach@openplace.net>
…ine (#7912)

* Add support for instrument-specific metric tags in the incoming pipeline

* Introduce support for optional instrument names in metric tag methods and improve metric handling logic

* Simplify metric tagging logic and remove `IncomingPipelineMetricsTagBagConventionTests`.

* Simplify `RecordSerializeTime` by removing unnecessary context dependency and refactoring metric tagging logic.

* fix to outgoing pipeline metric tagging

* Improve outgoing pipeline metric tagging and refactor `RecordSerializeTime` to handle context-based tags.

* Remove unused metric tags in `IncomingPipelineMetrics`.

* Refactor metric tagging by replacing `Get<IncomingPipelineMetricTags>` with `MetricTags` extension property. Remove IncomingPipelineMetrics dependency from fakes

* Introduce `IMetricsTags` interface as the public api for changing metric tags.

* Refactor metric tagging logic in `IncomingPipelineMetrics` to simplify tag application and enforce consistent usage of `TagList`.

* Refactor `criticalTime` calculation to ensure `completedAt` is only set when needed

* Fix incorrect usage of `messageDeserializeTime` to `messageSerializeTime` in metric tagging logic

* Rename `Add` to `AddOrOverride` in `IMetricsTags

* Remove unnecessary call to `GetOrCreate<IncomingPipelineMetricTags>` in `MessageContext` constructor. All calls are now GetOrCreate via extension property.

* change the base type for MetricTagsExtensions to concrete type from ExtensionBag. Makes accessing tags requrie less nesting

* Refactor metric tagging tests to use constants for tag keys and simplify assertions

* fixing formatting errors
@tmasternak
tmasternak added this pull request to stack #7948 September 23, 2026 10:06
@tmasternak
tmasternak removed this pull request from stack #7948 September 24, 2026 10:43
irinascurtu and others added 5 commits September 24, 2026 12:47
…truments (#7936)

* Rename `IncomingPipelineMetrics` to `PipelineMetrics`

The class now serves both incoming and outgoing pipelines (e.g. serialization),
so the `Incoming` prefix no longer applies.

* Add `nservicebus.queue` and `nservicebus.discriminator` default metric tags with send-only support

Include queue name and discriminator as default metric tags on PipelineMetrics
whenever they are available. For send-only endpoints, which never create an
input queue, both tags are omitted to avoid misleading values.

Also adds serialize time metric tagging with queue, discriminator, and message
type, with new acceptance tests covering both regular and send-only scenarios.

* Rename `UseMessageDestinationInSpanNames` to `UseMessageTypeNamesInSpanNames`

The new name better describes the intent: span names include the message type
name, not the transport destination address.

* Make `RecordError` accept a nullable `Activity?`

Allow callers to pass a null activity when tracing is disabled, avoiding
null-checks at every call site.

* Remove `TracingExtensions` and inline pipeline tracing into `MessageOperations`

Replace the TracingExtensions.Invoke extension method with explicit try/catch
blocks in each MessageOperations method and MainPipelineExecutor. This makes
tracing flow visible at each call site.

Also converts MessageOperations to a primary constructor for consistency.

* Extend span display names to include message type for all outgoing operations

When UseMessageTypeNamesInSpanNames is enabled, subscribe, unsubscribe, send,
and reply spans now include the event/message type name in their display name,
matching the existing behavior for publish spans.

* Rename `IncomingPipelineMetricTags` to `PipelineMetricTags`

* Prepare new opt-in configuration options for proper obsoletion

* introduce pre-v11 defaults for backward compatibility of trace modes

---------

Co-authored-by: Ramon Smits <ramon.smits@gmail.com>
Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
* refactor receiver activity creation

* fix string comparison for StartNewTrace header and improve comments

* Introduce `NServiceBus.TraceParent` header to prevent the clash with native header set by the SDK

* remove redundant context propagation compatibility tests
)

* ✨ Opt-in switch to parent the incoming span on an ambient transport SDK span

When a transport SDK (Azure Service Bus, RabbitMQ, SQS) has its own OpenTelemetry
instrumentation, its receive span is Activity.Current when the incoming pipeline
starts. With the `NServiceBus.Core.OpenTelemetry.UseTransportSpanAsParent` AppContext
switch enabled the incoming message span becomes a child of that ambient span and
links to the NServiceBus sender span, matching the existing behavior for a transport
activity supplied via the ContextBag.

The switch is off by default so the v10 trace shape (incoming span is a child of the
sender span) is unchanged on a minor upgrade. The switch and the pre-v11 default test
live in obsolete_v11.cs / TransportParentSpanDefaultBehaviorTests.cs and are deleted
in v11; ActivityFactoryTests already runs with the v11 default enabled.

* ♻️ Remove the unused transport Activity-from-ContextBag parenting path

No transport supplies its receive Activity through the ContextBag; a transport SDK's
receive span reaches the incoming pipeline as Activity.Current instead, which the
UseTransportSpanAsParent switch now handles. Drop the ContextBag lookup, the unused
ContextBag parameter and the four unit tests that exercised it.

* ✅ Prove receive-side trace header backwards compatibility

Cover receiving with only the W3C traceparent header (sender on a version that predates
NServiceBus.TraceParent), with only the NServiceBus header, and that an invalid
NServiceBus header does not fall back to the W3C one.

* make names of the activity factory tests more descriptive

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@gmail.com>
* 🐛 Propagate trace context only in the outgoing message pipeline

Move context propagation from RoutingToDispatchConnector to
OutgoingPhysicalToRoutingConnector so it only runs for messages sent
through the outgoing pipeline. The routing to dispatch stage is also
used by the move to error and audit pipelines, which caused the current
activity to be written into the headers of forwarded messages.

* ✅ Add regression tests for trace context preservation in forwarding paths

Messages moved to the error queue, delayed retries, audit copies and
ForwardCurrentMessageTo enter the pipeline at the routing stage. Each
test runs the real action through RoutingToDispatchConnector under an
unrelated ambient activity and asserts the received traceparent is kept
and no baggage leaks in. All four fail on the parent commit.

* 🐛 Propagate trace context to ServiceControl retry acknowledgements

The acknowledgement is a control message forked straight into the
routing stage, so it no longer receives trace headers from the dispatch
connector. Propagate explicitly, as the subscribe and unsubscribe
terminators do, to keep it correlated to the processing of the retried
message.

* Update trace parent header in tests to the new nsb header

* Fix header key for diagnostics trace parent

---------

Co-authored-by: Tomasz Masternak <tomasz.masternak@particular.net>
Resolve conflicts between the OpenTelemetry work and the master changes:

- TransportReceiveToPhysicalMessageConnector: keep the outbox-skip paths from master and the outbox fetch/store/duplicate metrics from otel. The dispatching events now respect EmitMessageDispatchingEvents again.
- ReceiveComponent: pass PipelineMetrics, InstrumentationOptions and the ILogger to the connector.
- RetryAcknowledgementBehavior: keep the trace context propagation on the acknowledgement message.
- Test package versions from master, OpenTelemetry package from otel.
- Re-approve the public API and the LogManager usage list (ActivityFactory still uses LogManager).
tmasternak and others added 12 commits October 5, 2026 13:13
The inlined event gating in the with-outbox path had an inverted condition and
duplicated the fork block. Both paths now call Dispatch, which forks directly
when the dispatching events are off and goes through DispatchWithEvents when
they are on. Removes the unused WriteStartDispatchingEvent helper.
The factory is created before the container exists, so it cannot take an
ILogger in its constructor. RecordError now receives the behavior context's
service provider instead of the unused ContextBag and resolves
ILogger<ActivityFactory> on first use. The message is emitted through a
LoggerMessage method. Removes the last LogManager usage added by the otel
branch.
…ontext

Trace state and baggage headers are only meaningful together with a trace
parent. Apply them only when the message has NServiceBus trace context.
Without it, NServiceBus has nothing to propagate. The incoming span then
inherits what an ambient activity has through the parent chain.

Baggage from the headers is always applied, also when a transport SDK
receive span is the parent. None of the supported SDKs (Azure Service Bus,
RabbitMQ, SQS) propagate baggage yet. For when they do, a key the adopted
parent already has is skipped. Activity.Baggage reads through the parent
chain, so adding it again would double the baggage on every hop.

Trace state from the headers is not applied when the message starts a new
trace. It describes the trace of the sender, which the new trace is
detached from.

Start the incoming activity before the headers are read. Activity.Start()
sets Activity.Parent, so the skip can read the parent chain of the runtime
instead of tracking the adopted parent separately. The id format is still
set to W3C before the start.

Split ContextPropagation.PropagateContextFromHeaders into a trace-state
part and a baggage part on both propagator paths. The combined method stays
for callers that need both.
The switch selects the parent Activity of the incoming span. Rename it and
update all references, tests and comments to the Activity terminology.
Records why NServiceBus applies trace state and baggage from the message
headers on receive only when its own trace context is present, also under
an ambient transport SDK span, and why keys the adopted parent already has
are skipped. Captures the verified SDK behavior and the Activity.Parent
mechanics the decision rests on, and the alternatives that were rejected.
Apply incoming trace state and baggage only with NServiceBus trace context
Matches the root obsoletes-v10.cs convention where obsoletes-vN.cs holds
code that is obsolete in vN and removed in vN+1. The ExceptionRecordingMode
environment variable scaffold is removed in v12, so it belongs in obsoletes-v11.cs.
Several OpenTelemetry behaviors become the only behavior in v11 with no
option left to opt in or out. Until then they are gated behind one
AppContext switch, NServiceBus.Core.OpenTelemetry.UseV11Behavior, so an
endpoint adopts the v11 behavior as a whole instead of per feature:

- DistributedContextPropagator-based trace context and baggage propagation
- handler spans from the NServiceBus.Core.Handler ActivitySource
- ambient transport SDK receive span as parent of the incoming span
- destination and message type in span names
- no Start/Finished dispatching events on the incoming span
- no execution.result metric tag
- no legacy otel.status_code/otel.status_description tags and no
  exception.escaped attribute

Replaces the UseDistributedContextPropagator, UseHandlerActivitySource and
UseTransportActivityAsParent switches and removes the unreleased
UseMessageTypeNamesInSpanNames, EmitMessageDispatchingEvents and
Meters.EmitExecutionResultTags options. PublishTraceMode keeps its public
option in v11 and is not governed by the switch.

All v11 scaffolding now lives in the root obsoletes-v10.cs, replacing
OpenTelemetry/Tracing/obsolete_v11.cs.
Single AppContext switch for the OpenTelemetry v11 behaviors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants