Different behavior between Sentry Open Telemetry and Sentry .NET SDK

[yasmoradi]

Description

I'm enabling Open Telemetry by following code:

services.AddLogging(loggingBuilder =>
{
    loggingBuilder.ConfigureLoggers(configuration);

    loggingBuilder.AddOpenTelemetry(options =>
    {
        options.IncludeScopes = true;
        options.IncludeFormattedMessage = true;
        configuration.Bind("Logging:OpenTelemetry", options);
    });
});

var openTelemetry = services.AddOpenTelemetry()
    .WithMetrics()
    .WithTracing()
    .ConfigureResource(resource =>
    {
        resource.AddAttributes([new("service.name", Application.ProductName!)]);
    });

var useOtlpExporter = string.IsNullOrWhiteSpace(configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]) is false
    || string.IsNullOrEmpty(configuration["OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"]) is false;

if (useOtlpExporter)
{
    openTelemetry.UseOtlpExporter();
}

It shows logs in https://xxx.sentry.io/explore/logs/ and shows traces in https://xxx.sentry.io/explore/traces/.

My question is, why it doesn't show errors in https://xxx.sentry.io/issues/ (Which basically is a home page)
Before switching to Open Telemetry, I could see the in home page and this is so confusing for people who expects to see something in the home page.

The other question that I've is as I can see, there's a metrics page https://xxx.sentry.io/explore/metrics/

But it seems OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is not supported at the moment as it's not documented in https://xxx.sentry.io/settings/projects/dotnet/keys/

[linear-code]

NET-727

[jamescrosswell]

It shows logs in https://xxx.sentry.io/explore/logs/ and shows traces in https://xxx.sentry.io/explore/traces/.

My question is, why it doesn't show errors in https://xxx.sentry.io/issues/

OpenTelemetry itself is a three legged stool that covers: Logs, Metrics and Traces.

There isn't any first class support for Errors in OpenTelemetry.

• You could log errors using the logging API... and these would show up in the Logs but the data structures used by OpenTelemetry logs don't have anywhere to store things like stack traces etc. so you'd get pretty watered down exceptions in this case (string representations of exceptions - not structured representations of exceptions).
• Older versions of the OpenTelemetry APIs did have very limited support for attaching exceptions to traces (aka Activities in .NET) but this has been deprecated and the current guidance is to use the logging APIs.

Open Telemetry itself also won't link logs or metrics to traces - the three concepts are all independent in OTEL

So whilst you can just pipe data directly to the OTLP endpoint for you Sentry project, you'll definitely get a better experience if you use our OTLP integration... That integration augments telemetry piped into the OTLP endpoint to ensure logs, metrics, traces and errors are all connected.

It also automatically captures any unhandled exceptions (or lets you manually capture exceptions) with full enhanced stack traces etc.

The other question that I've is as I can see, there's a metrics page https://xxx.sentry.io/explore/metrics/

But it seems OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is not supported at the moment as it's not documented in https://xxx.sentry.io/settings/projects/dotnet/keys/

That's complex, but I think this provides the full story:

• OTLP metrics ingestion endpoint sentry#111136

[yasmoradi]

Thanks!
OTel exporters I've seen so far, either:
1- Provide extension method for OpenTelemetryBuilder
AND/OR
2- Provide extension methods for TraceProviderBuilder, MeterProviderBuilder and LoggerProviderBuilder

And that would end-up with having traces, logs and metrics easily.

I've added samples for Azure App Insights, Open Telemetry Exporter (Works just fine with Aspire Dashboard) and Console Exporter and as you can see, from the 4 methods that I'm expecting from Sentry.OpenTelemetry.Exporter, there's only 1 provided:

services.AddLogging(loggingBuilder =>
{
    loggingBuilder.ConfigureLoggers(configuration);

    loggingBuilder.AddOpenTelemetry(options =>
    {
        options.IncludeScopes = true;
        options.IncludeFormattedMessage = true;
        configuration.Bind("Logging:OpenTelemetry", options);
    });
});

var openTelemetry = services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.AddConsoleExporter(); // Console
        metrics.AddSentryExporter(); // ⚠️Not supported
    })
    .WithTracing(tracing=>
    {
        tracing.AddConsoleExporter(); // Console
        tracing.AddSentryExporter(); // ✅Supported
    )
    .WithLogging(logging =>
    {
        logging.AddConsoleExporter(); // Console
        logging.AddSentryExporter(); //⚠️Not supported
    })
    .ConfigureResource(resource =>
    {
        resource.AddAttributes([new("service.name", Application.ProductName!)]);
    });

openTelemetry.UseOtlpExporter(); // Aspire Dashboard

openTelemetry.UseAzureMonitorExporter(); // Azure Application Insights

openTelemetry.UseSentryExporter(); //⚠️ Not supported

Would you recommend someone using Sentry.OpenTelemetry.* instead of Sentry.AspNetCore? To me, this part of the products seems to be in-complete.

[jamescrosswell]

Your comment above helps understand what people would ideally like in terms of a public API - thank you 🙏🏻 .

There are various technical challenges - but it's something to strive for at least.

To me, this part of the products seems to be in-complete.

I think that's fair - we added OTLP exporters for tracing around 2 months ago, so it'll take a while to flesh things out. Prior to that our OpenTelemetry integration was based on a custom span processor that captured OTEL spans and converted them to Sentry ones. We did build something to support OTEL metrics about 2 years ago but that product was ultimately not sustainable so was retired and replaced with v2 of our metrics product which, as yet, doesn't have any OTLP endpoint you can send metrics to... Structured Logs are also relatively new (maybe 18 months old) and, again, no OTLP endpoint.

One of the biggest technical challenges is that Sentry aims to provide a connected experience - you should be able to navigate seamlessly between things like Sentry Replays (a concept that doesn't exist in OTEL), Errors (limited support in OTEL), Metrics (slightly different shape in OTEL), Logs and Traces... It's very hard (maybe even impossible) to send all of that via OTLP because the protocol simply wasn't designed to model some of these things.

Would you recommend someone using Sentry.OpenTelemetry.* instead of Sentry.AspNetCore?

For tracing, you could potentially. It depends what other things you've got in your stack. Sentry provides quite robust automated tracing for some database technologies OOTB but for others (e.g. MongoDB that I've been working on in the last couple of days) you really need to leverage OTEL. So if your ASP.NET Core application uses MongoDB, you might be better off using OTEL across the board.

Also if you're using any 3rd party libraries that we don't have a dedicated integration for, but which are instrumented with OTEL, you can use Sentry.OpenTelemetry.* to capture the instrumentation built into those libraries... and that's ultimately a more scalable way of instrumenting applications (Sentry is never going to be able to build integrations for every NuGet package you could ever want to use - even if those packages did expose the hooks and APIs we needed to do so).

Whether Sentry's OTEL integration is ready for or makes sense for you and your team is obviously your call to make - but the above are some examples of why someone might chose to go the OpenTelemetry route.

In any event, thank you very much for the feedback/suggestion. It gives us some interesting things to consider in the team.