Sentry metadata is scrambled when using falcon

sentry (https://github.com/getsentry/sentry-ruby) stores metadata in thread-local variables to try to isolate requests. but falcon uses fibers. the end result is that the metadata stored with a given error is some random sampling of the last fiber to record it. it's a design issue with sentry, for sure, but i was wondering if you had any ideas about how to isolate it better, short of waiting for them to add an option to support fibers?

I'm sorry to hear that, I could imagine it's frustrating.

One solution would be to bypass sentry's instrumentation and use the traces gem. A possible option is to implement traces-backend-sentry and report the data. That data is captured and can be reported per-fiber (the default). We'd need a way to emit spans to sentry, and we'd need a way to specify the parent when emitting those spans. It wouldn't be the simplest hack, but at least I could say, provided we have the above interface, it would definitely work and be a reasonable solution.

https://github.com/socketry/traces-backend-datadog/blob/main/lib/traces/backend/datadog/interface.rb for reference.

Now, to get a bit more specific. The Datadog backend is already fiber aware, so there is no problem using their tracing primitives directly: ::Datadog::Tracing.trace(name, resource: resource, tags: attributes, &block).

I'm assuming sentry isn't so we'd need a slightly augmented version of the above method:

::Sentry.trace(name, resource: resource, tags: attributes, parent: parent_trace, &block)

Then, we'd need to store per-fiber the current trace, so in other words, we'd need to do something like this:

parent_span = Fiber.sentry_current_span
::Sentry.trace(name, resource: resource, tags: attributes, parent: parent_span) do |span|
  Fiber.sentry_current_span = span
  yield
ensure
  Fiber.sentry_current_span = parent_span
end

Basically, if you can coax enough of this interface out of Sentry, and disable their "automatic" instrumentation, this might be a middle ground in order to get things working. For reference, some of the built in trace backends (test/capture/console) use a similar approach, e.g. https://github.com/socketry/traces/blob/main/lib/traces/backend/capture.rb#L68-L76.

Another area to explore is whether sentry has open telemetry support - we have a backend for that and it's per-fiber: https://github.com/socketry/traces-backend-open_telemetry

we are only using sentry for error reporting. we use datadog for tracing, which works great. i've instead monkey-patched sentry to store exception context in fibers instead of threads. seems to be working!

module Sentry
  class ThreadStorageOverride
    def thread_variable_get(key)
      Fiber[key]
    end

    def thread_variable_set(key, value)
      Fiber[key] = value
    end

    def method_missing(name, *args, &block)
      return super unless ::Thread.current.respond_to?(name)
      ::Thread.current.send(name, *args, &block)
    end

    def respond_to_missing?(name, include_private = false)
      ::Thread.current.respond_to?(name)
    end
  end

  class Thread < ::Thread
    def self.current
      @_current ||= ThreadStorageOverride.new
    end
  end
end

Clever.

@jakeonfire

@jakeonfire my sentry seems fine without that patch for errors. What are you seeing?

Also are you still at IFTTT? Samuel would love to know how you're using async/falcon, I'm sure!

I just caught up with Erich and Siddharth at Rubyconf and it was fun!

when manually adding custom metadata via e.g.

Sentry.configure_scope do |scope|
  scope.set_tag(:remote_endpoint, URI.join(base_url, endpoint).to_s)
end

the tags/extras/etc do not match the request stack trace. e.g. the request hits a controller for e.g. netatmo but reports a remote_endpoint that could only come from a different controller. i assume this is because sentry stores a "hub" local to each thread, but falcon uses several fibers in the same thread, so those fibers overwrite each others' metadata, and what gets reported to sentry is thus a mishmash of metadata from concurrently executing fibers.

yes, still at IFTTT! we converted shimmy to use falcon (instead of unicorn). shimmy is one of our service implementation platforms (using ruby/rails) so it runs Triggers/Queries/Actions for a subset of our services, dishing out API calls. so it's spending a lot of time running and waiting for responses to calls to a few dozen external APIs and doing a lot of waiting on them. with falcon, the throughput matches our finagle/scala server for the same amount of infra! (and no one at IFTTT is very good with finagle/scala, hence the move back to shimmy as our primary service implementation platform.)

so cool you got to catch up with old coworkers 😊

@jakeonfire

@jakeonfire Can we write a blog post about your migration?

sure! what would you need from me?

Thanks for the explanation, Jake!

Also, sorry, Samuel I know I'm like 2 years overdue for a blog post on my end..! 😭

what would you need from me?

a blog post? 😏

It would be good to write about some success stories. Probably the best way is to characterise the problem, describe the original setup, show some metrics/numbers/problems, then describe the implementation of Falcon, and then report on how the problems were addressed/mitigated/resolved. I'd be happy to work on that if you are able to give me information.

i've been sitting on this as i wasn't sure i could collect sufficient info as we migrated just over a year ago. we converted one of our more lightly-used servers, and have been moving more and more service implementations to it since. here's what i've found, though:

the original problem:

we have a 12 year old multi-application platform, part of which is to manage several (300+) service implementations each of which sits between our main application (ifttt.com) and a 3rd party API, providing a uniform interface to each. over the years we've tried different platforms to implement these service implementations. originally we used rails. then we switched to scala/finatra/finagle (which runs on JVM) for better performance, which it had in spades. but with normal engineer turnover, we eventually found ourselves in a situation where we had no engineers with sufficient experience (nor desire to learn) the scala/finatra/finagle platform, so we wanted to see if we could get the same performance from rails. enter falcon.

original setup:

unicorn
12 k8s pods, each with 0.5 CPU, 1G memory, and 6 worker processes allocated (0.22 vcores CPU, 0.6G memory used on average)
instantiating a new faraday connection for each external request
k8s round-robin routing strategy
~6m requests per day

falcon setup:

mostly the same, but with falcon using the rack environment config, plus per-hostname faraday connection caching via Thread::Local, falcon virtual (initially)
Possible memory leak? async-http-faraday#35 (comment)

we initially saw these resource changes:

~20% less CPU, ~40% more memory, and p50 latency down ~30%. we were able to lower the deployment from 12 to 6 pods once we added the connection caching.

eventual falcon setup after several months' tweaks:

falcon-host
dropped async-http-faraday as we weren't properly taking advantage of it
2 processes per pod
16 pods
each instance/pod has 1.1 CPU, 1.5G memory allocated (0.4 vcores CPU, 0.75G memory used on average)
~60% reduction in p50 latency compared to unicorn setup
~22m requests per day

what am i missing? 🙂

ok, done editing the above 😅

regarding the original issue: sentry now has a built-in fiber isolation level setting 🥳

https://github.com/getsentry/sentry-ruby/blob/3bf6552c5cc3bd351e6e4439ab2e60d7f23d6796/sentry-ruby/lib/sentry/configuration.rb#L399-L415

getsentry/sentry-ruby#3018

Nice, thanks for the update!

Navigation Menu

Repository navigation

Sentry metadata is scrambled when using falcon #274

the original problem:

original setup:

falcon setup:

eventual falcon setup after several months' tweaks:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sentry metadata is scrambled when using falcon #274

Description

Activity

ioquatix commented on Feb 20, 2025

jakeonfire commented on Feb 22, 2025

ioquatix commented on Feb 22, 2025

trevorturk commented on Feb 22, 2025

jakeonfire commented on Feb 24, 2025

ioquatix commented on Feb 24, 2025

jakeonfire commented on Feb 24, 2025

trevorturk commented on Feb 24, 2025

jakeonfire commented on Feb 24, 2025

ioquatix commented on Feb 24, 2025

jakeonfire commented on Mar 15, 2025

the original problem:

original setup:

falcon setup:

eventual falcon setup after several months' tweaks:

jakeonfire commented on Mar 15, 2025

jakeonfire commented on Aug 31, 2026

ioquatix commented on Aug 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions