feat: Runtime metrics integration by aldy505 · Pull Request #1400 · getsentry/sentry-go

aldy505 · 2026-08-29T11:14:46Z

Description

This is a proof of concept that I talked really briefly on Slack. Might or might not be merged. NodeJS has this integration, and I thought, this is easy and really possible in Go. Let's just have it!

To use or test this feature/integration, just copy and paste the code into your codebase, and fix all the import bugs (stuff like SetHubOnContext should be sentry.SetHubOnContext, etc). Then use it, like so:

// optional, but recommended. otherwise how would you handle graceful shutdown?
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()

if err := sentry.Init(sentry.ClientOptions{}); err != nil {
  panic("yada yada yada")
}
defer sentry.FlushWithContext(ctx)
go sentry.StartRuntimeMetrics(sentry.RuntimeMetricsConfig{Context: ctx})

// the rest of your program

Issues

Changelog Entry Instructions

To add a custom changelog entry, uncomment the section above. Supports:

Single entry: just write text
Multiple entries: use bullet points
Nested bullets: indent 4+ spaces

For more details: custom changelog entries

Reminders

Add GH Issue ID & Linear ID
PR title should use conventional commit style (feat:, fix:, ref:, meta:)
For external contributors: CONTRIBUTING.md, Sentry SDK development docs, Discord community

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-08-30T04:22:02Z

+
+	onceRuntimeMetrics.Do(func() {
+		runtimeMetricsRunning = true
+	})


Broken single-start race

High Severity

StartRuntimeMetrics moved the collector loop out of sync.Once and now relies on an unsynchronized runtimeMetricsRunning check. Concurrent callers can both pass that check, then both start loops that share runtimeMetricsSamples, which runtime/metrics.Read forbids.

Additional Locations (1)

runtime_metrics.go#L151-L183

^{Reviewed by Cursor Bugbot for commit 0aeba9f. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit e35aa47. Configure here.}

cursor · 2026-09-19T00:06:58Z

+					case MetricTypeGauge:
+						meter.Gauge(runtimeMetricsKeys[i].Key, value, WithUnit(runtimeMetricsKeys[i].Unit))
+					case MetricTypeCounter:
+						meter.Count(runtimeMetricsKeys[i].Key, int64(value), WithUnit(runtimeMetricsKeys[i].Unit))


Cumulative counters sent as increments

High Severity

/gc/heap/allocs:bytes and /gc/cycles/total:gc-cycles are lifetime totals, but each scrape passes the raw reading into Count, which increments. Sentry then sums those values, so go.memory.allocated and go.memory.gc.cycles inflate far past the real totals.

Additional Locations (2)

runtime_metrics.go#L196-L198

runtime_metrics.go#L52-L53

^{Reviewed by Cursor Bugbot for commit e35aa47. Configure here.}

cursor · 2026-09-19T00:06:58Z

+						for j := uint64(0); j < count; j++ {
+							meter.Distribution(runtimeMetricsKeys[i].Key, value, WithUnit(runtimeMetricsKeys[i].Unit))
+						}
+					}


Histogram re-emits all historical pauses

High Severity

/sched/pauses/total/gc:seconds is a cumulative histogram, but every interval expands every bucket count into individual Distribution samples. That re-reports every GC pause since start and can emit an unbounded number of metrics as counts grow.

Additional Locations (1)

runtime_metrics.go#L148-L155

^{Reviewed by Cursor Bugbot for commit e35aa47. Configure here.}

sentry · 2026-09-19T00:07:35Z

+						for j := uint64(0); j < count; j++ {
+							meter.Distribution(runtimeMetricsKeys[i].Key, value, WithUnit(runtimeMetricsKeys[i].Unit))
+						}


Bug: The GC histogram collection loop re-emits all historical data points on every tick, causing unbounded memory and CPU usage over time as the cumulative count grows.
_{Severity: CRITICAL}

Suggested Fix

The implementation should track the delta between metric readings. Store the previous cumulative counts for each histogram bucket. On each tick, calculate the difference between the new cumulative count and the stored previous count. Iterate only new_count - old_count times to emit only the new events that have occurred since the last collection. Update the stored counts after processing.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: runtime_metrics.go#L242-L244 Potential issue: The loop processing GC histogram metrics at `runtime_metrics.go:242` iterates `count` times, where `count` is a cumulative total of GC events since program start. Because this count is never reset, the loop re-emits the entire history of GC pauses on every collection tick. For a long-running service, this leads to unbounded growth in loop iterations, memory allocations for `attrs` maps, and CPU usage, eventually causing the metrics goroutine to hang and consume excessive memory.

sentry · 2026-09-21T10:35:21Z

+	now func() time.Time
+}
+
+var maxProcs = float64(runtime.GOMAXPROCS(-1))


Bug: The maxProcs variable is cached at startup and not updated. If GOMAXPROCS changes at runtime, CPU utilization metrics will become incorrect.
_{Severity: HIGH}

Suggested Fix

Instead of caching the value in a package-level variable, call runtime.GOMAXPROCS(-1) inside the GetCPUUtilization function on each invocation. This ensures the calculation always uses the current number of available processors.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: runtime_metrics.go#L98 Potential issue: The `maxProcs` variable is initialized once at package load time on line 98 and is never updated. However, in containerized environments, the Go runtime (version 1.25+) can automatically adjust the `GOMAXPROCS` value at runtime if CPU limits change. When this happens, the CPU utilization calculation will use the stale `maxProcs` value, leading to significantly under- or over-reported metrics. This makes monitoring and capacity planning based on this metric unreliable in modern deployment environments.

Also affects:

runtime_metrics.go:119~122

giortzisg · 2026-09-21T11:16:56Z

+	// Interval is the interval at which the runtime metrics are collected.
+	// Default is 30 seconds. You don't want to set this too low, as it will
+	// trigger a lot of "stop-the-world" activity.
+	Interval time.Duration


This is misleading since the runtime/metrics library doesn't actually stop the world. Also the only thing this changes is collection frequency, and I don't think the the overhead is that high in general. Let's still keep the default at 30s, but I don't think that setting it to even 5s would be a problem.

giortzisg · 2026-09-21T11:23:05Z

+// To ensure that the runtime metrics integration is only started once.
+// From the Go docs:
+//
+// > It is safe to execute multiple Read calls concurrently, but their arguments
+// > must share no underlying memory. When in doubt, create a new []Sample from
+// > scratch, which is always safe, though may be inefficient.
+var onceRuntimeMetrics = sync.Once{}
+
+// A simple marker to guarantee that the runtime metrics integration is only
+// started once.
+var runtimeMetricsRunning = false


I agree that we should have only a single collector running at once, but an atomic.Bool is better here to allow restarts.

giortzisg · 2026-09-21T11:24:10Z

+	// CollectGCMetrics enables the collection of GC metrics.
+	// Default is false (disabled).
+	CollectGCMetrics bool


I would name this something along the lines of CollectOptionalMetrics. We might add more than gc metrics in the future.

giortzisg · 2026-09-21T11:31:34Z

+	// Handle opt-in metrics
+	if config.CollectGCMetrics {
+		runtimeMetricsKeys = append(
+			runtimeMetricsKeys,
+			runtimeMetricKeyMap{"go.memory.gc.cycles", "cycles", MetricTypeCounter},
+			runtimeMetricKeyMap{"go.memory.gc.pause", UnitSecond, MetricTypeDistribution},
+		)
+
+		runtimeMetricsSamples = append(
+			runtimeMetricsSamples,
+			runtime_metrics.Sample{Name: "/gc/cycles/total:gc-cycles"},
+			runtime_metrics.Sample{Name: "/sched/pauses/total/gc:seconds"},
+		)
+	}


If we allow collector restarts that I recommend above, this becomes harder to manage and seem kinda hacky. I'd prefer we explicitly handle both, or at least clear the memory first rather than append.

giortzisg · 2026-09-21T11:31:49Z

+
+	if config.Context == nil {
+		config.Context = SetHubOnContext(context.Background(), hub)
+	}


We need to set the hub for both contexts

giortzisg · 2026-09-21T11:50:40Z

+	{"go.memory.used.total", UnitByte, MetricTypeGauge},
+	{"go.memory.used.heap.objects", UnitByte, MetricTypeGauge},
+	{"go.memory.used.heap.free", UnitByte, MetricTypeGauge},
+	{"go.memory.used.heap.unused", UnitByte, MetricTypeGauge},
+	{"go.memory.used.heap.stacks", UnitByte, MetricTypeGauge},
+	{"go.memory.used.other", UnitByte, MetricTypeGauge},


These should be go.memory.used and have attributes that define the detailed_type.

giortzisg · 2026-09-21T11:54:10Z

+var maxProcs = float64(runtime.GOMAXPROCS(-1))
+
+func (t *cpuUtilizationTracker) GetCPUUtilization(currentCPUSeconds float64) float64 {
+	now := t.now
+	if now == nil {
+		now = time.Now
+	}
+	nowTime := now()
+
+	// First sample — can't calculate yet
+	if t.lastSampleTime.IsZero() {
+		t.lastCPUSeconds = currentCPUSeconds
+		t.lastSampleTime = nowTime
+		return 0.0
+	}
+
+	// Calculate deltas
+	cpuDelta := currentCPUSeconds - t.lastCPUSeconds
+	wallClockDelta := nowTime.Sub(t.lastSampleTime).Seconds()
+
+	// Normalize by GOMAXPROCS to get utilization percentage
+	// cpuDelta represents CPU-seconds consumed across all GOMAXPROCS goroutines
+	// wallClockDelta is real time that passed
+	// Divide by GOMAXPROCS to account for parallel CPUs
+	utilization := cpuDelta / (wallClockDelta * maxProcs)
+
+	// Clamp to [0.0, 1.0] (shouldn't exceed unless there's jitter)
+	if utilization > 1.0 {
+		utilization = 1.0
+	}
+	if utilization < 0.0 {
+		utilization = 0.0
+	}
+
+	// Update state for next sample
+	t.lastCPUSeconds = currentCPUSeconds
+	t.lastSampleTime = nowTime
+
+	return utilization
+}


We can just get this from /cpu/classes/total:cpu-seconds and /cpu/classes/idle:cpu-seconds

giortzisg · 2026-09-21T12:00:28Z

+					for j := 0; j < len(hist.Counts); j++ {
+						count := hist.Counts[j]


Both bugbot comments are correct here, and I don't think it's that trivial to properly support the histograms here.

giortzisg · 2026-09-21T12:03:05Z

+					case MetricTypeCounter:
+						meter.Count(runtimeMetricsKeys[i].Key, int64(value), WithUnit(runtimeMetricsKeys[i].Unit))


Counters need deltas.

aldy505 added 2 commits August 29, 2026 18:08

feat: runtime metrics integration

9441ae9

fix: interval should never be negative

5eafdcc

sentry Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread runtime_metrics.go Outdated

Comment thread runtime_metrics.go Outdated

cursor Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread runtime_metrics.go Outdated

Comment thread runtime_metrics.go Outdated

fix(runtime-metrics): correct calculation for cpu utilization

0aeba9f

sentry Bot reviewed Aug 30, 2026

View reviewed changes

Comment thread runtime_metrics.go

cursor Bot reviewed Aug 30, 2026

View reviewed changes

dingsdax mentioned this pull request Sep 3, 2026

Add Go runtime metrics integration #1414

Open

aldy505 added 2 commits September 18, 2026 21:16

Merge branch 'master' into feat/runtime-metrics

23adb21

feat: use agreed conventoinal naming

e35aa47

cursor Bot reviewed Sep 19, 2026

View reviewed changes

sentry Bot reviewed Sep 19, 2026

View reviewed changes

aldy505 added 2 commits September 19, 2026 08:11

test: runtime metrics

b84468c

test: don't use synctest

d040d38

sentry Bot reviewed Sep 21, 2026

View reviewed changes

giortzisg reviewed Sep 21, 2026

View reviewed changes

		for j := 0; j < len(hist.Counts); j++ {
		count := hist.Counts[j]

		case MetricTypeCounter:
		meter.Count(runtimeMetricsKeys[i].Key, int64(value), WithUnit(runtimeMetricsKeys[i].Unit))

Conversation

aldy505 commented Aug 29, 2026

Description

Issues

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 30, 2026

Choose a reason for hiding this comment

Broken single-start race

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 19, 2026

Choose a reason for hiding this comment

Cumulative counters sent as increments

Uh oh!

cursor Bot Sep 19, 2026

Choose a reason for hiding this comment

Histogram re-emits all historical pauses

Uh oh!

sentry Bot Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sentry Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cursor Bot left a comment •

edited

Loading