Skip to content

feat: route normalized OTLP signals to project destinations - #5939

Merged
tgmendes merged 7 commits into
mainfrom
feat/normalized-otel-data-export-routing
Sep 2, 2026
Merged

feat: route normalized OTLP signals to project destinations#5939
tgmendes merged 7 commits into
mainfrom
feat/normalized-otel-data-export-routing

Conversation

@tgmendes

@tgmendes tgmendes commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Builds on the data-export management API merged in #5792.

Summary

  • Add an active-route lookup to the dataexports repository and reuse its project-scoped destination model.
  • Relay normalized OTLP logs, metrics, and traces through the selected destination, including encrypted headers and destination-level sensitive-data policy.
  • Treat enabled product-telemetry routes as the authoritative replacement for the removed global PostHog forwarding kill switch.
  • Count normalized records with missing or malformed project IDs as invalid and drop them before route lookup or export.

Sensitive data policy

This is, for now, a somewhat naive policy, but I think will suffice. Over time we might need to revisit how we do proper redaction.

  • include preserves the normalized OTLP payload.
  • exclude replaces classified content and identity values with [REDACTED] across log, metric, and trace containers while preserving attribute keys.
  • Destinations configured to include sensitive data are reloaded for every delivery, so an include-to-exclude policy change cannot remain stale in the destination cache.

Motivation

Normalized telemetry must follow the project-scoped routes managed by the data export API. Resolving destinations at the normalized signal boundary keeps logs, metrics, and traces consistent while preserving project isolation and destination-specific disclosure policy.

NOTE

Once this is merged the old configs will stop working: this is ok. Only one customer has it configured other than us, and it's not working (all datadog logs are showing 403 failures). I have shared this with them so they are aware.

@tgmendes
tgmendes requested review from a team as code owners September 1, 2026 14:51
@tgmendes tgmendes added the enhancement New feature or request label Sep 1, 2026
@changeset-bot

changeset-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 561090f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Running ultrareview automatically — This large change introduces project-scoped OpenTelemetry exports with new authorization, write-only header secret handling, and a sensitive-data telemetry relay—security and data-integrity bugs here would have broad blast radius.. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview completed in 14m 45s

All reported issues were addressed across 119 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread server/internal/otel/handler_span_relay.go Outdated
Comment thread client/dashboard/src/pages/data-exports/DataExports.tsx Outdated
Comment thread server/internal/otel/handler_span_relay.go Outdated
Comment thread client/dashboard/src/pages/data-exports/DataExports.tsx Outdated
Comment thread client/dashboard/src/lib/write-only-headers.ts Outdated
Comment thread client/dashboard/src/pages/data-exports/ConfigureExportSheet.tsx Outdated
Comment thread client/dashboard/src/pages/data-exports/ConfigureExportSheet.tsx Outdated
Comment thread server/internal/otel/handler_metric_relay_test.go Outdated
Comment thread server/internal/dataexports/destinations.go
Comment thread client/dashboard/src/pages/data-exports/DataExports.tsx Outdated
@tgmendes tgmendes changed the title feat: add project-scoped OpenTelemetry data exports feat: route normalized OTLP signals to project destinations Sep 1, 2026
@tgmendes
tgmendes force-pushed the feat/normalized-otel-data-export-routing branch from 64b1f9d to fbfc623 Compare September 1, 2026 15:16

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 21 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread server/internal/otel/signal_relay.go
Comment thread server/internal/otel/signal_relay.go
@tgmendes
tgmendes force-pushed the feat/normalized-otel-data-export-routing branch 2 times, most recently from e4d954c to 5d60dfd Compare September 1, 2026 16:36
@tgmendes
tgmendes changed the base branch from main to feat/otel-data-forwarding-api September 1, 2026 16:36
Base automatically changed from feat/otel-data-forwarding-api to main September 2, 2026 08:33
@tgmendes
tgmendes force-pushed the feat/normalized-otel-data-export-routing branch from 3c32c05 to 1769516 Compare September 2, 2026 09:20
@tgmendes
tgmendes added this pull request to the merge queue Sep 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 2, 2026
@tgmendes
tgmendes added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit fe2346c Sep 2, 2026
47 of 48 checks passed
@tgmendes
tgmendes deleted the feat/normalized-otel-data-export-routing branch September 2, 2026 14:05
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 2, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants