Skip to content

feat(otel): unit metadata + decouple OTLP event mirroring from the webhook - #335

Merged
KIvanow merged 3 commits into
masterfrom
feat/otlp-export-polish
Jul 29, 2026
Merged

feat(otel): unit metadata + decouple OTLP event mirroring from the webhook#335
KIvanow merged 3 commits into
masterfrom
feat/otlp-export-polish

Conversation

@KIvanow

@KIvanow KIvanow commented Jul 23, 2026

Copy link
Copy Markdown
Member

🔗 Upstream Valkey issue: valkey-io/valkey#4078 — OpenTelemetry (OTLP) metrics export

What

Three OTLP-mirror polish items (valkey#4078) so an OTLP-only deployment — collector configured, no webhook — gets complete, well-typed telemetry rather than a subset gated behind the Pro webhook.

Changes

1. Unit metadata on mirrored metrics. planInstruments now derives a UCUM unit from the prom-client name suffix, and the exporter passes it to createObservable{Gauge,Counter}:

suffix unit
_bytes By
_seconds s
_milliseconds ms
_ratio 1
_percent %
_total / unknown (none)

The mirrored metric name is never rewritten — the unit lives in OTLP metadata, so downstream backends can convert/label. unit is omitted from the instrument options entirely when nothing is derivable (an empty-string unit is meaningful noise).

2. Decouple anomaly OTLP dispatch from the Pro webhook gate. addAnomaly no longer wraps the OTLP emit in webhookEventsProService.isEnabled(). OTLP is its own opt-in channel, gated only by the OTEL_* env vars inside the dispatcher (which no-ops when disabled). A licensed operator can now ship anomalies to their collector without also configuring a webhook.

3. Widen event coverage. cluster.failover (PrometheusService) and the instance.down / instance.up availability edges (HealthService) now mirror to OTLP on the state transition itself, independent of webhook config. The availability edges previously only advanced their state when a webhook dispatcher was wired; the transition + OTLP emit now happen regardless, with the webhook dispatch kept as a separate best-effort step.

Test plan

  • deriveUnit suffix table + "unit only when derivable" on the exporter (fake meter captures instrument options).
  • Anomaly mirror fires with the Pro webhook disabled (isEnabled → false), via the data-loss Rule A path.
  • prom-otel-bridge, otel-metrics-exporter (15 passed) and anomaly.service (128 passed) suites green; apps/api tsc clean.

Addresses valkey#4078.


Note

Medium Risk
Changes when operational events are emitted and can increase OTLP traffic for deployments that previously relied on webhook gates; behavior is still bounded by OTEL env config and existing transition logic.

Overview
Improves OTLP-only deployments so telemetry is complete and better typed without depending on Pro webhook configuration.

Mirrored metrics now attach UCUM unit metadata derived from Prometheus name suffixes (_bytesBy, _secondss, etc.); metric names are unchanged and empty units are omitted from OTel instrument options.

OTLP event mirroring is decoupled from the Pro webhook gate: anomaly detection always calls otelEvents.dispatch (still no-ops unless OTEL_* is set). Instance up/down in HealthService and cluster.failover in PrometheusService emit on state transitions regardless of webhook wiring; webhooks remain a separate optional step. Cluster failover state is advanced even when only OTLP fires.

Reviewed by Cursor Bugbot for commit 60ca87e. Bugbot is set up for automated code reviews on this repo. Configure here.

…e webhook

Three polish items on the OTLP mirror (valkey#4078), so an OTLP-only deployment
(collector configured, no webhook) gets complete, well-typed telemetry:

1. Unit metadata. planInstruments now derives a UCUM unit from the prom-client
   name suffix (_bytes -> By, _seconds -> s, _milliseconds -> ms, _ratio -> 1,
   _percent -> %; _total and unknowns carry none) and the exporter passes it to
   createObservable{Gauge,Counter}. The mirrored name is never rewritten — the
   unit lives in OTLP metadata, so downstream backends can convert/label.

2. Decouple anomaly OTLP dispatch from the Pro webhook gate. addAnomaly no longer
   wraps the OTLP emit in webhookEventsProService.isEnabled(); OTLP is its own
   opt-in channel, gated only by OTEL_* inside the dispatcher (which no-ops when
   disabled). A licensed operator can now ship anomalies to their collector
   without also configuring a webhook.

3. Widen event coverage. cluster.failover (PrometheusService) and the
   instance.down / instance.up availability edges (HealthService) now mirror to
   OTLP on the state transition itself, independent of webhook config. The
   availability edges previously only advanced state when a webhook dispatcher
   was wired; the transition + OTLP emit now happen regardless, with the webhook
   dispatch kept as a separate best-effort step.

Tests: deriveUnit suffix table + unit-only-when-derivable on the exporter; the
anomaly mirror fires with the Pro webhook disabled. All prom-otel-bridge,
otel-metrics-exporter and anomaly.service suites green; api tsc clean.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8cde829. Configure here.

Comment thread apps/api/src/prometheus/prometheus.service.ts Outdated
…hook

Bugbot (Medium): unlike instance.down/up, the cluster.failover edge detection,
state advance, and OTLP emit all stayed inside `if (this.webhookEventsProService)`
— only the inner isEnabled() check was dropped. So an OTLP-only deployment
without the Pro webhook service still never mirrored failover, and the tracked
previousClusterState/previousSlotsFail never advanced there.

Hoist the edge detection, the OTLP dispatch, and the state advance out of the
webhook-service gate; only the dispatchClusterFailover webhook call remains gated
on the Pro service being present. This matches the availability-edge handling and
delivers the PR's stated goal of webhook-independent OTLP mirroring.
@KIvanow
KIvanow requested a review from jamby77 July 24, 2026 07:16

@jamby77 jamby77 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — logic traced clean. The health/cluster state-machine restructure preserves edge semantics (one event per transition, no double-fires), OTLP failures can't suppress webhooks (separate try/catch, and OtelEventDispatcherService.dispatch is synchronous so the catch is real), and the @Optional() injection resolves since OtelEventsModule is @Global(). Two follow-ups worth doing:

  1. Test gap on the riskiest part. The HealthService instance.down/instance.up and PrometheusService cluster.failover transitions now advance without webhook wiring, and neither path has coverage — health.service.spec.ts never touches handleInstanceDown/handleInstanceUp, and there's no PrometheusService unit spec at all. A future refactor that reintroduces the webhook guard would silently kill OTLP-only availability events again (exactly the regression class this PR fixes) with nothing going red.

  2. deriveUnit() misses _seconds_total. The suffix match bails on _total before deriving, so betterdb_cpu_sys_seconds_total / betterdb_cpu_user_seconds_total get no unit even though they measure seconds. The OTel Prometheus convention derives from the token preceding _total — checking endsWith('_seconds_total') before the _total bail-out (or stripping _total first) fixes it. Same family: _us (e.g. betterdb_commandstats_latency_us) is unmapped, though I see the PR scopes to unambiguous suffixes deliberately.

The bridge/exporter half is well-tested (suffix table, unknown/_total cases, exporter omitting the key when underivable, and the anomaly mirror firing with isEnabled() → false).

…lish

# Conflicts:
#	apps/api/src/health/health.service.ts
@KIvanow
KIvanow merged commit 0275cfb into master Jul 29, 2026
3 checks passed
@KIvanow
KIvanow deleted the feat/otlp-export-polish branch July 29, 2026 08:45
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants