feat(otel): unit metadata + decouple OTLP event mirroring from the webhook - #335
Conversation
…e webhook
Three polish items on the OTLP mirror (valkey#4078), so an OTLP-only deployment
(collector configured, no webhook) gets complete, well-typed telemetry:
1. Unit metadata. planInstruments now derives a UCUM unit from the prom-client
name suffix (_bytes -> By, _seconds -> s, _milliseconds -> ms, _ratio -> 1,
_percent -> %; _total and unknowns carry none) and the exporter passes it to
createObservable{Gauge,Counter}. The mirrored name is never rewritten — the
unit lives in OTLP metadata, so downstream backends can convert/label.
2. Decouple anomaly OTLP dispatch from the Pro webhook gate. addAnomaly no longer
wraps the OTLP emit in webhookEventsProService.isEnabled(); OTLP is its own
opt-in channel, gated only by OTEL_* inside the dispatcher (which no-ops when
disabled). A licensed operator can now ship anomalies to their collector
without also configuring a webhook.
3. Widen event coverage. cluster.failover (PrometheusService) and the
instance.down / instance.up availability edges (HealthService) now mirror to
OTLP on the state transition itself, independent of webhook config. The
availability edges previously only advanced state when a webhook dispatcher
was wired; the transition + OTLP emit now happen regardless, with the webhook
dispatch kept as a separate best-effort step.
Tests: deriveUnit suffix table + unit-only-when-derivable on the exporter; the
anomaly mirror fires with the Pro webhook disabled. All prom-otel-bridge,
otel-metrics-exporter and anomaly.service suites green; api tsc clean.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8cde829. Configure here.
…hook Bugbot (Medium): unlike instance.down/up, the cluster.failover edge detection, state advance, and OTLP emit all stayed inside `if (this.webhookEventsProService)` — only the inner isEnabled() check was dropped. So an OTLP-only deployment without the Pro webhook service still never mirrored failover, and the tracked previousClusterState/previousSlotsFail never advanced there. Hoist the edge detection, the OTLP dispatch, and the state advance out of the webhook-service gate; only the dispatchClusterFailover webhook call remains gated on the Pro service being present. This matches the availability-edge handling and delivers the PR's stated goal of webhook-independent OTLP mirroring.
jamby77
left a comment
There was a problem hiding this comment.
Approving — logic traced clean. The health/cluster state-machine restructure preserves edge semantics (one event per transition, no double-fires), OTLP failures can't suppress webhooks (separate try/catch, and OtelEventDispatcherService.dispatch is synchronous so the catch is real), and the @Optional() injection resolves since OtelEventsModule is @Global(). Two follow-ups worth doing:
-
Test gap on the riskiest part. The HealthService
instance.down/instance.upand PrometheusServicecluster.failovertransitions now advance without webhook wiring, and neither path has coverage —health.service.spec.tsnever toucheshandleInstanceDown/handleInstanceUp, and there's no PrometheusService unit spec at all. A future refactor that reintroduces the webhook guard would silently kill OTLP-only availability events again (exactly the regression class this PR fixes) with nothing going red. -
deriveUnit()misses_seconds_total. The suffix match bails on_totalbefore deriving, sobetterdb_cpu_sys_seconds_total/betterdb_cpu_user_seconds_totalget no unit even though they measure seconds. The OTel Prometheus convention derives from the token preceding_total— checkingendsWith('_seconds_total')before the_totalbail-out (or stripping_totalfirst) fixes it. Same family:_us(e.g.betterdb_commandstats_latency_us) is unmapped, though I see the PR scopes to unambiguous suffixes deliberately.
The bridge/exporter half is well-tested (suffix table, unknown/_total cases, exporter omitting the key when underivable, and the anomaly mirror firing with isEnabled() → false).
…lish # Conflicts: # apps/api/src/health/health.service.ts

What
Three OTLP-mirror polish items (valkey#4078) so an OTLP-only deployment — collector configured, no webhook — gets complete, well-typed telemetry rather than a subset gated behind the Pro webhook.
Changes
1. Unit metadata on mirrored metrics.
planInstrumentsnow derives a UCUMunitfrom the prom-client name suffix, and the exporter passes it tocreateObservable{Gauge,Counter}:_bytesBy_secondss_millisecondsms_ratio1_percent%_total/ unknownThe mirrored metric name is never rewritten — the unit lives in OTLP metadata, so downstream backends can convert/label.
unitis omitted from the instrument options entirely when nothing is derivable (an empty-string unit is meaningful noise).2. Decouple anomaly OTLP dispatch from the Pro webhook gate.
addAnomalyno longer wraps the OTLP emit inwebhookEventsProService.isEnabled(). OTLP is its own opt-in channel, gated only by theOTEL_*env vars inside the dispatcher (which no-ops when disabled). A licensed operator can now ship anomalies to their collector without also configuring a webhook.3. Widen event coverage.
cluster.failover(PrometheusService) and theinstance.down/instance.upavailability edges (HealthService) now mirror to OTLP on the state transition itself, independent of webhook config. The availability edges previously only advanced their state when a webhook dispatcher was wired; the transition + OTLP emit now happen regardless, with the webhook dispatch kept as a separate best-effort step.Test plan
deriveUnitsuffix table + "unit only when derivable" on the exporter (fake meter captures instrument options).isEnabled → false), via the data-loss Rule A path.prom-otel-bridge,otel-metrics-exporter(15 passed) andanomaly.service(128 passed) suites green;apps/apitsc clean.Addresses valkey#4078.
Note
Medium Risk
Changes when operational events are emitted and can increase OTLP traffic for deployments that previously relied on webhook gates; behavior is still bounded by OTEL env config and existing transition logic.
Overview
Improves OTLP-only deployments so telemetry is complete and better typed without depending on Pro webhook configuration.
Mirrored metrics now attach UCUM unit metadata derived from Prometheus name suffixes (
_bytes→By,_seconds→s, etc.); metric names are unchanged and empty units are omitted from OTel instrument options.OTLP event mirroring is decoupled from the Pro webhook gate: anomaly detection always calls
otelEvents.dispatch(still no-ops unlessOTEL_*is set). Instance up/down inHealthServiceand cluster.failover inPrometheusServiceemit on state transitions regardless of webhook wiring; webhooks remain a separate optional step. Cluster failover state is advanced even when only OTLP fires.Reviewed by Cursor Bugbot for commit 60ca87e. Bugbot is set up for automated code reviews on this repo. Configure here.