feat: sanitize "." in descriptor keys/values before publishing metrics - #1210
Open
yusofg2 wants to merge 1 commit into
Open
feat: sanitize "." in descriptor keys/values before publishing metrics#1210yusofg2 wants to merge 1 commit into
yusofg2 wants to merge 1 commit into
Conversation
Add an opt-in SANITIZE_DESCRIPTOR_METRIC_DOTS setting (default false, so behavior is unchanged for existing deployments). When enabled, "." in a descriptor key or value is replaced with "_" as the metric name is built, so dotted values (hostnames, gRPC paths, etc.) don't inject extra statsd hierarchy segments that break the Prometheus statsd_exporter metric-name -> label mapping. Sanitization is applied only to the fragments written into metric names in the config key-builders (descriptorKey, detailed_metric, value_to_metric, and share_threshold paths); the descriptor-map lookup key and the rate-limit cache key are left verbatim, so rate-limit matching is unaffected. This is distinct from the pre-existing SanitizeStatName, which only strips dots inside IPv4 addresses and runs unconditionally. The flag is threaded through the stats Manager (which already receives Settings), avoiding churn in the config loader chain and its ~40 call sites. Tests: config-layer unit tests for flag on/off across the value, non-IPv4 value, key, detailed_metric, value_to_metric, and matching-still-works cases; a SanitizeStatKeyValue helper unit test; a manager accessor test; and an end-to-end integration test that drives a dotted value through a real running server and asserts the sanitized metric name for both the value_to_metric and detailed_metric builders. Signed-off-by: Yusof Ganji <yganji@salesforce.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In statsd/gostats,
.is the metric hierarchy separator. Rate limit builds a metric name by joining the domain and descriptor entries with dots (e.g.domain.source_ip_<value>.total_hits). When a descriptor value or key contains a.(a hostname likefoo.bar, a gRPC path like/helloworld.Greeter/SayHello, etc.), it injects unintended extra dot-segments into the metric name.This primarily breaks the Prometheus sink: names are piped through the statsd_exporter mapper (
examples/prom-statsd-exporter/conf.yaml), whose match rules use dots as the field separator (ratelimit.service.rate_limit.*.*.total_hits). An extra dot makes a different rule match, so the value gets split across the wrong labels (or the series is dropped entirely).The only existing sanitization,
utils.SanitizeStatName, replaces:→_,|→_, and dots only inside IPv4 addresses — a bare non-IPv4 dot in a value slips through.What this changes
Adds an opt-in setting
SANITIZE_DESCRIPTOR_METRIC_DOTS(defaultfalse, so existing behavior is unchanged). When enabled,.in a descriptor key or value is replaced with_as the metric name is built.Key correctness point: sanitization is applied only to the fragments written into metric names in the config key-builders (
descriptorKey, and thedetailed_metric,value_to_metric, andshare_thresholdpaths). The descriptor-map lookup key and the rate-limit cache key are left verbatim, so rate-limit matching is unaffected — this changes metric names only, not limiting behavior.The flag is threaded through the stats
Manager(which already receivesSettings), avoiding churn in the config loader chain and its call sites.Before / after
With a descriptor value
foo.barsent through a real server + the real statsd_exporter mapper conf:Flag off (today) — the dot creates an extra segment, so the value is split across two labels:
Flag on — the value stays intact in a single label:
Testing
foo.bar), dotted key,detailed_metric,value_to_metric, and a "matching still works with a dotted value" case (guards the lookup-vs-metric split).SanitizeStatKeyValuehelper unit test and a manager accessor test.value_to_metricanddetailed_metricbuilders.Docs
README.mddocumentsSANITIZE_DESCRIPTOR_METRIC_DOTSnear the other stats env vars.