Introduce config parameter limit label expansion

[kenhys]

In the previous versions, there is no mechanism to limit label expansion. That causes a possibility of cardinality OOM DoS.

To mitigate such situation, introduced the following parameters:

• max_label_value_length: The maximum length of a label value.
• max_series_per_metric: The maximum number of label sets a metric can hold.

The above parameter is configurable for filter_prometheus and out_prometheus.

For example, if about 8 million records are loaded without cardinality
limitation, RSS increased from 64MB to 582MB. It might cause OOM DoS.
In contrast to that case with cardinality limitation, RSS increased
from 64MB to 84 MB in similar case.

Additionally the following counter is introduced too.

• fluentd_prometheus_dropped_label_sets_total

[Watson1978]

Seems that once a record has raised an exception, valid records sent afterwards no longer show up in the metric.

Please try attached file to reproduce.
repro-261.tar.gz

[Copilot]

Pull request overview

Introduces configurable safeguards to bound Prometheus label cardinality and label value growth in filter_prometheus and out_prometheus, mitigating cardinality-driven memory exhaustion risks.

Changes:

• Add max_label_value_length (truncate label values) and max_series_per_metric (drop new label sets beyond a cap) with per-<metric> overrides.
• Add shared LogThrottle and use it to throttle repeated “dropped label set” warnings (and refactor in_prometheus error throttling to use it).
• Add/extend specs and documentation for the new limiting behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
spec/fluent/plugin/prometheus/log_throttle_spec.rb	Adds unit tests for the new shared log throttling utility.
spec/fluent/plugin/filter_prometheus_spec.rb	Adds coverage for max_series_per_metric behavior and throttled warning logging in the filter plugin.
README.md	Documents new label expansion limiting parameters and behavior details.
lib/fluent/plugin/prometheus.rb	Implements label truncation, series limiting, label-set limit warnings, and shared LogThrottle.
lib/fluent/plugin/out_prometheus.rb	Passes plugin-level metric limit options into metric construction.
lib/fluent/plugin/in_prometheus.rb	Replaces bespoke throttling with shared LogThrottle.
lib/fluent/plugin/filter_prometheus.rb	Passes plugin-level metric limit options into metric construction.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Watson1978]

Both limits are on by default, so upgrading changes the exported metrics of every existing user — and the change is silent

max_label_value_length defaults to 256 and max_series_per_metric defaults to 10000, so a user who upgrades the gem without touching their configuration gets both. Two different kinds of damage follow, and neither of them is visible.

Truncation merges label sets that used to be distinct

Verified with the default configuration (no limit set anywhere):

<filter test.**>
  @type prometheus
  <metric>
    name test_truncated
    type counter
    desc test
    key val
    <labels>
      path $.path
    </labels>
  </metric>
</filter>

Feeding two records whose path differs only after the 256th character produces one series instead of two, and their values are summed:

series = 1
path length = 256
value = 2.0
log lines = 0

Label values longer than 256 characters are not exotic — URLs with query strings, Kubernetes annotations, SQL statements and exception messages all reach that length routinely. For those users, upgrading makes existing series disappear and a new merged series appear in their place. Prometheus sees the old series go stale, so recording rules, dashboards and alerts built on them break, and the counter values are wrong rather than merely missing.

The cap drops records once a metric is saturated

A deployment that legitimately runs above 10000 label sets today starts losing everything past the 10001st after the upgrade, with no configuration change on their side.

Suggestion

Please consider defaulting both to 0 (unlimited) and letting operators opt in. The feature is valuable, but it changes the meaning of data that already exists, and that is the kind of change that should be a deliberate act rather than a side effect of bundle update. If the defaults stay on, this needs to be called out in the ChangeLog explicitly as a breaking change, and ideally the release should be a minor/major bump rather than a patch.

---

A dropped label set leaves almost no trace

This is what makes the previous point serious: when a record is dropped, there is essentially no way for an operator to find out.

• rescue LabelSetLimitError in instrument_single and instrument does not call router.emit_error_event, so @ERROR never sees the record. That is a reasonable choice on its own — dropping is intended here, not an error — but it means the record is gone with no route to inspect it.
• warn_label_set_limit is throttled by ignore_error_log_interval, which defaults to 3600. One log.warn per metric per hour, no matter how many records are discarded in between.
• suppressed_log_count is only reported when the next warning fires. If fluentd is restarted or the configuration is reloaded before the interval elapses, the accumulated count is never emitted at all, so the operator sees a single warning and has no way to learn how much was discarded.
• Truncation is not logged at all, at any rate.
• Nothing changes on the Prometheus side. The scrape target stays up == 1, the endpoint returns 200, and the input plugin keeps accepting records normally. The metric simply has fewer series than it should, which no alert can express unless absent() was already written in advance for that exact series.

In an earlier reproduction of a related problem, 161 dropped records produced exactly one log line while every POST returned HTTP 200. Data loss at that ratio of signal is very hard to attribute after the fact.

Suggestion

Self-instrument the drops so they are visible in Prometheus itself rather than only in logs — for example a counter such as fluentd_prometheus_dropped_label_sets_total{name="<metric name>"}, and a companion counter for truncated label values. That gives operators something they can alert on and graph, which is exactly what this plugin exists to provide for everything else.

If both limits are going to stay on by default, this feels like a prerequisite rather than a nice-to-have: the defaults are what make the loss possible, and this is what would make it noticeable.

---

This comment was written by Claude (Claude Code). The behaviour described above was verified by running the plugin at 39a9ae3.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lib/fluent/plugin/prometheus.rb:485

• release_series deletes the entry unconditionally, which can under-count series when another thread successfully instrumented the same label set while this (reserved) call failed. That can allow subsequent new label sets to pass the @series.size check and exceed max_series_per_metric.

If you adopt a reserved/confirmed state (e.g. :reserved vs :confirmed), only delete when the label set is still reserved, and update the comment above this method (it currently describes the unsafe behavior as intended).

        def release_series(label)
          @series_mutex.synchronize do
            @series.delete(label)
          end
        end

lib/fluent/plugin/prometheus.rb:408

• A failing instrumentation can free a series slot that another concurrent instrumentation of the same new label set has effectively “confirmed” by succeeding. This happens because the label set is reserved before yielding, but nothing marks it as confirmed on success, so release_series cannot distinguish an in-flight reservation from an established series.

Confirm the label set after a successful yield, so release_series can safely decide whether deletion is still allowed.

          reserved = reserve_series!(label)
          begin
            yield label
          rescue
            release_series(label) if reserved

lib/fluent/plugin/prometheus.rb:473

• reserve_series! records a newly taken slot as true. To make release_series safe under concurrent instrumentation of the same label set (one success + one failure), store a distinct reserved state (e.g. :reserved) and have successful instrumentations mark it confirmed.

            @series[label] = true
            next true
          end

[Watson1978]

This explanation may not be sufficient.

Setting max_series_per_metric 2 doesn't actually cap the metric at 2 series
when two <metric> blocks (or two plugin instances) share the same name:

<filter test.**>
  @type prometheus
  max_series_per_metric 2
  <metric>
    name dup_total
    type counter
    key val
    <labels>
      path $.path
    </labels>
  </metric>
  <metric>
    name dup_total
    type counter
    key val
    <labels>
      path $.path
    </labels>
  </metric>
</filter>

Feed 4 records with distinct path values (2 through each <metric> block) and
dup_total ends up with 4 series, not 2 — each <metric> block gets its own
Metric Ruby object with its own @series Hash, but both wrap the same
underlying ::Prometheus::Client::Counter (the registry dedupes by name
alone, prometheus.rb:462-474), so neither @series knows about the other.

require 'spec_helper'

describe 'max_series_per_metric with a metric name shared by two Metric instances' do
  let(:registry) { ::Prometheus::Client::Registry.new }
  let(:max_series_per_metric) { 2 }

  def build_metric
    element = Fluent::Config::Element.new(
      'metric', '',
      {
        'name' => 'dup_total',
        'type' => 'counter',
        'desc' => 'Something foo.',
        'key' => 'val',
        'max_series_per_metric' => max_series_per_metric.to_s,
      },
      [Fluent::Config::Element.new('labels', '', { 'path' => '$.path' }, [])]
    )
    Fluent::Plugin::Prometheus::Counter.new(element, registry, {}, {})
  end

  let(:expander) { double('expander') }

  def instrument(metric, path, value = 1)
    metric.instrument({ 'val' => value, 'path' => path }, expander)
  end

  it 'lets the shared client-side counter exceed max_series_per_metric' do
    metric_a = build_metric
    metric_b = build_metric

    expect(metric_a.instance_variable_get(:@counter))
      .to equal(metric_b.instance_variable_get(:@counter))

    # each wrapper independently accepts up to its own max_series_per_metric
    instrument(metric_a, '/a1')
    instrument(metric_a, '/a2')
    expect { instrument(metric_a, '/a3') }
      .to raise_error(Fluent::Plugin::Prometheus::LabelSetLimitError)

    instrument(metric_b, '/b1')
    instrument(metric_b, '/b2')
    expect { instrument(metric_b, '/b3') }
      .to raise_error(Fluent::Plugin::Prometheus::LabelSetLimitError)

    # ... yet the metric actually exposed to Prometheus holds FOUR series,
    # twice the configured max_series_per_metric of 2
    client_counter = registry.get(:dup_total)

    ##########################################################################
    # Configured `max_series_per_metric: 2`,
    # however 4 metrics existed.
    p client_counter.values.keys.size
    # => 4
    p client_counter.values.keys
    # => [{path: "/a1"}, {path: "/a2"}, {path: "/b1"}, {path: "/b2"}]
  end
end