Skip to content

scrape-manager: wire target discovery relabeling, honor_labels, and metric_relabel_configs - #64

Draft
bwplotka wants to merge 2 commits into
bwplotka/arrow-relabelfrom
bwplotka/scrape-manager-relabel
Draft

bwplotka wants to merge 2 commits into
bwplotka/arrow-relabelfrom
bwplotka/scrape-manager-relabel

Conversation

@bwplotka

Copy link
Copy Markdown
Member

Stacked on #63. Fixes #35.

Wires target discovery relabeling (relabel_configs), target resource attributes (TargetAttributes), honor_labels, and metric_relabel_configs into prometheus-scrape-manager.

ScrapeConfig gains metrics_path, scheme, params, honor_labels, relabel_configs, and metric_relabel_configs, plus ScrapeConfig::populate_targets and ScrapeTarget::{from_populated, from_populated_with_params} to run prometheus_relabel::populate_labels over discovered TargetGroups and construct target URLs matching Prometheus Target.URL() query parameter merging.

ScrapeManager::new compiles metric_relabel_configs once into a shared Arc<MetricRelabelPlan>. During Arrow decoding, ScrapeManager attaches honor_labels, the compiled plan, and cached per-target TargetAttributes (job -> service.name, instance -> service.instance.id, plus public target labels). ScrapeTarget itself stays free of interior mutability so it remains usable as a HashMap/HashSet key without clippy::mutable_key_type; TargetSchedule caches TargetAttributes per active slot in run_arrow, while direct scrape_once_arrow / run_once_arrow calls use a bounded cache (<= 1,024 entries).

Commits in reading order:

  1. 0848bbd (scrape-manager: wire target discovery relabeling, honor_labels, and metric_relabel_configs) extends ScrapeConfig and ScrapeTarget, caches TargetAttributes in TargetSchedule and ScrapeManager, and wires honor_labels and MetricRelabelPlan into cooperative Arrow decoding.
  2. 3673bd5 (scrape-manager: add discovery relabeling, honor_labels, and metric_relabel_configs tests and benchmark) adds unit tests for target population and URL/param rules, an end-to-end HTTP integration test, a relabeling ScrapeCycle benchmark, and specs/SPEC.md updates.
Benchmarks (before: 87efe0a vs after: 3673bd5)

Comparing 87efe0a (base PR #63) against 3673bd5. The +48 allocs/op (+0.57%) and +0.21% B/op on the 6 existing cases correspond to 1 Vec<ArrayRef> clone per emitted chunk (16 targets x 3 chunks) now that each ScrapeTarget attaches its cached ResourceAttrs batch.

cargo bench -p prometheus-scrape-manager --bench scrape -- -bench '^BenchmarkScrape' -benchtime 500ms -count 6 -cpu 2 -benchmem
goos: linux
goarch: amd64
pkg: prometheus-scrape-manager
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
                                                                           │ before_87efe0a.txt │         after_3673bd5.txt          │
                                                                           │       sec/op       │    sec/op     vs base              │
ScrapeCycle/targets=16/header_wait=none/mode=sequential_tick-2                     19.66m ±  1%   19.73m ±  9%       ~ (p=0.180 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1-2                 19.79m ±  1%   19.40m ±  1%  -1.97% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b8_d4-2                 19.47m ±  2%   19.33m ±  1%       ~ (p=0.132 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=sequential_tick-2                      55.70m ± 24%   56.47m ±  2%       ~ (p=0.132 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b2_d1-2                  33.50m ± 25%   31.31m ± 28%       ~ (p=0.937 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b8_d4-2                  21.73m ±  1%   21.74m ±  1%       ~ (p=0.589 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1_relabel-2                        21.90m ±  5%
geomean                                                                            25.98m         25.07m        -1.27%

                                                                           │ before_87efe0a.txt │          after_3673bd5.txt          │
                                                                           │        B/s         │      B/s       vs base              │
ScrapeCycle/targets=16/header_wait=none/mode=sequential_tick-2                    147.6Mi ±  1%   147.1Mi ±  8%       ~ (p=0.180 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1-2                146.6Mi ±  1%   149.6Mi ±  1%  +2.01% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b8_d4-2                149.1Mi ±  2%   150.1Mi ±  1%       ~ (p=0.132 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=sequential_tick-2                     52.10Mi ± 20%   51.39Mi ±  2%       ~ (p=0.121 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b2_d1-2                 86.64Mi ± 34%   94.46Mi ± 36%       ~ (p=0.937 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b8_d4-2                 133.6Mi ±  1%   133.5Mi ±  1%       ~ (p=0.589 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1_relabel-2                        132.5Mi ±  5%
geomean                                                                           111.7Mi         116.1Mi        +1.60%

                                                                           │ before_87efe0a.txt │         after_3673bd5.txt          │
                                                                           │        B/op        │     B/op      vs base              │
ScrapeCycle/targets=16/header_wait=none/mode=sequential_tick-2                     1.651Mi ± 0%   1.654Mi ± 0%  +0.18% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1-2                 1.792Mi ± 0%   1.796Mi ± 0%  +0.22% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b8_d4-2                 1.792Mi ± 0%   1.796Mi ± 0%  +0.22% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=sequential_tick-2                      1.651Mi ± 0%   1.654Mi ± 0%  +0.18% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b2_d1-2                  1.792Mi ± 0%   1.796Mi ± 0%  +0.22% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b8_d4-2                  1.792Mi ± 0%   1.796Mi ± 0%  +0.22% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1_relabel-2                        1.533Mi ± 0%
geomean                                                                            1.743Mi        1.715Mi       +0.21%

                                                                           │ before_87efe0a.txt │         after_3673bd5.txt         │
                                                                           │     allocs/op      │  allocs/op   vs base              │
ScrapeCycle/targets=16/header_wait=none/mode=sequential_tick-2                      8.385k ± 0%   8.433k ± 0%  +0.57% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1-2                  8.404k ± 0%   8.452k ± 0%  +0.57% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b8_d4-2                  8.404k ± 0%   8.452k ± 0%  +0.57% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=sequential_tick-2                       8.385k ± 0%   8.433k ± 0%  +0.57% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b2_d1-2                   8.403k ± 0%   8.451k ± 0%  +0.57% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=2ms/mode=concurrent_k8_b8_d4-2                   8.403k ± 0%   8.451k ± 0%  +0.57% (p=0.002 n=6)
ScrapeCycle/targets=16/header_wait=none/mode=concurrent_k8_b2_d1_relabel-2                        7.988k ± 0%
geomean                                                                             8.397k        8.378k       +0.57%

…etric_relabel_configs

Add metrics_path, scheme, params, honor_labels, relabel_configs, and
metric_relabel_configs to ScrapeConfig, along with
ScrapeConfig::populate_targets and
ScrapeTarget::{from_populated, from_populated_with_params} matching
Prometheus Target.URL() parameter merging.

Compile metric_relabel_configs once into Arc<MetricRelabelPlan> inside
ScrapeManager::new and attach honor_labels and cached TargetAttributes
per target during Arrow decoding while keeping ScrapeTarget free of
interior mutability (per-slot OnceLock in TargetSchedule and a bounded
cache for direct scrape_once_arrow calls).

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…label_configs tests and benchmark

Add unit tests for ScrapeConfig::populate_targets, Target.URL()
query parameter merging, IPv6/scheme/address validation, and relabel
config validation.

Add an end-to-end HTTP integration test exercising discovery
relabel_configs, target ResourceAttrs emission, honor_labels = false
and true, exemplar attribute preservation, Stage 1 and Stage 2
metric_relabel_configs, empty post-relabel scrapes, and
MissingMetricName failure reporting.

Add a relabeling scrape benchmark case and update specs/SPEC.md.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant