Skip to content

arrow: wire honor_labels and two-stage metric_relabel_configs into ArrowDecoder - #63

Draft
bwplotka wants to merge 3 commits into
bwplotka/arrow-resource-attrsfrom
bwplotka/arrow-relabel
Draft

bwplotka wants to merge 3 commits into
bwplotka/arrow-resource-attrsfrom
bwplotka/arrow-relabel

Conversation

@bwplotka

Copy link
Copy Markdown
Member

Stacked on #62. Towards #35.

Wires honor_labels collision resolution and two-stage metric_relabel_configs execution into ArrowDecoder. The arrow feature of prometheus-expfmt enables prometheus-relabel and re-exports it as prometheus_expfmt::relabel, while default-features = false keeps the text parser dependency-free.

When honor_labels is false and exposed metric labels collide with target labels, colliding keys are remapped in O(unique_keys) across the batch key dictionary to exported_<key> without touching exemplar attributes. Notice that exported_<key> occupancy is checked batch-wide across the dictionary rather than per series, which matches specs/SPEC.md and avoids per-sample key lookups.

MetricRelabelPlan splits leading __name__-only filter rules into Stage 1 (evaluated at wire series-name transitions before float parsing or interning) and executes remaining rules in Stage 2 over finished chunks (dictionary-vectorized for labeldrop, labelkeep, and single-label keep/drop/replace/lowercase/uppercase/hashmod, with a general scalar fallback for multi-source, labelmap, or __name__-rewriting rules). Chunks whose data points are all dropped are skipped by decode_stream. Dropping __name__ while other labels remain returns DecodeError::MissingMetricName, matching Prometheus errNameLabelMandatory.

Commits in reading order:

  1. 373a832 (relabel: add str-slice and iterator dictionary helpers for arena interners) adds iterator and &str entry points plus retained-memory accounting on prometheus-relabel caches so ArrowDecoder can feed its arena interners without copying keys into Vec<String>.
  2. 8a19730 (arrow: wire honor_labels and two-stage metric_relabel_configs into ArrowDecoder) adds MetricRelabelPlan, DecodeOptions::{with_honor_labels, with_metric_relabel_plan, with_metric_relabel_configs}, DecodeError::MissingMetricName, Stage 1 pushdown, Stage 2 relabeling, and table compaction.
  3. 87efe0a (tests, benches, docs: verify ArrowDecoder honor_labels and two-stage relabeling) adds unit tests (including a prometheus-relabel oracle), OTAP/OTLP round-trip coverage, dhat bounded-memory verification with active relabeling, a Stage 1 + Stage 2 decode benchmark, and specs/SPEC.md updates.
Benchmarks (before: 16ce7f9 vs after: 87efe0a)

Comparing 16ce7f9 (base PR #62) against 87efe0a:

cargo bench -p prometheus-expfmt --bench decode -- -bench '^BenchmarkDecode' -benchtime 1s -count 6 -cpu 2 -benchmem
goos: linux
goarch: amd64
pkg: prometheus-expfmt
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
                                                                        │ before_16ce7f9.txt │         after_87efe0a.txt          │
                                                                        │       sec/op       │    sec/op     vs base              │
Decode/case=alltypes.237mfs.nometa/mode=decode-2                                919.0µ ±  2%   897.1µ ±  2%  -2.38% (p=0.026 n=6)
Decode/case=alltypes.237mfs.nometa/mode=decode_cooperative-2                    933.1µ ±  1%   915.3µ ±  2%  -1.91% (p=0.041 n=6)
Decode/case=alltypes.237mfs.nometa/mode=stream_16k-2                            919.9µ ±  2%   927.0µ ±  5%       ~ (p=0.937 n=6)
Decode/case=alltypes.237mfs.nometa/mode=stream_256-2                            959.5µ ±  1%   940.8µ ±  1%  -1.95% (p=0.004 n=6)
Decode/case=alltypes.237mfs.nometa/mode=stream_256_cooperative-2                976.2µ ±  2%   951.9µ ±  3%  -2.49% (p=0.026 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=decode-2                         330.0µ ±  1%   316.8µ ±  2%  -4.00% (p=0.002 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=decode_cooperative-2             340.0µ ±  1%   330.8µ ±  5%       ~ (p=0.065 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_16k-2                     332.4µ ±  1%   323.7µ ±  3%       ~ (p=0.065 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256-2                     342.5µ ±  1%   334.6µ ±  1%  -2.31% (p=0.002 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256_cooperative-2         353.0µ ± 31%   348.1µ ±  7%       ~ (p=0.394 n=6)
Decode/case=alltypes.237mfs/mode=decode-2                                       1.136m ±  8%   1.104m ±  2%  -2.84% (p=0.026 n=6)
Decode/case=alltypes.237mfs/mode=decode_cooperative-2                           1.138m ±  1%   1.136m ±  2%       ~ (p=1.000 n=6)
Decode/case=alltypes.237mfs/mode=stream_16k-2                                   1.128m ±  1%   1.102m ±  1%  -2.25% (p=0.004 n=6)
Decode/case=alltypes.237mfs/mode=stream_256-2                                   1.171m ±  1%   1.140m ±  1%  -2.66% (p=0.002 n=6)
Decode/case=alltypes.237mfs/mode=stream_256_cooperative-2                       1.186m ±  1%   1.191m ±  3%       ~ (p=0.818 n=6)
Decode/case=alltypes.237mfs/mode=relabel_stage1_stage2-2                                       1.566m ± 36%
geomean                                                                         716.6µ         739.0µ        -1.90%

                                                                        │ before_16ce7f9.txt │          after_87efe0a.txt          │
                                                                        │        B/s         │      B/s       vs base              │
Decode/case=alltypes.237mfs.nometa/mode=decode-2                               153.7Mi ±  2%   157.5Mi ±  2%  +2.44% (p=0.026 n=6)
Decode/case=alltypes.237mfs.nometa/mode=decode_cooperative-2                   151.4Mi ±  1%   154.3Mi ±  2%  +1.95% (p=0.041 n=6)
Decode/case=alltypes.237mfs.nometa/mode=stream_16k-2                           153.6Mi ±  2%   152.4Mi ±  5%       ~ (p=0.937 n=6)
Decode/case=alltypes.237mfs.nometa/mode=stream_256-2                           147.2Mi ±  1%   150.2Mi ±  1%  +1.99% (p=0.004 n=6)
Decode/case=alltypes.237mfs.nometa/mode=stream_256_cooperative-2               144.7Mi ±  2%   148.4Mi ±  3%  +2.56% (p=0.026 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=decode-2                        211.2Mi ±  1%   220.0Mi ±  2%  +4.17% (p=0.002 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=decode_cooperative-2            204.9Mi ±  1%   210.6Mi ±  5%       ~ (p=0.065 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_16k-2                    209.7Mi ±  1%   215.3Mi ±  3%       ~ (p=0.065 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256-2                    203.4Mi ±  1%   208.3Mi ±  1%  +2.37% (p=0.002 n=6)
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256_cooperative-2        197.4Mi ± 24%   200.2Mi ±  6%       ~ (p=0.394 n=6)
Decode/case=alltypes.237mfs/mode=decode-2                                      159.6Mi ±  7%   164.3Mi ±  2%  +2.92% (p=0.026 n=6)
Decode/case=alltypes.237mfs/mode=decode_cooperative-2                          159.4Mi ±  1%   159.6Mi ±  2%       ~ (p=1.000 n=6)
Decode/case=alltypes.237mfs/mode=stream_16k-2                                  160.9Mi ±  1%   164.6Mi ±  1%  +2.30% (p=0.004 n=6)
Decode/case=alltypes.237mfs/mode=stream_256-2                                  154.9Mi ±  1%   159.2Mi ±  1%  +2.73% (p=0.002 n=6)
Decode/case=alltypes.237mfs/mode=stream_256_cooperative-2                      152.9Mi ±  1%   152.2Mi ±  3%       ~ (p=0.777 n=6)
Decode/case=alltypes.237mfs/mode=relabel_stage1_stage2-2                                       118.1Mi ± 28%
geomean                                                                        169.3Mi         168.5Mi        +1.94%

                                                                        │ before_16ce7f9.txt │          after_87efe0a.txt           │
                                                                        │        B/op        │     B/op      vs base                │
Decode/case=alltypes.237mfs.nometa/mode=decode-2                                102.1Ki ± 0%   102.1Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=decode_cooperative-2                    102.1Ki ± 0%   102.1Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=stream_16k-2                            102.1Ki ± 0%   102.1Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=stream_256-2                            136.9Ki ± 0%   136.9Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=stream_256_cooperative-2                136.9Ki ± 0%   136.9Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=decode-2                         50.58Ki ± 0%   50.58Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=decode_cooperative-2             50.58Ki ± 0%   50.58Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_16k-2                     50.58Ki ± 0%   50.58Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256-2                     56.39Ki ± 0%   56.39Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256_cooperative-2         56.39Ki ± 0%   56.39Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=decode-2                                       82.63Ki ± 0%   82.63Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=decode_cooperative-2                           82.63Ki ± 0%   82.63Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=stream_16k-2                                   82.63Ki ± 0%   82.63Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=stream_256-2                                   105.6Ki ± 0%   105.6Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=stream_256_cooperative-2                       105.6Ki ± 0%   105.6Ki ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=relabel_stage1_stage2-2                                       81.00Ki ± 0%
geomean                                                                         82.07Ki        82.00Ki       +0.00%
¹ all samples are equal

                                                                        │ before_16ce7f9.txt │         after_87efe0a.txt          │
                                                                        │     allocs/op      │ allocs/op   vs base                │
Decode/case=alltypes.237mfs.nometa/mode=decode-2                                  76.00 ± 0%   76.00 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=decode_cooperative-2                      76.00 ± 0%   76.00 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=stream_16k-2                              76.00 ± 0%   76.00 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=stream_256-2                              601.0 ± 0%   601.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs.nometa/mode=stream_256_cooperative-2                  601.0 ± 0%   601.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=decode-2                           91.00 ± 0%   91.00 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=decode_cooperative-2               91.00 ± 0%   91.00 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_16k-2                       91.00 ± 0%   91.00 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256-2                       181.0 ± 0%   181.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.209mfs.counter_gauge/mode=stream_256_cooperative-2           181.0 ± 0%   181.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=decode-2                                         182.0 ± 0%   182.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=decode_cooperative-2                             182.0 ± 0%   182.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=stream_16k-2                                     182.0 ± 0%   182.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=stream_256-2                                     524.0 ± 0%   524.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=stream_256_cooperative-2                         524.0 ± 0%   524.0 ± 0%       ~ (p=1.000 n=6) ¹
Decode/case=alltypes.237mfs/mode=relabel_stage1_stage2-2                                       226.0 ± 0%
geomean                                                                           179.5        182.1       +0.00%
¹ all samples are equal

…rners

Adds resolve_conflicting_exposed_keys_iter, eval_key_keep_mask_iter, SingleLabelKeepCache::keep_str, SingleLabelTransformCache::transform_str, SingleLabelTransformCache::eval_single_str, and retained_memory_bytes helpers so arena-backed interners like ArrowDecoder's StringInterner can run Stage 2 relabeling without building temporary key/value slices.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…rowDecoder

Adds MetricRelabelPlan, DecodeOptions::honor_labels (default false), and DecodeOptions::metric_relabel_plan / with_metric_relabel_configs to ArrowDecoder.

- Stage 1 pushes leading __name__-only keep/drop/keepequal/dropequal rules into ArrowDecoder::ingest_entries, caching per contiguous wire series name so dropped series runs skip label unescaping, string interning, and family assembly while preserving cooperative budget accounting.
- Stage 2 resolves honor_labels = false collisions once across key_interner (leaving exemplar attributes untouched), executes labeldrop/labelkeep and single-label keep/drop/replace/lowercase/uppercase/hashmod rules on dictionary interners, and falls back to exact prometheus_relabel::process_in_place for general multi-label, labelmap, keepequal/dropequal, dynamic target_label, and __name__-mutating rules.
- Compacts dropped data points, child exemplars, and unused metric families in place, skips empty chunks in decode_stream, and updates ResourceAttrs only when Stage 2 mutates or drops target labels.

Towards #35.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…relabeling

Add unit tests covering honor_labels collision handling and exemplar
preservation, Stage 1 __name__ pushdown (including histograms and
cooperative budgets), Stage 2 dictionary-vectorized rules, general
scalar fallback with a prometheus-relabel oracle, and empty-chunk
dropping.

Add OTAP/OTLP round-trip coverage and enable MetricRelabelPlan in the
dhat bounded-memory test, add a Stage 1 + Stage 2 decode benchmark case,
and update specs/SPEC.md.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant