Skip to content

Document real world examples of using relabel_config #341

Open

Description

@davidmankin

WRT http://prometheus.io/docs/operating/configuration/#%3Crelabel_config%3E

The documentation for relabel_config is very clinical and hard to understand. (At least it is accurate, as far as I can tell.)

In both relabel_configs and metric_relabel_configs some real-world examples would go a long way. (In fact, relabel_configs doesn't even have a paragraph. metric_relabel_configs says "One use for this is to blacklist time series that are too expensive to ingest" but gives no hint at how to accomplish that.)

It would be great if the examples gave some typical uses for relabeling, and included examples of relabeling a kubernetes_sd_config or similar.

I found the relabelling much easier to understand after I found this blog post which contained a few sparse examples: https://prometheus.io/blog/2015/06/01/advanced-service-discovery/ but, being a blog post instead of docs, it's already out of date.

Activity

  1. fabxc commented on Mar 3, 2016

    @fabxc
    Contributor

    I agree that more examples would are necessary and that blog posts come with the issue of going out of date.
    I'm strongly in favor for keeping the core docs clinical and precise though.

    If I want to look something up and know what I'm looking for, this shouldn't be disturbed by all the examples. One doesn't want to scan through a lot of text that may or may not contain feature specification.

    A guide section would probably be the way to go for having up-to-date examples with more detailed explanation.

    As so often I see the issue that this (as the current documentation) will need versioning at some point.
    Any suggestions about decent software for that are welcome.

  2. davidmankin commented on Mar 3, 2016

    @davidmankin
    Author

    Further, it'd be great if there could be some suggestions or howto on developing, testing, and debugging relabel configs. At least point out that the applied relabels appear the /status page and the original input labels appear in a tooltip. (I have no idea how to go about testing/debugging metric relabelling. Is there a similar feature?)

  3. davidmankin commented on Mar 3, 2016

    @davidmankin
    Author

    The Kubernetes documentation is versioned with the release. In github they have a header that says "warning this is the HEAD documentation" and then they strip it out when snapshotting the docs for a release.

  4. davidmankin commented on Mar 3, 2016

    @davidmankin
    Author

    I don't mind clinical & precise reference docs as long as they include a reference to a guide section of the docs.

  5. marekbruchaty commented on Aug 8, 2016

    @marekbruchaty

    I'm finding it very difficult to understand the relabeling functionality from the official docs. Few straight forward examples would be extremely useful.

  6. grobie commented on Aug 8, 2016

    @grobie
    Member

    The lack of examples and readability of our docs is one of the most common things I hear from people at meetups.

    Fortunately with interactive web pages it's not a black/white or concise=no-examples question. I don't see any reason we can't provide additional examples on the same page which only get shown upon click or something like that.

  7. brian-brazil commented on Aug 8, 2016

    @brian-brazil
    Contributor

    The challenge with relabel examples is that they quickly get into configuration management philosophy, even when hidden they'd quickly become unwieldy as there's just so many possibilities.

    I'm for a strong separation between reference docs and guides. I've seen where they've gotten mixed for the sake of getting people started, and it ended up making everyone unhappy.

    being a blog post instead of docs, it's already out of date.

    Our blog posts are meant to be up to date. Please file a bug for any inaccuracies you find.

  8. leonerd commented on Sep 20, 2016

    @leonerd
    Contributor

    If it's any help I just stumbled upon this bug while struggling with a lack of docs. After much fiddling and flailing I've managed to make my use-case work.

    My problem was I wanted to relabel bridge_JOBNAME_METRIC_NAME_HERE into bridge_METRIC_NAME_HERE{job=JOBNAME}. I've solved it by:

        metric_relabel_configs:
          # Extract the bridge name into the job label
          - source_labels: ["__name__"]
            regex: "bridge_([^_]+)_.*"
            target_label: "job"
            replacement: "$1"
          # Rewrite the label name to remove the bridge name out of it
          - source_labels: ["__name__"]
            regex: "bridge_[^_]+_(.*)"
            target_label: "__name__"
            replacement: "$1"
  9. brian-brazil commented on Sep 20, 2016

    @brian-brazil
    Contributor

    You might want to throw that up on http://tips.robustperception.io/

  10. buchanae commented on Feb 1, 2018

    @buchanae

    I'm still dumbfounded by using relabeling and service discovery. Normally I would just hit the +1 button, but this issue is nearly 2 years old.

  11. discordianfish commented on Feb 3, 2018

    @discordianfish
    Member

    Well, about time to look into this again :)
    I agree with @grobie, we can both have a concise reference docs and some more practical guide/tips/examples. I remember vividly when I introduced Prometheus at Docker, the biggest concern was lack of examples/snippets to get people started. From a UX perspective, I think some sidebar you can open additionally to the reference, which shows further tips/examples.

    I think it would be neat to have a separate repo for this with lower review requirements, so changes to the example don't need to be vetted by maintainers. A wiki might be even better, but something that is mapped to the docs and viable side by side (so you see right away when things don't add up and examples need updating). Something that gets loaded via js, so doesn't make the reference docs slower/bigger?

  12. bekir-murat commented on Feb 19, 2018

    @bekir-murat

    Afaik, there is no need for fancy UI stuff, even simple links to examples in a git repo would go a long way.
    We decided to use Prometheus for our monitoring infrastructure but we are questioning our choice now that we are digging deeper and using more advanced features. Our main gripes are with configuration and discovery.

  13. jamtur01 commented on Feb 20, 2018

    @jamtur01
    Contributor

    I wrote the getting started as a guide - https://prometheus.io/docs/prometheus/latest/getting_started/ (#895) and added a guide section - it'd be very easy to add further guides: using exporters, relabelling, service discovery, and querying all spring to mind as ideas.

  14. bekir-murat commented on Feb 20, 2018

    @bekir-murat

    Thank you for this, I used the getting started guide for our initial setup.
    Would it still be a getting started guide though, if these subjects were added ?

  15. 6 remaining items

  16. ravishivt commented on Aug 24, 2018

    @ravishivt

    Here's another good blog post for others struggling with these concepts. https://prometheus.io/blog/2015/06/01/advanced-service-discovery/

    This is another example of a blog post that should be converted to the official docs. The post itself is three years old and it reads partially as a v0.14.0 release note instead of a guide. Even with those issues, the post contains some of the most helpful information regarding relabelling that I was able to find.

  17. RichiH commented on Aug 26, 2018

    @RichiH
    Member

    @lucperkins Coming up with a few specific examples and collating all information would be worthwhile.

  18. retrogradeorbit commented on Sep 27, 2018

    @retrogradeorbit

    One thing that confused me was the name 'relabel' is itself misleading. I was under the assumption that it was a tool for changing the labels, not for creating new labels with new metric data, and not for processing the metric values themselves that are under those labels into new labels. When the documentation is improved this kind of high level overview of what can be done, and why, should be up front.

  19. SuperQ commented on Sep 27, 2018

    @SuperQ
    Member

    I have suggested calling it target_relabel_configs to differentiate it from metric_relabel_configs.

    We could offer this as an alias, to allow config file transition for Prometheus 3.x.

  20. carlbordum commented on Apr 20, 2020

    @carlbordum

    Is the Guides section of the docs intended for stuff like this?

    https://prometheus.io/docs/guides/

  21. brian-brazil commented on Apr 20, 2020

    @brian-brazil
    Contributor

    Yes, guides is where things like this would go.

  22. AndyFHAF commented on Aug 21, 2020

    @AndyFHAF

    good

  23. gillg commented on May 20, 2021

    @gillg

    Just a related idea (not trivial !)... If someone see how to create a JS webapp or a webassembly in go to test relabeling_rules it could be the graal 😅

    I know lot of devs not confortable with regex, and lot of sites like https://www.debuggex.com/ help them a lot to create / test regex.
    I know also lot of Prometheus adopters (including me) not very comfortable with relabel configs ! It would be awsome to put a <scrape_config> on it, a list a metrics (copied from /metrics endpoint) and see the result.

    That was my 2cents

  24. saliceti commented on Jan 10, 2022

    @saliceti
  25. jan--f commented on Mar 18, 2026

    @jan--f
    Contributor

    Hello from the bug scrub!
    This still seems worthwhile to have.

  26. lasomethingsomething commented on Nov 18, 2026

    @lasomethingsomething

    A few comments on this issue signal interest and usefulness, but this issue has been open for a long time. What could be a first step to getting it underway?

    • an outline listing examples/types of the examples worth documenting?
    • could AI produce a draft from available sources of these examples?
    • will there be special review rqts/a specific person or persons who MUST sign off?

    Just some ideas to try and break this task down -- +1 if anyone has other, better ideas in mind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions