You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Document real world examples of using relabel_config #341
The documentation for relabel_config is very clinical and hard to understand. (At least it is accurate, as far as I can tell.)
In both relabel_configs and metric_relabel_configs some real-world examples would go a long way. (In fact, relabel_configs doesn't even have a paragraph. metric_relabel_configs says "One use for this is to blacklist time series that are too expensive to ingest" but gives no hint at how to accomplish that.)
It would be great if the examples gave some typical uses for relabeling, and included examples of relabeling a kubernetes_sd_config or similar.
I agree that more examples would are necessary and that blog posts come with the issue of going out of date.
I'm strongly in favor for keeping the core docs clinical and precise though.
If I want to look something up and know what I'm looking for, this shouldn't be disturbed by all the examples. One doesn't want to scan through a lot of text that may or may not contain feature specification.
A guide section would probably be the way to go for having up-to-date examples with more detailed explanation.
As so often I see the issue that this (as the current documentation) will need versioning at some point.
Any suggestions about decent software for that are welcome.
Reacted by Christopher Hanson, Wojciech C., zeenatl, Utkarsh Mani Tripathi, Bekir Murat, George V, pulord, Merritt Krakowitzer, Sahil Gupta, Chen Li and 51 more
Further, it'd be great if there could be some suggestions or howto on developing, testing, and debugging relabel configs. At least point out that the applied relabels appear the /status page and the original input labels appear in a tooltip. (I have no idea how to go about testing/debugging metric relabelling. Is there a similar feature?)
The Kubernetes documentation is versioned with the release. In github they have a header that says "warning this is the HEAD documentation" and then they strip it out when snapshotting the docs for a release.
I'm finding it very difficult to understand the relabeling functionality from the official docs. Few straight forward examples would be extremely useful.
Reacted by Phil Porada, orinciog, Florian Schmidt, Jonas Mellquist, Yukun Sun, Patrick, Ahsan Shafiq, realdimas, efficks, Yueming Liu and 40 more
The lack of examples and readability of our docs is one of the most common things I hear from people at meetups.
Fortunately with interactive web pages it's not a black/white or concise=no-examples question. I don't see any reason we can't provide additional examples on the same page which only get shown upon click or something like that.
Reacted by Johannes 'fish' Ziemke, Chen Li, Di Xu, Phil Porada, Ulrich Petri, Jonas Mellquist, Patrick, realdimas, Yueming Liu, Paul Makepeace and 17 more
The challenge with relabel examples is that they quickly get into configuration management philosophy, even when hidden they'd quickly become unwieldy as there's just so many possibilities.
I'm for a strong separation between reference docs and guides. I've seen where they've gotten mixed for the sake of getting people started, and it ended up making everyone unhappy.
being a blog post instead of docs, it's already out of date.
Our blog posts are meant to be up to date. Please file a bug for any inaccuracies you find.
If it's any help I just stumbled upon this bug while struggling with a lack of docs. After much fiddling and flailing I've managed to make my use-case work.
My problem was I wanted to relabel bridge_JOBNAME_METRIC_NAME_HERE into bridge_METRIC_NAME_HERE{job=JOBNAME}. I've solved it by:
metric_relabel_configs:
# Extract the bridge name into the job label
- source_labels: ["__name__"]regex: "bridge_([^_]+)_.*"target_label: "job"replacement: "$1"# Rewrite the label name to remove the bridge name out of it
- source_labels: ["__name__"]regex: "bridge_[^_]+_(.*)"target_label: "__name__"replacement: "$1"
Reacted by Jens Rantil, Sean, Delweng, Rob Figueiredo, Dan Fredell, Olivier Bierlaire, David Barri, Yurii Matsiuk, Frank lin Piat, Marko Kruljac and 19 more
Reacted by Jens Rantil, Ash McKenzie and Bartlomiej (Bart) Rekke
Reacted by DK, Crispin Wellington, Denis Krivak, Musin Vitaly, Dennis, Aleksei Krasnoperov, Johannes Innerbichler, Alex Silcock, Rafael Lima Joia, ScrumpyJack and 14 more
Well, about time to look into this again :)
I agree with @grobie, we can both have a concise reference docs and some more practical guide/tips/examples. I remember vividly when I introduced Prometheus at Docker, the biggest concern was lack of examples/snippets to get people started. From a UX perspective, I think some sidebar you can open additionally to the reference, which shows further tips/examples.
I think it would be neat to have a separate repo for this with lower review requirements, so changes to the example don't need to be vetted by maintainers. A wiki might be even better, but something that is mapped to the docs and viable side by side (so you see right away when things don't add up and examples need updating). Something that gets loaded via js, so doesn't make the reference docs slower/bigger?
Reacted by Danny Kulchinsky, Bekir Murat and Witt A.
Afaik, there is no need for fancy UI stuff, even simple links to examples in a git repo would go a long way.
We decided to use Prometheus for our monitoring infrastructure but we are questioning our choice now that we are digging deeper and using more advanced features. Our main gripes are with configuration and discovery.
I wrote the getting started as a guide - https://prometheus.io/docs/prometheus/latest/getting_started/ (#895) and added a guide section - it'd be very easy to add further guides: using exporters, relabelling, service discovery, and querying all spring to mind as ideas.
Thank you for this, I used the getting started guide for our initial setup.
Would it still be a getting started guide though, if these subjects were added ?
This is another example of a blog post that should be converted to the official docs. The post itself is three years old and it reads partially as a v0.14.0 release note instead of a guide. Even with those issues, the post contains some of the most helpful information regarding relabelling that I was able to find.
One thing that confused me was the name 'relabel' is itself misleading. I was under the assumption that it was a tool for changing the labels, not for creating new labels with new metric data, and not for processing the metric values themselves that are under those labels into new labels. When the documentation is improved this kind of high level overview of what can be done, and why, should be up front.
I have suggested calling it target_relabel_configs to differentiate it from metric_relabel_configs.
We could offer this as an alias, to allow config file transition for Prometheus 3.x.
Reacted by Nicolai von Neudeck, Anthony DeChiaro, realdimas, Shyukri Shyukriev, Christian Lipphardt, Adriano Carmezim, zm, alik-agaev, Viktor Dorotovič, Brandon Duffany and 2 more
Just a related idea (not trivial !)... If someone see how to create a JS webapp or a webassembly in go to test relabeling_rules it could be the graal 😅
I know lot of devs not confortable with regex, and lot of sites like https://www.debuggex.com/ help them a lot to create / test regex.
I know also lot of Prometheus adopters (including me) not very comfortable with relabel configs ! It would be awsome to put a <scrape_config> on it, a list a metrics (copied from /metrics endpoint) and see the result.
A few comments on this issue signal interest and usefulness, but this issue has been open for a long time. What could be a first step to getting it underway?
an outline listing examples/types of the examples worth documenting?
could AI produce a draft from available sources of these examples?
will there be special review rqts/a specific person or persons who MUST sign off?
Just some ideas to try and break this task down -- +1 if anyone has other, better ideas in mind.
Activity
fabxc commentedon Mar 3, 2016
I agree that more examples would are necessary and that blog posts come with the issue of going out of date.
I'm strongly in favor for keeping the core docs clinical and precise though.
If I want to look something up and know what I'm looking for, this shouldn't be disturbed by all the examples. One doesn't want to scan through a lot of text that may or may not contain feature specification.
A guide section would probably be the way to go for having up-to-date examples with more detailed explanation.
As so often I see the issue that this (as the current documentation) will need versioning at some point.
Any suggestions about decent software for that are welcome.
davidmankin commentedon Mar 3, 2016
Further, it'd be great if there could be some suggestions or howto on developing, testing, and debugging relabel configs. At least point out that the applied relabels appear the /status page and the original input labels appear in a tooltip. (I have no idea how to go about testing/debugging metric relabelling. Is there a similar feature?)
davidmankin commentedon Mar 3, 2016
The Kubernetes documentation is versioned with the release. In github they have a header that says "warning this is the HEAD documentation" and then they strip it out when snapshotting the docs for a release.
davidmankin commentedon Mar 3, 2016
I don't mind clinical & precise reference docs as long as they include a reference to a guide section of the docs.
marekbruchaty commentedon Aug 8, 2016
I'm finding it very difficult to understand the relabeling functionality from the official docs. Few straight forward examples would be extremely useful.
grobie commentedon Aug 8, 2016
The lack of examples and readability of our docs is one of the most common things I hear from people at meetups.
Fortunately with interactive web pages it's not a black/white or concise=no-examples question. I don't see any reason we can't provide additional examples on the same page which only get shown upon click or something like that.
brian-brazil commentedon Aug 8, 2016
The challenge with relabel examples is that they quickly get into configuration management philosophy, even when hidden they'd quickly become unwieldy as there's just so many possibilities.
I'm for a strong separation between reference docs and guides. I've seen where they've gotten mixed for the sake of getting people started, and it ended up making everyone unhappy.
Our blog posts are meant to be up to date. Please file a bug for any inaccuracies you find.
leonerd commentedon Sep 20, 2016
If it's any help I just stumbled upon this bug while struggling with a lack of docs. After much fiddling and flailing I've managed to make my use-case work.
My problem was I wanted to relabel
bridge_JOBNAME_METRIC_NAME_HEREintobridge_METRIC_NAME_HERE{job=JOBNAME}. I've solved it by:brian-brazil commentedon Sep 20, 2016
You might want to throw that up on http://tips.robustperception.io/
buchanae commentedon Feb 1, 2018
I'm still dumbfounded by using relabeling and service discovery. Normally I would just hit the +1 button, but this issue is nearly 2 years old.
discordianfish commentedon Feb 3, 2018
Well, about time to look into this again :)
I agree with @grobie, we can both have a concise reference docs and some more practical guide/tips/examples. I remember vividly when I introduced Prometheus at Docker, the biggest concern was lack of examples/snippets to get people started. From a UX perspective, I think some sidebar you can open additionally to the reference, which shows further tips/examples.
I think it would be neat to have a separate repo for this with lower review requirements, so changes to the example don't need to be vetted by maintainers. A wiki might be even better, but something that is mapped to the docs and viable side by side (so you see right away when things don't add up and examples need updating). Something that gets loaded via js, so doesn't make the reference docs slower/bigger?
bekir-murat commentedon Feb 19, 2018
Afaik, there is no need for fancy UI stuff, even simple links to examples in a git repo would go a long way.
We decided to use Prometheus for our monitoring infrastructure but we are questioning our choice now that we are digging deeper and using more advanced features. Our main gripes are with configuration and discovery.
jamtur01 commentedon Feb 20, 2018
I wrote the getting started as a guide - https://prometheus.io/docs/prometheus/latest/getting_started/ (#895) and added a guide section - it'd be very easy to add further guides: using exporters, relabelling, service discovery, and querying all spring to mind as ideas.
bekir-murat commentedon Feb 20, 2018
Thank you for this, I used the getting started guide for our initial setup.
Would it still be a getting started guide though, if these subjects were added ?
6 remaining items
ravishivt commentedon Aug 24, 2018
Here's another good blog post for others struggling with these concepts. https://prometheus.io/blog/2015/06/01/advanced-service-discovery/
This is another example of a blog post that should be converted to the official docs. The post itself is three years old and it reads partially as a v0.14.0 release note instead of a guide. Even with those issues, the post contains some of the most helpful information regarding relabelling that I was able to find.
RichiH commentedon Aug 26, 2018
@lucperkins Coming up with a few specific examples and collating all information would be worthwhile.
retrogradeorbit commentedon Sep 27, 2018
One thing that confused me was the name 'relabel' is itself misleading. I was under the assumption that it was a tool for changing the labels, not for creating new labels with new metric data, and not for processing the metric values themselves that are under those labels into new labels. When the documentation is improved this kind of high level overview of what can be done, and why, should be up front.
SuperQ commentedon Sep 27, 2018
I have suggested calling it
target_relabel_configsto differentiate it frommetric_relabel_configs.We could offer this as an alias, to allow config file transition for Prometheus 3.x.
Update feature description in overview and readme (prometheus#341)
carlbordum commentedon Apr 20, 2020
Is the Guides section of the docs intended for stuff like this?
https://prometheus.io/docs/guides/
brian-brazil commentedon Apr 20, 2020
Yes, guides is where things like this would go.
AndyFHAF commentedon Aug 21, 2020
good
gillg commentedon May 20, 2021
Just a related idea (not trivial !)... If someone see how to create a JS webapp or a webassembly in go to test relabeling_rules it could be the graal 😅
I know lot of devs not confortable with regex, and lot of sites like https://www.debuggex.com/ help them a lot to create / test regex.
I know also lot of Prometheus adopters (including me) not very comfortable with relabel configs ! It would be awsome to put a <scrape_config> on it, a list a metrics (copied from /metrics endpoint) and see the result.
That was my 2cents
saliceti commentedon Jan 10, 2022
@gillg this is very useful: https://relabeler.promlabs.com/
jan--f commentedon Mar 18, 2026
Hello from the bug scrub!
This still seems worthwhile to have.
lasomethingsomething commentedon Nov 18, 2026
A few comments on this issue signal interest and usefulness, but this issue has been open for a long time. What could be a first step to getting it underway?
Just some ideas to try and break this task down -- +1 if anyone has other, better ideas in mind.