Multi-Value Prometheus Query Grafana Multi-Value Prometheus Query Grafana kubernetes kubernetes

Multi-Value Prometheus Query Grafana


Try it like this:

sum(rate(container_cpu_usage_seconds_total{id="/", instance=~"($ip_test):10250"}[1m]))


From multiple values formating documentation, Prometheus variables are expanded as regex:

InfluxDB and Prometheus uses regex expressions, so the same variable would be interpolated as (host1|host2|host3). Every value would also be regex escaped if not, a value with a regex control character would break the regex expression.

Therefore your variable ip_test = ['127.0.0.1', '127.0.0.2',...] is supposed to be transformed into: (127\.0\.0\.1|127\.0\.0\.2).

This means that your expression =~$ip_test:10250 should be transformed into =~"(127\.0\.0\.1|127\.0\.0\.2):10250" so you don't need the multiple expansion you are asking for.

The reason it is not working is that either the documentation is incorrect or there is a bug in Grafana (tested with version v6.7.2). From my tests, I suspect, the Prometheus expansion doesn't expand with the enclosing () and you end up with the expression =~"127\.0\.0\.1|127\.0\.0\.2:10250" - which is not what you want.

The workaround is to use the regex notation =~"${ip_test:regex}:10250".