Auto-generate alert rules in Prometheus in Kubernetes? Auto-generate alert rules in Prometheus in Kubernetes? kubernetes kubernetes

Auto-generate alert rules in Prometheus in Kubernetes?


As long as you are able to express the extent of volumes to alert on through PromQL wildcards, you can have a single rule for all volumes in question.

Prometheus is actually quite smart about doing arithmetic operations on values with matching value labels, even when you use wildcards.

E.g., a rule to cover all instances of loki persistent volumes (i.e. not just the -0) would look like this:

additionalPrometheusRulesMap:    rule-name:     groups:     - name: storage       rules:        - alert: lokiStorageAt70%          expr: ( sum(kubelet_volume_stats_capacity_bytes{job="kubelet", namespace="kube-logging", persistentvolumeclaim=~"storage-loki-.+"})                 - sum(kubelet_volume_stats_available_bytes{job="kubelet", namespace="kube-logging", persistentvolumeclaim=~"storage-loki-.+"}))                 / sum(kubelet_volume_stats_capacity_bytes{job="kubelet", namespace="kube-logging", persistentvolumeclaim=~"storage-loki-.+"})> 0.7          for: 15m          labels:            severity: warning          annotations:            summary: The Storage of Loki is 70% full. Maybe increase the storage size?

This being just an example, you can of course make the wildcard even wilder to cover more of the volumes you want to alert on.