Auto-generate alert rules in Prometheus in Kubernetes?
As long as you are able to express the extent of volumes to alert on through PromQL wildcards, you can have a single rule for all volumes in question.
Prometheus is actually quite smart about doing arithmetic operations on values with matching value labels, even when you use wildcards.
E.g., a rule to cover all instances of loki persistent volumes (i.e. not just the -0
) would look like this:
additionalPrometheusRulesMap: rule-name: groups: - name: storage rules: - alert: lokiStorageAt70% expr: ( sum(kubelet_volume_stats_capacity_bytes{job="kubelet", namespace="kube-logging", persistentvolumeclaim=~"storage-loki-.+"}) - sum(kubelet_volume_stats_available_bytes{job="kubelet", namespace="kube-logging", persistentvolumeclaim=~"storage-loki-.+"})) / sum(kubelet_volume_stats_capacity_bytes{job="kubelet", namespace="kube-logging", persistentvolumeclaim=~"storage-loki-.+"})> 0.7 for: 15m labels: severity: warning annotations: summary: The Storage of Loki is 70% full. Maybe increase the storage size?
This being just an example, you can of course make the wildcard even wilder to cover more of the volumes you want to alert on.