How to get notified or alerted on config reloader error in prometheus operator?
Prometheus exposes a /metric endpoint you can scrape.In particular, there is a metric indicating if the last reload suceeded:
# HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.# TYPE prometheus_config_last_reload_successful gaugeprometheus_config_last_reload_successful 0You can use it to alert on failed reload.
groups:- name: PrometheusAlerts rules: - alert: FailedReload expr: prometheus_config_last_reload_successful == 0 for: 5m labels: severity: warning annotations: description: Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{ $labels.pod}}. summary: Prometheus configuration reload has failed