Prometheus: Monitor all services without creating ServiceMonitor for each service?
Only if you have a common label on all services
# for example:org: "my-company"# ormonitoring: "true"# or app.kubernetes.io/managed-by: "Helm" # <- in most cases this represents all
Then you define a single, cross-namespace ServiceMonitor
, that covers all labeled services:
apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: common-monitor namespace: monitoringspec: endpoints: - port: metrics interval: 30s path: /metrics jobLabel: monitoring namespaceSelector: any: true # <- important selector: matchLabels: monitoring: "true" # <- should match what you've chosen as common
Then to make sure this ServiceMonitor
is discovered by the Prometheus Operator you either:
- Declare this
ServiceMonitor
via the built-in operator template: https://github.com/prometheus-community/helm-charts/blob/4164ad5fdb6a977f1aba7b65f4e65582d3081528/charts/kube-prometheus-stack/values.yaml#L2008 - Or, declare a
serviceMonitorSelector
that points to yourServiceMonitor
https://github.com/prometheus-community/helm-charts/blob/4164ad5fdb6a977f1aba7b65f4e65582d3081528/charts/kube-prometheus-stack/values.yaml#L1760
This additional explicit linkage between Prometheus Operator and ServiceMonitor is done intentionally - in this way, if you have 2 Prometheus instances on your cluster (say Infra and Product) you can separate which Prometheus will get which Pods to its scraping config.
From your question, it sounds like you already have a serviceMonitorSelector
based on release: prometheus
label - try adding that on your catch-all ServiceMonitor
as well.