Prometheus: Monitor all services without creating ServiceMonitor for each service? Prometheus: Monitor all services without creating ServiceMonitor for each service? kubernetes kubernetes

Prometheus: Monitor all services without creating ServiceMonitor for each service?


Only if you have a common label on all services

# for example:org: "my-company"# ormonitoring: "true"# or app.kubernetes.io/managed-by: "Helm"  # <- in most cases this represents all

Then you define a single, cross-namespace ServiceMonitor, that covers all labeled services:

apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  name: common-monitor  namespace: monitoringspec:  endpoints:  - port: metrics    interval: 30s    path: /metrics  jobLabel: monitoring  namespaceSelector:    any: true  # <- important  selector:    matchLabels:      monitoring: "true"  # <- should match what you've chosen as common

Then to make sure this ServiceMonitor is discovered by the Prometheus Operator you either:

This additional explicit linkage between Prometheus Operator and ServiceMonitor is done intentionally - in this way, if you have 2 Prometheus instances on your cluster (say Infra and Product) you can separate which Prometheus will get which Pods to its scraping config.

From your question, it sounds like you already have a serviceMonitorSelector based on release: prometheus label - try adding that on your catch-all ServiceMonitor as well.