How to create a ServiceMonitor for prometheus-operator?
Thanks to Peter who showed me that it idea in principle wasn't entirely incorrect I've found the missing link. As a servicemonitor
does monitor services (haha), I missed the part of creating a service which isn't part of the gitlab helm chart. Finally this yaml did the trick for me and the metrics appear in Prometheus:
# Service targeting gitlab instancesapiVersion: v1kind: Servicemetadata: name: gitlab-metrics labels: app: gitlab-runner-gitlab-runnerspec: ports: - name: metrics # expose metrics port port: 9252 # defined in gitlab chart targetPort: metrics protocol: TCP selector: app: gitlab-runner-gitlab-runner # target gitlab pods---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: gitlab-metrics-servicemonitor # Change this to the namespace the Prometheus instance is running in # namespace: default labels: app: gitlab-runner-gitlab-runner release: prometheusspec: selector: matchLabels: app: gitlab-runner-gitlab-runner # target gitlab service endpoints: - port: metrics interval: 15s
Nice to know: the metrics
targetPort
is defined in the gitlab runner chart.
I know this question is already answered. But I had a similar problem when Prometheus deployed in Kubernetes with Helm's stable/prometheus-operator chart couldn't find any active targets for my ServiceMonitor
. It turned out that my Service exposed a port that I didn't explicitly named:
- protocol: TCP port: 8080 targetPort: uwsgi
I could use it in Ingress by targeting uwsgi
port. But it seems that ServiceMonitor
needs an explicitly named port in Service
even if it has the same name as its own tagetPort:
- name: uwsgi protocol: TCP port: 8080 targetPort: uwsgi
I have written a blog post about this problem here