Monitor custom kubernetes pod metrics using Prometheus

You have to add this three annotation to your pods:

prometheus.io/scrape: 'true'prometheus.io/path: '/data/metrics'prometheus.io/port: '80'

How it will work?

Look at the kubernetes-pods job of config-map.yaml you are using to configure prometheus,

- job_name: 'kubernetes-pods'        kubernetes_sd_configs:        - role: pod        relabel_configs:        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]          action: keep          regex: true        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]          action: replace          target_label: __metrics_path__          regex: (.+)        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]          action: replace          regex: ([^:]+)(?::\d+)?;(\d+)          replacement: $1:$2          target_label: __address__        - action: labelmap          regex: __meta_kubernetes_pod_label_(.+)        - source_labels: [__meta_kubernetes_namespace]          action: replace          target_label: kubernetes_namespace        - source_labels: [__meta_kubernetes_pod_name]          action: replace          target_label: kubernetes_pod_name

Check this three relabel configuration

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]    action: keep    regex: true- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]    action: replace    target_label: __metrics_path__    regex: (.+)- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]    action: replace    regex: ([^:]+)(?::\d+)?;(\d+)    replacement: $1:$2    target_label: __address__

Here, __metrics_path__ and port and whether to scrap metrics from this pod are being read from pod annotations.

For, more details on how to configure Prometheus see here.

The link provided in the question refers to this ConfigMap for the prometheus configuration. It that ConfigMap is used then prometheus is already configured to scrape pods.

For that configuration (see relabel_configs) to have prometheus scrape the custom metrics exposed by pods at :80/data/metrics, add these annotations to the pods deployment configurations:

metadata:  annotations:    prometheus.io/scrape: 'true'    prometheus.io/path: '/data/metrics'    prometheus.io/port: '80'

See the configuration options for Kubernetes discovery in the prometheus docs (scroll down) for settings related to scraping over https and more.

Edit:I saw Emruz Hossain's answer only after I posted mine. His answer currently lacks the prometheus.io/scrape: 'true' annotation and specified = instead of : as annotations' name/value separator which is invalid in yaml or json.