prometheus node-exporter on kubernetes

The previous advice to use Helm is highly valid, I would also recommend that.

Regarding your issue: thing is that you are not scraping nodes directly, you're using node-exporter for that. So role: node is incorrect, you should instead use role: endpoints. For that you also need to create service for all pods of your DaemonSet.

Here is working example from my environment (installed by Helm):

- job_name: monitoring/kube-prometheus-exporter-node/0  scrape_interval: 15s  scrape_timeout: 10s  metrics_path: /metrics  scheme: http  kubernetes_sd_configs:  - role: endpoints    namespaces:      names:      - monitoring  relabel_configs:  - source_labels: [__meta_kubernetes_service_label_app]    separator: ;    regex: exporter-node    replacement: $1    action: keep  - source_labels: [__meta_kubernetes_endpoint_port_name]    separator: ;    regex: metrics    replacement: $1    action: keep  - source_labels: [__meta_kubernetes_namespace]    separator: ;    regex: (.*)    target_label: namespace    replacement: $1    action: replace  - source_labels: [__meta_kubernetes_pod_name]    separator: ;    regex: (.*)    target_label: pod    replacement: $1    action: replace  - source_labels: [__meta_kubernetes_service_name]    separator: ;    regex: (.*)    target_label: service    replacement: $1    action: replace  - source_labels: [__meta_kubernetes_service_name]    separator: ;    regex: (.*)    target_label: job    replacement: ${1}    action: replace  - separator: ;    regex: (.*)    target_label: endpoint    replacement: metrics    action: replace

How did you deploy Prometheus? Whenever i used the helm-chart (https://github.com/helm/charts/tree/master/stable/prometheus) the node-exporter had been deployed. Maybe this is a simpler solution.

I was stuck at the similar place. But here my node-exporters are not part of helm deployment since we have got the add-on node exporter from Tanzu kubernetes grid(k8s cluster). So I have created the service monitor and now I can see the service discovery and the count is what should be. But in the target section it is saying 0/4 count. Not able to see the metrics of nodes but when I can curl the localhost:9100/metrics I can see the data. Some where I missing the logic.

I checked the helm deployed node-exporter data, it looks same but what am I missing here?

Please ignore the indentation as they are missed while copy paste in mobile.

 - job_name: node-exporter   scrape_interval: 15s   scrape_timeout: 10s   metrics_path: /metrics   scheme: http   kubernetes_sd_configs:     - role: endpoints       namespaces:       names:        - monitoring  relabel_configs:   - source_labels:        [__meta_kubernetes_service_label_app]      separator: ;      regex: exporter-node      replacement: $1      action: keep   - source_labels:        [__meta_kubernetes_endpoint_port_name]      separator: ;      regex: metrics      replacement: $1      action: keep  - source_labels: [__meta_kubernetes_namespace]     separator: ;     regex: (.*)     target_label: namespace     replacement: $1     action: replace - source_labels: [__meta_kubernetes_pod_name]   separator: ;   regex: (.*)   target_label: pod   replacement: $1   action: replace- source_labels: [__meta_kubernetes_service_name]   separator: ;   regex: (.*)   target_label: service   replacement: $1   action: replace - source_labels: [__meta_kubernetes_service_name]   separator: ;    regex: (.*)   target_label: job   replacement: ${1}   action: replace  - separator: ;    regex: (.*)    target_label: endpoint    replacement: metrics    action: replace