Set up Prometheus with docker-compose to get metrics of existing Kubernetes pods
You are running the Prometheus at the local and server level both.
server one working fine and getting the metrics of Kubernetes containers as it's on kubernetes.
while docker-compose one not working due to you are running it locally on docker not on kubernetes cluster.
it's the issue of the Target your Prometheus is not getting the metrics of your Kubernetes cluster.
for example, you are running the Prometheus locally but want to monitor the external Kubernetes cluster you have to expose your Kube-state-metrics service using IP.
in that case your Prometheus config will have job like
scrape_configs: - job_name: 'kube-state-metrics' static_configs: - targets: ['address'] //address of the k8s service IPin your case you have to do something like
kubernetes_sd_configs:- api_server: https://<ip>:6443 role: node bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: insecure_skip_verify: truethis will fetch the metrics of kubernetes cluster and you can see data locally.
you can read this nice gist : https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450