can't collect docker metrics using prometheus

In respect to your question

What can I do next?

after changing the configuration in /etc/docker/daemon.json it will be necessary to

systemctl daemon-reloadsystemctl restart docker 

You may check after this if the port 9323 is in state LISTEN

lsof -Pi TCP -a -c dockerd 

As you mentioned in your comment that

curl http://$(hostname):9323/metrics  

is working properly on your local host, it indicates a problem with(in) your network.

As the Prometheus service is usually running somewhere else you may check if there is a connection from the remote machine to your Docker host. For this you could use something like

root@prometheusHost:/# nc -vz dockerHost 9323

It will give you a hint if the connection is refused, e.g. by a firewall.

Get docker ip addres

ip addr show docker0    

Enter docker ip addres in prometherus.yml configuration

    static_configs:  - targets: ['172.17.0.1:9323']

Configure the Docker daemon

NOTE: 0.0.0.0 i.e. all IP addresses on the local machine

/etc/docker/daemon.json

{  "metrics-addr" : "0.0.0.0:9323",  "experimental" : true}

Open firewall port 9323 --permanently

$ sudo firewall-cmd --zone=public --permanent --add-port=9323/tcpsuccess

A few checks

Check, remember to list --permanent configuration, or it won't show

$ sudo firewall-cmd --zone=public --permanent --list-allpublic  target: default  icmp-block-inversion: no  interfaces:   sources:   services: dhcpv6-client http https ssh  ports: 9323/tcp                            <= OPEN  protocols:   masquerade: no  forward-ports:   source-ports:   icmp-blocks:   rich rules: 

$ sudo lsof -Pi TCP -a -c dockerdCOMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAMEdockerd-c 28157 root   14u  IPv6 30732963      0t0  TCP *:9323 (LISTEN)...

Check connection to localhost

nc -vz localhost 9323Ncat: Version 7.50 ( https://nmap.org/ncat )Ncat: Connected to ::1:9323.Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check connection to host IP e.g. 10.223.37.14

$ nc -vz 10.223.37.14 9323Ncat: Version 7.50 ( https://nmap.org/ncat )Ncat: Connected to 10.223.37.14:9323.Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

Check the metrics acccess using curl

curl http://localhost:9323/metrics

or using the IP from another machine

curl http://10.223.37.14:9323/metrics

You will get a connection refused if the port is not open

curl http://10.223.37.14:9323/metricscurl: (7) Failed to connect to 10.223.37.14 port 9323: Connection refused

but once it's open, you will be able to see the metrics

$ curl http://10.223.37.14:9323/metrics# HELP engine_daemon_container_actions_seconds The number of seconds it takes to process each container action# TYPE engine_daemon_container_actions_seconds histogramengine_daemon_container_actions_seconds_bucket{action="changes",le="0.005"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="0.01"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="0.025"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="0.05"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="0.1"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="0.25"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="0.5"} 1engine_daemon_container_actions_seconds_bucket{action="changes",le="1"} 1...