Prometheus cAdvisor docker monitoring Prometheus cAdvisor docker monitoring docker docker

Prometheus cAdvisor docker monitoring


It seems to be related to the amount of time cAdvisor stores the data in memory.

While cAdvisor keeps the data in memory, you still have a valid date in container_last_seen metric. So the count_scalar instruction still 'sees' the container as it has a valid value.

In my test setup, cAdvisor keeps the data during 5 minutes. After this duration, I get the right information out of your formula because the container_last_seen metric has disappeared.

You can change this cAdvisor configuration with the --storage_duration flag.

--storage_duration=2m0s: How long to store data.

As an alternative if you wan't quick alerting, you could also consider running a query that would compare last seen date with current date:

count_scalar(time()-container_last_seen{name=~"container1|container2"}<=60)