Prometheus cAdvisor docker monitoring
It seems to be related to the amount of time cAdvisor stores the data in memory.
While cAdvisor keeps the data in memory, you still have a valid date in container_last_seen
metric. So the count_scalar
instruction still 'sees' the container as it has a valid value.
In my test setup, cAdvisor keeps the data during 5 minutes. After this duration, I get the right information out of your formula because the container_last_seen
metric has disappeared.
You can change this cAdvisor configuration with the --storage_duration
flag.
--storage_duration=2m0s: How long to store data.
As an alternative if you wan't quick alerting, you could also consider running a query that would compare last seen date with current date:
count_scalar(time()-container_last_seen{name=~"container1|container2"}<=60)