Resource Usage by stopped Docker containers
I am not an expert about docker core but I will try to answer some of these questions.
- I suppose the fact that the file system state is preserved means that the container still does consume some space on the host's file system?
Yes. Docker save all the container and image data in /var/lib/docker. The default way to save the container and image data is using aufs. The data of each layer is saved under /var/lib/docker/aufs/diff. When a new container is created, a new layer is also created with is folder, and there the changes from the layers of the source image are stored.
- Is there a performance hit (other than host disk space consumption) associated with having 10s, or even 100s, of stopped containers in the system? For instance, does it make it any harder for Docker to startup and manage new containers?
As far as I know, it should not be any performace hit. When you stop a container, docker daemon sends SIGTERM and SIGKILL to all the process of that container, as described in docker CLI documentation:
Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop a running container by sending SIGTERM and then SIGKILL after a grace period
-t, --time=10 Number of seconds to wait for the container to stop before killing it. Default is 10 seconds.
3.And finally, if Paused containers retain their memory state when Unpaused - as demonstrated by their ability to remember memcached keys - do they have a different impact on CPU and memory?
As @Usman said, docker implements pause/unpause using the cgroup freezer. If I'm not wrong, when you put a process in the freezer (or its cgroup), you block the execution of new task of that process from the kernel task scheduler (i.e.: it stops the process), but you don't kill them and they keep consuming the memory they are using (although the Kernel may move that memory to swap or to solid disk). And the CPU resources used by a paused container I would consider insignificant. For more information about this I would check the pull request of this feature, Docker issue #5948