Docker long-running command memory consumption grows over time

Docker processes are not the one who are using the memory, docker is the one who is reporting the use. The memory is used by the the processes in the container and the kernel modules. Based on the information in the question, I don't see that you have problem, rather you are confused by the way that the memory is managed by the kernel, so I will try to explain on a very high level what is going on.

Kernel Namespaces

The modern Kernel allows creation of partitions (namespaces) where the processes that are assigned to this partitions can work in semi-isolation (isolated from all other processes in the user space, but not from all kernel subsystems/modules). Docker is using this feature of the kernel to create containers.

The container is running in a so called kernel namespace. The kernel namespace has little to do with the docker's memory space in which the docker application is running. The docker application is responsible only for the orchestration of the namespaces (creating new namespaces, assigning containers to namespaces, setting the resource limits, provisioning network infrastructure, etc.), so Docker, after creating the kernel namespace is not involved if further memory management of it.

So when you start a container, Docker creates a new namespace for the container, prapares all of the resource limitation, prepares the network and starts the processes in that name space. I have deliberately oversimplified the creation of the container for this conversation, so if you need more information you can check this posts:

• Understanding the Docker Internals
• Build Containers From Scratch in Go (Part 1: Namespaces)
• Demystifying Containers - Part I: Kernel Space

Are the Buffers and Cache really a problem

The kernel will try to use as much as possible from the "unused memory" as a cache. This makes a good sense. The unused memory is sitting idle, so using it as a cache to speed-up the processes is a really good strategy. This helps to make a huge difference for the processes that are I/O bound (mainly when processes have lot's of IO read operations). The article Linux Page Cache Basics provides good text as entry point to learn more about the kernel cache.

The kernel will tend repurpose the "free" memory for caches and buffers. As mentioned, this is very good feature and helps all processes to run faster. Once the memory pressure starts growing in the namespace, meaning the processes running in the namespace need more memory, the kernel will start shrinking the cache by finding the least used regions and releasing them so they can be allocated to the processes. This is described in the kernel documentation.

The kernel will not rush to release the cache unless there is a memory pressure. So unless there is a need for the memory, once allocated to the cache, the kernel will keep it. This also is a very good strategy from kernel perspective since it avoids additional work for releasing the cache and potentially repopulating it in future.

So, unless we talk about a defect in the kernel or in a kernel module, in general, the memory being used by the buffers/cache is a good thing and not an issue.

By using the sync && echo 3 > /proc/sys/vm/drop_caches which is explained in the kernel documentation section Documentation for /proc/sys/vm/ the system is forced to drop as much as possible cache objects which will lead toward I/O performance issues until the cache is repopulated. This is useful for debugging and profiling, but not as a good practice in a normal production. Once cache is cleaned, the kernel will need to repopulate (since there are other processes who really need this cache), which leads to the additional IO.

Managing the memory

Docker provides mechanism to manage how the container will use the memory and the limits associated with the container. You can check more in docker's documentation Runtime options with Memory, CPUs, and GPUs

Unless there is a special use case which demands special handing of the memory, the features that Docker provides for resource management are more than sufficient to assure smooth operations of the containers.

Troubleshooting the memory

Sometimes we can deal with memory leak problems, either by the kernel / kernel modules or by the applications in the container.

First step in troubleshooting is to identify who is really using the memory and how it's growing. To identify the problematic code, we need to troubleshoot the memory allocation and to understand if there are regions that are constantly growing and are not released when the memory pressure is reached. Apart from the free and ps the better information about the memory lifecycle can be found on the sysfs.

The sysfs exposes internals of the kernel as a file system for reading and sending signals. The /proc/sys/vm is also part of the same mechanism known as procfs. The /sys/fs/cgroup/memory is the folder where the information and the statistics about memory allocation. Docker creates it's own cgroup called Docker, so the statistics of all docker containers will be found in /sys/fs/cgroup/memory/docker/. Further more, docker will create a cgroup for for each container under /sys/fs/cgroup/memory/docker/<<container id>>.

To find the kernel statistics about the memory usage of a container you can inspect the files in the sysfs as follows:

• /sys/fs/cgroup/memory/docker/<<container id>>/memory.stat statistics about the memory usage (similar to free but with more details
• /sys/fs/cgroup/memory/docker/<<container id>>/memory.limit_in_bytes current set limits in bytes for the container
• /sys/fs/cgroup/memory/docker/<<container id>>/memory.kmem.slabinfo regions of the kernel memory and the usage for the cache/buffer objects