Memory usage discrepancy: cgroup memory.usage_in_bytes vs. RSS inside docker container

One thing I didn't see you check here is kernel memory. This is also accounted for in the memory.usage_in_bytes figure, but doesn't appear in memory.stat. You can find that by looking at /sys/fs/cgroup/memory/memory.kmem.usage_in_bytes.

I saw a similar thing happening for one of our .NET core applications, once, and I couldn't figure out what exactly was happening (perhaps a memory leak in .NET core since it's unmanaged memory our app doesn't control).

Perhaps it's another breadcrumb for you. It would depend on your application whether that use was normal or not, but in terms of cgroups I believe kernel memory use is unconstrained by default.

It looks like it's using VSZ, Virtual Memory Size, not RSS. Kubernetes sees this:

% kubectl top pods -l app=myappNAME                             CPU(cores)   MEMORY(bytes)myapp-7b4b4f84f8-fxbns           39m          7136Mi

ps inside the container, when summing the 5th column (VSZ), says:

meme@myapp-7b4b4f84f8-fxbns:/app# kb=$(ps aux | grep -v grep | grep -v 'ps aux' | grep -v bash | grep -v awk | grep -v RSS | awk '{print $5}' | awk '{s+=$1} END {printf "%.0f", s}'); mb=$(expr $kb / 1024); printf "Kb: $kb\nMb: $mb\n"Kb: 7032172Mb: 6867

VSZ is in the neighborhood of 7GB (doesn't match exactly, but it's close), whereas RSS says 665MB, so this makes me believe Kubernetes and /sys/fs/cgroup/memory/memory.usage_in_bytes are using something along the lines of VSZ, at least in this case.

I don't know if you already find your answer or not but let me give you some information that may help.

• cAdvisor extract many memory-related metrics. We will focus on:

  1. container_memory_usage_bytes -> gets the usage of the memory. From memory.usage_in_bytes file.

  2. container_memory_working_set_bytes -> is calculated in cAdvisor. It is <= usage. In other words, usage = working_set_bytes + total_inactive_file

  3. container_memory_rss -> equals total_rss from memory.state file.

• Now you know how those metrics are gathered, you need to know that when you use the kubectl top pods command, you get the value of container_memory_working_set_bytes not usage metric.

  so from your values:

  5039Mi "working set fro kubectl command" ~= 5064 "from memory.usage file" - 28 "total_inactive_file from Memory section from docker's container stats API"

• It is also worth to mention that when the value of container_memory_usage_bytes reaches to the limits, your pod will NOT get oom-killed. BUT if container_memory_working_set_bytes or container_memory_rss reached to the limits, the pod will be killed.