Memory usage discrepancy: cgroup memory.usage_in_bytes vs. RSS inside docker container
One thing I didn't see you check here is kernel memory. This is also accounted for in the memory.usage_in_bytes
figure, but doesn't appear in memory.stat
. You can find that by looking at /sys/fs/cgroup/memory/memory.kmem.usage_in_bytes
.
I saw a similar thing happening for one of our .NET core applications, once, and I couldn't figure out what exactly was happening (perhaps a memory leak in .NET core since it's unmanaged memory our app doesn't control).
Perhaps it's another breadcrumb for you. It would depend on your application whether that use was normal or not, but in terms of cgroups I believe kernel memory use is unconstrained by default.
It looks like it's using VSZ, Virtual Memory Size, not RSS. Kubernetes sees this:
% kubectl top pods -l app=myappNAME CPU(cores) MEMORY(bytes)myapp-7b4b4f84f8-fxbns 39m 7136Mi
ps inside the container, when summing the 5th column (VSZ), says:
meme@myapp-7b4b4f84f8-fxbns:/app# kb=$(ps aux | grep -v grep | grep -v 'ps aux' | grep -v bash | grep -v awk | grep -v RSS | awk '{print $5}' | awk '{s+=$1} END {printf "%.0f", s}'); mb=$(expr $kb / 1024); printf "Kb: $kb\nMb: $mb\n"Kb: 7032172Mb: 6867
VSZ is in the neighborhood of 7GB (doesn't match exactly, but it's close), whereas RSS says 665MB, so this makes me believe Kubernetes and /sys/fs/cgroup/memory/memory.usage_in_bytes
are using something along the lines of VSZ, at least in this case.
I don't know if you already find your answer or not but let me give you some information that may help.
cAdvisor extract many memory-related metrics. We will focus on:
container_memory_usage_bytes
-> gets the usage of the memory. From memory.usage_in_bytes file.container_memory_working_set_bytes
-> is calculated in cAdvisor. It is <=usage
. In other words,usage
=working_set_bytes
+total_inactive_file
container_memory_rss
-> equalstotal_rss
from memory.state file.
Now you know how those metrics are gathered, you need to know that when you use the
kubectl top pods
command, you get the value ofcontainer_memory_working_set_bytes
notusage
metric.so from your values:
5039Mi "working set fro kubectl command" ~= 5064 "from memory.usage file" - 28 "total_inactive_file from Memory section from docker's container stats API"
It is also worth to mention that when the value of
container_memory_usage_bytes
reaches to the limits, your pod will NOT get oom-killed. BUT ifcontainer_memory_working_set_bytes
orcontainer_memory_rss
reached to the limits, the pod will be killed.