Are the container in a kubernetes pod part of same cgroup? Are the container in a kubernetes pod part of same cgroup? docker docker

Are the container in a kubernetes pod part of same cgroup?


Cgroups

Container in a pod share part of cgroup hierarchy but each container get's it's own cgroup. We can try this out and verify ourself.

  1. Start a multi container pod.
# cat mc2.yamlapiVersion: v1kind: Podmetadata:  name: two-containersspec:  restartPolicy: Never  containers:  - name: container1    image: ubuntu    command: [ "/bin/bash", "-c", "--" ]    args: [ "while true; do sleep 30; done;" ]  - name: container2    image: ubuntu    command: [ "/bin/bash", "-c", "--" ]    args: [ "while true; do sleep 30; done;" ]
# kubectl apply -f mc2.yamlpod/two-containers created
  1. Find the process cgroups on the host machine
# ps -ax | grep while | grep -v grep19653 ?        Ss     0:00 /bin/bash -c -- while true; do sleep 30; done;19768 ?        Ss     0:00 /bin/bash -c -- while true; do sleep 30; done;
# cat /proc/19653/cgroup12:hugetlb:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe199711:memory:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe199710:perf_event:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19979:freezer:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19978:cpuset:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19977:net_cls,net_prio:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19976:cpu,cpuacct:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19975:blkio:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19974:pids:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19973:devices:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19972:rdma:/1:name=systemd:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19970::/
# cat /proc/19768/cgroup12:hugetlb:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce8976511:memory:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce8976510:perf_event:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897659:freezer:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897658:cpuset:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897657:net_cls,net_prio:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897656:cpu,cpuacct:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897655:blkio:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897654:pids:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897653:devices:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897652:rdma:/1:name=systemd:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/e10fa18a63cc26de27f3f79f46631cd814efa3ef7c2f5ace4b84cf5abce897650::/

As you can see the containers in the pods share the cgroup hierarchy until /kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011 and then they get their own cgroup. (These containers are under besteffort cgroup because we have not specified the resource requests)

Another clue that containers run in their own cgroup is that kubernetes allows you to set resource requests at the container level.

You can also find the cgroups of the container by logging into the container and viewing /proc/self/cgroup file. (This may not work in recent versions of kubernetes if cgroup namespace is enabled)

# kubectl exec -it two-containers -c container2 bash# root@two-containers:# cat /proc/self/cgroup12:hugetlb:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe199711:memory:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe199710:perf_event:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19979:freezer:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19978:cpuset:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19977:net_cls,net_prio:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19976:cpu,cpuacct:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19975:blkio:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19974:pids:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19973:devices:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19972:rdma:/1:name=systemd:/kubepods/besteffort/poda9c80282-3f6b-4d5b-84d5-a137a6668011/ed89697807a981b82f6245ac3a13be232c1e13435d52bc3f53060d61babe19970::/

Namespaces

Containers in pod also share network and IPC namespaces by default.

# cd /proc/19768/ns/# /proc/19768/ns# ls -lrttotal 0lrwxrwxrwx 1 root root 0 Jul  4 01:41 uts -> uts:[4026536153]lrwxrwxrwx 1 root root 0 Jul  4 01:41 user -> user:[4026531837]lrwxrwxrwx 1 root root 0 Jul  4 01:41 pid_for_children -> pid:[4026536154]lrwxrwxrwx 1 root root 0 Jul  4 01:41 pid -> pid:[4026536154]lrwxrwxrwx 1 root root 0 Jul  4 01:41 net -> net:[4026536052]lrwxrwxrwx 1 root root 0 Jul  4 01:41 mnt -> mnt:[4026536152]lrwxrwxrwx 1 root root 0 Jul  4 01:41 ipc -> ipc:[4026536049]lrwxrwxrwx 1 root root 0 Jul  4 01:41 cgroup -> cgroup:[4026531835]
# cd /proc/19653/ns# /proc/19653/ns# ls -lrttotal 0lrwxrwxrwx 1 root root 0 Jul  4 01:42 uts -> uts:[4026536150]lrwxrwxrwx 1 root root 0 Jul  4 01:42 user -> user:[4026531837]lrwxrwxrwx 1 root root 0 Jul  4 01:42 pid_for_children -> pid:[4026536151]lrwxrwxrwx 1 root root 0 Jul  4 01:42 pid -> pid:[4026536151]lrwxrwxrwx 1 root root 0 Jul  4 01:42 net -> net:[4026536052]lrwxrwxrwx 1 root root 0 Jul  4 01:42 mnt -> mnt:[4026536149]lrwxrwxrwx 1 root root 0 Jul  4 01:42 ipc -> ipc:[4026536049]lrwxrwxrwx 1 root root 0 Jul  4 01:42 cgroup -> cgroup:[4026531835]

As you can see the containers share the network and IPC namespaces. You can also make the container share pid namespace using shareProcessNamespace field in the pod spec.

https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace


cgroup:[4026531835] is same for both the containers. Is this(cgroup namespace) different from the cgroups they (containers) are part of.

cgroups limits the resources(cpu, memory etc) which a process(or group of processes) can use.

namespaces isolate and limit the visibility a process(or a group of processes) has over system resources like network, process trees etc. There are different namespace groups like network, IPC etc. One of such namespace is cgroup namespace. Using cgroup namespace you can limit the visibility of other cgroups from a process(or group of processes)

cgroup namespace virtualises the view of a process's cgroups. Currently if you try cat /proc/self/cgroup from within the container, you would be able to see the full cgroup hierarchy starting from the global cgroup root. This can be avoided using cgroup namespaces and is available from kubernetes v1.19. Docker also supports this from version 20.03. When cgroup namespace is used while creating the container, you would see the cgroup root as / inside the container instead of seeing the global cgroups hierarchy.

https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html