Kubespray disable 'swapoff' command failed with returning 'non-zero return code'

I divided this answer on 2 parts:

• TL;DR Why Kubespray fails on swapoff -a
• How to install Kubernetes with Kubespray on LXC containers

TL;DR

Kubespray fails because he gets non exit zero code (255) when running swapoff -a.

A non-zero exit status indicates failure. This seemingly counter-intuitive scheme is used so there is one well-defined way to indicate success and a variety of ways to indicate various failure modes.

Gnu.org: Exit Status

Even if you set limits.memory.swap: "false" in the profile associated with the containers it will still produce this error.

There is a workaround for it by disabling swap in your host system. You can do it by:

• $ swapoff -a
• delete line associated with swap in /etc/fstab
• $ reboot

After that your container should produce zero exit code when issuing$ swapoff -a

How to install Kubernetes with Kubespray on LXC containers

Assuming that you created your lxc containers and have full ssh access to them, there are still things to take into consideration before running kubespray.

I ran kubespray on lxc containers and stumbled upon issues with:

• storage space
• docker packages
• kmsg
• kernel modules
• conntrack

Storage space

Please make sure you have enough storage within your storage pool as lack of it will result in failure to provision the cluster. Default storage pool size could be not big enough to hold 5 nodes.

Docker packages

When provisioning the cluster please make sure that you have the newest kubespray version available as the older ones had an issue with docker packages not compatible with each other.

Kmsg

The /dev/kmsg character device node provides userspace access to the kernel's printk buffer.

Kernel.org: Documentation: dev-kmsg

By default kubespray will fail to provision the cluster when the /dev/kmsg is not available on the node (lxc container).

/dev/kmsg is not available on lxc container and this will cause a failure of kubespray provisioning.

There is a workaround for it. In each lxc container run:

# Hack required to provision K8s v1.15+ in LXC containersmknod /dev/kmsg c 1 11chmod +x /etc/rc.d/rc.localecho 'mknod /dev/kmsg c 1 11' >> /etc/rc.d/rc.local

Github.com: Justmeandopensource: lxd-provisioning: bootstrap-kube.sh

I tried other workarounds like:

• add lxc.kmsg = 1 to /etc/lxc/default.conf - deprecated
• running echo 'L /dev/kmsg - - - - /dev/console' > /etc/tmpfiles.d/kmsg.conf inside the container and then restarting is causing the systemd-journald to sit at 100% usage of a core.

Kernel modules

The LXC/LXD system containers do not load kernel modules for their own use. What you do, is get the host it load the kernel module, and this module could be available in the container.

Linuxcontainers.org: How to add kernel modules to LXC container

Kubespray will check if certain kernel modules are available within your nodes.

You will need to add following modules on your host:

• ip_vs
• ip_vs_sh
• ip_vs_rr
• ip_vs_wrr

You can add above modules with $ modprobe MODULE_NAME or follow this link: Cyberciti.biz: Linux how to load a kernel module automatically.

Conntrack

You will need to install conntrack and load a module named nf_conntrack:

• $ apt install conntrack -y
• modprobe nf_conntrack

Without above commands kubespray will fail on step of checking the availability of conntrack.

With this change in place you should be able to run Kubernetes cluster with kubespray within lxc environment and get output of nodes similar to this:

root@k8s1:~# kubectl get nodes -o wideNAME   STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIMEk8s1   Ready    master   14h   v1.18.2   10.224.47.185   <none>        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7k8s2   Ready    master   14h   v1.18.2   10.224.47.98    <none>        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7k8s3   Ready    <none>   14h   v1.18.2   10.224.47.46    <none>        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7k8s4   Ready    <none>   14h   v1.18.2   10.224.47.246   <none>        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7