Kubelet - failed to "CreatePodSandbox" for coredns; failed to set bridge addr: could not add ip addr to "cni0": permission denied

Holy Hand Grenade of Antioch! I finally fixed it! It only took me, what, about a bazillion years and a restless-night. Sweet Victory! Well... ehm. On to the solution.

I finally understand the comments by @Arghya Sadhu and @Piotr Malec and they were right. I didn't configure my CNI-plugin correctly. I am using Flannel as a network provider and they require a 10.244.0.0/16 subnet. In my crio-bridge.conf found in /etc/cni/net.d/ the default subnet was different (10.85.0.0/16 or something). I thought it would be enough to specify the CIDR on the kubeadm init command but I was wrong. You need to set the correct CIDR in the crio-bridge.conf and podman.conflist (or similar files in the directory). I also thought those files that were installed with CRI-O were configured with reasonable defaults and, to be honest, I didn't fully understand what they were for.

Also something strange happened: According to Flannel the subnet for CRI-O should be /16 but when I checked the logs with journalctl -u kubelet it mentioned a /24 subnet.

failed to set bridge addr: \"cni0\" already has an IP address different from 10.244.0.1/24"

So I had to change the subnet in crio.conf to /24 and it worked. I probably have to change the subnet in the podman.conflist too, but I am not sure.

Anyway, thanks to Arghya and Piotr for their help!

To setup a cluster with Calico network plugin and cri-o container runtime, I had to:

Add to /etc/crio/crio.conf

[crio.network]network_dir = "/etc/cni/net.d/"plugin_dirs = [    "/opt/cni/bin/",    "/usr/libexec/cni/",]

Add --cgroup-driver=systemd in /var/lib/kubelet/kubeadm-flags.env

KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --pod-infra-container-image=k8s.gcr.io/pause:3.5"

Restart kubelet and crio

systemctl daemon-reload && systemctl restart kubelet crio

Initialize cluster

kubeadm init --pod-network-cidr='10.85.0.0/16'

Install calico network plugin

kubectl create -f https://docs.projectcalico.org/manifests/calico.yaml