Adding the below lines to the calico yaml did the magic.

Specify interface

        - name: IP_AUTODETECTION_METHOD          value: "interface=ens."

What Sanjay M. P. shared worked for me, however I want to clarify what caused the problem, and why the solution work with some more detail.

First of all, I am running an ubuntu env, so what Piknik shared does not work, firewalld is only on centos / rhel systems. Even still, ufw was disabled on all nodes.

I was able to narrow down the exact error I was receiving to cause this problem by doing a kubectl describe pod calico-node-*****. What I found was the calico BIRD service could not connect to peers. What also showed was the IP addresses the calico-node was trying to use to pair to for it's BGP peers. It was using the wrong interface, thereby wrong ips.

To define the problem for myself, all of my node host vms have multiple interfaces. If you don't explicitly specify which interface to use, calico "automatically" picks one, weather you want that interface or not.

The solution was to specify the specific interface when you build your calico overlay network in the calico.yaml file. Sanjay M. P. uses a regex, which MAY work if you have different named interfaces, however, as I am running Ubuntu Server, the string "ens" starts for all interfaces, so the same problem happens.

I have stripped out most of the calico.yaml file to show the exact location of where this setting should be (~line 675) Add the setting there, I also left the CALICO_IPV4POOL_CIDR as well as this setting needs to be set appropriately to the same subnet range specified on kubeadm initialization:

spec:  template:    spec:      containers:        - name: calico-node          image: calico/node:v3.14.2          env:            - name: CALICO_IPV4POOL_CIDR              value: "192.168.0.0/22"            - name: IP_AUTODETECTION_METHOD              value: "interface=ens224"

Unfortunately I did not find a way to roll back older configurations, so I just rebuilt the whole cluster, and redeployed the calico overlay (Thank god for VM snapshots).

kubeadm init your cluster.Then run kubectl create -f calico.yaml with the setting added to build out the overlay network.

Confirm overlay network is working

• run watch -n1 kube-system get pods -o wide, and then add your nodes. Make sure all calico-nodes being build on newly added kube nodes come up as "1/1 Running".
• Download and install calicoctl, and run calicoctl node status, make sure the correct network is being used for BGP.

You can read more about IP_AUTODETECTION_METHOD here.

Activate the network interface used by calico through firewall-cmd.

Recently, while operating a node using a virtual machine, the interface of the virtual machine was deactivated, causing the same phenomenon.