Kubernetes dial tcp myIP:10250: connect: no route to host

It seems that a firewall is blocking ingress traffic from port 10250 on the 10.2.67.203 node.

You can open it by running the commands below (I'm assuming firewalld is installed or you can run the commands of the equivalent firewall module):

sudo firewall-cmd --add-port=10250/tcp --permanentsudo firewall-cmd --reloadsudo firewall-cmd --list-all  # you should see that port `10250` is updated

tl;dr; It looks like your cluster itself is fairly broken and should be repaired before looking at Ceph specifically

Get https://10.2.67.203:10250/containerLogs/ceph/ceph-mon-744f6dc9d6-mqwgb/ceph-mon?tailLines=5000&timestamps=true: dial tcp 10.2.67.203:10250: connect: no route to host

10250 is the port that the Kubernetes API server uses to connect to a node's Kubelet to retrieve the logs.

This error indicates that the Kubernetes API server is unable to reach the node. This has nothing to do with your containers, pods or even your CNI network. no route to host indicates that either:

1. The host is unavailable
2. A network segmentation has occurred
3. The Kubelet is unable to answer the API server

Before addressing issues with the Ceph pods I would investigate why the Kubelet isn't reachable from the API server.

After you have solved the underlying network connectivity issues I would address the crash-looping Calico pods (You can see the logs of the previously executed containers by running kubectl logs -n kube-system calico-node-dwm47 -p).

Once you have both the underlying network and the pod network sorted I would address the issues with the Kubernetes Dashboard crash-looping, and finally, start to investigate why you are having issues deploying Ceph.