Cert-manager fails on kubernetes with webhooks

If you experience this problem, check the logs of CoreDNS(Or KubeDNS) and you may see lots of errors related to contacting services. Unfortunately, I no longer have the errors.But this is how I figured out that my network setup was invalid.

I'm using Calico(Will apply for other networks as well) and its network was not set to the same as the POD_CIDR network that I initialized my Kubernetes with.

Example1. Set up K8:

kubeadm init --pod-network-cidr=10.244.0.0/16

2. Configure Calico.yaml:

   - name: CALICO_IPV4POOL_CIDR  value: "10.244.0.0/16"

I also tried cert-manager v0.8.0 a very similar setup on Ubuntu 18.04 and k8s v1.14.1 and I began to get the same error when i tore down cert-manager using kubectl delete and reinstalled it, after experiencing some network issues on the cluster.

I stumbled on a solution that worked. On the master node, simply restart the apiserver container:

$ sudo docker ps -a | grep apiserveraf99f816c7ec        gcr.io/google_containers/kube-apiserver@sha256:53b987e5a2932bdaff88497081b488e3b56af5b6a14891895b08703129477d85               "/bin/sh -c '/usr/loc"   15 months ago       Up 19 hours                                     k8s_kube-apiserver_kube-apiserver-ip-xxxxxc_040f3a18050c3        gcr.io/google_containers/pause-amd64:3.0                                                                                      "/pause"                 15 months ago       Up 15 months                                    k8s_POD_kube-apiserver-ip-xxxc_0$ sudo docker restart af99f816c7ecaf99f816c7ec$ 

Then try applying the test-resources.yaml again:

$ kubectl apply -f test-resources.yamlnamespace/cert-manager-test unchangedissuer.certmanager.k8s.io/test-selfsigned createdcertificate.certmanager.k8s.io/selfsigned-cert created

If that does not work, this github issue mentions that the master node might need firewall rules to be able to reach the cert-manager-webhook pod. The exact steps to do so will depend on which cloud platform you are on.