How can I get CoreDNS to resolve on my Raspberry Pi Kubernetes cluster?

1. Create a simple Pod to use as a test environment for DNS diagnosing:

apiVersion: v1kind: Podmetadata:  name: dnsutils  namespace: defaultspec:  containers:  - name: dnsutils    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3    command:      - sleep      - "3600"    imagePullPolicy: IfNotPresent  restartPolicy: Always

kubectl apply -f dnsutils.yaml

2. Check the status of Pod

$ kubectl get pods dnsutilsNAME      READY     STATUS    RESTARTS   AGEdnsutils   1/1       Running   0          <some-time>

Once that Pod is running, you can exec nslookup in that environment. If you see something like the following, DNS is working correctly.

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.defaultServer:    10.0.0.10Address 1: 10.0.0.10Name:      kubernetes.defaultAddress 1: 10.0.0.1

If the nslookup command fails, check the following:

3. Take a look inside the resolv.conf file.

kubectl exec -ti dnsutils -- cat /etc/resolv.conf

Verify that the search path and name server are set up like the following (note that search path may vary for different cloud providers):

search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internalnameserver 10.0.0.10options ndots:5

Errors such as the following indicate a problem with the CoreDNS (or kube-dns) add-on or with associated Services:

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.defaultServer:    10.0.0.10Address 1: 10.0.0.10nslookup: can't resolve 'kubernetes.default'ORServer:    10.0.0.10Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.localnslookup: can't resolve 'kubernetes.default'

4. Check if the DNS pod is running

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dnsNAME                       READY     STATUS    RESTARTS   AGE...coredns-7b96bf9f76-5hsxb   1/1       Running   0           1hcoredns-7b96bf9f76-mvmmt   1/1       Running   0           1h...

5. Check for errors in the DNS podHere is an example of a healthy CoreDNS log:

$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns.:532018/08/15 14:37:17 [INFO] CoreDNS-1.2.22018/08/15 14:37:17 [INFO] linux/amd64, go1.10.3, 2e322f6CoreDNS-1.2.2linux/amd64, go1.10.3, 2e322f62018/08/15 14:37:17 [INFO] plugin/reload: Running configuration MD5 = 24e6c59e83ce706f07bcc82c31b1ea1c

6. Verify that the DNS service is up by using the kubectl get service command.

$ kubectl get svc --namespace=kube-systemNAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE...kube-dns     ClusterIP   10.0.0.10      <none>        53/UDP,53/TCP        1h...

7. You can verify that DNS endpoints are exposed by using the kubectl get endpoints command.

$ kubectl get endpoints kube-dns --namespace=kube-systemNAME       ENDPOINTS                       AGEkube-dns   10.180.3.17:53,10.180.3.17:53    1h

8. You can verify if queries are being received by CoreDNS by adding the log plugin to the CoreDNS configuration (aka Corefile). The CoreDNS Corefile is held in a ConfigMap named coredns. To edit it, use the command:

$ kubectl -n kube-system edit configmap coredns

Then add log in the Corefile section per the example below:

apiVersion: v1kind: ConfigMapmetadata:  name: coredns  namespace: kube-systemdata:  Corefile: |    .:53 {        log        errors        health        kubernetes cluster.local in-addr.arpa ip6.arpa {          pods insecure          upstream          fallthrough in-addr.arpa ip6.arpa        }        prometheus :9153        forward . /etc/resolv.conf        cache 30        loop        reload        loadbalance    }

After saving the changes, it may take up to minute or two for Kubernetes to propagate these changes to the CoreDNS pods.Next, make some queries and view the logs per the sections above in this document. If CoreDNS pods are receiving the queries, you should see them in the logs.

Here is an example of a query in the log:

.:532018/08/15 14:37:15 [INFO] CoreDNS-1.2.02018/08/15 14:37:15 [INFO] linux/amd64, go1.10.3, 2e322f6CoreDNS-1.2.0linux/amd64, go1.10.3, 2e322f62018/09/07 15:29:04 [INFO] plugin/reload: Running configuration MD5 = 162475cdf272d8aa601e6fe67a6ad42f2018/09/07 15:29:04 [INFO] Reloading complete172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s

As pointed out in the comments: The configuration of kubeadm seems fine.
Your pods have the correct /etc/resolv.conf and they should work.

It's pretty hard to clarily determine the problem - many things can be happend here.
My guess: There something not right with ufw.
You can easily proof it: Disable ufw on all nodes (with ufw disable).

I'm not hundred percent sure which ports are needed. I'm using iptables for my single node k8s and at the start I had many problems FORWARD vs INPUT rules. In docker all ports are forwarded.
So I guess there is something wrong with FORWARD-rules and/or the dns-ports (53/udp and 53/tcp).

Good luck.