How can I debug why a Kubernetes load balancer service isn't responding on a port?

Here is the primary debugging document for Services:

http://kubernetes.io/docs/user-guide/debugging-services/

LoadBalancer creates an external resource. What exactly that resource is depends on your Cloud Provider - some of them don't support it at all (in this case, you might want to try NodePort instead).

Both Google and Amazon support external load balancers.

Overall, when asking these questions it's extremely helpful to know if you are running on Google Container Engine, Google Compute Engine, Amazon Web Services, Digital Ocean, Vagrant, or whatever, because the answer depends on that. Showing all your configs and all your existing Kubnernetes resources (kubectl get pods, kubectl get services) along with your Dockerfiles or which images you are using will also help.

For Google (GKE or GCE), you would verify the load balancer exists:

gcloud compute forwarding-rules list

The external load balancer will map port 80 to an arbitrary Node, but then the Kubernetes proxy will map that to an ephemeral port on the correct node that actually has a Pod with that label, then it will map to the container port. So you have to figure out which step along the way isn't working. Unfortunately all those kube-proxy and iptables jumps are quite difficult to follow, so usually I would first double check all my Pods exist and have labels that match the selector of the Service. I would double check that my container is exposing the right port, I am using the right name for the port, etc. You might want to create some other Pods that just make calls to the Service (using the environment variables or KubeDNS, see the Kubernetes service documentation if you don't know what I'm referring to) and verify it's accessible internally before debugging the load balancer.

Some other good debugging steps:

Verify that your Kubernetes Service exists:

kubectl get serviceskubectl get pods

Check your logs of your pod

kubectl logs <pod name>

Check that your service is created internally by printing the environment variable for it

kubectl exec <pod name> -- printenv GUESTBOOK_SERVICE_HOST

try creating a new pod and see if the service can be reached internally through GUESTBOOK_SERVICE_HOST and GUESTBOOK_SERVICE_PORT.

kubectl describe pod <pod name>

will give the instance id of the pod, you can SSH to it and run Docker and verify your container is running, attach to it, etc. If you really want to get into the IP tables debugging, try

sudo iptables-save

The target port of the LoadBalancer needs to be the port ​INSIDE​ the container. So in my case I need to set the targetPort to 3000 instead of 80, on the LoadBalancer. Even though on the pod itself I have already mapped port 80 to 3000.

This is very counter intuitive to me, and not mentioned in all the LoadBalancer docs.