Debugging uWSGI in kubernetes

Debugging in kubernetes is not very different from debugging outside, there's just some concepts that need to be overlaid for the kubernetes world.

A Pod in kubernetes is what you would conceptually see as a host in the VM world. Every container running in a Pod will see each others services on localhost. From there, a Pod to anything else will have a network connection involved (even if the endpoint is node local). So start testing with services on localhost and work your way out through pod IP, service IP, service name.

Some complexity comes from having the debug tools available in the containers. Generally containers are built slim and don't have everything available. So you either need to install tools while a container is running (if you can) or build a special "debug" container you can deploy on demand in the same environment. You can always fall back to testing from the cluster nodes which also have access.

Where you have python available you can test with uswgi_curl

pip install uwsgi-toolsuwsgi_curl hostname:port /path

Otherwise nc/curl will suffice, to a point.

Pod to localhost

First step is to make sure the container itself is responding. In this case you are likely to have python/pip available to use uwsgi_curl

kubectl exec -ti my-app-XXXX-XXXX shnc -v localhost 5000uwsgi_curl localhost:5000 /path

Pod to Pod/Service

Next include the kubernetes networking. Start with IP's and finish with names.

Less likely to have python here, or even nc but I think testing the environment variables is important here:

kubectl exec -ti nginx-XXXX-XXXX shnc -v my-app-pod-IP 5000nc -v my-app-service-IP 5000nc -v my-app-service-name 5000echo $APP_SERVERecho $FK_SERVER_NAMEnc -v $APP_SERVER 5000# or uwsgi_curl $APP_SERVER:5000 /path

Debug Pod to Pod/Service

If you do need to use a debug pod, try and mimic the pod you are testing as much as possible. It's great to have a generic debug pod/deployment to quickly test anything, but if that doesn't reveal the issue you may need to customise the deployment to mimic the pod you are testing more closely.

In this case the environment variables play a part in the connection setup, so that should be emulated for a debug pod.

Node to Pod/Service

Pods/Services will be available from the cluster nodes (if you are not using restrictive network policies) so usually the quick test is to check Pods/Services are working from there:

nc -v <pod_ip> <container_port>nc -v <service_ip> <service_port>nc -v <service__dns> <service_port>

In this case:

nc -v <my_app_pod_ip> 5000nc -v <my_app_service_ip> 5000nc -v my-app.svc.<namespace>.cluster.local 5000