Kubernetes Readiness probe failed: dial tcp 10.244.0.10:5000: connect: connection refused

TL;DR:

I've made some tests, your docker image and deployment seems really fine ,I was able to log into the pod, it was running and listening on the port.

• The reason why your readiness probe was returning Warning Unhealthy...: connection refused was because it was not given enough time for the pod to start.

I edited your deployment with the following lines:

          readinessProbe:            tcpSocket:              port: 5000            initialDelaySeconds: 300            periodSeconds: 30

Explanation:

• From Configuring Probes Documentation:

initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.

periodSeconds: How often (in seconds) to perform the probe. Default to 10s. Minimum value is 1s.

NOTE: During my tests I noticed that the pod takes about 5 minutes to be running, way longer than the default 10s, that's why I set it as 300 seconds.

Meaning that after 5 minutes the pod was serving on port 5000.

Add the initialDelaySeconds line to your deployment and you should be fine.

Here is my Reproduction:

• Edited Deployment:

apiVersion: v1kind: Servicemetadata:  name: keystone-apispec:  selector:    app: keystone  ports:    - protocol: TCP      port: 5000      targetPort: 5000      name: public    - protocol: TCP      port: 35357      targetPort: 35357      name: admin---apiVersion: apps/v1kind: Deploymentmetadata:  name: keystone  labels:    app: keystonespec:  replicas: 1  selector:    matchLabels:      app: keystone  template:    metadata:      labels:        app: keystone    spec:      containers:        - name: keystone          image: openio/openstack-keystone          readinessProbe:            tcpSocket:              port: 5000            initialDelaySeconds: 300            periodSeconds: 30          env:            - name: OS_IDENTITY_ADMIN_PASSWD              value: password            - name: IPADDR              valueFrom:                fieldRef:                  fieldPath: status.podIP          ports:            - containerPort: 5000              name: public            - containerPort: 35357              name: admin

• Create the resource and wait:

$ kubectl get pods  -wNAME                        READY   STATUS    RESTARTS   AGEkeystone-7fd895cfb5-kqnnn   0/1     Running   0          3m28subuntu                      1/1     Running   0          113mkeystone-7fd895cfb5-kqnnn   1/1     Running   0          5m4s

• After 5min4s the container was running 1/1 and I describe the pod:

$ kubectl describe pod keystone-586b8948d5-c4lpqName:         keystone-586b8948d5-c4lpqNamespace:    defaultPriority:     0Node:         minikube/192.168.39.39Start Time:   Mon, 20 Apr 2020 15:02:24 +0000Labels:       app=keystone              pod-template-hash=586b8948d5Annotations:  <none>Status:       RunningIP:           172.17.0.7IPs:  IP:           172.17.0.7Controlled By:  ReplicaSet/keystone-586b8948d5Containers:  keystone:    Container ID:   docker://8bc14d2b6868df6852967c4a68c997371006a5d83555c500d86060e48c549165    Image:          openio/openstack-keystone    Image ID:       docker-pullable://openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320    Ports:          5000/TCP, 35357/TCP    Host Ports:     0/TCP, 0/TCP    State:          Running      Started:      Mon, 20 Apr 2020 15:02:26 +0000    Ready:          True    Restart Count:  0    Readiness:      tcp-socket :5000 delay=300s timeout=1s period=30s #success=1 #failure=3    Environment:      OS_IDENTITY_ADMIN_PASSWD:  password      IPADDR:                     (v1:status.podIP)    Mounts:      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kcw8c (ro)Conditions:  Type              Status  Initialized       True   Ready             True   ContainersReady   True   PodScheduled      True Volumes:  default-token-kcw8c:    Type:        Secret (a volume populated by a Secret)    SecretName:  default-token-kcw8c    Optional:    falseQoS Class:       BestEffortNode-Selectors:  <none>Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s                 node.kubernetes.io/unreachable:NoExecute for 300sEvents:  Type    Reason     Age        From               Message  ----    ------     ----       ----               -------  Normal  Scheduled  <unknown>  default-scheduler  Successfully assigned default/keystone-586b8948d5-c4lpq to minikube  Normal  Pulling    7m12s      kubelet, minikube  Pulling image "openio/openstack-keystone"  Normal  Pulled     7m11s      kubelet, minikube  Successfully pulled image "openio/openstack-keystone"  Normal  Created    7m11s      kubelet, minikube  Created container keystone  Normal  Started    7m11s      kubelet, minikube  Started container keystone

As you can see now there is no error.

Let me know in the comments if you have any doubt.

I checked with the docker instructions, and it did run, but somehow the app was not getting deployed correctly. To check with docker it is straight forward because they are using host network, so from you host you can do netstat, and you will see that there is nothing listening on port 5000.

I accessed the container and ran the init script (keystone-v3.sh) again, and it started to work. I did the same on kubernetes Deployment, and it worked too.

So, this is your functional Deployment:

apiVersion: apps/v1kind: Deploymentmetadata:  name: keystone  labels:    app: keystonespec:  replicas: 1  selector:    matchLabels:      app: keystone  template:    metadata:      labels:        app: keystone    spec:      containers:        - name: keystone          image: openio/openstack-keystone          command: ["./keystone-v3.sh"]     #<- you add this line          readinessProbe:            tcpSocket:              port: 5000          env:            - name: OS_IDENTITY_ADMIN_PASSWD              value: password            - name: IPADDR              valueFrom:                fieldRef:                  fieldPath: status.podIP          ports:            - containerPort: 5000              name: public            - containerPort: 35357              name: admin

~$ kubectl get poNAME                        READY   STATUS    RESTARTS   AGEalpine-786c6d498d-dsxfh     1/1     Running   1          11dcurler-755cc7cfff-fwz4g     1/1     Running   1          11dkeystone-6d997f4f8c-5kkxc   1/1     Running   0          26mnginx-6db489d4b7-jlhql      1/1     Running   1          11d~$ kubectl logs --tail 5 keystone-6d997f4f8c-5kkxc********************************************************************************STARTING test server keystone.server.wsgi.initialize_public_applicationAvailable at http://keystone-6d997f4f8c-5kkxc:5000/DANGER! For testing only, do not use in production********************************************************************************~$ 

Or you try to fix it from the image, but I guess that's not your repo right?

UPDATE

Check on this:

~$ sudo docker run -d --net=host -e IPADDR=192.168.56.102 openio/openstack-keystoneUnable to find image 'openio/openstack-keystone:latest' locallylatest: Pulling from openio/openstack-keystoneab5ef0e58194: Pull complete ca37595f2b63: Pull complete 878ef80688be: Pull complete Digest: sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320Status: Downloaded newer image for openio/openstack-keystone:latest703a05b8fdc8b7294895122b6f369a4d0a6b4582104ed360d6be68d012ea5b3c~$ netstat -tlpn | grep 5000NOTE: NOTHING LISTENING ON PORT 5000~$ sudo docker ps | grep openio703a05b8fdc8        openio/openstack-keystone   "/keystone-v3.sh"        34 seconds ago      Up 32 seconds                           quizzical_swartz~$ sudo docker exec -it 703a05b8fdc8 bash[root@v1-17-master /]# ls   anaconda-post.log  bin  dev  etc  home  keystone-v3.log  keystone-v3.sh  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var[root@v1-17-master /]# tail keystone-v3.sh openstack endpoint create --region "$OS_OBJECTSTORE_SERVICE_REGION" 'object-store' internal "$OS_OBJECTSTORE_URL_INTERNAL"openstack endpoint create --region "$OS_OBJECTSTORE_SERVICE_REGION" 'object-store' admin    "$OS_OBJECTSTORE_URL_ADMIN"# Demo useropenstack domain create "$OS_USER_DEMO_DOMAIN"openstack project create "$OS_USER_DEMO_PROJECT"openstack user create --password "$OS_USER_DEMO_PASSWD" --project "$OS_USER_DEMO_PROJECT" "$OS_USER_DEMO_USERNAME"openstack role add --user "$OS_USER_DEMO_USERNAME" --project "$OS_USER_DEMO_PROJECT" "$OS_USER_DEMO_ROLE"echo '> Starting Keystone public service ...'/usr/bin/keystone-wsgi-public --port 5000[root@v1-17-master /]# /usr/bin/keystone-wsgi-public --port 5000 &[1] 172[root@v1-17-master /]# exit~$ sudo netstat -tlpn | grep 5000tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      10207/python2