Problem:

Looking at the pod details for logstash, I was able to identify the issue.. I an entry similar to the following.

I0414 19:41:24.402257    3338 prober.go:104] Liveness probe for "mypod:mycontainer" failed (failure): Get http://10.168.0.3:80/: dial tcp 10.168.0.3:80: connection refused

It specified a "connection refused" for liveness probe, and after 50~60 seconds of uptime restarted the pod.

Cause:

looking at the liveness probe in the helm chart Values.yaml it shows the following settings.

...livenessProbe:  httpGet:    path: /    port: monitorinitialDelaySeconds: 20# periodSeconds: 30# timeoutSeconds: 30# failureThreshold: 6# successThreshold: 1...

Only InitialDelaySeconds is set, so the others should be Kubernetes defaults as shown here to the following.

# periodSeconds: 10# timeoutSeconds: 1# failureThreshold: 1# successThreshold: 3

This indicates the following give or take a few seconds:

+------+-----------------------------+| Time |            Event            |+------+-----------------------------+| 0s   | Container created           || 20s  | First liveness probe        || 21s  | First liveness probe fails  || 31s  | Second liveness probe       || 32s  | Second liveness probe fails || 42s  | Third liveness probe        || 43s  | Third liveness probe fails  || 44s  | Send SIGTERM to application |+------+-----------------------------+

Solution:

After some troubleshooting to find the correct InitialDelaySeconds value, I put the following into my override.yaml file to fix the issue.

livenessProbe:  initialDelaySeconds: 90

It seems that depending on the plugins being used, Logstash may not respond to HTTP requests for upwards of 100s.