Create JVM heapdump when K8s healthcheck restarts the pod - no OOM occur Create JVM heapdump when K8s healthcheck restarts the pod - no OOM occur kubernetes kubernetes

Create JVM heapdump when K8s healthcheck restarts the pod - no OOM occur


The question is, whether the preStop hook is fired when the pod isrestarted because of health check failure?

Yes. As per the definition, perStop hook runs immediately before a container is terminated due to an API request or management event such as liveness probe failure, preemption, resource contention and others.


Should I use preStop hook to capture Java Heap Dump before podtermination?

Yes. But you need to be careful, a call to the preStop hook fails if the container is already in terminated or completed state. When the pod is terminated, it waits for default 30 second grace period (with additional 2 seconds if PerStop hook is not completed) before sending KILL signal. If the preStop hook needs longer to complete than the default grace period allows, you must modify terminationGracePeriodSeconds to suit this.


Any more elegant solution to this?

Not I am aware of. I guess by adding an empty dir volume to the pod, and configuring the JVM to do the heap dumps to that directory command: ["java", "-XX:+HeapDumpOnOutOfMemoryError", "-XX:HeapDumpPath=/dumps/oom.bin", "-jar", "yourapp.jar"] should work.

Why the above solution will work?

When kubernetes kills your container because it is not responding to the health check, the kubernetes will just restart the container, but it will not reschedule the pod, so it will not move it to another node. Hence the empty dir volume is not deleted until the pod is moved to another node. Hence when the container is restarted, the new container will mount the same empty dir, which will contain the heap dump from the previous run. So you can kubectl cp those files at any time after the event. There might be other challenges to copy the heap dump files but they are solvable. Check this for more info.