How can kubernetes recover Pods stuck in Error or Terminating
This is typically related to the metadata.finalizers on your objects (pod, deployment, etc)
You can also read more about Foreground Cascading Deleting and how it uses metadata.finalizers.
If not it could be a networking issue, you could check the kubelet logs, typically:
journalctl -xeu kubelet You can also check the docker daemon logs, typically:
cat /var/log/syslog | grep dockerd
I had to restart all the nodes. I noticed one minion was slow and unresponsive, probably that one was the culprit. After restart all Terminating pods disappeared.
Removing the finalizers is a workaround by running the kubectl patch. This can happen to different type of resources like persistentvolume or deployment. More common to PV/PVC in my experience.
# for pods$ kubectl patch pod pod-name-123abc -p '{"metadata":{"finalizers":null}}' -n your-app-namespace# for pvc$ kubectl patch pvc pvc-name-123abc -p '{"metadata":{"finalizers":null}}' -n your-app-namespace