How to get history of Pods run on Kubernetes Node?
You can now use kube-state-metrics kube_pod_container_status_terminated_reason to detect OOM events
kube_pod_container_status_terminated_reason{reason="OOMKilled"}kube_pod_container_status_terminated_reason{container="addon-resizer",endpoint="http-metrics",instance="100.125.128.3:8080",job="kube-state-metrics",namespace="monitoring",pod="kube-state-metrics-569ffcff95-t929d",reason="OOMKilled",service="kube-state-metrics"}
We use Prometheus to monitor OOM events.
This expression should report the number of times that memory usage has reached the limits:
rate(container_memory_failcnt{pod_name!=""}[5m]) > 0FYI: this is the next best thing to proper docs, the code
Event history for your particular namespace, ordered by creationTimestamp:
kubectl get events -n YOURNAMESPACE -o wide --sort-by=.metadata.creationTimestampOr if you want to check the event history for all namespaces, ordered by creationTimestamp:
kubectl get events --all-namespaces -o wide --sort-by=.metadata.creationTimestamp