Kubernetes eviction manager evicting control plane pods to reclaim ephemeral storage

I had this same problem and solved it by changing the threshold for evictionHard.

Looking at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf I have:

[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamicallyEnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.EnvironmentFile=-/etc/default/kubeletExecStart=ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

so I see my config file for kubelet is /var/lib/kubelet/config.yaml

Opening that I changed evitionHard settings to be (I think they were 10 or 15% before):

...evictionHard:  imagefs.available: 1%  memory.available: 100Mi  nodefs.available: 1%  nodefs.inodesFree: 1%...

There is also the --experimental-allocatable-ignore-eviction (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) setting which should completely disable eviction.

This is because of your kubelet config setting for eviction nodefs and imagefs % too high, set it lower, then the issues will be resolved:Modify config in /var/lib/kubelet/config.yaml

Find out the section evction and set the percentage lower like this:

 evictionHard:   imagefs.available: 1%   memory.available: 100Mi   nodefs.available: 1%   nodefs.inodesFree: 1%