Rancher Kubernetes can't create persisten volume claim

For some reason, the DNS resolver of the kube-controller-manager pod on kubernet master was empty:

docker exec  kube-controller-manager cat /etc/resolv.conf# Generated by dhcpcd# /etc/resolv.conf.head can replace this line# /etc/resolv.conf.tail can replace this line

Since the host server resolv.confwas correct, I simply restarted the container:

docker restart kube-controller-manager

(an alternative ugly way would have been to edit the resolv.conf manually using docker restart kube-controller-manager, then do run the appropriate echo XXX >> /etc/resolv.conf ... bad idea )

Some other containers may have a similar issue on this node. This is a hacky way to identify and restart those containers:

cd  cd /var/lib/docker/containersls -1 "$(grep nameserver -L */resolv.conf)" | sed -e 's#/.*##' 0c10e1374644cc262c8186e28787f53e02051cc75c1f943678d7aeaa00e5d45070fd116282643406b72d9d782857bb7ec76dd85dc8a7c0a83dc7ab0e90d30966841def818a8b4df06a0d30b0b7a66b75a3b554fb5feffe78846130cdfeb39899ae356e26f1bf8fafe530d57d8c68a022a0ee0e13b4e177d3ad6d4e808d1b36dad593735a2f6d96bcab3addafcfe3d44b6d839d9f3775449247bdb8801e2e1692d9b0dfaa270d3f50169fb1aed064ca7a594583b9f71a111f653c71d704daf391

Restart affected containers:

cd /var/lib/docker/containers ; ls -1 $(grep nameserver -L */resolv.conf) | sed -e 's#/.*##' | xargs -n1 -r docker restart