cannot schedule kubernetes pods with request for nvidia.com/gpu
Have you installed NVIDIA plugin in K8S?
kubectl create -f nvidia.io/device-plugin.yml
Some devices are too old and cannot be healthchecked.So this option must be disabled:
containers: - image: nvidia/k8s-device-plugin:1.9 name: nvidia-device-plugin-ctr env: - name: DP_DISABLE_HEALTHCHECKS value: "xids"
Take a look at:
- Device plugin: https://kubernetes.io/docs/concepts/cluster-administration/device-plugins/
- NVIDIA github: https://github.com/NVIDIA/k8s-device-plugin