Spark 2.3 submit on Kubernetes error
You need to set ${SPARK_LOCAL_IP} environment variable to pod IP and pass to spark-submit this environment variable using --conf spark.driver.host=${SPARK_LOCAL_IP}.
See the spark docs for more information regarding these variables.
My ubuntu18.04 PC also encountered this problem, but it has been resolved. The general steps are as follows:
- Get the url through the
minikube dashboard -urland enter the dashboard. Search for the coredns service to start normally - Check if the coredns service log is caused by
Loop ... detected ... - If this problem is caused, vim /etc/resolv.conf,Modify nameserver 127.0.0.53 to nameserver 8.8.8.8.
- restart
minikube stop && minikube start
If coredns can't be started, you can check it out here: https://github.com/coredns/coredns/tree/master/plugin/loop