Cannot launch SparkPi example on Kubernetes Spark 2.4.0

Spark on Kubernetes has a bug.

During Spark job submission to the Kubernetes cluster we first create Spark Driver Pod: https://github.com/apache/spark/blob/02c5b4f76337cc3901b8741887292bb4478931f3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L130 .

Only after that we create all other resources (eg.: Spark Driver Service), including ConfigMap: https://github.com/apache/spark/blob/02c5b4f76337cc3901b8741887292bb4478931f3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L135 .

We do that so to be able to set Spark Driver Pod as the ownerReference to all of those resources (which cannot be done before we create the owner Pod): https://github.com/apache/spark/blob/02c5b4f76337cc3901b8741887292bb4478931f3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L134.

It helps us to delegate the deletion of all those resources to the Kubernetes responsibility, which is useful for collecting the unused resources more easily in the cluster. All we need to cleanup in that case is just delete Spark Driver Pod. But there is a risk that Kubernetes will instantiate Spark Driver Pod creation before the ConfigMap is ready, which will cause your issue.

This is still true for 2.4.4.

I think the problem is mostly with the fact that my docker 'latest' tag was pointing at an image that was for the previous version of spark (v2.3.2). It seems that way the container receives parameters from spark-submit and kubernetes has changed a bit. My remaining problems launch spark pipelines seem to be with serviceAccounts (and probably belong in another question).