Spark Kubernetes - FileNotFoundException when copying config files from driver to executors using --files or spark.files
I've figured it out. spark-submit sends a request to kubernetes master's api-server to create a driver pod. A configmap volume is mounted to the driver's pod at mountPath: /opt/spark/conf, which overrides my config files located at that path in the docker container.Workaround : editing /opt/spark/conf to /opt/spark/config in Dockerfile so that my configuration files are copied from the latter.