Authentication for Spark standalone cluster
there are 2 parts to enable support of authentication:
- setting the secret on the master an all the slaves
- using the same secret when submitting jobs to the cluster
master and slaves
on each server in your cluster, add the following config to conf/spark-defaults.conf:
spark.authenticate.secret SomeSecretKeysubmitting jobs
when you initialize the spark context, you should add the same config to it as well, ie:
val conf = new SparkConf() .set("spark.authenticate.secret", "SomeSecretKey")val sc = new SparkContext(conf)or if you are using SparkSession:
val spark = SparkSession.builder() .conf("spark.authenticate.secret", "SomeSecretKey") .getOrCreate()