PySpark SparkSession Builder with Kubernetes Master
pyspark client mode works on Spark's latest version 2.4.0
This is how I did it (in Jupyter lab):
import osos.environ['PYSPARK_PYTHON']="/usr/bin/python3.6"os.environ['PYSPARK_DRIVER_PYTHON']="/usr/bin/python3.6"from pyspark import SparkContext, SparkConffrom pyspark.sql import SparkSessionsparkConf = SparkConf()sparkConf.setMaster("k8s://https://localhost:6443")sparkConf.setAppName("KUBERNETES-IS-AWESOME")sparkConf.set("spark.kubernetes.container.image", "robot108/spark-py:latest")sparkConf.set("spark.kubernetes.namespace", "playground")spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()sc = spark.sparkContextNote: I am running kubernetes locally on Mac with Docker Desktop.