Spark classpath in HDFS
From the SparkContext documentation:
def addJar(path: String):Unit
Adds a JAR dependency for all tasks tobe executed on this SparkContext in the future. The path passed can beeither a local file, a file in HDFS (or other Hadoop-supportedfilesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file onevery worker node.
So I think it is enough to just add this in your sparkContext initialization:
sc.addJar("hdfs://your/path/to/whatever.jar")If you want to add just a file, there is a relevant addFile() method.
See docs for more.