Spark (Java) to Elasticsearch Spark (Java) to Elasticsearch elasticsearch elasticsearch

Spark (Java) to Elasticsearch


Spark has executors and driver process. Executor runs in different node apart from driver node. Spark computes the rdd graph in various stages depending up on the transformations. And these stages have tasks that is executed on executors. So you need to pass the dependent jars to both executors and driver if you are using the library methods to compute rdd.

You should pass the dependent jars in --jars options in spark-submit

    spark-submit --jars $JARS \     --driver-class-path $JARS_COLON_SEP \     --class $CLASS_NAME $APP_JAR  

In your case it would be

    spark-submit --jars elasticsearch-hadoop-2.3.2.jar \    --master local[4]\     --driver-class-path elasticsearch-hadoop-2.3.2.jar \     --class "SimpleApp" target/simple-project-1.0.jar