Spark streaming elasticsearch dependencies

Cascading and its dependencies are not available in Maven central but in their own repo (which es-hadoop cannot specify through its pom).

I resolved the problem by using elasticsearch-spark_2.10

http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/install.html

Sbt can't resolve some dependencies because they aren't part of the Maven repository.However, you can find them on clojars and conjars.You need to add the following lines so sbt can resolve them:

resolvers += "clojars" at "https://clojars.org/repo"resolvers += "conjars" at "http://conjars.org/repo"

Also, the dependency elasticsearch-hadoop "2.1.0" doesn't exist (yet ?), you should use "2.1.0.Beta4" (or whatever the latest version is when you read this)

Your sbt file should look like this:

name := "twitter-sparkstreaming-elasticsearch"version := "0.0.1"scalaVersion := "2.10.4"libraryDependencies ++= Seq(    "org.apache.spark" %% "spark-core" % "1.1.0",    "org.apache.spark" %% "spark-streaming" % "1.1.0",    "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta4")resolvers += "clojars" at "https://clojars.org/repo"resolvers += "conjars" at "http://conjars.org/repo"

This has been tested (with spark-core 1.3.1 and without spark-streaming but it should work for you).Hope it helps.

That is because cascading and its dependencies are not in Maven.You have to add resolvers to get them

Add this line to your build.sbt

resolvers += "conjars.org" at "http://conjars.org/repo"

Your build.sbt should look like this:

name := "twitter-sparkstreaming-elasticsearch"version := "0.0.1"scalaVersion := "2.10.4"// additional librarieslibraryDependencies ++= Seq(  "org.apache.spark" %% "spark-core" % "1.1.0",  "org.apache.spark" %% "spark-streaming" % "1.1.0",  "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0")resolvers += "conjars.org" at "http://conjars.org/repo"

NOTE: This issue has been raised and closed here https://github.com/elasticsearch/elasticsearch-hadoop/issues/304 with the same solution as above