Elasticsearch for spark 3.0
Spark 3.0.0 relies on Scala 2.12, which is not yet supported by Elasticsearch-hadoop. This and a few further issues prevent us using Spark 3.0.0 together with Elasticsearch. If you want to compile it yourself, there is a pull-request on elasticsearch-hadoop (https://github.com/elastic/elasticsearch-hadoop/pull/1308) which should at least allow using scala 2.12. Not sure if it will fix the other issues as well.
It is not official for now, but you can compile the dependency onhttps://github.com/elastic/elasticsearch, the steps are
- git clone https://github.com/elastic/elasticsearch.git
- cd elasticsearch-hadoop/
- vim ~/.bashrc
- export JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
- source ~/.bashrc
- ./gradlew elasticsearch-spark-30:distribution --console=plain
and finally you can find .jar package in folder: "elasticsearch-hadoop\spark\sql-30\build\distributions", elasticsearch-spark-30_2.12-8.0.0-SNAPSHOT.jar is the es packages
It's officially released for spark 3.0
Enhancements:https://www.elastic.co/guide/en/elasticsearch/hadoop/7.12/eshadoop-7.12.0.html
Maven Repository:https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-30_2.12/7.12.0