What is version library spark supported SparkSession What is version library spark supported SparkSession hadoop hadoop

What is version library spark supported SparkSession


you need both core and SQL artifacts

<repositories>    <repository>        <id>cloudera</id>        <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>    </repository></repositories><dependencies>    <dependency>        <groupId>org.apache.spark</groupId>        <artifactId>spark-core_2.11</artifactId>        <version>2.0.0-cloudera1-SNAPSHOT</version>    </dependency>    <dependency>        <groupId>org.apache.spark</groupId>        <artifactId>spark-sql_2.11</artifactId>        <version>2.0.0-cloudera1-SNAPSHOT</version>    </dependency></dependencies> 


You need Spark 2.0 to use SparkSession. It's available in Maven central snapshot repository as for now:

groupId = org.apache.sparkartifactId = spark-core_2.11version = 2.0.0-SNAPSHOT

The same version have to be specified for other Spark artifacts. Note, that 2.0 is still in beta and expected to be stable in about a month, AFAIK.

Update. Alternatively, you can use Cloudera fork of Spark 2.0:

groupId = org.apache.sparkartifactId = spark-core_2.11version = 2.0.0-cloudera1-SNAPSHOT

Cloudera repository has to be specified in your Maven repositories list:

<repository>   <id>cloudera</id>   <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url></repository>