Spark Unable to load native-hadoop library for your platform
Steps to fix:
- download Hadoop binaries
- unpack to directory of your choice
- set
HADOOP_HOMEto point to that directory. - add
$HADOOP_HOME/lib/nativetoLD_LIBRARY_PATH.
- Download hadoop binary (link) and put it in your home directory(you can choose a different hadoop version if you like and change the next steps accordingly)
- Unzip the folder in your home directory using the following command.tar -zxvf hadoop_file_name
- Now add export HADOOP_HOME=~/hadoop-2.8.0 to your .bashrc file. Open a new terminal and try again.
Source: Install PySpark on ubuntu