load a local file to spark using sc.textFile() load a local file to spark using sc.textFile() hadoop hadoop

load a local file to spark using sc.textFile()


I checked all the dependencies and the environment variables again. The actual path "file:///home/..../.. .txt" would fetch the data from the local file system as the hadoop env.sh file has its default file system set to fs.defaultFs.If we leave the Spark-env.sh to its defaults without any change it takes the local file system when it encounters "file://..." and the hdfs when the path is "hdfs://.." If you specifically need any file system export HADOOP_CONF_DIR to the spark-env.shAnd it would support any file system supported by Hadoop.This was my observation. Any corrections or suggestions accepted. Thank you


Try changing

val inputFile = sc.textFile("file///C:/Users/swaapnika/Desktop/to do list")

to this:

val inputFile = sc.textFile("file:///Users/swaapnika/Desktop/to do list")

I'm also fairly new to hadoop and spark, but from what I gather, when running spark locally on Windows, the string file:/// when passed to sc.textFile already refers to C:\.


The file path you have defined is incorrect.

Try changing

sc.textFile("file///C:/Users/swaapnika/Desktop/to do list")

to

sc.textFile("file://C:/Users/swaapnika/Desktop/to do list")

or

sc.textFile("C:/Users/swaapnika/Desktop/to do list")