load a local file to spark using sc.textFile()
I checked all the dependencies and the environment variables again. The actual path "file:///home/..../.. .txt" would fetch the data from the local file system as the hadoop env.sh file has its default file system set to fs.defaultFs.If we leave the Spark-env.sh to its defaults without any change it takes the local file system when it encounters "file://..." and the hdfs when the path is "hdfs://.." If you specifically need any file system export HADOOP_CONF_DIR to the spark-env.shAnd it would support any file system supported by Hadoop.This was my observation. Any corrections or suggestions accepted. Thank you
Try changing
val inputFile = sc.textFile("file///C:/Users/swaapnika/Desktop/to do list")
to this:
val inputFile = sc.textFile("file:///Users/swaapnika/Desktop/to do list")
I'm also fairly new to hadoop and spark, but from what I gather, when running spark locally on Windows, the string file:///
when passed to sc.textFile
already refers to C:\
.
The file path you have defined is incorrect.
Try changing
sc.textFile("file///C:/Users/swaapnika/Desktop/to do list")
to
sc.textFile("file://C:/Users/swaapnika/Desktop/to do list")
or
sc.textFile("C:/Users/swaapnika/Desktop/to do list")