Spark - failed on connection exception: java.net.ConnectException - localhost

Sorry... actually namenode is running in a different port which I found from core-site.xml. It works for me after using the port 54310.

<property>  <name>fs.default.name</name>  <value>hdfs://localhost:54310</value>  <description>The name of the default file system.  A URI whose  scheme and authority determine the FileSystem implementation.  The  uri's scheme determines the config property (fs.SCHEME.impl) naming  the FileSystem implementation class.  The uri's authority is used to  determine the host, port, etc. for a filesystem.</description></property></configuration>

Could someone please tell me what does 3079 means here, which is the output of JPS command.

That's a good answer to yourself ! :)

3079 is the pid ( process id ) of namenode. See more details about jps's output here: Jps doc. BTW, you can verify the listening ports of namenode using the pid with a command such as:

lsof -p 3079 -a -i

In your command output:

java    3079 hduser   89u  IPv4  59998      0t0  TCP localhost:54310->localhost:46507 (ESTABLISHED)

That line indicates that other process established TCP connection with Namenode at port 46507. Possible processes could be resoure manager(RM) or data nodes etc.