Spark Job Keep on Running

You can achieve this by couple of ways

1)You can run the spark submit driver process in background using nohupEg:

nohup  ./spark-submit --class  customer.core.classname \  --master yarn --numexecutors 2 \  --driver-memory 2g --executor-memory 2g --executor-cores 1 \  /home/hdfs/Test/classname-0.0.1-SNAPSHOT-SNAPSHOT.jar \  newdata host:6667 &

2)Run in deploy mode as cluster so that driver process runs in different node.

I think this question is more about shell than spark,

To keep an application running, even when closing the shell, tou should add & at the end of your command. So your spark-submit command will be (just add the & to the end)

./spark-submit --class  customer.core.classname --master yarn --numexecutors 2 --driver-memory 2g --executor-memory 2g --executor-cores 1 /home/hdfs/Test/classname-0.0.1-SNAPSHOT-SNAPSHOT.jar newdata host:6667 &[1] 28299

You still get the logs and output messages, unless you redirected them

hope I understand the question. In general, if you want a process to keep running you can create a process file that will run in the background. in your case, the job will continue running until you specifically kill it using yarn -kill. so even if you kill the spark submit it will continue to run since yarn is managing it after submission.