Error (application returned with exitcode 1) when running Spark in yarn-cluster mode
I was finally able to fix the problem. What was happening was that my bash script that calls spark-submit was passing an invalid argument to it. When a job starts, a script called launch_container.sh would be executing org.apache.spark.deploy.yarn.ApplicationMaster with the arguments passed to spark-submit and the ApplicationMaster returns with an exit code of 1 when any argument to it is invalid.
More information here