Spark 2.2.0 FileOutputCommitter
I have been hit by this issue. Spark is discouraging the usage of DirectFileOutputCommitter as it might lead to data loss in case of race situation. The algorithm version 2 doesn't help a lot.
I have tried to use the gzip to save the data in s3 instead of snappy compression which gave some benefit.
The real issue here is that spark writes in the s3://<output_directory>/_temporary/0 first then copies the data from temporary to the output. This process is pretty slow in s3,(Generally 6MBPS) So if you get lot of data you will get considerable slowdown.
The alternative is to write to HDFS first then use distcp / s3distcp to copy the data to s3.
Also , You could look for a solution Netflix provided.
I haven't evaluated that.
EDIT:
The new spark2.4 version has solved the problem of slow s3 write. I have found the s3 write performance of spark2.4 with hadoop 2.8 in the latest EMR version (5.24) is almost at par with HDFS write.
See the documents