distcp from Hadoop to S3 fails with "No space available in any of the local directories"

We ran into a similar exception while trying to save directly the results of a run from Apache Spark (version 1.5.2) to S3. The exception is the same though. I'm not really sure what the core issue is - somehow S3 upload doesn't seem to "play nice" with Hadoop's LocalDirAllocator class (version 2.7).

What finally solved it for us was a combination of:

1. enabling S3's "fast upload" - by setting "fs.s3a.fast.upload" to "true" in Hadoop configuration. This uses S3AFastOutputStream instead of S3AOutputStream and uploads data directly from memory, instead of first allocating local storage

2. merging the results of the job to a single part before saving to s3 (in Spark that's called repartitioning/coalescing)

Some caveats though:

1. S3's fast upload is apparently marked "experimental" in Hadoop 2.7

2. this work-around only applies to the newer s3a file-system ("s3a://..."). it won't work for the older "native" s3n file-system ("s3n://...")

hope this helps

Ideally you should use s3a rather than s3n, as s3n is deprecated.

With s3a, there is a parameter:

<property>  <name>fs.s3a.buffer.dir</name>  <value>${hadoop.tmp.dir}/s3a</value>  <description>Comma separated list of directories that will be used to buffer fileuploads to. No effect if fs.s3a.fast.upload is true.</description></property>

When you are getting the local file error, it most likely because the buffer directory has no space.

While you can change this setting to point at a directory with more space, a better solution may be to set (again in S3a):

fs.s3a.fast.upload=true

This avoids buffering the data on local disk and should actually be faster too.

The S3n buffer directory parameter should be:

fs.s3.buffer.dir

So if you stick with s3n, ensure it has plenty of space and it should hopefully resolve this issue.

I had this error for some days and did not get what was happening, all nodes have PLENTY space (around 400GB).After some research I found this:2019-01-09 17:31:30,326 WARN [main] org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: Failed to create /mnt/hadoop/tmp/s3a

The exception says about space but the real error is permission, the message could be improved.