HDFS LeaseExpiredException HDFS LeaseExpiredException hadoop hadoop

HDFS LeaseExpiredException


So Finally I could determine what is going on. So from the source, S3, our application was downloading files like

 /xyz/2014/01/week1/abc /xyz/2014/01/week1/def /xyz/2014/01/week2/abc /xyz/2014/01/week2/def /xyz/2014/01/week3/abc /xyz/2014/01/week3/def 

Notice the same file names across different weeks. And then each of these files were being written to HDFS using the DFSClient. So essentially multiple mappers were trying to write the "same file" (because of the same file name like abc, def) even though the files were actually different. As the client has to acquire a lease before writing the file and as the client writing the first of the "abc" file was not releasing the lease while during the writing process, the other client trying to write the other "abc" file was throwing the LeaseExpriedException with the Lease Mismatch Message.

But this still does not explain why the client which first acquired the lease for the write did not succeed. I mean I would expect in such a case that the first writers of every such files to succeed. Any explanation ?