How does HDFS with append works How does HDFS with append works hadoop hadoop

How does HDFS with append works


According to the latest design document in the Jira issue mentioned before, we find the following answers to your question:

  1. HDFS will append to the last block, not create a new block and copy the data from the old last block. This is not difficult because HDFS just uses a normal filesystem to write these block-files as normal files. Normal file systems have mechanisms for appending new data. Of course, if you fill up the last block, you will create a new block.
  2. Only one single write or append to any file is allowed at the same time in HDFS, so there is no concurrency to handle. This is managed by the namenode. You need to close a file if you want someone else to begin writing to it.
  3. If the last block in a file is not replicated, the append will fail. The append is written to a single replica, who pipelines it to the replicas, similar to a normal write. It seems to me like there is no extra risk of dataloss as compared to a normal write.


Here is a very comprehensive design document about append and it contains concurrency issues.

Current HDFS docs gives a link to that document, so we can assume that it is the recent one. (Document date is 2009)

And the related issue.


Hadoop Distributed File System supports appends to files, and in this case it should add the 20 MB to the 2nd block in your example (the one with 2 MB in it initially). That way you will end up with two blocks, one with 128 MB and one with 22 MB.

This is the reference to the append java docs for HDFS.