Will there be Shuffle and sort in Map only task? Will there be Shuffle and sort in Map only task? hadoop hadoop

Will there be Shuffle and sort in Map only task?


When you have a map-only task, there is not shuffling at all, which means that mappers will write the final output directly to the HDFS.

On the other hand, when you have a whole Map-Reduce program, with mappers and reducers, yes, shuffling can start before reduce-phase start.

Quoting this very nice answer in SO:

First of all shuffling is the process of transfering data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn't be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to save some time. That's why you can see a reduce status greater than 0% (but less than 33%) when the map status is not yet 100%.

Hope this answer had clarified your confusion.