Hadoop performance Hadoop performance hadoop hadoop

Hadoop performance


This depends on a large number of factors, including your configuration, your machine, memory config, JVM settings, etc. You also need to subtract JVM startup time.

It runs much more quickly for me. That said, of course it will be slower on small data sets than a dedicated C program--consider what it's doing "behind the scenes".

Try it on a terabyte of data spread across a few thousand files and see what happens.


As Dave said Hadoop is optimized to handle large amounts of data not toy examplesThere's a tax for "waking up the elephant" to get things going which is not needed when you work on smaller sets.You can take a look at "About the performance of Map Reduce Jobs" for some details on what's going on


Your input data was small and so you observed that hadoop took long time. Job creation process in hadoop is heavy as it involves lot of things. Had the input data being large, then you would see hadoop doing better over wc.