Hadoop Streaming with SequenceFile (on AWS)

The answer to this is to specify the input format as a command line argument to Hadoop.

-inputformat SequenceFileAsTextInputFormat

The chances are that you want the SequenceFile as text, but there is also SequenceFileAsBinaryInputFormat if thats more appropriate.

Not sure if this is what you're asking for, but the command to use ruby map reduce scripts with the hadoop command line would look something like this:

% hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-*-streaming.jar \  -input input/ncdc/sample.txt \  -output output \  -mapper ch02/src/main/ruby/max_temperature_map.rb \  -reducer ch02/src/main/ruby/max_temperature_reduce.rb

You can (and should) use a combiner with big data sets. Add it with the -combiner option. The combiner output will feed directly into your mapper (but no guarantee how many times this will be called, if at all). Otherwise your input is split (according to the standard hadoop protocal) and feeds directly into your mapper. The example is from O'Reily's Hadoop: The Definitive Guide 3rd Edition. It has some very good information on streaming, and a section dedicated to streaming with ruby.