With Hadoop, can I create a tasktracker on a machine that isn't running a datanode? With Hadoop, can I create a tasktracker on a machine that isn't running a datanode? hadoop hadoop

With Hadoop, can I create a tasktracker on a machine that isn't running a datanode?


I'm imagining that, when scheduling a job on one of the "disposable" machines, the jobtracker will send a line of input over the network to the tasktracker, which then takes that line of input and feeds it directly to a Mapper, without writing anything to the disc. Is this what happens?

Not quite, the Job tracker tasks a task tracker to run a map task to process the input split. The JobTracker does not pass the data to the task tracker, more is passes the serialized split information (file name, start offset and length). The TaskTracker runs the MapTask, and it is the MapTask that instantiates the InputFormat and associated RecordReader for the split information - passing the input Key/Values to the Mapper.

In the case where you don't have a local data node, OR you do have a local data node, but the data is not replicated on the local data node, the data will be read across the network from another data node (hopefully rack local, but could still come from somewhere else).

You can see the stats for how often a data block was local to the task or local to the rack in the Hadoop counters output.