Read multiple files using scalding and output a SINGLE file Read multiple files using scalding and output a SINGLE file hadoop hadoop

Read multiple files using scalding and output a SINGLE file


You can use groupAll to send all the map outputs (the job is a map only job) to a single reducer, considering the data is small, then do a write. The output will be written to a single file.

....filter(a => !a._3).map (that => MyScaldingType (that._1, that._2)).groupAll.write(MyScaldingType.typedSink(typedArgs))