Compress Output Scalding / Cascading TsvCompressed

Anyway after much faffification I managed to write a TsvCompressed output which seems to do the job (you still need to set the hadoop job system configuration properties, i.e. set compress to true, and set the codec to something sensible or it defaults to crappy deflate)

import com.twitter.scalding._import cascading.tuple.Fieldsimport cascading.scheme.localimport cascading.scheme.hadoop.{TextLine, TextDelimited}import cascading.scheme.Schemeimport org.apache.hadoop.mapred.{OutputCollector, RecordReader, JobConf}case class TsvCompressed(p: String) extends FixedPathSource(p) with DelimitedSchemeCompressedtrait DelimitedSchemeCompressed extends Source {  val types: Array[Class[_]] = null  override def localScheme = new local.TextDelimited(Fields.ALL, false, false, "\t", types)  override def hdfsScheme = {    val temp = new TextDelimited(Fields.ALL, false, false, "\t", types)    temp.setSinkCompression(TextLine.Compress.ENABLE)    temp.asInstanceOf[Scheme[JobConf,RecordReader[_,_],OutputCollector[_,_],_,_]]  }}

I have also small project showing how to achieve compressed output from Tsv. WordCount-Compressed.

Scalding was setting null to the Cascading TextDelimeted parameter which disables compression.