Hive compression in ORC using Lz4
- The compression we can use in addition to ORC columnar compression is either of one of NONE, ZLIB, SNAPPY.
- The default compression codec is ZLIB.
- The compression codecs other than above are not allowed.
In general to get an idea about error, read the error log completely to find the issue to some extent. The error log said -
"org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.hive.ql.io.orc.CompressionKind.Lz4"
Now you can manually substitute the jar file for lz4 on official spark repo.