Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie hadoop hadoop

Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie


Hive unfortunately overrides this capability by setting it's own NullOutputComitter:

conf.setOutputCommitter(NullOutputCommitter.class);

see

src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.javasrc/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java

Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.


You can add 'dfs' command to your hive script, like

dfs -touchz '$table_base_path'/dt='${partition}'/_SUCCESS

https://archive.cloudera.com/cdh4/cdh/4/hive/language_manual/cli.html


I ran into the same issue and ended up using a shell action to create the flag.

Here's a full example: http://nathan.vertile.com/blog/2014/09/02/oozie-data-pipeline-done-flag/