Oozie shell action - running sqoop command and need logging details Oozie shell action - running sqoop command and need logging details hadoop hadoop

Oozie shell action - running sqoop command and need logging details


You have a parameter hinting that Log4J should use a specific properties file, but Sqoop is not instructed to use it (no -Dlog4j.configuration=...file name without path... on command line).

OK, let's assume that is is done on purpose (?); the problem is that

  • Log4J searchs for a properties file w/ default name i.e. log4j.properties
  • search is done in the directories present in CLASSPATH, stopping on first match
  • default CLASSPATH for an Oozie shell is Hadoop JARs, then Hadoop conf dir, then current working dir (the place where all <file> dependencies are dumped along with Oozie JARs) and all these app/Oozie JARs
  • the first match for log4j.properties happens to be a file that Log4J cannot open
  • thus Log4J does not log anything anywhere (????????)

A possible workaround would be

  1. create a custom log4j.properties -- cf. 1st example in that post to log anything tagged INFO and above (i.e. INFO, WARN, ERROR but not DEBUG) to StdOut
  2. upload that file to HDFS somewhere, then tell the Oozie Action to download it to the container with a <file> element
  3. tell the Oozie Action to request that its CLASSPATH starts with current working dir, by setting a property such as oozie.launcher.mapreduce.task.classpath.first to true (actual property may depend on your Hadoop version, see that post and that JIRA)

Note that step 3 is only necessary because of the default name being present elsewhere in the CLASSPATH; if Sqoop was instructed to use a different file with a different name then there would be no ambiguity.