Yarn mini-cluster container log directories don't contain syslog files
OK, finally happened to be about classpath, client configuration and packaging.
- Client configuration SHALL include proper classpath for YARN applications. In my case I have added the following lines to
yarn-site.xml(please note$HADOOP_COMMON_HOMEsubstitution):
<property> <name>yarn.application.classpath</name> <value>$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*</value></property>
- I have added the following variable definition to mini-cluster start-up script (it worth to note I have all mini-cluster server-side JARS into
./librelatively to mini-cluster startup script:
BASE_PATH="
pwd" export HADOOP_COMMON_HOME=${BASE_PATH}
The root cause of not working logging was client map-reduce job starting inside new VM on YARN without knowledge where to locate hadoop-yarn-server-nodemanager.jar which contains container-log4j.properties file which is in turn responsible for container default logging configuration. Now everything is working fine.