Client cannot authenticate via:[TOKEN, KERBEROS]

Alright, well after hours and hours and hours we have this figured out. For all following generations of coders, forever plagued by hadoop's lack of documentation:

You must grab the tokens from UserGroupInformation object with a call to get credentials. Then you must set the tokens on the ContainerLaunchContext.

You will also get this error if you are using the actual name node instead of logical URI of HA in any of the hdfs path.

This is because if it finds a namenode uri instead of logical uri then it will create non-HA file system which will try to use simple UGI instead of kerberos UGI.

The same error with incompatible hadoop artifact versions.

Working example:

public static final String CONF_CORE_SITE = "/etc/hadoop/conf/core-site.xml";public static final String CONF_HDFS_SITE = "/etc/hadoop/conf/hdfs-site.xml";/** * Pick the config files from class path */private static Configuration getHdfsConfiguration() throws IOException {    Configuration configuration = new Configuration();    configuration.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());    configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());    File hadoopCoreConfig = new File(CONF_CORE_SITE);    File hadoopHdfsConfig = new File(CONF_HDFS_SITE);    if (! hadoopCoreConfig.exists() || ! hadoopHdfsConfig.exists()) {        throw new FileNotFoundException("Files core-site.xml or hdfs-site.xml are not found. Check /etc/hadoop/conf/ path.");    }    configuration.addResource(new Path(hadoopCoreConfig.toURI()));    configuration.addResource(new Path(hadoopHdfsConfig.toURI()));    //Use existing security context created by $ kinit    UserGroupInformation.setConfiguration(configuration);    UserGroupInformation.loginUserFromSubject(null);    return configuration;}

pom.xml

<properties>  <hadoop.version>2.6.0</hadoop.version>  <hadoop.release>cdh5.14.2</hadoop.release></properties><dependency>  <groupId>org.apache.hadoop</groupId>  <artifactId>hadoop-core</artifactId>  <version>${hadoop.version}-mr1-${hadoop.release}</version></dependency><dependency>  <groupId>org.apache.hadoop</groupId>  <artifactId>hadoop-hdfs</artifactId>  <version>${hadoop.version}-${hadoop.release}</version></dependency><dependency>  <groupId>org.apache.hadoop</groupId>  <artifactId>hadoop-common</artifactId>  <version>${hadoop.version}-${hadoop.release}</version></dependency>