Configure SnakeBite HDFS clients to work with high availability mode Configure SnakeBite HDFS clients to work with high availability mode hadoop hadoop

Configure SnakeBite HDFS clients to work with high availability mode


It turns out that Snakebite has not one, but two solutions to this problem: AutoConfigClient, which will take its configs from the hadoop configs, and HAClient which takes two namenodes.

In my case, I was actually using snakebite through airflow. It turns out that airflow's HDFSHook is smart enough to cope with two namenodes being provided in one connection, and will then use the HAClient.