How to load typesafe configFactory from file on hdfs? How to load typesafe configFactory from file on hdfs? hadoop hadoop

How to load typesafe configFactory from file on hdfs?


Using ConfigFactory.parseReader should work (but I haven't tested it):

val reader = new InputStreamReader(file)val config = try {  ConfigFactory.parseReader(reader)} finally {  reader.close()}


You should be able to load .conf file in hdfs using the following code:

ConfigFactory.parseFile(new File("application.conf"));

Please keep in mind that the .conf file should be placed on the same directory as your app file (e.g. jar file in spark).


Here is what I did with Spark application:

  /**    * Load typesafe's configuration from hdfs file location    * @param sparkContext    * @param confHdfsFileLocation    * @return    */  def loadHdfsConfig(sparkContext: SparkContext, confHdfsFileLocation: String) : Config = {    // Array of 1 element (fileName, fileContent)    val appConf: Array[(String, String)] = sparkContext.wholeTextFiles(confHdfsFileLocation).collect()    val appConfStringContent = appConf(0)._2    ConfigFactory.parseString(appConfStringContent)  }

Now in the code, just use

val config = loadHdfsConfig(sparkContext, confHdfsFileLocation)config.getString("key-here")

I hope it helps.