How to load typesafe configFactory from file on hdfs?
Using ConfigFactory.parseReader
should work (but I haven't tested it):
val reader = new InputStreamReader(file)val config = try { ConfigFactory.parseReader(reader)} finally { reader.close()}
You should be able to load .conf file in hdfs using the following code:
ConfigFactory.parseFile(new File("application.conf"));
Please keep in mind that the .conf file should be placed on the same directory as your app file (e.g. jar file in spark).
Here is what I did with Spark application:
/** * Load typesafe's configuration from hdfs file location * @param sparkContext * @param confHdfsFileLocation * @return */ def loadHdfsConfig(sparkContext: SparkContext, confHdfsFileLocation: String) : Config = { // Array of 1 element (fileName, fileContent) val appConf: Array[(String, String)] = sparkContext.wholeTextFiles(confHdfsFileLocation).collect() val appConfStringContent = appConf(0)._2 ConfigFactory.parseString(appConfStringContent) }
Now in the code, just use
val config = loadHdfsConfig(sparkContext, confHdfsFileLocation)config.getString("key-here")
I hope it helps.