无法读取火花scala中的conf文件

我想读取我的火花应用程序中的conf文件。 conf文件位于Hadoop边缘节点目录中。无法读取火花scala中的conf文件

omega.conf

username = "surrender" 
location = "USA"

我的星火代码：

package com.test.spark 

import org.apache.spark.{SparkConf, SparkContext} 
import java.io.File 
import com.typesafe.config.{ Config, ConfigFactory } 


object DemoMain { 

def main(args: Array[String]): Unit = { 
println("Lets Get Started ") 
val conf = new SparkConf().setAppName("SIMPLE") 
val sc = new SparkContext(conf) 
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) 
val conf_loc = "/home/cloudera/localinputfiles/omega.conf" 
loadConfigFile(conf_loc) 

} 

    def loadConfigFile(loc:String):Unit ={ 
     val config = ConfigFactory.parseFile(new File(loc)) 
     val username = config.getString("username") 
     println(username) 

} 

}

我使用运行该应用程序的火花火花提交

spark-submit --class com.test.spark.DemoMain --master local /home/cloudera/dev/jars/spark_examples.jar

星火工作启动，但它抛出我说下面的错误。它说，找不到配置设置的关键'用户名'

17/03/29 12:57:37 INFO SparkContext: Created broadcast 0 from textFile at DemoMain.scala:25 
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'username' 
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155) 
    at com.typesafe.config.impl.SimpleConfig.getString (SimpleConfig.java:197) 
    at com.test.spark.DemoMain$.loadConfigFile(DemoMain.scala:53) 
    at com.test.spark.DemoMain$.main(DemoMain.scala:27) 
    at com.test.spark.DemoMain.main(DemoMain.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

请帮我

来源

2017-03-29 Surender Raja

我只是尝试它工作正常我用下面的代码

val config=ConfigFactory.parseFile(new File("/home/sandy/my.conf")) 
    println("::::::::::::::::::::"+config.getString("username"))

和的conf文件

用户名=“投降测试此修复此问题“

location =”USA“

请打印它，检查文件的位置。

来源

2017-03-29 12:50:08

是的，你是正确的，我从一个不正确的位置读取..现在纠正它，它工作正常 –

无法读取火花scala中的conf文件

回答

相关问题