SparkContext：初始化SparkContext时出错。 java.net.UnknownHostException：master

我是新的Spark和Intellij。下面我有我的build.sbt文件：SparkContext：初始化SparkContext时出错。 java.net.UnknownHostException：master

name := "TestSpark" 
version := "1.0" 
scalaVersion := "2.11.8" 
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.11" % "1.6.2","org.apache.hadoop" % "hadoop-client" % "2.6.2")

我也有TestMain.scala中的src/main /斯卡拉2.11/TestMain.scala

import org.apache.spark.{SparkContext,SparkConf} 
/** 
    * Created by tuannv5 on 24/08/2016. 
    */ 
object TestMain { 
    def main(args: Array[String]) { 
    val conf = new SparkConf().setMaster("local[*]").setAppName("Test Spark").set("spark.executor.memory","2g") 
    val sc = new SparkContext(conf) 
    val data = sc.parallelize(1 to 1000000).filter(_<10000) 
    data.foreach(println) 

    } 
}

出于某种原因，当我运行的应用程序中的IntelliJ。我得到的错误：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
16/08/24 10:29:28 INFO SparkContext: Running Spark version 1.6.2 
16/08/24 10:29:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/08/24 10:29:29 ERROR SparkContext: Error initializing SparkContext. 
java.net.UnknownHostException: master: master: unknown error 
    at java.net.InetAddress.getLocalHost(InetAddress.java:1505) 
    at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:789) 
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:782) 
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:782) 
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839) 
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839) 
    at scala.Option.getOrElse(Option.scala:121) 
    at org.apache.spark.util.Utils$.localHostName(Utils.scala:839) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:420) 
    at TestMain$.main(TestMain.scala:8) 
    at TestMain.main(TestMain.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) 
Caused by: java.net.UnknownHostException: master: unknown error 
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) 
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) 
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) 
    at java.net.InetAddress.getLocalHost(InetAddress.java:1500) 
    ... 15 more 
16/08/24 10:29:29 INFO SparkContext: Successfully stopped SparkContext 
Exception in thread "main" java.net.UnknownHostException: master: master: unknown error 
    at java.net.InetAddress.getLocalHost(InetAddress.java:1505) 
    at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:789) 
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:782) 
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:782) 
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839) 
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839) 
    at scala.Option.getOrElse(Option.scala:121) 
    at org.apache.spark.util.Utils$.localHostName(Utils.scala:839) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:420) 
    at TestMain$.main(TestMain.scala:8) 
    at TestMain.main(TestMain.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) 
Caused by: java.net.UnknownHostException: master: unknown error 
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) 
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) 
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) 
    at java.net.InetAddress.getLocalHost(InetAddress.java:1500) 
    ... 15 more 

Process finished with exit code 1

有人请向我解释在这个错误中说明的问题？这是因为我的依赖没有正确安装，还是因为另一个原因？

来源

2016-08-24 TuanMang

你可以运行'hostname'和检查输出？看起来你的主机名称设置不正确。 – zsxwing

当我运行hostname命令时，输出是：master。但我不知道修复这个。你能向我解释一下吗？ – TuanMang

你可以在'/ etc/hosts'中添加一行：'127.0.0.1 master' – zsxwing

您的主机名称不正确。

类型的主机名和检查它是否是相同的/etc/hosts：

例如：

[email protected]:/home/makbul# hostname

谷歌已经同在/etc/hosts：

127.0.1.1  google

来源

2017-01-18 00:20:09 user1464878

SparkContext：初始化SparkContext时出错。 java.net.UnknownHostException：master

回答

相关问题