我使用火花0.7.2在独立模式下使用以下驱动程序来处理〜90GB:使用7-工人和1个不同的主logdata的(压缩19GB):火花独立模式:连接到127.0.1.1:<PORT>拒绝
System.setProperty("spark.default.parallelism", "32")
val sc = new SparkContext("spark://10.111.1.30:7077", "MRTest", System.getenv("SPARK_HOME"), Seq(System.getenv("NM_JAR_PATH")))
val logData = sc.textFile("hdfs://10.111.1.30:54310/logs/")
val dcxMap = logData.map(line => (line.split("\\|")(0),
line.split("\\|")(9)))
.reduceByKey(_ + " || " + _)
dcxMap.saveAsTextFile("hdfs://10.111.1.30:54310/out")
所有阶段1的ShuffleMapTasks
完成后:
Stage 1 (reduceByKey at DcxMap.scala:31) finished in 111.312 s
它提交阶段0:
Submitting Stage 0 (MappedRDD[6] at saveAsTextFile at DcxMap.scala:38), which is now runnable
有些系列化后,它打印
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host23
spark.MapOutputTracker - Size of output statuses for shuffle 0 is 2008 bytes
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host21
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host22
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host26
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host24
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host27
spark.MapOutputTrackerActor - Asked to send map output locations for shuffle 0 to host28
在此之后,什么都不会发生了,也top
表明,现在工人都处于闲置状态。 如果我查看日志在工人的机器,在他们每个人的同样的事情发生:
13/06/21 07:32:25 INFO network.SendingConnection: Initiating connection to [host27/127.0.1.1:34288]
13/06/21 07:32:25 INFO network.SendingConnection: Initiating connection to [host27/127.0.1.1:36040]
13/06/21 07:32:25 INFO network.SendingConnection: Initiating connection to [host27/127.0.1.1:50467]
13/06/21 07:32:25 INFO network.SendingConnection: Initiating connection to [host27/127.0.1.1:60833]
13/06/21 07:32:25 INFO network.SendingConnection: Initiating connection to [host27/127.0.1.1:49893]
13/06/21 07:32:25 INFO network.SendingConnection: Initiating connection to [host27/127.0.1.1:39907]
然后,这些“发起连接”尝试每个,它在每个抛出同样的错误工作人员(显示host27的日志为例,只是第一次出现错误):
13/06/21 07:32:25 WARN network.SendingConnection: Error finishing connection to host27/127.0.1.1:49893
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at spark.network.SendingConnection.finishConnect(Connection.scala:221)
at spark.network.ConnectionManager.spark$network$ConnectionManager$$run(ConnectionManager.scala:127)
at spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:70)
为什么会发生这种情况?这似乎是工作人员可以互相沟通很好,唯一的问题似乎当他们想将消息发送到自己发生;在上面的例子中,host27尝试发送6条消息给自己,但是失败了6次。发送消息给其他工作正常。 有人有想法吗?
编辑:也许它有使用127.0火花做。 0.1,而不是127.0。 0.1? /etc/hosts
如下所示:
127.0.0.1 localhost
127.0.1.1 host27.<ourdomain> host27