当python运行spark示例kmeans时出现ClassNotFoundException

Question

我有一个6个从属的spark集群。并且火花默认示例pi.py可以在我的环境中成功运行。

但运行spark默认示例kmeans.py时出现以下错误。

./bin/spark-submit --master spark://master_ip:7077 examples/src/main/python/mllib/kmeans.py data/mllib/kmeans_data.txt 2 

的错误信息是：

17/03/03 10:21:21 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.xx.xx.xx:42586 (size: 6.5 KB, free: 366.3 MB) 
17/03/03 10:21:22 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 7772374377312901948 
java.lang.ClassNotFoundException: org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$RetrieveSparkAppConfig$ 
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
     at java.lang.Class.forName0(Native Method) 
     at java.lang.Class.forName(Class.java:348) 
     at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67) 
     at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) 
     at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521) 
     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) 
     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) 
     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) 
     at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) 
     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) 
     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) 
     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) 
     at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) 
     at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108) 
     at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1$$anonfun$apply$1.apply(NettyRpcEnv.scala:259) 
     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) 
     at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:308) 
     at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1.apply(NettyRpcEnv.scala:258) 
     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) 
     at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:257) 
     at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:578) 
     at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:563) 
     at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:159) 
     at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:107) 
     at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119) 
     at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) 
     at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) 
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) 
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) 
     at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) 
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) 
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) 
     at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) 
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) 
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) 
     at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85) 
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) 
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) 
     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) 
     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
     at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
     at java.lang.Thread.run(Thread.java:745) 

示例kmeans.py由蟒写入并在Python环境跑去。但是我有一个Java错误。在这个问题上的任何帮助？

Answer 1

运行到相同的错误 - 原来是奴隶和主人之间的版本差异。几个细节：

• 硕士（星火2.0.2）
• SLAVE1（火花2.0.2）
• SLAVE2（星火2.0.2/2.1.1星火）

简单地说，这里的奇怪：在Slave2上有两个版本的火花。如果一个进程试图单独使用Spark 211，我会得到你的错误，并且作业不能完成（必须被杀死）。

相反，如果一个进程运行正确的202版本，一切工作正常。此外，如果202点火过程正在运行，211版本也可以无误地运行（奇怪）。

基本上，我的“答案”是检查各处的版本（包括Spark和Python）。希望有所帮助。