为什么Spark 1.5.2在独立模式下抛出“local class incompatible”？

今天与Spark 1.5.2得到了本地类不兼容的错误。为什么Spark 1.5.2在独立模式下抛出“local class incompatible”？

我知道，这种类型的错误与混合星火的几个版本有关，但我完全不知道在那里我可以做到这一点...

我的应用程序是一个简单的例子，当我处理用JDBC连接到mysql数据库。在那里，我想用一些数据获取DataFrame，这可以在加入两个表后接收。

我下载了Spark的Hadoop 2.6的预生成版本。

和往常一样，我使用Maven构建fat jar，并在SparkConf中将路径设置为jar。

从那以后，我开始主节点，然后 - 此命令：

./spark-class org.apache.spark.deploy.worker.Worker spark://dima:7077

下水工人。

然后我从IDE启动应用程序。

这里是应用程序的代码：

public class Main implements Serializable { 

public static void main(String[] args) throws ClassNotFoundException { 

    SparkConf conf = new SparkConf() 
          .setAppName("sparkproject") 
          .setMaster("spark://dima:7077") 
          .setJars(new String[] {"target/sparkproject-1.0-SNAPSHOT-jar-with-dependencies.jar"}) 
          .set("spark.home","/home/dima/PROGRA~1/spark-1.5.2"); 
    JavaSparkContext sc = new JavaSparkContext(conf); 

    SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc); 

    Map<String, String> options = new HashMap<String, String>(); 
    options.put("driver", "com.mysql.jdbc.Driver"); 
    options.put("url", "jdbc:mysql://localhost:3306/db?user=root"); 
    options.put("dbtable", "db.enreg_karta"); 

    sqlContext.read().format("jdbc").options(options).load() 
              .toDF() 
              .registerTempTable("enreg_karta"); 

    options.put("dbtable", "db.entem_rubr"); 

    sqlContext.read().format("jdbc").options(options).load() 
              .toDF() 
              .registerTempTable("entem_rubr"); 

    DataFrame jdbcDF = sqlContext.sql("select rk.enf_namew, rk.enf_referatw from enreg_karta rk join entem_rubr tr on rk.syrecordidw = tr.enrel_inf_tehnw").toDF(); 
    jdbcDF.show(); 
}}

这里是的pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
<modelVersion>4.0.0</modelVersion> 
<groupId>dima.study.spark</groupId> 
<artifactId>sparkproject</artifactId> 
<version>1.0-SNAPSHOT</version> 
<packaging>jar</packaging> 
<properties> 
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
    <maven.compiler.source>1.8</maven.compiler.source> 
    <maven.compiler.target>1.8</maven.compiler.target> 
</properties> 
<dependencies> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.10</artifactId> 
     <version>1.5.2</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.10</artifactId> 
     <version>1.5.1</version> 
    </dependency> 
    <dependency> 
     <groupId>mysql</groupId> 
     <artifactId>mysql-connector-java</artifactId> 
     <version>5.1.23</version> 
    </dependency> 
</dependencies> 
<build> 
    <plugins> 
     <plugin> 
      <groupId>org.apache.maven.plugins</groupId> 
      <artifactId>maven-assembly-plugin</artifactId> 
      <version>2.2</version> 
      <configuration> 
       <descriptorRefs> 
        <descriptorRef>jar-with-dependencies</descriptorRef> 
       </descriptorRefs> 
      </configuration> 
      <executions> 
       <execution> 
        <id>assemble-all</id> 
        <phase>package</phase> 
        <goals> 
         <goal>single</goal> 
        </goals> 
       </execution> 
      </executions> 
     </plugin> 
    </plugins> 
</build>

在jdbcDF.show我的应用程序崩溃（）;命令。

这里是堆栈跟踪：

5/11/29 23:07:23 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.205.17.81): java.io.InvalidClassException: org.apache.spark.sql.catalyst.expressions.AttributeReference; local class incompatible: stream classdesc serialVersionUID = 370695178000872136, local class serialVersionUID = -8877631944444173448 
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) 
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623) 
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) 
at scala.collection.immutable.$colon$colon.readObject(List.scala:362) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:497) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) 
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72) 
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98) 
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:64) 
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) 
at org.apache.spark.scheduler.Task.run(Task.scala:88) 
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)

附：我的笔记本电脑上有唯一的JVM。

有人知道最新错了吗？

谢谢。

来源

2015-11-29 dimson

夫妇的事情，你可以尝试： -

删除 “.setMaster”， “.setJars”，从火花塞的conf “.SET”。
设置环境变量SPARK_HOME =
确保您的计算机上只安装了1个JDK/JVM（首选为JDK7）。并使用相同的JDK编译并创建一个包含所有项目依赖项（Spark或Scala除外）的胖jar文件。
最后尝试“spark-submit”提交你的工作。

这是一个更清洁的方式。试试看看它是否有效。

谢谢， Sumit

来源

2015-11-30 11:06:03 Sumit

Thanx，Sumit！ spark-submit解决问题。奇怪的是，但我记得，在早期的Spark版本（例如1.0.0）中，我只是使用Spark类运行Workers，然后可以从IDE启动应用程序。你知道为什么它现在不工作吗？ – dimson

在集群模式下（Standalone，Mesos或Yarn），我们需要“提交”应用程序，以便它可以分布在Spark节点/工作者上。我从来没有尝试过IDE，但尝试使用本地[线程]，看看它是否工作。 – Sumit

为什么Spark 1.5.2在独立模式下抛出“local class incompatible”？

回答

相关问题