Spark：线程“main”中的异常java.lang.VerifyError：class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer

Question

我想对一个大项目中的某些数据做一个简单的Spark并行处理，但即使最简单的例子，我得到这个错误

Exception in thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase; 

的错误出现任何简单的并行化，即使这个简单的一个。我没有在那里这个错误甚至

val conf: SparkConf = new SparkConf().setAppName("IEEG Spark").setMaster("local") 
    val sc: SparkContext = new SparkContext(conf) 
    val data = Array(1, 2, 3, 4, 5) 
    val distVals = sc.parallelize(data) 
    distVals.foreach(println) 

附带的任何想法，下面是我的Maven pom.xml文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 

    <groupId>astiefel</groupId> 
    <artifactId>ieeg-spark</artifactId> 
    <modelVersion>4.0.0</modelVersion> 
    <name>Spark IEEG</name> 
    <parent> 
     <groupId>edu.upenn.cis.ieeg</groupId> 
     <artifactId>ieeg</artifactId> 
     <version>1.15-SNAPSHOT</version> 
    </parent> 
    <properties> 
     <scala.version>2.10.4</scala.version> 
    </properties> 
    <dependencies> 
     <dependency> 
      <groupId>edu.upenn.cis.ieeg</groupId> 
      <artifactId>ieeg-client</artifactId> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.10</artifactId> 
      <version>1.5.0</version> 
     </dependency> 
     <dependency> 
      <groupId>org.scala-lang</groupId> 
      <artifactId>scala-compiler</artifactId> 
      <version>${scala.version}</version> 
      <scope>compile</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.scalanlp</groupId> 
      <artifactId>breeze_2.10</artifactId> 
      <version>0.10</version> 
     </dependency> 
    </dependencies> 

    <build> 
     <sourceDirectory>src/main/scala</sourceDirectory> 
     <plugins> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <executions> 
        <execution> 
         <goals> 
          <goal>compile</goal> 
          <goal>testCompile</goal> 
         </goals> 
        </execution> 
       </executions> 
       <configuration> 
        <scalaVersion>${scala.version}</scalaVersion> 
        <args> 
         <arg>-target:jvm-1.5</arg> 
        </args> 
       </configuration> 
      </plugin> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-eclipse-plugin</artifactId> 
       <configuration> 
        <downloadSources>true</downloadSources> 
        <buildcommands> 
         <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand> 
        </buildcommands> 
        <additionalProjectnatures> 
         <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature> 
        </additionalProjectnatures> 
        <classpathContainers> 
         <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer> 
         <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer> 
        </classpathContainers> 
       </configuration> 
      </plugin> 
     </plugins> 
    </build> 
    <reporting> 
     <plugins> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <configuration> 
        <scalaVersion>${scala.version}</scalaVersion> 
       </configuration> 
      </plugin> 
     </plugins> 
    </reporting> 
</project> 

我这样失去了这个错误，任何提示，以哪里开始？

Answer 1

你可以使用下面的依赖关系做一些简单的Spark并行测试，它直接使用 spark-assembly-xxx.jar作为jar依赖项。

<profile> 
     <id>apache-spark/scala</id> 
     <dependencies> 
      <!--Apache Spark --> 
      <dependency> 
       <groupId>org.apache.spark</groupId> 
       <artifactId>spark-assembly</artifactId> 
       <version>1.5.2</version> 
       <scope>system</scope> 
       <systemPath>${spark.home}/lib/spark-assembly-1.5.2-hadoop2.6.0.jar</systemPath> 
      </dependency> 
      <dependency> 
       <groupId>org.scala-lang</groupId> 
       <artifactId>scala-library</artifactId> 
       <version>2.10.2</version> 
      </dependency> 
     </dependencies> 
    </profile>