提交弹簧启动应用程序罐火花提交

Question

我很新的火花，我想尝试火花提交。我在spring boot中创建了一个应用程序，使用mvn package来创建一个jar。但是，当我试图提交罐子​​，它是无法找到主类。但是主要类是存在于罐子里。

spark-submit --class com.dip.sparkapp.SparkappApplication --master local target/sparkapp-0.0.1-SNAPSHOT.jar 

Answer 1

我们遇到了同样的问题，实际上，在您发布此消息的同一天。我们的解决方案是使用Maven的阴影插件来编辑我们的构建。我们发现，当使用spring-boot-maven插件进行打包时，它将我们的类嵌套在BOOT-INF/classes类中，而spark类不喜欢这些类。我将粘贴相关部分，以便您可以在自己的应用程序中试用 - 祝您好运！

<build> 
    <plugins> 
     <plugin> 
      <groupId>org.apache.maven.plugins</groupId> 
      <artifactId>maven-shade-plugin</artifactId> 
      <dependencies> 
       <dependency> 
        <groupId>org.springframework.boot</groupId> 
        <artifactId>spring-boot-maven-plugin</artifactId> 
        <version>${spring-boot-version}</version> 
       </dependency> 
      </dependencies> 
      <configuration> 
       <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope> 
       <createDependencyReducedPom>false</createDependencyReducedPom> 
       <filters> 
        <filter> 
         <artifact>*:*</artifact> 
         <excludes> 
          <exclude>META-INF/*.SF</exclude> 
          <exclude>META-INF/*.DSA</exclude> 
          <exclude>META-INF/*.RSA</exclude> 
         </excludes> 
        </filter> 
       </filters> 
       <transformers> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> 
         <resource>META-INF/spring.handlers</resource> 
        </transformer> 
        <transformer 
          implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer"> 
         <resource>META-INF/spring.factories</resource> 
        </transformer> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> 
         <resource>META-INF/spring.schemas</resource> 
        </transformer> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> 
        <transformer 
          implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> 
         <mainClass>${start-class}</mainClass> 
        </transformer> 
       </transformers> 
      </configuration> 
      <executions> 
       <execution> 
        <phase>package</phase> 
        <goals> 
         <goal>shade</goal> 
        </goals> 
       </execution> 
      </executions> 
     </plugin> 
    </plugins> 
</build>