sbt运行，但./spark-submit不

Question

我想用lift-json解析器使用sbt构建。我built.sbt文件具有以下内容：

name := "MyProject" 

version := "1.0" 

scalaVersion := "2.10.0" 
// https://mvnrepository.com/artifact/net.liftweb/lift-json_2.10 
libraryDependencies += "net.liftweb" % "lift-json_2.10" % "3.0-M1" 
val lift_json = "net.liftweb" %% "lift-json_2.10" % "3.0-M1" 
//val json4sNative = "org.json4s" %% "json4s-native" % "3.3.0" 
//libraryDependencies += "org.scala-lang" % "scala-library" % "2.9.1" 
lazy val gitclonefile = "/root/githubdependencies/lift" 
lazy val g = RootProject(file(gitclonefile)) 
lazy val root = project in file(".") dependsOn g 

我的代码是这样的：

package org.inno.parsertest 
import net.liftweb.json._ 
//import org.json4s._ 
//import org.json4s.native.JsonMethods._ 
object parser { 
    def main (args: Array[String]){ 
    val x = parse(""" { "numbers" : [1, 2, 3, 4] } """) 
    println(x) 
    val x1 = "jaimin is awesome" 
    println(x1) 
} 
} 

SBT包，然后运行SBT作品。但是，当我想用​​火花提交运行此，我收到以下错误：

Error: application failed with exception 
java.lang.NoClassDefFoundError: net/liftweb/json/package$ 
    at org.inno.parsertest.parser$.main(jsonparser.scala:7) 
    at org.inno.parsertest.parser.main(jsonparser.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: java.lang.ClassNotFoundException: net.liftweb.json.package$ 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
    ... 9 more 

我怎样才能让./spark-submit工作？

Answer 1

一旦火花驱动程序开始在您的应用程序上工作（当您提交它时），它必须处理import net.liftweb.json._行，这意味着它将在其类路径中查找此类。

但是Spark不附带liftweb的罐子，所以这是一个小姐，最后你会得到一个ClassNotFoundException。

所以你需要提供你的应用程序所需的罐子。有很多方法可以做到这一点，详细讨论。

您可能会从spark documentation开始。

Bundling Your Application’s Dependencies
If your code depends on other projects, you will need to package them alongside your application in order to distribute the code to a Spark cluster. To do this, create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins. When creating assembly jars, list Spark and Hadoop as provided dependencies; these need not be bundled since they are provided by the cluster manager at runtime. Once you have an assembled jar you can call the bin/spark-submit script as shown here while passing your jar.

有人可能会提出：

1. 应用程序打包成是通常所说的“超级罐子”或“胖罐子”，例如用sbt's "assembly"插件，或maven shade，这取决于您的偏好。该策略将所有依赖项的所有类和资源合并到单个JAR中，即您提交的那个。

2. 将参数添加到spark-submit调用。有几种方法，一个简单的方法是使用--jars参数，然后是需要的jar文件列表（以逗号分隔）。这些罐子将火花启动您的工作

   之前加入到实际的驱动程序/工人类路径

3. 泰尔火花提交“绑定”到Maven仓库

   Users may also include any other dependencies by supplying a comma-delimited list of maven coordinates with --packages. All transitive dependencies will be handled when using this command. Additional repositories (or resolvers in SBT) can be added in a comma-delimited fashion with the flag --repositories.

但充分讨论所有的选项都是相当长的一个，我建议你谷歌“包火花应用程序”或搜索这些主题StackOverflow获得更好的概述。

旁注：提交给Spark一个不使用SparkContext的应用程序似乎毫无意义，但我想你只是在这一点上做实验。