1
我正在使用与Kafka集成的Spark流,当我在本地模式下从我的IDE运行流应用程序时,一切都可以作为魅力。但是当我将其提交到群集我保持具有以下错误:Spark Streaming Kafka java.lang.ClassNotFoundException:org.apache.kafka.common.serialization.StringDeserializer
java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.StringDeserializer
我使用SBT组件来构建我的项目。
我SBT是这样的:
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" % Provided,
"org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided,
"org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided,
"org.marc4j" % "marc4j" % "2.8.2",
"net.sf.saxon" % "Saxon-HE" % "9.7.0-20"
)
run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated
mainClass in assembly := Some("EstimatorStreamingApp")
我也试过用--package选项
尝试1
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0
尝试2
--packages org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:2.2.0
所有没有成功。有没有人有什么建议