当主设置为纱线客户端时由Spark引发NoClassDefFoundError

我有一个简单的Spark（1.4.1版本）应用程序，用Scala编写，它使用kinesis流中的数据。如果我运行该应用程序，使用spark-submit命令，将主设备的值设置为local [*]一切正常。如果让我选择作为主纱的客户端使用我有以下异常：当主设置为纱线客户端时由Spark引发NoClassDefFoundError

15/11/24 14:22:09 ERROR ReceiverTracker: Deregistered receiver for stream 1: Error starting receiver 1 - java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat 
    at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
    at java.lang.Class.newInstance(Class.java:442) 
    at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119) 
    at com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105) 
    at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78) 
    at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307) 
    at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280) 
    at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160) 
    at com.amazonaws.services.kinesis.AmazonKinesisClient.setEndpoint(AmazonKinesisClient.java:2102) 
    at com.amazonaws.services.kinesis.AmazonKinesisClient.init(AmazonKinesisClient.java:216) 
    at com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:202) 
    at com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:175) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:106) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.<init>(Worker.java:92) 
    at org.apache.spark.streaming.kinesis.KinesisReceiver.onStart(KinesisReceiver.scala:133) 
    at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:125) 
    at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:109) 
    at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:308) 
    at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:300) 
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) 
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) 
    at org.apache.spark.scheduler.Task.run(Task.scala:70) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
    ... 31 more

很明显，我已经使用SBT组装插件创建脂肪罐子，其中包括火花流，室壁运动，asl_2.10具有joda-time-2.9.1.jar作为依赖关系的库。我列出了包含在我的胖罐中的文件，并且该类存在。为了确保它的存在，我也尝试从主类中使用DateTimeFormat，并且我没有任何问题。

我希望有人能帮我解决这个问题。谢谢。

来源

2015-11-24 Adam Reith

你能分享spark-submit命令吗？ – Kaushal

spark-submit --master yarn-client --class主app.jar –

我建议从火花“应用程序详细信息UI”=>“环境”选项卡中检查类路径条目，并检查是否在那里看到任何有效时间。

来源

2015-12-02 16:04:11

在环境页面中，我没有看到joda时间条目，因为库包含在我的胖jar中，显然我将其视为classpath条目。 –

当主设置为纱线客户端时由Spark引发NoClassDefFoundError

回答

相关问题