java.util.concurrent.ExecutionException：java.lang.NoSuchMethodError当运行kinesis spark作业时出现异常

我很难试图找出我的spark（2.1.0）作业scala依赖关系。java.util.concurrent.ExecutionException：java.lang.NoSuchMethodError当运行kinesis spark作业时出现异常

我build.sbt文件：

name := "test" 

version := "0.0.1" 

scalaVersion := "2.11.0" 

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "2.1.0" 
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.5.1" 

assemblyJarName in assembly := "test.jar" 


mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => 
    { 
    case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard 
    case m if m.startsWith("META-INF") => MergeStrategy.discard 
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first 
    case PathList("org", "apache", xs @ _*) => MergeStrategy.first 
    case PathList("org", "jboss", xs @ _*) => MergeStrategy.first 
    case "about.html" => MergeStrategy.rename 
    case "reference.conf" => MergeStrategy.concat 
    case _ => MergeStrategy.first 
    } 
} 

exportJars:= true 

mainClass in assembly := Some("test.Job") 
```

当我跑我的工作，它会抛出异常java.lang.NoSuchMethodError。

17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 87 
17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 86 
17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 85 
17/03/08 05:19:15 ERROR worker.Worker: Worker.run caught exception, sleeping for 1000 milli seconds! 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long; 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:156) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.consumeShard(ShardConsumer.java:125) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:335) 
    at org.apache.spark.streaming.kinesis.KinesisReceiver$$anon$1.run(KinesisReceiver.scala:174) 
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long; 
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) 
    at java.util.concurrent.FutureTask.get(FutureTask.java:192) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:136) 
    ... 3 more 
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long; 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResultAndRecordMillisBehindLatest(ProcessTask.java:291) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResult(ProcessTask.java:249) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:120) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) 
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745)

来源

2017-03-08 mikeklein

你检查了这个http://stackoverflow.com/questions/38237345/kinesis-stream-with-empty-records-in-google-dataproc-with-spark-1-6-1-hadoop-2-7 ？ – semsorock

你在aws emr上运行这个吗？如果是以下链接可以帮助：

Spark streaming 1.6.1 is not working with Kinesis asl 1.6.1 and asl 2.0.0-preview http://www.waitingforcode.com/apache-spark/shading-solution-dependency-hell-spark/read

基本AWS EMR支持protobuf的2.5，但是火花核心，火花流和火花流，室壁运动-ASL版本都依赖的protobuf 2.6.1，当我们遇到这个问题的时候，我们通过一个阴影的jar来解决这个问题，上面的两个链接给出了如何设置up的好例子。

来源

2017-10-18 03:19:02

java.util.concurrent.ExecutionException：java.lang.NoSuchMethodError当运行kinesis spark作业时出现异常

回答

相关问题