2017-07-19 19 views
0

我现在使用mongo-spark-connector_2.11-2.0.0.jar从mongodb中读取数据,这是一个具有5个配置服务器,5个分片服务器和1个分片服务器的分片群集mongos。我的代码是这样的:带火花的mongodb有错误代码-5

val rdd = MongoSpark.builder().sparkSession(spark).build.toRDD() 
rdd.foreach{ x => { 
    try{ 
     dosomething(x) 
    }catch{ 
     case e: Throwable => e.printStackTrace() 
    } 
}} 

和我的火花配置为:

.config("spark.cores.max", 60)  
.config("spark.executor.cores", 12) 
.config("spark.executor.memory", "32g") 
.config("spark.mongodb.input.uri", "mongodb://192.168.12.161:27017/datab.origin2") 

有集合中的27,000,000文档,当火花应用程序启动RDD有2500分。运行一段时间后,我得到了一个错误代码-5在我的司机:

Caused by: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 2639909050433532364 not found on server 192.168.12.161:27017' on server 192.168.12.161:27017 at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27) at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:213) at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:103) at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:918) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:918) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)

我读了工人火花日志,这是第一次433任务开始并在第一时间就返回错误:

17/07/17 19:14:23 INFO CoarseGrainedExecutorBackend: Got assigned task 433

17/07/17 19:14:23 INFO Executor: Running task 433.0 in stage 0.0 (TID 433)

17/07/17 19:37:31 ERROR Executor: Exception in task 433.0 in stage 0.0 (TID 433) com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 2639909048849185072 not found on server 192.168.12.161:27017' on server 192.168.12.161:27017

而这mongs登录:

2017-07-17T19:24:49.677+0800 I QUERY [ClusterCursorCleanupJob] Marking cursor id 2639909048849185072 for deletion, idle since 2017-07-17T19:14:46.055+0800

我要寻找的错误代码-5和知道它发生时光标不是10分钟使用,但其他分区只需要3-4分钟,完成这个过程唱。 当我使用java驱动程序时,我可以使用noCursorTimeout()来避免此问题,当我使用mongo-spark-connector时,如何解决此问题?或者我可以用我的分片群来解决它?

回答

0

当我在配置spark.master=local[16]的本地运行spark时,我得到了同样的错误。我花了很多时间在互联网上寻找解决方案,但没有发现。最后,我试图设置spark.master=local[1],它工作!