我现在使用mongo-spark-connector_2.11-2.0.0.jar从mongodb中读取数据,这是一个具有5个配置服务器,5个分片服务器和1个分片服务器的分片群集mongos。我的代码是这样的:带火花的mongodb有错误代码-5
val rdd = MongoSpark.builder().sparkSession(spark).build.toRDD()
rdd.foreach{ x => {
try{
dosomething(x)
}catch{
case e: Throwable => e.printStackTrace()
}
}}
和我的火花配置为:
.config("spark.cores.max", 60)
.config("spark.executor.cores", 12)
.config("spark.executor.memory", "32g")
.config("spark.mongodb.input.uri", "mongodb://192.168.12.161:27017/datab.origin2")
有集合中的27,000,000文档,当火花应用程序启动RDD有2500分。运行一段时间后,我得到了一个错误代码-5在我的司机:
Caused by: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 2639909050433532364 not found on server 192.168.12.161:27017' on server 192.168.12.161:27017 at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27) at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:213) at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:103) at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:918) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:918) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)
我读了工人火花日志,这是第一次433任务开始并在第一时间就返回错误:
17/07/17 19:14:23 INFO CoarseGrainedExecutorBackend: Got assigned task 433
17/07/17 19:14:23 INFO Executor: Running task 433.0 in stage 0.0 (TID 433)
17/07/17 19:37:31 ERROR Executor: Exception in task 433.0 in stage 0.0 (TID 433) com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 2639909048849185072 not found on server 192.168.12.161:27017' on server 192.168.12.161:27017
而这mongs登录:
2017-07-17T19:24:49.677+0800 I QUERY [ClusterCursorCleanupJob] Marking cursor id 2639909048849185072 for deletion, idle since 2017-07-17T19:14:46.055+0800
我要寻找的错误代码-5和知道它发生时光标不是10分钟使用,但其他分区只需要3-4分钟,完成这个过程唱。 当我使用java驱动程序时,我可以使用noCursorTimeout()
来避免此问题,当我使用mongo-spark-connector时,如何解决此问题?或者我可以用我的分片群来解决它?