Hadoop最后地图作业卡住了 - 需要帮助

Question

我在做一些使用hadoop map-reduce作业的文本处理。我的工作完成了99.2％，并停留在最后的地图工作上。

地图输出的最后几行显示如下。最后一次，当这个问题发生时，我试图打印出从地图上删除的键值，并注意到其中一个关键字有大量值与它相关联，我认为它在排序这些值时出现卡住现象。然后，我停止从地图作业中发现该密钥，并且它工作正常。

我认为，同样的问题再次出现，打印出键值对是一项繁琐的工作，因为这项工作需要时间。有更好的选择吗？像配置hadoop，如果他们在分类上花费太多时间，就会忘记几个键。有没有这样的事情。

 
2010-10-20 14:43:32,274 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 14:43:32,274 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 79698262; bufvoid = 99614720 
2010-10-20 14:43:32,274 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 6601; length = 327680 
2010-10-20 14:43:33,272 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 
2010-10-20 14:50:44,113 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 14:50:44,113 INFO org.apache.hadoop.mapred.MapTask: bufstart = 79698262; bufend = 59800449; bufvoid = 99614720 
2010-10-20 14:50:44,113 INFO org.apache.hadoop.mapred.MapTask: kvstart = 6601; kvend = 9039; length = 327680 
2010-10-20 14:50:44,864 INFO org.apache.hadoop.mapred.MapTask: Finished spill 1 
2010-10-20 14:58:33,105 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 14:58:33,105 INFO org.apache.hadoop.mapred.MapTask: bufstart = 59800449; bufend = 39893455; bufvoid = 99614720 
2010-10-20 14:58:33,105 INFO org.apache.hadoop.mapred.MapTask: kvstart = 9039; kvend = 11228; length = 327680 
2010-10-20 14:58:33,817 INFO org.apache.hadoop.mapred.MapTask: Finished spill 2 
2010-10-20 15:06:48,675 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 15:06:48,675 INFO org.apache.hadoop.mapred.MapTask: bufstart = 39893455; bufend = 20000988; bufvoid = 99614720 
2010-10-20 15:06:48,675 INFO org.apache.hadoop.mapred.MapTask: kvstart = 11228; kvend = 13286; length = 327680 
2010-10-20 15:06:49,395 INFO org.apache.hadoop.mapred.MapTask: Finished spill 3 
2010-10-20 15:15:23,514 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 15:15:23,514 INFO org.apache.hadoop.mapred.MapTask: bufstart = 20000988; bufend = 78879; bufvoid = 99614720 
2010-10-20 15:15:23,514 INFO org.apache.hadoop.mapred.MapTask: kvstart = 13286; kvend = 15265; length = 327680 
2010-10-20 15:15:24,230 INFO org.apache.hadoop.mapred.MapTask: Finished spill 4 
2010-10-20 15:24:35,797 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 15:24:35,797 INFO org.apache.hadoop.mapred.MapTask: bufstart = 78879; bufend = 79807573; bufvoid = 99614720 
2010-10-20 15:24:35,797 INFO org.apache.hadoop.mapred.MapTask: kvstart = 15265; kvend = 17188; length = 327680 
2010-10-20 15:24:36,500 INFO org.apache.hadoop.mapred.MapTask: Finished spill 5 
2010-10-20 15:33:33,391 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 15:33:33,391 INFO org.apache.hadoop.mapred.MapTask: bufstart = 79807573; bufend = 59907680; bufvoid = 99614720 
2010-10-20 15:33:33,391 INFO org.apache.hadoop.mapred.MapTask: kvstart = 17188; kvend = 19074; length = 327680 
2010-10-20 15:33:34,114 INFO org.apache.hadoop.mapred.MapTask: Finished spill 6 
2010-10-20 15:42:39,913 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 15:42:39,913 INFO org.apache.hadoop.mapred.MapTask: bufstart = 59907680; bufend = 40011208; bufvoid = 99614720 
2010-10-20 15:42:39,913 INFO org.apache.hadoop.mapred.MapTask: kvstart = 19074; kvend = 20926; length = 327680 
2010-10-20 15:42:40,597 INFO org.apache.hadoop.mapred.MapTask: Finished spill 7 
2010-10-20 15:51:49,668 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 15:51:49,668 INFO org.apache.hadoop.mapred.MapTask: bufstart = 40011208; bufend = 20111383; bufvoid = 99614720 
2010-10-20 15:51:49,668 INFO org.apache.hadoop.mapred.MapTask: kvstart = 20926; kvend = 22759; length = 327680 
2010-10-20 15:51:50,378 INFO org.apache.hadoop.mapred.MapTask: Finished spill 8 
2010-10-20 16:01:05,893 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 16:01:05,893 INFO org.apache.hadoop.mapred.MapTask: bufstart = 20111383; bufend = 196929; bufvoid = 99614720 
2010-10-20 16:01:05,894 INFO org.apache.hadoop.mapred.MapTask: kvstart = 22759; kvend = 24572; length = 327680 
2010-10-20 16:01:06,634 INFO org.apache.hadoop.mapred.MapTask: Finished spill 9 
2010-10-20 16:10:25,000 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 16:10:25,000 INFO org.apache.hadoop.mapred.MapTask: bufstart = 196929; bufend = 79900267; bufvoid = 99614720 
2010-10-20 16:10:25,000 INFO org.apache.hadoop.mapred.MapTask: kvstart = 24572; kvend = 26370; length = 327680 
2010-10-20 16:10:25,776 INFO org.apache.hadoop.mapred.MapTask: Finished spill 10 
2010-10-20 16:19:48,283 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full= true 
2010-10-20 16:19:48,283 INFO org.apache.hadoop.mapred.MapTask: bufstart = 79900267; bufend = 59993676; bufvoid = 99614720 
2010-10-20 16:19:48,284 INFO org.apache.hadoop.mapred.MapTask: kvstart = 26370; kvend = 28152; length = 327680 
2010-10-20 16:19:49,042 INFO org.apache.hadoop.mapred.MapTask: Finished spill 11 

谢谢

Answer 1

没有什么在Hadoop中，将知道地图的特定invokation（）被发射键值对的了大量资金。我猜你在map（）函数中有某种循环发出这些键值对。如果发射超过N对，您可以简单地将环路编码为短路。

另一种方法是找出某种方式对输入值进行分区，以便映射器处理更多粒度块，以便所有映射器执行大致相同的工作量。

我不确定你想要做什么，所以这些建议可能不适用。希望这可以帮助。