Mahout RecommenderJob不收敛

Question

这是我的第一篇SO帖子，如果我错过了任何重要的东西，请让我知道。我是Mahout/Hadoop初学者，并且正在尝试将分布式推荐引擎放在一起。

为了模拟在远程集群上的工作，我在我的机器上设置了hadoop以与位于我的机器上的Ubuntu VM（使用VirtualBox）通信，该机器上安装了hadoop。这个设置似乎工作正常，我现在试图在一个非常小的试用数据集上运行Mahout的RecommenderJob作为测试。

输入包括含有格式大约50用户偏好的.csv文件（保存在Hadoop的DFS）的：userID, itemID, preference ......我正在运行的命令是：

hadoop jar /Users/MyName/src/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -Dmapred.input.dir=/user/MyName/Recommendations/input/TestRatings.csv -Dmapred.output.dir=/user/MyName/Recommendations/output -s SIMILARITY_PEARSON_CORELLATION 

其中TestRatings.csv是包含首选项的文件和output是所需的输出目录。

起初的工作看起来像它的运行良好，我也得到了以下的输出：

12/12/11 12:26:21 INFO common.AbstractJob: Command line arguments: {--booleanData=[false], --endPhase=[2147483647], --maxPrefsPerUser=[10], --maxPrefsPerUserInItemSimilarity=[1000], --maxSimilaritiesPerItem=[100], --minPrefsPerUser=[1], --numRecommendations=[10], --similarityClassname=[SIMILARITY_PEARSON_CORELLATION], --startPhase=[0], --tempDir=[temp]} 
12/12/11 12:26:21 INFO common.AbstractJob: Command line arguments: {--booleanData=[false], --endPhase=[2147483647], --input=[/user/Naaman/Delphi/input/TestRatings.csv], --maxPrefsPerUser=[1000], --minPrefsPerUser=[1], --output=[temp/preparePreferenceMatrix], --ratingShift=[0.0], --startPhase=[0], --tempDir=[temp]} 
12/12/11 12:26:21 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 
12/12/11 12:26:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
12/12/11 12:26:22 INFO input.FileInputFormat: Total input paths to process : 1 
12/12/11 12:26:22 WARN snappy.LoadSnappy: Snappy native library not loaded 
12/12/11 12:26:22 INFO mapred.JobClient: Running job: job_local_0001 
12/12/11 12:26:22 INFO mapred.Task: Using ResourceCalculatorPlugin : null 
12/12/11 12:26:22 INFO mapred.MapTask: io.sort.mb = 100 
12/12/11 12:26:22 INFO mapred.MapTask: data buffer = 79691776/99614720 
12/12/11 12:26:22 INFO mapred.MapTask: record buffer = 262144/327680 
12/12/11 12:26:22 INFO mapred.MapTask: Starting flush of map output 
12/12/11 12:26:22 INFO compress.CodecPool: Got brand-new compressor 
12/12/11 12:26:22 INFO mapred.MapTask: Finished spill 0 
12/12/11 12:26:22 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 
12/12/11 12:26:22 INFO mapred.LocalJobRunner: 
12/12/11 12:26:22 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done. 
12/12/11 12:26:22 INFO mapred.Task: Using ResourceCalculatorPlugin : null 
12/12/11 12:26:22 INFO mapred.ReduceTask: ShuffleRamManager: MemoryLimit=1491035776, MaxSingleShuffleLimit=372758944 
12/12/11 12:26:22 INFO compress.CodecPool: Got brand-new decompressor 
12/12/11 12:26:22 INFO compress.CodecPool: Got brand-new decompressor 
12/12/11 12:26:22 INFO compress.CodecPool: Got brand-new decompressor 
12/12/11 12:26:22 INFO compress.CodecPool: Got brand-new decompressor 
12/12/11 12:26:22 INFO compress.CodecPool: Got brand-new decompressor 
12/12/11 12:26:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Thread started: Thread for merging on-disk files 
12/12/11 12:26:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Thread started: Thread for merging in memory files 
12/12/11 12:26:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Thread waiting: Thread for merging on-disk files 
12/12/11 12:26:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Need another 1 map output(s) where 0 is already in progress 
12/12/11 12:26:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Thread started: Thread for polling Map Completion Events 
12/12/11 12:26:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0 dup hosts) 
12/12/11 12:26:23 INFO mapred.JobClient: map 100% reduce 0% 
12/12/11 12:26:28 INFO mapred.LocalJobRunner: reduce > copy > 
12/12/11 12:26:31 INFO mapred.LocalJobRunner: reduce > copy > 
12/12/11 12:26:37 INFO mapred.LocalJobRunner: reduce > copy > 

但随后的最后三行无限重复（我把它一夜之间...），与两行：

12/12/11 12:27:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Need another 1 map output(s) where 0 is already in progress 
12/12/11 12:27:22 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0 dup hosts) 

重复每十行。

我不确定我的输入是否有问题，或者试验数据的细小尺寸是否让事情变得糟糕。任何帮助和/或建议的最佳途径去这件事将不胜感激。

p.s.我试图按照https://www.box.com/s/041rdjeh7sny128r2uki的说明操作

Answer 1

这实际上是一个Hadoop或群集问题。它正在等待未到来的映射器输出。在映射阶段寻找早期的失败。