0
使用MLlib功能ALS我已经从文件中读取这样的:错误在星火
val ratingText = sc.textFile("/home/cloudera/rec_data/processed_data/ratings/000000_0")
使用下面的函数来分析这样的数据:
def parseRating(str: String): Rating= {
val fields = str.split(",")
Rating(fields(0).toInt, fields(1).trim.toInt, fields(2).trim.toDouble)
}
而且创造了一个RDD,这是然后分裂成不同的RDDS
val ratingsRDD = ratingText.map(x=>parseRating(x)).cache()
val splits = ratingsRDD.randomSplit(Array(0.8, 0.2), 0L)
val trainingRatingsRDD = splits(0).cache()
使用的训练RDD创建模型如下:
val model = (new ALS().setRank(20).setIterations(10) .run(trainingRatingsRDD))
我得到以下错误在最后的命令
16/10/28 01:03:44 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
16/10/28 01:03:44 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
16/10/28 01:03:46 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
16/10/28 01:03:46 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
编辑:T. Gaweda的建议,帮助在消除错误,但我仍然得到以下警告:
16/10/28 01:53:59 WARN Executor: 1 block locks were not released by TID = 60:
[rdd_420_0]
16/10/28 01:54:00 WARN Executor: 1 block locks were not released by TID = 61:
[rdd_421_0]
我认为这导致了一个空的模型,因为下一步导致以下错误:
VAL topRecsForUser = model.recommendProducts(4276736,3)
错误是:
java.util.NoSuchElementException: next on empty iterator at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
请帮帮忙!
感谢@T。 Gaweda,在删除警告方面效果很好,但以下错误持续存在。你可以请看一看::val model =(new ALS()。setRank(20).setIterations(10).run(trainingRatingsRDD)) 16/10/28 01:53:59 WARN Executor:1 block locks were不是由TID = 60发布: [rdd_420_0] 16/10/28 01:54:00警告执行程序:1个块锁不是由TID = 61发布的: [rdd_421_0] model:org.apache.spark.mllib .recommendation.MatrixFactorizationModel = [email protected]a853a7d – Kuwali
另外,在此之后,当我使用val topRecsForUser = model.recommendProducts(4276736,3)时,我得到一个错误,说java.util.NoSuchElementException:next在空迭代器 在scala.collection.Iterator $$ anon $ 2.next(Iterator.scala:39) 这是否意味着该模型是空的?请帮忙! – Kuwali
这是你的代码有问题 - 你试图用用户标识预测模型中不存在 - http://stackoverflow.com/questions/32488328/mllib-matrixfactorizationmodel-recommendproductsuser-num-failing-on-some-user –