1
我正在使用spark 1.4.1。当我试图播放随机林模型就说明我这个错误:PySpark中的广播随机森林模型
Traceback (most recent call last):
File "/gpfs/haifa/home/d/a/davidbi/codeBook/Nice.py", line 358, in <module>
broadModel = sc.broadcast(model)
File "/opt/apache/spark-1.4.1-bin-hadoop2.4_doop/python/lib/pyspark.zip/pyspark/context.py", line 698, in broadcast
File "/opt/apache/spark-1.4.1-bin-hadoop2.4_doop/python/lib/pyspark.zip/pyspark/broadcast.py", line 70, in __init__
File "/opt/apache/spark-1.4.1-bin-hadoop2.4_doop/python/lib/pyspark.zip/pyspark/broadcast.py", line 78, in dump
File "/opt/apache/spark-1.4.1-bin-hadoop2.4_doop/python/lib/pyspark.zip/pyspark/context.py", line 252, in __getnewargs__
Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
例子的代码我试图执行:
sc = SparkContext(appName= "Something")
model = RandomForest.trainRegressor(sc.parallelize(data), categoricalFeaturesInfo=categorical, numTrees=100, featureSubsetStrategy="auto", impurity='variance', maxDepth=4)
broadModel= sc.broadcast(model)
如果有人能与我会帮助我非常感谢! 非常感谢!
有你为什么需要广播整个模型的一个原因?该模型可以对输入的RDD进行预测。 – Magsol
有多个模型(在我的情况下,每个模型定义组)。每个样本需要从每个模型中获取预测,以了解他最擅长的组。我正在处理大数据,因此我需要将模型广播到映射器。 – dadibiton