PySpark：许多功能，标记点RDD

Question

新星火，和所有的例子我已阅读处理小数据集，如：

RDD = sc.parallelize([ 
LabeledPoint(1, [1.0, 2.0, 3.0]), 
LabeledPoint(2, [3.0, 4.0, 5.0]), 

但是，我有50多个功能的大型数据集。一排

u'2596,51,3,258,0,510,221,232,148,6279,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,5' 

的

例子我想快速创建一个Labeledpoint RDD在PySpark。我尝试将最后一个位置作为Labeledpoint RDD中的第一个数据点索引，然后将第一个n-1位置索引为稠密向量。但是，我收到以下错误。任何指导表示赞赏！注意：如果在创建标记点时将[]更改为（），则会出现“无效语法”错误。

df = myDataRDD.map(lambda line: line.split(',')) 
data = [ 
    LabeledPoint(df[54], df[0:53]) 
] 
TypeError: 'PipelinedRDD' object does not support indexing 
--------------------------------------------------------------------------- 
TypeError         Traceback (most recent call last) 
<ipython-input-67-fa1b56e8441e> in <module>() 
     2 df = myDataRDD.map(lambda line: line.split(',')) 
     3 data = [ 
----> 4  LabeledPoint(df[54], df[0:53]) 
     5 ] 

TypeError: 'PipelinedRDD' object does not support indexing 

Answer 1

当你的错误状态，你无法通过索引访问RDD。 你需要一个第二map语句将序列转化为LabeledPoint小号

rows = [u'2596,51,3,258,0,510,221,232,148,6279,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,5', u'2596,51,3,258,0,510,221,232,148,6279,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,5'] 

rows_rdd = sc.parallelize(rows) # create RDD with given rows 

labeled_points_rdd = rows_rdd\ 
        .map(lambda row: row.split(','))\     # split rows into sequences 
        .map(lambda seq: LabeledPoint(seq[-1],seq[:-2])) # create Labeled Points from these sequences with last Item as label 

print labeled_points_rdd.take(2) 
# prints [LabeledPoint(5.0, [2596.0,51.0,3.0,258.0,0.0,510.0,221.0,...]), 
#   LabeledPoint(5.0,[2596.0,51.0,3.0,258.0,0.0,510.0,221.0,...]) 

注意的是Python负指数让你访问序列倒退。

随着.take(n)你然后从你的RDD获得第一个n元素。

希望这会有所帮助。

Answer 2

您不能使用索引，而必须使用Spark API中可用的方法。所以：

data = [ LabeledPoint(myDataRDD.take(RDD.count()), #Last element 
         myDataRDD.top(RDD.count()-1)) #All but last ] 

（未经测试，不过，这是一般的想法）