我想找到一种有效的方法来使用数据框在PySpark中创建备用矢量。稀疏矢量pyspark
比方说,考虑到交易输入:
df = spark.createDataFrame([
(0, "a"),
(1, "a"),
(1, "b"),
(1, "c"),
(2, "a"),
(2, "b"),
(2, "b"),
(2, "b"),
(2, "c"),
(0, "a"),
(1, "b"),
(1, "b"),
(2, "cc"),
(3, "a"),
(4, "a"),
(5, "c")
], ["id", "category"])
+---+--------+
| id|category|
+---+--------+
| 0| a|
| 1| a|
| 1| b|
| 1| c|
| 2| a|
| 2| b|
| 2| b|
| 2| b|
| 2| c|
| 0| a|
| 1| b|
| 1| b|
| 2| cc|
| 3| a|
| 4| a|
| 5| c|
+---+--------+
在总结格式:
df.groupBy(df["id"],df["category"]).count().show()
+---+--------+-----+
| id|category|count|
+---+--------+-----+
| 1| b| 3|
| 1| a| 1|
| 1| c| 1|
| 2| cc| 1|
| 2| c| 1|
| 2| a| 1|
| 1| a| 1|
| 0| a| 2|
+---+--------+-----+
我的目标是通过ID来获得这个输出:
+---+-----------------------------------------------+
| id| feature |
+---+-----------------------------------------------+
| 2|SparseVector({a: 1.0, b: 3.0, c: 1.0, cc: 1.0})|
请您指点我正确的方向?在Java中使用mapreduce对我来说似乎更容易一些。