Spark将列组合为嵌套数组

如何将spark中的列组合为嵌套数组？Spark将列组合为嵌套数组

val inputSmall = Seq(
    ("A", 0.3, "B", 0.25), 
    ("A", 0.3, "g", 0.4), 
    ("d", 0.0, "f", 0.1), 
    ("d", 0.0, "d", 0.7), 
    ("A", 0.3, "d", 0.7), 
    ("d", 0.0, "g", 0.4), 
    ("c", 0.2, "B", 0.25)).toDF("column1", "transformedCol1", "column2", "transformedCol2")

以类似

+-------+---------------+---------------+------- + 
|column1|transformedCol1|transformedCol2|combined| 
+-------+---------------+---------------+------ -+ 
|  A|   0.3|   0.3[0.3, 0.3]| 
+-------+---------------+---------------+-------+

来源

2016-12-20 Georg Heiler

如果你想多列合并成数组类型的新列，您可以使用array功能的东西：

import org.apache.spark.sql.functions._ 
val result = inputSmall.withColumn("combined", array($"transformedCol1", $"transformedCol2")) 
result.show() 

+-------+---------------+-------+---------------+-----------+ 
|column1|transformedCol1|column2|transformedCol2| combined| 
+-------+---------------+-------+---------------+-----------+ 
|  A|   0.3|  B|   0.25|[0.3, 0.25]| 
|  A|   0.3|  g|   0.4| [0.3, 0.4]| 
|  d|   0.0|  f|   0.1| [0.0, 0.1]| 
|  d|   0.0|  d|   0.7| [0.0, 0.7]| 
|  A|   0.3|  d|   0.7| [0.3, 0.7]| 
|  d|   0.0|  g|   0.4| [0.0, 0.4]| 
|  c|   0.2|  B|   0.25|[0.2, 0.25]| 
+-------+---------------+-------+---------------+-----------+

来源

2016-12-20 10:19:52

Spark将列组合为嵌套数组

回答

相关问题