从数据框中创建地图火花斯卡拉

Question

我有一个JSON字符串，如下面一个数据帧

aaa | bbb | ccc |ddd | eee   
-------------------------------------- 
    100 | xxxx | 123 |yyy|2017 
    100 | yyyy | 345 |zzz|2017 
    200 | rrrr | 500 |qqq|2017 
    300 | uuuu | 200 |ttt|2017 
    200 | iiii | 500 |ooo|2017 

我想要得到的结果作为

{100,[{xxxx:{123,yyy}},{yyyy:{345,zzz}}],2017} 
{200,[{rrrr:{500,qqq}},{iiii:{500,ooo}}],2017} 
{300,[{uuuu:{200,ttt}}],2017} 

请帮助

Answer 1

这工作：

val df = data 
    .withColumn("cd", array('ccc, 'ddd)) // create arrays of c and d 
    .withColumn("valuesMap", map('bbb, 'cd)) // create mapping 
    .withColumn("values", collect_list('valuesMap) // collect mappings 
       .over(Window.partitionBy('aaa))) 
    .withColumn("eee", first('eee) // e is constant, just get first value of Window 
       .over(Window.partitionBy('aaa))) 
    .select("aaa", "values", "eee") // select only columns that are in the question selected 
    .select(to_json(struct("aaa", "values", "eee")).as("value")) // create JSON 

Ma确保你有进口org.apache.spark.sql.functions._和org.apache.spark.sql.expression._