在Spark中总结多列

Question

如何在Spark中总结多个列？例如，在SparkR中，以下代码用于获取一列的总和，但如果我尝试获取df中两列的总和，则会出现错误。

# Create SparkDataFrame 
df <- createDataFrame(faithful) 

# Use agg to sum total waiting times 
head(agg(df, totalWaiting = sum(df$waiting))) 
##This works 

# Use agg to sum total of waiting and eruptions 
head(agg(df, total = sum(df$waiting, df$eruptions))) 
##This doesn't work 

无论SparkR或PySpark代码将工作。

Answer 1

org.apache.spark.sql.functions.sum(Column e) 

聚合函数：返回表达式中所有值的总和。

正如你所看到的，sum只需一个栏为输入，从而sum(df$waiting, df$eruptions)不会work.Since你婉总结数字字段，你可以做sum(df("waiting") + df("eruptions"))。如果你婉总结了各个列的值，那么，你可以df.agg(sum(df$waiting),sum(df$eruptions)).show

Answer 2

你可以做一些像下面pyspark

>>> from pyspark.sql import functions as F 
>>> df = spark.createDataFrame([("a",1,10), ("b",2,20), ("c",3,30), ("d",4,40)], ["col1", "col2", "col3"]) 
>>> df.groupBy("col1").agg(F.sum(df.col2+df.col3)).show() 
+----+------------------+ 
|col1|sum((col2 + col3))| 
+----+------------------+ 
| d|    44| 
| c|    33| 
| b|    22| 
| a|    11| 
+----+------------------+ 

Answer 3

sparkR代码：

library(SparkR) 
df <- createDataFrame(sqlContext,faithful) 
w<-agg(df,sum(df$waiting)),agg(df,sum(df$eruptions)) 
head(w[[1]]) 
head(w[[2]]) 

Answer 4

对于PySpark，如果你不希望明确键入出列：

from operator import add 
from functools import reduce 
new_df = df.withColumn('total',reduce(add, [F.col(x) for x in numeric_col_list]))