1
你好我的具体要求,我想写一个UDAF,它只是收集所有的输入行。如何编写一个简单的行集合的Spark UDAF?
输入是一个两列的行,Double Type;
中级架构,“我想”,是的ArrayList(纠正我,如果我错了)
返回的数据类型是ArrayList的
我写我的UDAF的“想法”,但我希望有人帮我完成它。
class CollectorUDAF() extends UserDefinedAggregateFunction {
// Input Data Type Schema
def inputSchema: StructType = StructType(Array(StructField("value", DoubleType), StructField("y", DoubleType)))
// Intermediate Schema
def bufferSchema = util.ArrayList[Array(StructField("value", DoubleType), StructField("y", DoubleType)]
// Returned Data Type .
def dataType: DataType = util.ArrayList[Array(StructField("value", DoubleType), StructField("y", DoubleType)]
// Self-explaining
def deterministic = true
// This function is called whenever key changes
def initialize(buffer: MutableAggregationBuffer) = {
}
// Iterate over each entry of a group
def update(buffer: MutableAggregationBuffer, input: Row) = {
}
// Called after all the entries are exhausted.
def evaluate(buffer: Row) = {
}
def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
}
}