访问WrappedArray元素

Question

我有一个火花数据帧，这里是架构：

|-- eid: long (nullable = true) 
|-- age: long (nullable = true) 
|-- sex: long (nullable = true) 
|-- father: array (nullable = true) 
| |-- element: array (containsNull = true) 
| | |-- element: long (containsNull = true) 

和行的样本:.

df.select(df['father']).show() 
+--------------------+ 
|    father| 
+--------------------+ 
|[WrappedArray(-17...| 
|[WrappedArray(-11...| 
|[WrappedArray(13,...| 
+--------------------+ 

和类型是

DataFrame[father: array<array<bigint>>] 

怎样可以访问内部数组的每个元素？例如第一行中的-17？ 我尝试了像df.select(df['father'])(0)(0).show()这样的不同的东西，但没有运气。

Answer 1

如果我没有记错，在Python语法是

df.select(df['father'])[0][0].show() 

或

df.select(df['father']).getItem(0).getItem(0).show() 

看一些例子在这里：http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=column#pyspark.sql.Column

Answer 2

在阶溶液应尽可能

import org.apache.spark.sql.functions._ 
val data = sparkContext.parallelize("""{"eid":1,"age":30,"sex":1,"father":[[1,2]]}""" :: Nil) 
val dataframe = sqlContext.read.json(data).toDF() 

数据帧看起来

+---+---+---+--------------------+ 
|eid|age|sex|father    | 
+---+---+---+--------------------+ 
|1 |30 |1 |[WrappedArray(1, 2)]| 
+---+---+---+--------------------+ 

溶液应

dataframe.select(col("father")(0)(0) as("first"), col("father")(0)(1) as("second")).show(false) 

输出应该是

+-----+------+ 
|first|second| 
+-----+------+ 
|1 |2  | 
+-----+------+ 

Answer 3

另一个阶的答案是这样的：

df.select(col("father").getItem(0) as "father_0", col("father").getItem(1) as "father_1")