修改火花数据帧列

Question

我有一个火花数据帧，我想添加一个具有特定值的新列。我曾尝试使用withcolumn函数，但它不按预期工作。我想与特定值的新列或我要替换现有列

Answer 1

见这个例子

我有一个数据帧：

>>> df.show() 
+-------+----+-----+---+ 
| name|year|month|day| 
+-------+----+-----+---+ 
| Ali|2014| 9| 1| 
| Matei|2015| 10| 26| 
|Michael|2015| 10| 25| 
|Reynold|2015| 10| 25| 
|Patrick|2015| 9| 1| 
+-------+----+-----+---+ 

我想补充一个信息，对于每一行，我可以使用​​3210做

>>> from pyspark.sql.functions import lit 
>>> df.withColumn('my_new_column', lit('testing info for all')).show() 
+-------+----+-----+---+--------------------+ 
| name|year|month|day|  my_new_column| 
+-------+----+-----+---+--------------------+ 
| Ali|2014| 9| 1|testing info for all| 
| Matei|2015| 10| 26|testing info for all| 
|Michael|2015| 10| 25|testing info for all| 
|Reynold|2015| 10| 25|testing info for all| 
|Patrick|2015| 9| 1|testing info for all| 
+-------+----+-----+---+--------------------+ 

如果你想添加不同的信息，每行一个列表，你可以使用explode：

>>> from pyspark.sql.functions import explode 
>>> df.withColumn('my_new_column', 
...    explode(array(lit('testing info for all'), 
...        lit('other testing again')))).show() 
+-------+----+-----+---+--------------------+ 
| name|year|month|day|  my_new_column| 
+-------+----+-----+---+--------------------+ 
| Ali|2014| 9| 1|testing info for all| 
| Ali|2014| 9| 1| other testing again| 
| Matei|2015| 10| 26|testing info for all| 
| Matei|2015| 10| 26| other testing again| 
|Michael|2015| 10| 25|testing info for all| 
|Michael|2015| 10| 25| other testing again| 
|Reynold|2015| 10| 25|testing info for all| 
|Reynold|2015| 10| 25| other testing again| 
|Patrick|2015| 9| 1|testing info for all| 
|Patrick|2015| 9| 1| other testing again| 
+-------+----+-----+---+--------------------+