如果一个数字存在于一个字符串中，请将该字符串替换为null - Spark

Question

我是Spark-Scala的新手。我正在尝试清理一些数据。我在清理FIRSTNAME和LASTNAME列时遇到了问题。字符串中有数字。如何识别数字并用空字符替换整个字符串。

Consider the following dataframe: 

+---------+--------+ 
|FIRSTNAME|LASTNAME| 
+---------+--------+ 
| Steve| 10 C| 
|  Mark| 9436| 
| Brian| Lara| 
+---------+--------+ 

How do I get this: 

+---------+--------+ 
|FIRSTNAME|LASTNAME| 
+---------+--------+ 
| Steve| null| 
|  Mark| null| 
| Brian| Lara| 
+---------+--------+ 

任何帮助将不胜感激。非常感谢你！

编辑：

scala> df2.withColumn("LASTNAME_TEMP", when(col("LASTNAME").contains("1"), null).otherwise(col("LASTNAME"))).show() 
+---------+--------+-------------+ 
|FIRSTNAME|LASTNAME|LASTNAME_TEMP| 
+---------+--------+-------------+ 
| Steve| 10 C|   null| 
|  Mark| 9436|   9436| 
| Brian| Lara|   Lara| 
+---------+--------+-------------+ 

但上面的代码将只在一个字符串。我宁愿它拿一个字符串列表。例如：

val numList = List("1", "2", "3", "4", "5", "6", "7", "8", "9", "0") 

我宣布上述名单，并运行下面的代码：

scala> df2.filter(col("LASTNAME").isin(numList:_*)).show() 

我得到了以下数据框：

+---------+--------+ 
|FIRSTNAME|LASTNAME| 
+---------+--------+ 
+---------+--------+ 

Answer 1

您可以使用正则表达式与rlike模式匹配：

val df = Seq(
    ("Steve", "10 C"), 
    ("Mark", "9436"), 
    ("Brian", "Lara") 
).toDF(
    "FIRSTNAME", "LASTNAME" 
) 

// Keep original LASTNAME in new column only if it doesn't consist of any digit 
val df2 = df.withColumn("LASTNAMEFIXED", when(! col("LASTNAME").rlike(".*[0-9]+.*"), col("LASTNAME"))) 

+---------+--------+-------------+ 
|FIRSTNAME|LASTNAME|LASTNAMEFIXED| 
+---------+--------+-------------+ 
| Steve| 10 C|   null| 
|  Mark| 9436|   null| 
| Brian| Lara|   Lara| 
+---------+--------+-------------+