我在AWS S3中有一个CSV文件,正在加载到AWS Glue,即用于对来自S3的源数据文件应用转换。它提供了PySpark脚本环境。数据看起来有点像这样:如何检查列的数值是否包含通过SQL查询的字母
"ID","CNTRY_CD","SUB_ID","PRIME_KEY","DATE"
"123","IND","25635525","11243749772","2017-10-17"
"123","IND","25632349","112322abcd","2017-10-17"
"123","IND","25635234","11243kjsd434","2017-10-17"
"123","IND","25639822","1124374343","2017-10-17"
预期结果应该是这样的:
"123","IND","25632349","112322abcd","2017-10-17"
"123","IND","25635234","11243kjsd434","2017-10-17"
在这里,我通过整型的名字“PRIME_KEY”工作的领域,可能包含英文字母,这导致数据格式不正确。
现在的需求是,我需要找出Integer类型的主键列是否包含任何使用SQL查询的数字值的字母数字字符。正则表达式到目前为止,我已经尝试了几个变种,以做到这一点像下面的一个,但没有运气:
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
# s3 output directory
output_dir = "s3://aws-glue-scripts../.."
# Data Catalog: database and table name
db_name = "sampledb"
glue_tbl_name = "sampleTable"
datasource = glueContext.create_dynamic_frame.from_catalog(database = db_name, table_name = glue_tbl_name)
datasource_df = datasource.toDF()
datasource_df.registerTempTable("sample_tbl")
invalid_primarykey_values_df = spark.sql("SELECT * FROM sample_tbl WHERE CAST(PRIME_KEY AS STRING) RLIKE '([a-z]+[0-9]+)|([0-9]+[a-z]+)'")
invalid_primarykey_values_df.show()
这个脚本的输出如下:
SELECT *
FROM table_name
WHERE column_name IS NOT NULL AND
CAST(column_name AS VARCHAR(100)) LIKE \'%[0-9a-z0-9]%\'
源脚本
+ --- + -------- + -------- + ------------ + ---------- + ----------- + --------------- +
| ID | CNTRY_CD | SUB_ID | PRIME_KEY | DATE |
+ --- + -------- + -------- + ------------ + ---------- + ----------- + --------------- +
| 123 | IND | 25635525 | [11243749772,null] | 2017-10-17 |
| 123 | IND | 25632349 | [null,112322ab .. | 2017-10-17 |
| 123 | IND | 25635234 | [null,11243kjsd .. | 2017-10-17 |
| 123 | IND | 25639822 | [1124374343,null] | 2017-10-17 |
+ -------- + -------- + -------------------- + ------ ---- + ----------- + --------------- +
我突出显示了我正在处理的字段的值。它看起来有点不同于源数据。
任何帮助,将不胜感激。由于