亚马逊雅典娜不解析云端日志

Question

我遵循Athena getting started guide并尝试解析我自己的Cloudfront日志。但是，这些字段没有被解析。

我用一个小测试文件，内容如下：

#Version: 1.0 
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type 
2016-02-02 07:57:45 LHR5 5001 86.177.253.38 GET d3g47gpj5mj0b.cloudfront.net /foo 404 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_10_5)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/47.0.2526.111%2520Safari/537.36 - - Error -tHYQ3YpojqpR8yFHCUg5YW4OC_yw7X0VWvqwsegPwDqDFkIqhZ_gA== d3g47gpj5mj0b.cloudfront.net https421 0.076 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Error 
2016-02-02 07:57:45 LHR5 1158241 86.177.253.38 GET d3g47gpj5mj0b.cloudfront.net /images/posts/cover/404.jpg 200 https://d3g47gpj5mj0b.cloudfront.net/foo Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_10_5)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/47.0.2526.111%2520Safari/537.36 - - Miss oUdDIjmA1ON1GjWmFEKlrbNzZx60w6EHxzmaUdWEwGMbq8V536O4WA== d3g47gpj5mj0b.cloudfront.net https 419 0.440 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss 

而与此SQL创建的表：

CREATE EXTERNAL TABLE IF NOT EXISTS cloudfront_logs (
    `Date` DATE, 
    Time STRING, 
    Location STRING, 
    Bytes INT, 
    RequestIP STRING, 
    Method STRING, 
    Host STRING, 
    Uri STRING, 
    Status INT, 
    Referrer STRING, 
    os STRING, 
    Browser STRING, 
    BrowserVersion STRING 
) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' 
    WITH SERDEPROPERTIES (
    "input.regex" = "^(?!#)([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+[^\(]+[\(]([^\;]+).*\%20([^\/]+)[\/](.*)$" 
) LOCATION 's3://test/athena-csv/' 

但没有数据回来：

我可以看到它返回4行，但前两个应该排除，因为他们sta rt与＃，所以这就像正则表达式不被正确解析。

我做错了什么？或者是正则表达式错误（似乎不太可能，因为它在文档中，对我来说看起来很好）？

Answer 1

这是我结束了：

CREATE EXTERNAL TABLE logs (
    `date` date, 
    `time` string, 
    `location` string, 
    `bytes` int, 
    `request_ip` string, 
    `method` string, 
    `host` string, 
    `uri` string, 
    `status` int, 
    `referer` string, 
    `useragent` string, 
    `uri_query` string, 
    `cookie` string, 
    `edge_type` string, 
    `edget_requiest_id` string, 
    `host_header` string, 
    `cs_protocol` string, 
    `cs_bytes` int, 
    `time_taken` string, 
    `x_forwarded_for` string, 
    `ssl_protocol` string, 
    `ssl_cipher` string, 
    `result_type` string, 
    `protocol` string 
) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' 
WITH SERDEPROPERTIES (
    'input.regex' = '^(?!#.*)(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s*(\\S*)' 
) LOCATION 's3://logs' 

注意双反斜线是故意的。

云端日志的格式在某些时候更改以添加。这处理较旧和较新的文件。

Answer 2

雅典娜是case insensitive and considers every column to be lowercase。尝试定义Athena表并使用小写字段名称进行查询。

Answer 3

该演示也不适用于我。玩了一下后，我得到了以下工作：

CREATE EXTERNAL TABLE IF NOT EXISTS DBNAME.TABLENAME (
    `date` date, 
    `time` string, 
    `location` string, 
    `bytes` int, 
    `request_ip` string, 
    `method` string, 
    `host` string, 
    `uri` string, 
    `status` int, 
    `referer` string, 
    `useragent` string, 
    `uri_query` string, 
    `cookie` string, 
    `edge_type` string, 
    `edget_requiest_id` string, 
    `host_header` string, 
    `cs_protocol` string, 
    `cs_bytes` int, 
    `time_taken` string, 
    `x_forwarded_for` string, 
    `ssl_protocol` string, 
    `ssl_cipher` string, 
    `result_type` string 
) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' 
WITH SERDEPROPERTIES (
    'serialization.format' = '1', 
    'input.regex' = '^(?!#.*)(?!#.*)([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)\s+([^\s]+)$' 
) LOCATION 's3://bucket/logs/'; 

用您的信息替换桶/日志和dbname.table。由于某些原因，它仍然为＃行插入空行，但我得到了其余的数据。

我认为下一步就是尝试为用户代理或cookie创建一个。

Answer 4

拉我的头发了这一点，并提高对@CoderDans后回答：

秘诀是使用的\ t值分离，而不是\ S为正则表达式。

CREATE EXTERNAL TABLE IF NOT EXISTS mytablename (
    `date` date, 
    `time` string, 
    `location` string, 
    `bytes` int, 
    `request_ip` string, 
    `method` string, 
    `host` string, 
    `uri` string, 
    `status` int, 
    `referer` string, 
    `useragent` string, 
    `uri_query` string, 
    `cookie` string, 
    `edge_type` string, 
    `edget_request_id` string, 
    `host_header` string, 
    `cs_protocol` string, 
    `cs_bytes` int, 
    `time_taken` int, 
    `x_forwarded_for` string, 
    `ssl_protocol` string, 
    `ssl_cipher` string, 
    `result_type` string, 
    `protocol_version` string 
) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' 
WITH SERDEPROPERTIES (
    'serialization.format' = '1', 
    'input.regex' = '^(?!#.*)(?!#.*)([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)$' 
) LOCATION 's3://mybucket/myprefix/'; 

Answer 5

实际上，这里的所有答案都有一个小错误：第4个字段必须是BIGINT，而不是INT。否则，您的> 2GB文件请求未正确解析。经过与AWS Business Support的长时间讨论，看起来正确的格式是：

CREATE EXTERNAL TABLE your_table_name (
    `Date` DATE, 
    Time STRING, 
    Location STRING, 
    SCBytes BIGINT, 
    RequestIP STRING, 
    Method STRING, 
    Host STRING, 
    Uri STRING, 
    Status INT, 
    Referrer STRING, 
    UserAgent STRING, 
    UriQS STRING, 
    Cookie STRING, 
    ResultType STRING, 
    RequestId STRING, 
    HostHeader STRING, 
    Protocol STRING, 
    CSBytes BIGINT, 
    TimeTaken FLOAT, 
    XForwardFor STRING, 
    SSLProtocol STRING, 
    SSLCipher STRING, 
    ResponseResultType STRING, 
    CSProtocolVersion STRING 
) 
    ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' 
    LOCATION 's3://path_to_your_data_directory'