我需要解析下面的日志文件,在脚本中应该考虑从时间戳记150324-21:06:32:937378
开始直到下一个时间戳开始为一条记录。我尝试使用库在猪中使用正则表达式解析日志文件
org.apache.pig.piggybank.storage.MyRegExLoader
加载自定义格式的记录。
150324-21:06:32:937378 [mod=STB, lvl=INFO ]
top - 21:06:33 up 3:41, 0 users, load average: 0.75, 0.95, 0.72
Tasks: 120 total, 3 running, 117 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.8%us, 12.9%sy, 2.9%ni, 60.7%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st
Mem: 317108k total, 232588k used, 84520k free, 25960k buffers
Swap: 0k total, 0k used, 0k free, 110820k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19122 root 20 0 456m 72m 37m R 72 23.5 85:50.22 Receiver
5859 root 20 0 349m 9128 6948 S 15 2.9 22:42.88 rmfStreamer
150324-21:06:32:937378 [mod=STB, lvl=INFO ]
top - 21:06:33 up 3:41, 0 users, load average: 0.75, 0.95, 0.72
Tasks: 120 total, 3 running, 117 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.8%us, 12.9%sy, 2.9%ni, 60.7%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st
Mem: 317108k total, 232588k used, 84520k free, 25960k buffers
Swap: 0k total, 0k used, 0k free, 110820k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19122 root 20 0 456m 72m 37m R 72 23.5 85:50.22 Receiver
5859 root 20 0 349m 9128 6948 S 15 2.9 22:42.88 rmfStreamer
这里是我的相关代码片段,我用
raw_logs = LOAD './main*/*top_log*' USING org.apache.pig.piggybank.storage.MyRegExLoader('(?m)(?s)\\d*-\\d{2}:\\d{2}:\\d{2}\\:\\d*.*') AS line:chararray ; DUMP raw_logs;
这里是我的输出:
(150325-05:47:26:253050 [mod=STB, lvl=INFO ])
(150325-05:57:27:294069 [mod=STB, lvl=INFO ])
(150325-06:07:28:235302 [mod=STB, lvl=INFO ])
(150325-06:17:29:124282 [mod=STB, lvl=INFO ])
(150325-06:27:30:036264 [mod=STB, lvl=INFO ])
(150325-06:37:30:941804 [mod=STB, lvl=INFO ])
(150325-06:47:31:909712 [mod=STB, lvl=INFO ])
应该是这样的2元组
(150324-21:06:32:937378 [mod=STB, lvl=INFO ]
top - 21:06:33 up 3:41, 0 users, load average: 0.75, 0.95, 0.72
Tasks: 120 total, 3 running, 117 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.8%us, 12.9%sy, 2.9%ni, 60.7%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st
Mem: 317108k total, 232588k used, 84520k free, 25960k buffers
Swap: 0k total, 0k used, 0k free, 110820k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19122 root 20 0 456m 72m 37m R 72 23.5 85:50.22 Receiver
5859 root 20 0 349m 9128 6948 S 15 2.9 22:42.88 rmfStreamer)
(150324-21:06:32:937378 [mod=STB, lvl=INFO ]
top - 21:06:33 up 3:41, 0 users, load average: 0.75, 0.95, 0.72
Tasks: 120 total, 3 running, 117 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.8%us, 12.9%sy, 2.9%ni, 60.7%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st
Mem: 317108k total, 232588k used, 84520k free, 25960k buffers
Swap: 0k total, 0k used, 0k free, 110820k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19122 root 20 0 456m 72m 37m R 72 23.5 85:50.22 Receiver
5859 root 20 0 349m 9128 6948 S 15 2.9 22:42.88 rmfStreamer)
请让我知道正则表达式我可以使用,所以米y脚本考虑时间戳的开始,直到下一个时间戳记录的开始。
谢谢你。我尝试过,但它不工作我没有得到任何使用上述表达式匹配。 – evoluguy
请参阅链接:https://regex101.com/r/mD7mE9/1 –