Logstash滤波

Question

我具有与以下格式写入JSON对象（逐行）到/var/log/myLog.json一个Python脚本：

{"timestamp":"2016-07-21T01:20:04.392799-0400","in_iface":"docker0","event_type":"alert","src_ip":"172.17.0.2","dest_ip":"172.17.0.3","proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","**gid**":2,"signature_id":2,"rev":0,"signature":"ICMP msg","category":"","severity":3},"payload":"hFuQVwAA","payload_printable":"kk"} 

我想使用Logstash，以便：

1. 阅读JSON对象，逐行从/var/log/myLog.json

2. 解析GID和转发到另一台机器作为UDP MSG（赋予了特定的IP地址+端口） - 例如：if gid == 2 th恩此JSON对象转发到172.123.10.3:10001

此外，我希望能够动态更新Logstash配置文件过滤器（又名，才能够添加另一个规则，如：“如果GID == x然后将这个json对象转换为另一个IP）。

我该怎么做？

Logstash配置文件应该如何显示？ 以及如何插入/删除动态过滤器的命令看起来像？

谢谢，伙计们。

Answer 1

您可以按照以下配置运行logstash。 和我已经测试了两个样本json数据。

{"timestamp":"2016-07-21T01:20:04.392799-0400","in_iface":"docker0","event_type":"alert","src_ip":"172.17.0.2","dest_ip":"172.17.0.3","proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","gid":2,"signature_id":2,"rev":0,"signature":"ICMP msg","category":"","severity":3},"payload":"hFuQVwAA","payload_printable":"kk"} 
{"timestamp":"2016-07-21T01:20:04.392799-0400","in_iface":"docker0","event_type":"alert","src_ip":"172.17.0.2","dest_ip":"172.17.0.3","proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","gid":3,"signature_id":2,"rev":0,"signature":"ICMP msg","category":"","severity":3},"payload":"hFuQVwAA","payload_printable":"kk"} 



input { 
    file { 
     path => "/etc/logstash/jsonSample.log" 
     start_position => "beginning" 
     sincedb_path => "/dev/null" 
    } 
} 

filter { 
       json { 
         source => "message" 
         target => "doc" 
         add_field => {"alert.gid" => "%{[doc][alert][gid]}"} 
         add_tag => ["tagName_%{[doc][alert][gid]}"] 
       } 


} 


output { 
if "tagName_2" in [tags] { 
stdout {codec => rubydebug} 
}else if "tagName_3" in [tags] { 
} 

} 

然后你就可以看到结果

{ 
     "message" => "{\"timestamp\":\"2016-07-21T01:20:04.392799-0400\",\"in_iface\":\"docker0\",\"event_type\":\"alert\",\"src_ip\":\"172.17.0.2\",\"dest_ip\":\"172.17.0.3\",\"proto\":\"ICMP\",\"icmp_type\":0,\"icmp_code\":0,\"alert\":{\"action\":\"allowed\",\"gid\":2,\"signature_id\":2,\"rev\":0,\"signature\":\"ICMP msg\",\"category\":\"\",\"severity\":3},\"payload\":\"hFuQVwAA\",\"payload_printable\":\"kk\"}", 
     "@version" => "1", 
    "@timestamp" => "2016-07-25T04:41:11.980Z", 
      "path" => "/etc/logstash/jsonSample.log", 
      "host" => "baklava", 
      "doc" => { 
       "timestamp" => "2016-07-21T01:20:04.392799-0400", 
       "in_iface" => "docker0", 
       "event_type" => "alert", 
        "src_ip" => "172.17.0.2", 
        "dest_ip" => "172.17.0.3", 
        "proto" => "ICMP", 
       "icmp_type" => 0, 
       "icmp_code" => 0, 
        "alert" => { 
        "action" => "allowed", 
        "gid" => 2, 
      "signature_id" => 2, 
        "rev" => 0, 
       "signature" => "ICMP msg", 
       "category" => "", 
       "severity" => 3 
     }, 
        "payload" => "hFuQVwAA", 
     "payload_printable" => "kk" 
    }, 
    "alert.gid" => 2, 
      "tags" => [ 
     [0] "tagName_2" 
    ] 
} 

你也可以改变施加在它上面的配置。

问候。

你可以参考事件和JSON过滤 https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html