Logstash没有将日志分为配置

Question

我正在使用elkb来处理我的访问日志。有一天，我发现kibana错过了一个日志。

然后，grep的filebeat日志，而且我能找到丢失的日志：

2017/03/01 10:19:20.096711 client.go:184: DBG Publish: { 
    "@timestamp": "2017-03-01T10:19:16.327Z", 
    "beat": { 
    "hostname": "kvm980156.jx.diditaxi.com", 
    "name": "kvm980156.jx.diditaxi.com", 
    "version": "5.0.0" 
    }, 
    "input_type": "log", 
    "message": "2017-03-01 18:19:11.699|10.94.104.169|17714317657896955-151|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|POST|/api/v1/answer/|com.didi.km.api.controller.api.v1.quest 
ion.AnswerController#post[2 args]|{\"questionId\":[\"145\"],\"content\":[\"\u003cp\u003e123123123123123\u003c/p\u003e\"]}|200|220", 
    "offset": 1723505, 
    "source": "/home/km/didi-km-api/logs/km-access.2017-03-01.log", 
    "type": "log" 
} 

而且，我用grep logstash日志，我可以找到它太：

{ 
    "@timestamp" => 2017-03-01T10:19:16.327Z, 
     "offset" => 1723505, 
     "@version" => "1", 
    "input_type" => "log", 
      "beat" => { 
     "hostname" => "kvm980156.jx.diditaxi.com", 
      "name" => "kvm980156.jx.diditaxi.com", 
     "version" => "5.0.0" 
    }, 
      "host" => "kvm980156.jx.diditaxi.com", 
     "source" => "/home/km/didi-km-api/logs/km-access.2017-03-01.log", 
     "message" => "2017-03-01 18:19:11.699|10.94.104.169|17714317657896955-151|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10 
_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|POST|/api/v1/answer/|com.didi.km.api.controller.api.v 
1.question.AnswerController#post[2 args]|{\"questionId\":[\"145\"],\"content\":[\"<p>123123123123123</p>\"]}|200|220", 
      "type" => "log", 
      "tags" => [ 
     [0] "beats_input_codec_plain_applied", 
     [1] "_grokparsefailure" 
    ] 
} 

但有一些不同在这个日志和其他人之间。这个日志并没有像我的配置文件那样分裂，而其他人也这么做。

下一个日志丢失的日志下：

{ 
    "controllerMethod" => "com.didi.km.api.controller.api.v1.question.AnswerController#answersOrderByHot[2 args]", 
       "offset" => 1723849, 
       "method" => "GET", 
      "input_type" => "log", 
       "source" => "/home/km/didi-km-api/logs/km-access.2017-03-01.log", 
      "message" => "2017-03-01 18:19:11.855|10.94.104.169|17714317657896955-152|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac O 
S X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|GET|/api/v1/answer/145|com.didi.km.api.controll 
er.api.v1.question.AnswerController#answersOrderByHot[2 args]|{\"order\":[\"hot\"],\"pager\":[\"1,100\"]}|200|60", 
       "type" => "log", 
        "ua" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 
Safari/537.36", 
       "uri" => "/api/v1/answer/145", 
       "tags" => [ 
     [0] "beats_input_codec_plain_applied" 
    ], 
       "uid" => 1, 
      "@timestamp" => 2017-03-01T10:19:11.855Z, 
       "param" => "{\"order\":[\"hot\"],\"pager\":[\"1,100\"]}", 
      "costTime" => 60, 
      "requestID" => "17714317657896955-152", 
      "host-ip" => "10.94.104.169", 
      "@version" => "1", 
       "beat" => { 
     "hostname" => "kvm980156.jx.diditaxi.com", 
      "name" => "kvm980156.jx.diditaxi.com", 
     "version" => "5.0.0" 
    }, 
       "host" => "kvm980156.jx.diditaxi.com", 
       "time" => "2017-03-01 18:19:11.855", 
      "username" => "wangziyi", 
      "statusCode" => 200 
} 

而且，这是我的logstash配置。使用grok来分割日志。

input { 
    beats { 
     port => "5043" 
    } 
} 
filter { 
    # TIME||HOST-IP||REQUEST-ID||UID||USERNAME||METHOD||URI||CONTROLLER-METHOD||PARAMS-MAP 
    grok { 
     match => { 
      "message" => "%{TIMESTAMP_ISO8601:time}\|%{IP:host-ip}\|(?<requestID>\d+-\d+)\|%{INT:uid:int}\|%{WORD:username}\|(?<ua>(\ 
w|\/|\.|\s|\(|;|\)|,)+)\|%{WORD:method}\|(?<uri>(\w|\/)+)\|(?<controllerMethod>(\w|\d|\s|\.|#|\[|\])+)\|(?<param>(\w|{|}|\"|\:|\[|\]| 
\,)+)\|%{NUMBER:statusCode:int}\|%{NUMBER:costTime:int}" 
     } 
    } 

    date { 
     match => ["time", "yyyy-MM-dd HH:mm:ss.SSS"] 
     target => "@timestamp" 
    } 
} 
output { 
    stdout { codec => rubydebug } 
    elasticsearch { 
     hosts => [ "10.94.66.193:9200" ] 
     index => "km-access-%{+YYYY.MM.dd}" 
    } 
} 

因为这个bug，我不能用Kibana来计算一些日志。

这是我原来的日志：

2017-03-01 18:19:11.699|10.94.104.169|17714317657896955-151|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|POST|/api/v1/answer/|com.didi.km.api.controller.api.v1.question.AnswerController#post[2 args]|{"questionId":["145"],"content":["<p>123123123123123</p>"]}|200|220 

Answer 1

我看得出来，你只是试图提取您的logline和match它的timestamp一部分。如果是这样的话，如果你有你的神交比赛本身没有使它更加复杂：

grok { 
    match => { 
     "message" => "%{TIMESTAMP_ISO8601:time}%{GREEDYDATA}" 
    } 
} 

date { 
    match => ["time", "yyyy-MM-dd HH:mm:ss.SSS"] 
    target => "@timestamp" 
}