测试自定义Logstash过滤器

Question

我们使用Ansible并拥有Logstash。

如何编写一些测试来覆盖我们的自定义logstash过滤器？我想要做的是这样的：

1. 给予Logstash配置有过滤器，
2. 传一个日志行（或多行日志记录），
3. 看到它成功地解析成片。

我知道这是 - https://github.com/elastic/logstash/wiki/Tips:Testing-your-filters，但我不明白这是如何有用 - 它似乎已过时。

Answer 1

回答行之有效高达logstash的1.5版本。此后，grok模式从核心中移除并放入logstash-core-patterns中。在logstash 2.2中对我有用的是：

# encoding: utf-8 

require 'spec_helper' 
require "logstash/patterns/core" 

# solution based on https://github.com/logstash-plugins/logstash-filter-grok/blob/master/spec/filters/grok_spec.rb 
module LogStash::Environment 
    # running the grok code outside a logstash package means 
    # LOGSTASH_HOME will not be defined, so let's set it here 
    # before requiring the grok filter 

    # the path that is set is the plugin root path 
    unless self.const_defined?(:LOGSTASH_HOME) 
    LOGSTASH_HOME = File.expand_path("../../../", __FILE__) 
    end 

    # also :pattern_path method must exist so we define it too 

    # method is called by logstash-filter-grok to create patterns_path array 
    # 
    # logstash-filter-grok/lib/logstash/filters/grok.rb(line ~230): 
    # 
    # @@patterns_path += [ 
    #  LogStash::Patterns::Core.path, 
    #  LogStash::Environment.pattern_path("*") 
    # 
    # patterns defined in spec/patterns/ will be joined to the array by grok 

    unless self.method_defined?(:pattern_path) 
    def pattern_path(path) 
     ::File.join(LOGSTASH_HOME, "spec", "patterns", path) 
    end 
    end 
end 

require "logstash/filters/grok" 
require "logstash/filters/<tested-plugin>" 

其余规范示例仍然有效。

随着新的依赖关系要求的到来，Gemfile也必须更改。我gemspec依赖关系是这样的：

# Gem dependencies 
s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0" 
s.add_runtime_dependency "<tested-plugin>" 
s.add_development_dependency 'logstash-devutils', '~> 0' 
s.add_development_dependency 'logstash-filter-grok', '~> 3.2' 
s.add_development_dependency 'logstash-patterns-core', '~> 4.0' 

一个完整的工作exapmle可以发现here

Answer 2

我发现this并结束了以下工作的测试代码：

# simple_filter_spec.rb 
# 
# run using: 
# bundle exec rspec simple_filter_spec.rb 

require "logstash/devutils/rspec/spec_helper" 

LogStash::Environment::LOGSTASH_HOME = `gem which logstash-core` 
module LogStash::Environment 
    unless self.method_defined?(:pattern_path) 
    def pattern_path(path) 
     ::File.join(LOGSTASH_HOME, "patterns", path) 
    end 
    end 
end 


require "logstash/filters/grok" 

describe LogStash::Filters::Grok do 
    config <<-CONFIG 
    filter { 
    grok { 
     match => { "message" => "%{SYSLOGLINE}" } 
     singles => true 
     overwrite => [ "message" ] 
    } 
    } 
    CONFIG 

    sample "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]" do 
    insist { subject["tags"] }.nil? 
    insist { subject["logsource"] } == "evita" 
    insist { subject["timestamp"] } == "Mar 16 00:01:25" 
    insist { subject["message"] } == "connect from camomile.cloud9.net[168.100.1.3]" 
    insist { subject["program"] } == "postfix/smtpd" 
    insist { subject["pid"] } == "1713" 
    end 
end 

而且我的Gemfile看起来像这样：

source 'https://www.rubygems.org' 

platform :jruby do 
    gem 'pry' 
    gem 'rspec' 
    gem 'logstash-core' 
    gem 'logstash-devutils' 
    gem 'logstash-filter-grok' 
end 

Answer 3

可以使用Logstash-Tester - 一个小工具，单元测试logstash过滤器和模式。您使用json编写测试用例，logstash-tester使用docker容器在logstash上运行它们。 （免责声明：我写的工具）

Answer 4

从this blog：

$ git clone https://github.com/elastic/logstash 
$ cd logstash 
$ git checkout 2.1 
$ rake bootstrap 
$ rake test:install-core 

而不是检查出来的2.1分支，你应该检查出logstash的标记的版本，你会实际运行，例如v2.3.2（注意前面的“v”）。

运行上述命令后，可以在logstash回购中运行bin/rspec /some/path/your_filter_spec.rb。

重要：我发现编码行# encoding: utf-8是必需的，否则匹配将失败。

样品测试文件：由gmile给出

# encoding: utf-8 

require "spec_helper" 

describe "simple request log" do 
    config (<<-CONFIG) 
    filter { 
    grok { 
     match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" } 
    } 
    } 
    CONFIG 

    sample '55.3.244.1 GET /index.html 15824 0.043' do 
    insist { subject['client'] } == '55.3.244.1' 
    insist { subject['method'] } == 'GET' 
    insist { subject['request'] } == '/index.html' 
    insist { subject['bytes'] } == '15824' 
    insist { subject['duration'] } == '0.043' 
    end 
end