logstash 在grok模式的空间上拆分日志消息

kxkpmulp  于 2022-12-09  发布在  Logstash
关注(0)|答案(1)|浏览(257)

我刚接触grokELK两天。我正在努力根据空间分解日志消息,使它们在logstash中显示为不同的字段。
我的输入模式是:2022-02-11 11:57:49 - app - INFO - function_name=add elapsed_time=0.0296 input_params=6_3
我希望在logstash/kibana中看到function_nameelapsed_timeinput_params的不同字段。
现在,我有下面的.conf

input{
  file{
  path => "/path/to/log/file"
  start_position => "beginning"
  }
}
filter{
  grok{
  match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{(?<function_name>[^.]*)\.(?<elapsed_time>[^.]*)\.(?<input>[^.]*)}"}
  }
    date {
    match => ["timestamp", "ISO8601"]
    }
    function_name {
    match => ["function_name", "DATA"]
    }
    elapsed_time {
    match => ["elapsed_time", "BASE16FLOAT"]
    }
    input {
    match => ["input", "DATA"]
    }
}
output{
  elasticsearch{
  hosts => ["localhost:9200"]
  index => "math_apis"
  }
  stdout{codec => rubydebug}
}

但这只会在logstash中生成以下消息

{
          "host" => "hostname",
    "@timestamp" => 2022-02-11T06:27:49.404Z,
       "message" => "2022-02-11 11:57:49 - app - INFO - function_name=add elapsed_time=0.0296 input_params=6_3",
          "path" => "path/to/log/file",
      "@version" => "1",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}
s5a0g9ez

s5a0g9ez1#

You can use the following pattern:

%{TIMESTAMP_ISO8601:timestamp} - \S+ - %{LOGLEVEL:log_level} - function_name=%{NOTSPACE:function_name} elapsed_time=%{NOTSPACE:elapsed_time} input_params=%{NOTSPACE:input}
  • Details*:
  • %{TIMESTAMP_ISO8601:timestamp} - timestamp field
  • - - a literal string
  • \S+ - any one or more non-whitespace chars
  • - - a literal string
  • %{LOGLEVEL:log_level} - LOGLEVEL pattern
  • - function_name= - a literal string
  • %{NOTSPACE:function_name} - function_name field of one or more non-whitespace chars
  • elapsed_time= - space and elapsed_time= string
  • %{NOTSPACE:elapsed_time} - elapsed_time field of one or more non-whitespace chars
  • input_params= - literal string
  • %{NOTSPACE:input} - input field of one or more non-whitespace chars.

See more about Grok patterns here .
Test output:

{
  "timestamp": [
    [
      "2022-02-11 11:57:49"
    ]
  ],
  "YEAR": [
    [
      "2022"
    ]
  ],
  "MONTHNUM": [
    [
      "02"
    ]
  ],
  "MONTHDAY": [
    [
      "11"
    ]
  ],
  "HOUR": [
    [
      "11",
      null
    ]
  ],
  "MINUTE": [
    [
      "57",
      null
    ]
  ],
  "SECOND": [
    [
      "49"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "log_level": [
    [
      "INFO"
    ]
  ],
  "function_name": [
    [
      "add"
    ]
  ],
  "elapsed_time": [
    [
      "0.0296"
    ]
  ],
  "input": [
    [
      "6_3"
    ]
  ]
}

相关问题