grokdebugger验证logstash最终拒绝的日志条目

jvlzgdj9 于 2022-12-16 发布在 Logstash

关注(0)|答案(1)|浏览(178)

我使用grokdebugger改编了我在Internet上找到的内容，以便第一次尝试处理logback、spring-boot类型的日志。
下面是发送到grokdebugger的日志条目：

2022-03-09 06:35:15,821 [http-nio-9090-exec-1] WARN  org.springdoc.core.OpenAPIService - found more than one OpenAPIDefinition class. springdoc-openapi will be using the first one found.

使用grok模式：
(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) \[(?<thread>(.*?)+)\] %{LOGLEVEL:level}\s+%{GREEDYDATA:class} - (?<logmessage>.*)
并且它按照希望发送其内容：

{
  "timestamp": [
    [
      "2022-03-09 06:35:15,821"
    ]
  ],
  "YEAR": [
    [
      "2022"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],
  "MONTHDAY": [
    [
      "09"
    ]
  ],
  "TIME": [
    [
      "06:35:15,821"
    ]
  ],
  "HOUR": [
    [
      "06"
    ]
  ],
  "MINUTE": [
    [
      "35"
    ]
  ],
  "SECOND": [
    [
      "15,821"
    ]
  ],
  "thread": [
    [
      "http-nio-9090-exec-1"
    ]
  ],
  "level": [
    [
      "WARN"
    ]
  ],
  "class": [
    [
      "org.springdoc.core.OpenAPIService"
    ]
  ],
  "logmessage": [
    [
      "found more than one OpenAPIDefinition class. springdoc-openapi will be using the first one found."
    ]
  ]
}

但是当我在logstash中请求相同的操作时，我在input声明的配置中设置：

input {
    file {
        path => "/home/lebihan/dev/Java/comptes-france/metier-et-gestion/dev/ApplicationMetierEtGestion/sparkMetier.log"

        codec => multiline {
           pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
           negate => "true"
           what => "previous"
        }
    }
}

对于filter声明：

filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }
 
 grok {
    match => [ "message",
               "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) \[(?<thread>(.*?)+)\] %{LOGLEVEL:level}\s+%{GREEDYDATA:class} - (?<logmessage>.*)"
             ]
  }
  
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
}

但是它解析它失败了，而且我不知道如何有关于_grokparsefailure提到的底层错误的额外内容。

logstash

来源：https://stackoverflow.com/questions/71405189/grokdebugger-validates-entries-of-a-log-that-logstash-eventually-refuses

1条答案

按热度按时间

ca1c2owp1#

我的麻烦的主要责任是：

grok {
      match => [

而不是：

grok {
      match => {

但在那之后，我不得不改变：

%{TIMESTAMP_ISO8601:timestamp}的时间戳定义
日期匹配
并在日期匹配中向其添加目标以避免

以避免_dateparsefailure。

@timestamp:
    Mar 16, 2022 @ 09:14:22.002
@version:
    1
class:
    f.e.service.AbstractSparkDataset
host:
    debian
level:
    INFO
logmessage:
    Un dataset a été sauvegardé dans le fichier parquet /data/tmp/balanceComptesCommunes_2019_2019.
thread:
    http-nio-9090-exec-10
timestamp:
    2022-03-16T06:34:09.394Z
_id:
    8R_KkX8BBIYNTaMw1Jfg
_index:
    ecoemploimetier-2022.03.16
_score:
    - 
_type:
    _doc

我最终更正了我的 logstash 配置文件，如下所示：

input {
    file {
        path => "/home/[...]/myLog.log"

        sincedb_path => "/dev/null"
        start_position => "beginning"

        codec => multiline {
           pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
           negate => "true"
           what => "previous"
        }
    }
}

filter {
   #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
   if [message] =~ "\tat" {
      grok {
         match => ["message", "^(\tat)"]
         add_tag => ["stacktrace"]
      }
   }
 
   grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[(?<thread>(.*?)+)\] %{LOGLEVEL:level} %{GREEDYDATA:class} - (?<logmessage>.*)" }
   }
 
   date {
      # 2022-03-16 07:32:24,860
      match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
      target => "timestamp"
    }

   # S'il n'y a pas d'erreur de parsing, supprimer le message d'origine, non parsé
   if "_grokparsefailure" not in [tags] {
      mutate {
         remove_field => [ "message", "path" ]
      }
   }
}

output {
    stdout { codec => rubydebug }

    elasticsearch {
        hosts => ["localhost:9200"]
        index => "ecoemploimetier-%{+YYYY.MM.dd}"
    }
}

赞(0）回复(0）举报 2022-12-16

我来回答

grokdebugger验证logstash最终拒绝的日志条目

1条答案

相关问题

热门标签

最新问答