Kibana 日志存储配置|删除空行

z3yyvxxp  于 2023-02-06  发布在  Kibana
关注(0)|答案(1)|浏览(253)

LogStash的conf文件的内容如下所示:

input {
    beats {
        port => 5044
    }
    
    file {
        path => "/usr/share/logstash/iway_logs/*"       
        start_position => "beginning"
        sincedb_path => "/dev/null"
        #ignore_older => 0
        codec => multiline {
         pattern => "^\[%{NOTSPACE:timestamp}\]"
         negate => true
         what => "previous"
         max_lines => 2500
        }
  }
}

filter {    
    
    grok {
        match => {  "message" => 
            ['(?m)\[%{NOTSPACE:timestamp}\]%{SPACE}%{WORD:level}%{SPACE}\(%{NOTSPACE:entity}\)%{SPACE}%{GREEDYDATA:rawlog}'         
            ]
        }
    }
                    
    date {
      match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSS"]
      target => "@timestamp"
    }
        
    grok {
        match => { "entity" =>          ['(?:W.%{GREEDYDATA:channel}:%{GREEDYDATA:inlet}:%{GREEDYDATA:listener}\.%{GREEDYDATA:workerid}|W.%{GREEDYDATA:channel}\.%{GREEDYDATA:workerid}|%{GREEDYDATA:channel}:%{GREEDYDATA:inlet}:%{GREEDYDATA:listener}\.%{GREEDYDATA:workerid}|%{GREEDYDATA:channel}:%{GREEDYDATA:inlet}:%{GREEDYDATA:listener}|%{GREEDYDATA:channel})']
        }
    }
    
    dissect {
        mapping => {
            "[log][file][path]" => "/usr/share/logstash/iway_logs/%{serverName}#%{configName}#%{?ignore}.log"
        }
    }
    
}

output {
    elasticsearch {
        hosts => "${ELASTICSEARCH_HOST_PORT}"
        index => "iway_"
        user => "${ELASTIC_USERNAME}"
        password => "${ELASTIC_PASSWORD}"
        ssl => true
        ssl_certificate_verification => false
        cacert => "/certs/ca.crt"
    }
}

可以看出,这个想法是使用多行提取来解析自定义日志。提取完成了它的工作。日志偶尔会包含一个空的第一行。因此:

[2022-11-29T12:23:15.073] DEBUG (manager) Generic XPath iFL functions use full XPath 1.0 syntax
[2022-11-29T12:23:15.074] DEBUG (manager) XPath 1.0 iFL functions use iWay's full syntax implementation

这自然会导致Kibana报告一个空行:

为了避免将此行发送到ES,我添加了以下内容作为最后一个筛选项:

if ![message] {
  drop { }
}
    
if [message] =~ /^\s*$/ {
  drop { }
}

将生成的JSON有效负载发送到ES:

{
    "@timestamp": [
        "2022-12-09T14:09:35.616Z"
    ],
    "@version": [
        "1"
    ],
    "@version.keyword": [
        "1"
    ],
    "event.original": [
        "\r"
    ],
    "event.original.keyword": [
        "\r"
    ],
    "host.name": [
        "xxx"
    ],
    "host.name.keyword": [
        "xxx"
    ],
    "log.file.path": [
        "/usr/share/logstash/iway_logs/localhost#iCLP#iway_2022-11-29T12_23_33.log"
    ],
    "log.file.path.keyword": [
        "/usr/share/logstash/iway_logs/localhost#iCLP#iway_2022-11-29T12_23_33.log"
    ],
    "message": [
        "\r"
    ],
    "message.keyword": [
        "\r"
    ],
    "tags": [
        "_grokparsefailure"
    ],
    "tags.keyword": [
        "_grokparsefailure"
    ],
    "_id": "oRc494QBirnaojU7W0Uf",
    "_index": "iway_",
    "_score": null
}

虽然这确实删除了空的第一行,但不幸的是,它也干扰了其他行上的多行操作。换句话说,多行操作不再起作用了。我做错了什么?

oxcyiej7

oxcyiej71#

使用以下变更解决了该问题:

if [message] =~ /\A\s*\Z/ {
  drop { }
}

此解决方案基于Badger提供的on the Logstash forums的答案,其中也提出了此问题。

相关问题