Logstash不会添加标记

11dmarpk  于 2022-12-09  发布在  Logstash
关注(0)|答案(1)|浏览(138)

我有一个奇怪的问题与logstash(v8.3.3)没有正确应用标签。
/conf.d/中当前只有两个配置文件,一个用于beats,一个用于syslog

01-节拍.conf

input {
     beats {
       port => 5044
       ssl => false
       tags => [ 'beat' ]
      }
    }
    

filter {
if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }

    date {
match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
if 'beat' in [tags] {
   elasticsearch {
     hosts => localhost
     index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   }
}
stdout {
  codec => rubydebug
  }
}

02-系统日志文件配置

input {
  tcp {
    port => 514
    type => syslog
    tags => [ 'syslog' ]
  }
  udp {
    port => 514
    type => syslog
    tags => [ 'syslog' ]

  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
  if [message] =~ /(?i)crc/ {
     mutate { add_tag => ["switch"] }
  }
}

output {
if 'syslog' in [tags] {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "syslog-%{+YYYY.MM}"
   }
}
stdout {
    codec => rubydebug
    }

如果我通过以下方式运行logstash,则会在02-syslog.conf文件中出现问题:/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/02-syslog.conf已成功添加switch标记。

范例:

root@elk-test ~ $ logger -n localhost -T -P 514 "crc error on port 48"

{
          "type" => "syslog",
          "tags" => [
        [0] "syslog",
        [1] "_grokparsefailure",
        [2] "switch"
    ],
    "@timestamp" => 2022-07-31T17:42:36.408104Z,
      "@version" => "1",
         "event" => {
        "original" => "<13>1 2022-07-31T19:42:36.314411+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"88217\"] crc error on port 48"
    },
       "message" => "<13>1 2022-07-31T19:42:36.314411+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"88217\"] crc error on port 48"
}

而当我通过以下方式启动服务时:systemctl start logstash.service不添加标记。

范例:

root@elk-test ~ $ logger -n localhost -T -P 514 "crc error on port 48"
    
    {
      "_index": "syslog-2022.07",
      "_id": "mcJQVYIBNEboR1irs878",
      "_version": 1,
      "_score": 0,
      "_source": {
        "@timestamp": "2022-07-31T17:32:08.058612Z",
        "message": [
          "<13>1 2022-07-31T19:32:07.935622+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48",
          "root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48"
        ],
        "timestamp": "2022-07-31T19:32:07.935622+02:00",
        "tags": [
          "syslog",
          "_dateparsefailure",
          "_grokparsefailure"
        ],
        "host": {
          "hostname": "srv-elk-hck.localdomain"
        },
        "@version": "1",
        "type": "syslog",
        "event": {
          "original": "<13>1 2022-07-31T19:32:07.935622+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48"
        }
      },
      "fields": {
        "event.original": [
          "<13>1 2022-07-31T19:32:07.935622+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48"
        ],
        "tags.keyword": [
          "syslog",
          "_dateparsefailure",
          "_grokparsefailure"
        ],
        "@version.keyword": [
          "1"
        ],
        "host.hostname": [
          "srv-elk-hck.localdomain"
        ],
        "type": [
          "syslog"
        ],
        "message": [
          "<13>1 2022-07-31T19:32:07.935622+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48",
          "root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48"
        ],
        "tags": [
          "syslog",
          "_dateparsefailure",
          "_grokparsefailure"
        ],
        "@timestamp": [
          "2022-07-31T17:32:08.058Z"
        ],
        "type.keyword": [
          "syslog"
        ],
        "message.keyword": [
          "<13>1 2022-07-31T19:32:07.935622+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48",
          "root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48"
        ],
        "event.original.keyword": [
          "<13>1 2022-07-31T19:32:07.935622+02:00 srv-elk-hck.localdomain root - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"15780\"] crc error on port 48"
        ],
        "host.hostname.keyword": [
          "srv-elk-hck.localdomain"
        ],
        "@version": [
          "1"
        ],
        "timestamp": [
          "2022-07-31T17:32:07.935Z"
        ]
      }
    }

我错过了什么?

eivnm1vs

eivnm1vs1#

服务必须使用不同的配置。在事件中,您显示的[message]是一个数组,这表明grok模式以%{GREEDYDATA:message}(message,而不是syslog_message)结尾。如果您修复了该问题,则会添加您的标记。
否则,请尝试

if [message][0] =~ /(?i)crc/ { mutate { add_tag => ["switch"] } }

相关问题