logstash 我要为日志创建自定义GROK

fslejnso  于 2022-12-09  发布在  Logstash
关注(0)|答案(1)|浏览(175)

日志示例:

30-10-22 20:35:36       [DEBUG] [Default]       [Worker]         Sleeping for 10 seconds...

我可以得到%{TIMESTAMP_ISO8601:timestamp}%{LOGLEVEL:log-level},但我只需要消息“Sleeping for 10 seconds...
%{GREEDYDATA:message}返回所有消息:

Sleeping for 10 seconds...

opensearch输出示例:

@timestamp
Nov 2, 2022 @ 12:30:45.074
    
@version
1
    
_id
Z_nkN4QBewLz0DWJsjKB
    
_index
logstash-logs-2022.11.02
    
_score
 - 
    
_type
_doc
    
host
ip-10-155-29-101
    
log-level_pvt
DEBUG
    
message
30-10-22 19:58:28   [DEBUG] [Default]   [Worker]     Sleeping for 10 seconds...
    
message_pvt
30-10-22 19:58:28   [DEBUG] [Default]   [Worker]     Sleeping for 10 seconds...
    
path
/home/ubuntu/logstashdir/WorkerLogs/EC2AMAZ-06N2AJA_2.log
    
timestamp_pvt
30-10-22 19:58:28

在json格式中:

{
  "_index": "logstash-logs-2022.11.02",
  "_type": "_doc",
  "_id": "Z_nkN4QBewLz0DWJsjKB",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "timestamp_pvt": "30-10-22 19:58:28",
    "message_pvt": "30-10-22 19:58:28\t[DEBUG]\t[Default]\t[Worker]\t Sleeping for 10 seconds... is 113\r",
    "@timestamp": "2022-11-02T10:30:45.074Z",
    "host": "ip-10-155-29-101",
    "path": "/home/ubuntu/logstashdir/WorkerLogs/EC2AMAZ-06N2AJA_2.log",
    "log-level_pvt": "DEBUG",
    "message": "30-10-22 19:58:28\t[DEBUG]\t[Default]\t[Worker]\t Sleeping for 10 seconds... 113\r"
  },
  "fields": {
    "@timestamp": [
      "2022-11-02T10:30:45.074Z"
    ]
  },
  "sort": [
    1667385045074
  ]
}

有什么建议吗?

filter
{
        grok
        {
                match => { "message" => "%{LOGLEVEL:log-level_pvt}" }
        }

        grok
        {
                match => { "message" => "%{GREEDYDATA:message_pvt}" }
        }
        
        grok
        {
                match => { "message" => "%{TIMESTAMP_ISO8601:timestamp_pvt}" }
        }
}

消息为:

30-10-22 19:58:28   [DEBUG] [Default]   [Worker]     Sleeping for 10 seconds...

我希望消息是:

Sleeping for 10 seconds...
ni65a41a

ni65a41a1#

您可以使用overwrite参数来覆盖原始消息。出于性能考虑,请合并grok语句并以^开头

grok {
    overwrite => "message"
    match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp_pvt}\t\[%{LOGLEVEL:log-level_pvt}\]\t\[%{WORD}\]\t\[%{WORD}\]%{GREEDYDATA:message}$" }
}

相关问题