如何删除filebeat元数据

ecbunoof  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(998)

我正在使用filebeat将传入的日志从haproxy转发到kafka主题,但是在转发之后,filebeat向kafka消息添加了太多的元数据,这消耗了我想要避免的更多内存。
filebeat向kafka发送的消息示例,其中添加了元数据、主机和许多其他内容:
{“@timestamp”:“2017-03-27t08:14:09.508z”,“beat”:{“hostname”:“stage-kube03”,“name”:“stage-kube03”,“version”:“5.2.1”},“input揕type”:“log”,“message”:{“message”:{“activitytype”:null},“offset”:3783008,“source”:“/var/log/audit.log”,“type”:“log”}如何控制/减少filebeat添加到kafka消息的额外元数据以及日志行负载?下面是我的filebeat.yml文件


###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

# 

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

# configuration file.

# =========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/haproxy.log
  #exclude_files: [".gz$"]
  #fields:
   # codec: plain
   # token: USER_TOKEN
   # type: haproxy_log
  #fields_under_root: true
    #- c:\programdata\elasticsearch\logs\*
  processors:
   - drop_event:
     # fields: ["prospector","event","dataset"]

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']
  exclude_lines: ['^source']
  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

# ============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

# ================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

# name:

# The tags of the shipper are included in their own field with each

# transaction published.

# tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the

# output.

# fields:

# env: staging

# ============================== Dashboards =====================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here, or by using the `-setup` CLI flag or the `setup` command.

# setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the artifacts.elastic.co

# website.

# setup.dashboards.url:

# ============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# ============================= Elastic Cloud ==================================

# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

# cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

# cloud.auth:

# ================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

# -------------------------- Elasticsearch output ------------------------------

# output.elasticsearch:

  # Array of hosts to connect to.
 # hosts: ["localhost:9200"]

  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

# ----------------------------- Logstash output --------------------------------

# output.logstash:

  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

# ================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

# logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["*"]. Examples of other selectors are "beat",

# "publish", "service".

# logging.selectors: ["*"]

# ============================== Xpack Monitoring ===============================

# filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

# xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well. Any setting that is not set is

# automatically inherited from the Elasticsearch output configuration, so if you

# have the Elasticsearch output configured, you can simply uncomment the

# following line.

# xpack.monitoring.elasticsearch:

output.kafka:
  hosts: ["10.12.0.90:9092"]
  topic: "data-meter-topic"
  codec.json:
    pretty: true
col17t5w

col17t5w1#

您需要删除附加的 add_host_metadata 以及 add_cloud_metadata 使用drop\u field processor显式添加并删除其余字段的元数据:
我测试了您的配置并更改了以下内容:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.console:
  pretty: true
processors:
  - drop_fields:
      fields: ["agent", "log", "input", "host", "ecs" ]
  #- add_host_metadata: ~
  #- add_cloud_metadata: ~

结果是:

{
  "@timestamp": "2020-11-27T15:55:17.098Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.0"
  },
  "message": "2020-11-27 00:29:58 status installed libc-bin:amd64 2.28-10"
}

根据文档,您不能删除某些元数据,即 @timestamp 以及 type (其中应包括 @metadata 字段)。
drop\u fields处理器指定在满足某个条件时要删除哪些字段。条件是可选的。如果缺少,则始终删除指定的字段。@timestamp和type字段不能删除,即使它们显示在drop\u字段列表中。
编辑:
由于您似乎运行的是filebeat 5.2.1,因此我尝试了以下配置,其成功率甚至超过了filebeat 7.x:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
output.console:
  pretty: true
processors:
  - drop_fields:
     fields: ["log_type", "input_type", "offset", "beat", "source"]

结果:

{
  "@timestamp": "2020-11-30T09:51:40.404Z",
  "message": "2020-11-27 00:29:58 status half-configured vim:amd64 2:8.1.0875-5",
  "type": "log"
}

编辑2:
相反,因为您发布了一个filebeat 6.8.0版本的输出,所以我也使用这个非常相同的版本进行了测试:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.console:
  pretty: true
processors:
  - drop_fields:
      fields: ["beat", "source", "prospector", "offset", "host", "log", "input", "event", "fileset" ]
  #- add_host_metadata: ~
  #- add_cloud_metadata: ~

输出:

{
  "@timestamp": "2020-11-30T10:08:26.176Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.8.0"
  },
  "message": "2020-11-27 00:29:58 status unpacked vim:amd64 2:8.1.0875-5"
}

相关问题