当我尝试在filebeat.yml中创建多个索引并输出到elasticsearch时,我得到临时的批量发送失败错误。只有当我将ilm设置为disable时才会出现这种情况。有人能帮忙吗
以下是filebeat配置
filebeat.inputs:
- type: filestream
id: denali
enabled: true
paths:
- /var/log/denali/denali.log
parsers:
- multiline:
type: pattern
pattern: '^(\d{4}-\d{2}-\d{2})'
negate: true
match: after
fields:
app_id: denali
- type: filestream
id: freeswitch
enabled: true
paths:
- /var/log/freeswitch/freeswitch.log
parsers:
- multiline:
type: pattern
pattern: '^((\d|[a-z]|-)+ \d{4}-\d{2}-\d{2}|\d{4}-\d{2}-\d{2})'
negate: true
match: after
fields:
app_id: freeswitch
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.enabled: true
setup.ilm.enabled: false
setup.template.overwrite: true
setup.template.name: "index-%{[agent.version]}"
setup.template.pattern: "index-%{[agent.version]}-*"
output.elasticsearch:
hosts: ["ip:port"]
index: "index-%{[agent.version]}-%{[fields.app_id]:other}-%{+yyyy.MM.dd}"
protocol: "http"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- drop_fields:
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.name", "agent.type", "agent.version", "cloud.account.id", "cloud.provider", "cloud.service.name", "container.id", "container.image.name", "container.labels.COMMIT", "container.labels.PIPELINE_URL", "container.labels.PROJECT_NAME", "container.labels.PROJECT_URL", "container.labels.SOURCE_BRANCH", "container.labels.TimeStamp", "container.labels.RELEASEARTIFACT_VERSION", "container.labels.com_docker_compose_config-hash", "container.labels.com_docker_compose_container-number", "container.labels.com_docker_compose_oneoff", "container.labels.com_docker_compose_project", "container.labels.com_docker_compose_project_config_files", "container.labels.com_docker_compose_project_working_dir", "container.labels.com_docker_compose_service", "container.labels.com_docker_compose_version", "ecs.version", "host.architecture", "host.containerized", "host.id", "host.mac", "host.os.codename", "host.os.family", "host.os.kernel", "host.os.name", "host.os.platform", "host.os.type", "host.os.version", "log.offset"]
1条答案
按热度按时间y1aodyip1#
@Ramanichandran您能提供filebeat的错误日志吗?另外,当filebeat尝试发送日志进行摄取时,您是否在ES日志上看到任何错误?
我不认为这是由于创建了多个索引,因为您实际上只创建了3个索引。在我的使用案例中,我配置了filebeat来创建大约15个索引,在我的配置与您的配置类似但禁用了ILM的情况下,它工作得很好。
值得尝试为output.elasticsearch设置以下属性: