logstash输出ElasticSearch带序号的索引

pod7payv  于 2022-12-09  发布在  Logstash
关注(0)|答案(1)|浏览(247)

我正在使用AWSElasticSearch(版本7.10)和Logstash 7.10。目的是将内容从logstash发送到ElasticSearch,并在特定大小或时间后使用策略滚动索引。

policy: {
    "policy_id": "Rollover_Policy",
    "description": "roller index",
    "last_updated_time": 1634910129219,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "1mb"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm"
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "replica_count": {
                        "number_of_replicas": 1
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "1h"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "products*"
            ],
            "priority": 100,
            "last_updated_time": 1634910129219
        }
    ]
}

当我试图在logstash输出插件中将ilm_enabled设置为true时,它无法与ElasticSearchxpack API连接。

    • 注意**:AWSElasticSearch不支持xpack和ILM。
elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => true
        ilm_rollover_alias => "products"
        ilm_pattern => "{now/d}-000001"
        ilm_policy => "Rollover_Policy"
}

因此,我已将ilm_enabled标志更改为false,并尝试了以下选项

elasticsearch {
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false
        index => "products-%{+YYYY.MM.dd}-000001"
}

现在的问题是,即使在翻转之后,logstash仍然将文档发送到001索引而不是新索引。如果我不在索引名称中指定-000001,则翻转将失败。

mxg2im7a

mxg2im7a1#

Create an index with following REST request in elastic. Since the index name is having date pattern, the rollover will create new index with current date.

PUT %3Cproducts-%7Bnow%2Fd%7D-000001%3E
{
  "settings":{
    "number_of_shards":1,
    "number_of_replicas":1
  },
  "aliases": {
    "products":  {
      "is_write_index": true
    }
  }

Create a template for index pattern along with rollover alias

PUT _index_template/products_logs
{
  "index_patterns": [
    "products*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "opendistro": {
        "index_state_management": {
          "rollover_alias": "products"
        }
      }
    }
  }
}

In logstash output plugin give the below details to send the data to elastic search

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false 
        index => "products"
}

Note : the index name represents alias name of the index.

相关问题