logstash输出ElasticSearch带序号的索引

pod7payv  于 2022-12-09  发布在  Logstash
关注(0)|答案(1)|浏览(246)

我正在使用AWSElasticSearch(版本7.10)和Logstash 7.10。目的是将内容从logstash发送到ElasticSearch,并在特定大小或时间后使用策略滚动索引。

  1. policy: {
  2. "policy_id": "Rollover_Policy",
  3. "description": "roller index",
  4. "last_updated_time": 1634910129219,
  5. "schema_version": 1,
  6. "error_notification": null,
  7. "default_state": "hot",
  8. "states": [
  9. {
  10. "name": "hot",
  11. "actions": [
  12. {
  13. "rollover": {
  14. "min_size": "1mb"
  15. }
  16. }
  17. ],
  18. "transitions": [
  19. {
  20. "state_name": "warm"
  21. }
  22. ]
  23. },
  24. {
  25. "name": "warm",
  26. "actions": [
  27. {
  28. "replica_count": {
  29. "number_of_replicas": 1
  30. }
  31. }
  32. ],
  33. "transitions": [
  34. {
  35. "state_name": "delete",
  36. "conditions": {
  37. "min_index_age": "1h"
  38. }
  39. }
  40. ]
  41. },
  42. {
  43. "name": "delete",
  44. "actions": [
  45. {
  46. "delete": {}
  47. }
  48. ],
  49. "transitions": []
  50. }
  51. ],
  52. "ism_template": [
  53. {
  54. "index_patterns": [
  55. "products*"
  56. ],
  57. "priority": 100,
  58. "last_updated_time": 1634910129219
  59. }
  60. ]
  61. }

当我试图在logstash输出插件中将ilm_enabled设置为true时,它无法与ElasticSearchxpack API连接。

    • 注意**:AWSElasticSearch不支持xpack和ILM。
  1. elasticsearch {
  2. hosts => "${elasticsearch_endpoint}"
  3. user => "${elasticsearch_user}"
  4. password => "${elasticsearch_password}"
  5. ilm_enabled => true
  6. ilm_rollover_alias => "products"
  7. ilm_pattern => "{now/d}-000001"
  8. ilm_policy => "Rollover_Policy"
  9. }

因此,我已将ilm_enabled标志更改为false,并尝试了以下选项

  1. elasticsearch {
  2. hosts => "${elasticsearch_endpoint}"
  3. user => "${elasticsearch_user}"
  4. password => "${elasticsearch_password}"
  5. ilm_enabled => false
  6. index => "products-%{+YYYY.MM.dd}-000001"
  7. }

现在的问题是,即使在翻转之后,logstash仍然将文档发送到001索引而不是新索引。如果我不在索引名称中指定-000001,则翻转将失败。

mxg2im7a

mxg2im7a1#

Create an index with following REST request in elastic. Since the index name is having date pattern, the rollover will create new index with current date.

  1. PUT %3Cproducts-%7Bnow%2Fd%7D-000001%3E
  2. {
  3. "settings":{
  4. "number_of_shards":1,
  5. "number_of_replicas":1
  6. },
  7. "aliases": {
  8. "products": {
  9. "is_write_index": true
  10. }
  11. }

Create a template for index pattern along with rollover alias

  1. PUT _index_template/products_logs
  2. {
  3. "index_patterns": [
  4. "products*"
  5. ],
  6. "template": {
  7. "settings": {
  8. "number_of_shards": 1,
  9. "number_of_replicas": 1,
  10. "opendistro": {
  11. "index_state_management": {
  12. "rollover_alias": "products"
  13. }
  14. }
  15. }
  16. }
  17. }

In logstash output plugin give the below details to send the data to elastic search

  1. elasticsearch {
  2. hosts => "${elasticsearch_endpoint}"
  3. user => "${elasticsearch_user}"
  4. password => "${elasticsearch_password}"
  5. ilm_enabled => false
  6. index => "products"
  7. }

Note : the index name represents alias name of the index.

展开查看全部

相关问题