docker compose重启后kafka主题配置丢失

cuxqih21  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(1262)

我用docker运行kafka,设置在卷上存储数据。我设置了一些源连接器,主题是用cleanup.policy自动创建的 delete . 使用Kafka管理器,我将策略更改为 compact .
问题是:
停止/启动docker compose主题出现后,但cleanup.policy恢复为 delete 问题:
重启后如何持久化主题配置?
附加信息
我用命令重新启动Kafka Docker : rm /kafka/data/1/meta.properties; docker-compose down && docker-compose up -d --no-recreate docker-compose.yml码:

  1. version: '2'
  2. services:
  3. zookeeper:
  4. image: debezium/zookeeper:${DEBEZIUM_VERSION}
  5. ports:
  6. - 2181:2181
  7. - 2888:2888
  8. - 3888:3888
  9. volumes:
  10. - /kafka/zookeeper_data:/zookeeper/data
  11. - /kafka/zookeeper_logs:/zookeeper/logs
  12. - /kafka/zookeeper_conf:/zookeeper/conf
  13. kafka:
  14. image: debezium/kafka:${DEBEZIUM_VERSION}
  15. ports:
  16. - 9092:9092
  17. links:
  18. - zookeeper
  19. environment:
  20. - ZOOKEEPER_CONNECT=zookeeper:2181
  21. - KAFKA_BROKER_ID=1
  22. - ADVERTISED_HOST_NAME=172.16.10.187
  23. volumes:
  24. - /kafka/data:/kafka/data
  25. - /kafka/config:/kafka/config
  26. schema-registry:
  27. image: confluentinc/cp-schema-registry
  28. ports:
  29. - 8181:8181
  30. - 8081:8081
  31. environment:
  32. - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181
  33. - SCHEMA_REGISTRY_HOST_NAME=schema-registry
  34. - SCHEMA_REGISTRY_LISTENERS=http://schema-registry:8081
  35. links:
  36. - zookeeper
  37. connect3:
  38. build:
  39. context: debezium-jdbc-es
  40. ports:
  41. - 8083:8083
  42. links:
  43. - kafka
  44. environment:
  45. - BOOTSTRAP_SERVERS=kafka:9092
  46. - GROUP_ID=1
  47. - CONFIG_STORAGE_TOPIC=my_connect_configs
  48. - OFFSET_STORAGE_TOPIC=my_connect_offsets
  49. - STATUS_STORAGE_TOPIC=my_connect_statuses
  50. - KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
  51. - VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
  52. - INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
  53. - INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
  54. - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
  55. - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
  56. - KAFKA_OPTS=-javaagent:/kafka/jmx_prometheus_javaagent.jar=8080:/kafka/config.yml
  57. - CONNECT_REST_ADVERTISED_HOST_NAME=connect3
  58. - JMX_PORT=1976
  59. prometheus:
  60. build:
  61. context: debezium-prometheus
  62. ports:
  63. - 9090:9090
  64. links:
  65. - connect3
  66. grafana:
  67. build:
  68. context: debezium-grafana
  69. ports:
  70. - 3000:3000
  71. links:
  72. - prometheus
  73. environment:
  74. - DS_PROMETHEUS=prometheus
  75. restproxy:
  76. image: confluentinc/cp-kafka-rest
  77. environment:
  78. KAFKA_REST_BOOTSTRAP_SERVERS: "kafka:9092"
  79. KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
  80. KAFKA_REST_HOST_NAME: restproxy
  81. KAFKA_REST_DEBUG: "true"
  82. KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
  83. ports:
  84. - 8082:8082
  85. kafka-ui:
  86. image: landoop/kafka-connect-ui:latest
  87. ports:
  88. - 8000:8000
  89. links:
  90. - connect3
  91. - schema-registry
  92. - zookeeper
  93. environment:
  94. - CONNECT_URL=http://connect3:8083/
  95. kafka-topic-ui:
  96. image: landoop/kafka-topics-ui
  97. links:
  98. - connect3
  99. ports:
  100. - 8001:8000
  101. environment:
  102. - KAFKA_REST_PROXY_URL=http://restproxy:8082
  103. - PROXY=true
  104. kafka_manager:
  105. image: hlebalbau/kafka-manager:stable
  106. ports:
  107. - "9000:9000"
  108. environment:
  109. ZK_HOSTS: "zookeeper:2181"
  110. links:
  111. - connect3
qyzbxkaa

qyzbxkaa1#

Kafka主题配置存储在zookeeper中。您可以使用

  1. bin/zookeeper-shell zookeeper_ip_or_fdqn:2181 get /config/topics/yourTopic

想到的一个潜在问题是zookeeper数据没有持久化。我看到您的zookeeper容器有3个卷,但可能需要检查zookeeper.properties中的datadir属性是否指向持久化文件夹,或者zookeeper数据没有持久化的另一个原因。
希望这有用。
致以最诚挚的问候。

相关问题