大家好,
早些时候,在生产环境中,我们有一个broker.id=0的kafka(和zookeeper)示例,服务器是prd-1。一切都很好,直到我们把它转换成集群恢复。
因此,在将kafka的单个工作示例转换为具有2个节点的集群时,我们在$kafka\u home/config/server.properties中做了如下更改,在server.1和server.2属性的zookeer config中也做了类似的更改。
prd-1:
port=9092
broker.id=1 #Note change in broker id here. This was a mistake.
advertised.host.name=PRD-1.KPT.COM
zookeeper.connect=PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181
prd-2:
port=9092
broker.id=2
advertised.host.name=PRD-2.KPT.COM
zookeeper.connect=PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181
在这个更改之后,我们提出了集群,它从代理1和代理2开始。它消耗来自多个生产者的信息。一切正常,集群正常运行。
但是,没有一个消费者能够连接到新创建的kafka集群来使用消息。当我检查存储所有使用者偏移量的内部主题(主题:\使用者\偏移量)时,我发现它是在我们将示例转换到集群之前创建的。我们发现了以下奇怪的细节:
./kafka-topics.sh --describe --zookeeper PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 0 Isr: 0
显然,内部topic\uu consumer\u offset仍然是用broker.id=0配置的(如果prd-1在没有集群的单机上运行的话)。
鉴于上述情况,我试图重新分配以下命令。
./kafka-topics.sh --describe --zookeeper PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 1,2,0 Isr: 0
但现在重新分配使情况更糟,我有3个副本(0,1,2),其中0是不存在的死代理。
现在,我的主题\uu consumer\u offset的复制因子为3,因为集群中只有2个节点(代理1和2)。
主题重新分配现在失败,错误消息如下。
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-22
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-30
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-8
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-21
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-4
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-27
.....
..
..
Reassignment of partition __consumer_offsets-22 failed
Reassignment of partition __consumer_offsets-30 failed
Reassignment of partition __consumer_offsets-8 failed
有人能帮上忙吗?怎样才能让主题消费者识别集群并相应地更新其详细信息?。
或者
我需要放下主题,弹开Kafka,让它自动创建?。
任何寻求帮助/解决方案/尝试方法的人都将不胜感激。
谢谢您。
暂无答案!
目前还没有任何答案,快来回答吧!