kafka内部主题“\uu消费者\u偏移量”使用错误的代理ID重新分配,现在已损坏

wgx48brx  于 2021-06-08  发布在  Kafka
关注(0)|答案(0)|浏览(160)

大家好,
早些时候,在生产环境中,我们有一个broker.id=0的kafka(和zookeeper)示例,服务器是prd-1。一切都很好,直到我们把它转换成集群恢复。
因此,在将kafka的单个工作示例转换为具有2个节点的集群时,我们在$kafka\u home/config/server.properties中做了如下更改,在server.1和server.2属性的zookeer config中也做了类似的更改。
prd-1:

port=9092
broker.id=1    #Note change in broker id here. This was a mistake.
advertised.host.name=PRD-1.KPT.COM
zookeeper.connect=PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181

prd-2:

port=9092
  broker.id=2
  advertised.host.name=PRD-2.KPT.COM
  zookeeper.connect=PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181

在这个更改之后,我们提出了集群,它从代理1和代理2开始。它消耗来自多个生产者的信息。一切正常,集群正常运行。
但是,没有一个消费者能够连接到新创建的kafka集群来使用消息。当我检查存储所有使用者偏移量的内部主题(主题:\使用者\偏移量)时,我发现它是在我们将示例转换到集群之前创建的。我们发现了以下奇怪的细节:

./kafka-topics.sh --describe --zookeeper PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181 --topic __consumer_offsets
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets   Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 2    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 3    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 4    Leader: 0   Replicas: 0 Isr: 0

显然,内部topic\uu consumer\u offset仍然是用broker.id=0配置的(如果prd-1在没有集群的单机上运行的话)。
鉴于上述情况,我试图重新分配以下命令。

./kafka-topics.sh --describe --zookeeper PRD-1.KPT.COM:2181,PRD-2.KPT.COM:2181 --topic __consumer_offsets
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets   Partition: 0    Leader: 0   Replicas: 1,2,0 Isr: 0
    Topic: __consumer_offsets   Partition: 1    Leader: 0   Replicas: 1,2,0 Isr: 0
    Topic: __consumer_offsets   Partition: 2    Leader: 0   Replicas: 1,2,0 Isr: 0
    Topic: __consumer_offsets   Partition: 3    Leader: 0   Replicas: 1,2,0 Isr: 0
    Topic: __consumer_offsets   Partition: 4    Leader: 0   Replicas: 1,2,0 Isr: 0

但现在重新分配使情况更糟,我有3个副本(0,1,2),其中0是不存在的死代理。
现在,我的主题\uu consumer\u offset的复制因子为3,因为集群中只有2个节点(代理1和2)。
主题重新分配现在失败,错误消息如下。

ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-22
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-30
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-8
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-21
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-4
ERROR: Assigned replicas (1,2,0) don't match the list of replicas for reassignment (1,2) for partition __consumer_offsets-27
.....
..
..
Reassignment of partition __consumer_offsets-22 failed
Reassignment of partition __consumer_offsets-30 failed
Reassignment of partition __consumer_offsets-8 failed

有人能帮上忙吗?怎样才能让主题消费者识别集群并相应地更新其详细信息?。
或者
我需要放下主题,弹开Kafka,让它自动创建?。
任何寻求帮助/解决方案/尝试方法的人都将不胜感激。
谢谢您。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题