druidkafka摄取(imple-2.2.3):kafka错误noreplicaonlineexception

igsr9ssn  于 2021-06-08  发布在  Kafka
关注(0)|答案(3)|浏览(309)

我使用DruidKafka索引服务从kafka加载我自己的流。
我使用来自kafka教程的load来实现它。
默认情况下,kafka具有所有设置(仅从tgz中提取)。
当我用空数据(var remove folder之后)启动imple-2.2.3(druid)时,所有的工作都正常。
但当我停止Kafka2.11-0.10.2.0并再次启动时,会发生错误,Druid-Kafka摄取不再工作,直到我停止暗示(druid)并删除所有数据(即删除var文件夹)。
有时Druid只是不接受Kafka的数据,甚至Kafka中没有错误。当我删除druid中的var文件夹时,所有的文件都被重新编写,直到下一个相同的错误。
错误:

kafka.common.NoReplicaOnlineException: No replica for partition [__consumer_offsets,19] is alive. Live brokers are: [Set()], Assigned replicas are: [List(0)]
    at kafka.controller.OfflinePartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:73) ~[kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:339) ~[kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:200) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:115) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:112) [kafka_2.11-0.10.2.0.jar:?]
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) [scala-library-2.11.8.jar:?]
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) [scala-library-2.11.8.jar:?]
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) [scala-library-2.11.8.jar:?]
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) [scala-library-2.11.8.jar:?]
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) [scala-library-2.11.8.jar:?]
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) [scala-library-2.11.8.jar:?]
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) [scala-library-2.11.8.jar:?]
    at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:112) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:67) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:342) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:160) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:85) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.ZookeeperLeaderElector$$anonfun$startup$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:51) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.ZookeeperLeaderElector$$anonfun$startup$1.apply(ZookeeperLeaderElector.scala:49) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.ZookeeperLeaderElector$$anonfun$startup$1.apply(ZookeeperLeaderElector.scala:49) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.ZookeeperLeaderElector.startup(ZookeeperLeaderElector.scala:49) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.KafkaController$$anonfun$startup$1.apply$mcV$sp(KafkaController.scala:681) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.KafkaController$$anonfun$startup$1.apply(KafkaController.scala:677) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.KafkaController$$anonfun$startup$1.apply(KafkaController.scala:677) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.controller.KafkaController.startup(KafkaController.scala:677) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.KafkaServer.startup(KafkaServer.scala:224) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.Kafka$.main(Kafka.scala:67) [kafka_2.11-0.10.2.0.jar:?]
    at kafka.Kafka.main(Kafka.scala) [kafka_2.11-0.10.2.0.jar:?]

我所做的步骤:
1开始暗示:

bin/supervise -c conf/supervise/quickstart.conf

2开始Kafka:

./bin/kafka-server-start.sh config/server.properties

三。创建主题:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic wikiticker

4启用DruidKafka摄取:

curl -XPOST -H'Content-Type: application/json' -d @quickstart/wikiticker-kafka-supervisor.json http://localhost:8090/druid/indexer/v1/supervisor

5将事件发布到Kafka主题,然后Kafka索引服务将这些事件吸收到Druid中
在所有.properties文件(common.runtime.properties、broker、coordinator、historical、middlemanager、overlord)中添加了属性:

druid.extensions.loadList=["druid-caffeine-cache", "druid-histogram", "druid-datasketches", "druid-kafka-indexing-service"]

其中包括“DruidKafka索引服务”提供摄取服务。
我相信这样的问题不应该发生在Druid·Kafka身上。
有办法解决这个问题吗?

nfeuvbwi

nfeuvbwi1#

看起来您有一个单节点kafka集群,并且唯一的代理节点已关闭。这不是一个非常容错的设置。您应该有3个kafka代理,并设置复制因子为3的所有主题,以便即使一个或两个kafka代理关闭,系统也能工作。单节点集群通常只用于开发。

wrrgggsh

wrrgggsh2#

该消息指示id为0的代理已关闭,因为它是唯一承载该分区的代理,所以您现在不能使用该分区。您必须确保代理0已启动并提供服务。

db2dz4w8

db2dz4w83#

我通过添加3个kafka代理修复了它,并设置了所有主题,其中复制因子为3,以保证kafka的稳定性。
在Druid我通过增加 druid.worker.capacity 在中间经理和减少 taskDurationioConfig 监理规范。
另一个问题的细节。

相关问题