消费者之间的主题共享

yks3o0rb  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(366)

我正在开发一个springboot应用程序,它在kafka主题中为不同的分区重新平衡(例如700个主题,每个主题有10个分区,即7000个分区)。但我想创建多个DockedApp示例,其中应用程序将包含所有700个主题名称,但它应该只选择前25个分区并取消订阅其他分区)

@KafkaListener( topics = "#{kafkaProperties.getTopics()}" )

kafkaproperties.gettopics()返回所有700个主题名

avwztpqn

avwztpqn1#

正如我在要求澄清的初始评论中所说的,对于这样的分区分配方案,您将需要实现一个自定义的 ConsumerPartitionAssignor .
每个示例应选择数量有限的线程,每个线程有一个主题/分区
示例不“选择”主题/分区;选择一个示例并决定哪个示例获得哪个主题/分区。
查看其javadocs。

/**
 * This interface is used to define custom partition assignment for use in
 * {@link org.apache.kafka.clients.consumer.KafkaConsumer}. Members of the consumer group subscribe
 * to the topics they are interested in and forward their subscriptions to a Kafka broker serving
 * as the group coordinator. The coordinator selects one member to perform the group assignment and
 * propagates the subscriptions of all members to it. Then {@link #assign(Cluster, GroupSubscription)} is called
 * to perform the assignment and the results are forwarded back to each respective members
 *
 * In some cases, it is useful to forward additional metadata to the assignor in order to make
 * assignment decisions. For this, you can override {@link #subscriptionUserData(Set)} and provide custom
 * userData in the returned Subscription. For example, to have a rack-aware assignor, an implementation
 * can use this user data to forward the rackId belonging to each member.
 */
public interface ConsumerPartitionAssignor {

要向示例添加更多线程(使用者),请增加容器并发性。
对于大量示例,我建议 COOPERATIVE 再平衡协议。

相关问题