kafka消费者接收开销?

fruv7luv  于 2021-06-07  发布在  Kafka
关注(0)|答案(2)|浏览(433)

我有一个Kafka消费者每10秒做一次民意调查。我正在使用wireshark监视我的网络活动。
我注意到,即使我没有执行任何fetch请求,代理和我的消费者之间仍然存在通信量。我还注意到,周期性地发送和接收的是相同的数据包(几乎相同,只是负载有一点变化)。
这是某种保命包吗?如何减少它们?
以下是这些数据包的截图:

ps:我使用cppkafka作为lib和kafka代理0.8.2.2
编辑:客户代码

bool running = true;

int main(int argc, char* argv[]) {
    string brokers;
    string topic_name;
    string group_id;

    po::options_description options("Options");
    options.add_options()
        ("help,h",     "produce this help message")
        ("brokers,b",  po::value<string>(&brokers)->required(), 
                       "the kafka broker list")
        ("topic,t",    po::value<string>(&topic_name)->required(),
                       "the topic in which to write to")
        ("group-id,g", po::value<string>(&group_id)->required(),
                       "the consumer group id")
        ;

    po::variables_map vm;

    try {
        po::store(po::command_line_parser(argc, argv).options(options).run(), vm);
        po::notify(vm);
    }
    catch (exception& ex) {
        cout << "Error parsing options: " << ex.what() << endl;
        cout << endl;
        cout << options << endl;
        return 1;
    }

    // Stop processing on SIGINT
    signal(SIGINT, [](int) { running = false; });

    // Construct the configuration
    Configuration config = {
        { "metadata.broker.list", brokers },
        { "api.version.request", false },
        { "broker.version.fallback", "0.8.2.2" },   
        { "group.id", group_id },
        // Disable auto commit
        { "enable.auto.commit", false }
    };

    // Create the consumer
    Consumer consumer(config);

    // Subscribe to the topic
    TopicPartitionList topicList;
    cppkafka::TopicPartition topPar(topic_name,0);
    topPar.set_offset(0);
    topicList.push_back(topPar);
    cout << "Consuming messages from topic " << topic_name << endl;

    consumer.assign(topicList);

    // Now read lines and write them into kafka
    while (running) {
        // Try to consume a message
        Message msg = consumer.poll();
        if (msg) {
            // If we managed to get a message
            if (msg.get_error()) {
                // Ignore EOF notifications from rdkafka
                if (!msg.is_eof()) {
                    cout << "[+] Received error notification: " << msg.get_error() << endl;
                } else {
                    std::this_thread::sleep_for(std::chrono::milliseconds(10000));
                }
            } else {
                // Print the key (if any)
                if (msg.get_key()) {
                    cout << msg.get_key() << " -> ";
                }
                // Print the payload
                cout << msg.get_payload() << endl;
            }
        }
    }
}
ajsxfq5m

ajsxfq5m1#

Kafka是建立在Kafka之上的。librdkafka尝试为所有分配的分区预取消息,因此调用poll()时消息立即可用。
默认情况下,librdkafka相当激进(以最佳性能为目标),因此每秒只能看到很少的fetchrequests。
有关更多详细信息,请参阅librdkafka的常见问题解答:
https://github.com/edenhill/librdkafka/wiki/faq#how-是否提取分区
https://github.com/edenhill/librdkafka/wiki/faq#my-否则-idle-consumer-is-taking-1-2-cpu

toe95027

toe950272#

您可能会看到心跳消息以保持消费群体的活力,您可以在此处找到有关它们的更多信息:https://cwiki.apache.org/confluence/display/kafka/a+guide+to+the+kafka+protocol#aguidetothekafkaprotocol-组成员身份API
可以通过修改heartbeat.interval.ms来调整心跳间隔,请检查librdkafka配置。

相关问题