未能运行kafka消费者性能测试

lf5gs5x2 于 2021-06-08 发布在 Kafka

关注(0)|答案(1)|浏览(362)

命令：

./bin/kafka-consumer-perf-test.sh --topic test_topic --messages 500000 --zookeeper zookeepernode --show-detailed-stats --consumer.config ./conf/connect-distributed.properties

导致

{metadata.broker.list=kafkanode1:9093,kafkanode2:9093,av3l338p.kafkanode3:9093, request.timeout.ms=30000, client.id=perf-consumer-68473, security.protocol=SASL_SSL}

WARN Fetching topic metadata with correlation id 0 for topics [Set(test_topic)] from broker [BrokerEndPoint(1003,kafkanode3,9093)] failed (kafka.client.ClientUtils$)
java.io.EOFException
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)

[...]
WARN Fetching topic metadata with correlation id 0 for topics [Set(test_topic)] from broker [BrokerEndPoint(1001,kafkanode1,9093)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedByInterruptException
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)

[...]
WARN Fetching topic metadata with correlation id 0 for topics [Set(test_topic)] from broker [BrokerEndPoint(1002,kafkanode2,9093)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:122)

apache-kafka kafka-consumer-api

来源：https://stackoverflow.com/questions/44282158/failing-to-run-kafka-consumer-performance-test

1条答案

按热度按时间

j1dl9f461#

尝试添加标志“--new consumer”：

./bin/kafka-consumer-perf-test.sh --new-consumer --broker-list <broker-host>:9093 --topic <topic-read-from> --messages <number-of-messages> --show-detailed-stats --consumer.config consumer.properties --zookeeper <zookeeper-host>:2181

consumer.properties如下所示（包括sasl\u ssl和kerberos配置）：

zookeeper.connect=<zookeeper-host>:2181

# timeout in ms for connecting to zookeeper

zookeeper.connection.timeout.ms=6000

# consumer group id

group.id=<consumer-group>

security.protocol=SASL_SSL
ssl.keystore.location=<path-to-keystore-file>
ssl.keystore.password=<password>
ssl.truststore.location=<path-to-truststore-file>
ssl.truststore.password=<password>

输出应如下所示：

time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2017-06-01 11:29:15:194, 0, 1,6726, 4,1816, 5000, 12500,0000
2017-06-01 11:29:15:207, 0, 3,3916, 859,4761, 10000, 2500000,0000
2017-06-01 11:29:15:208, 0, 5,1009, 1709,3201, 15000, 5000000,0000
2017-06-01 11:29:15:209, 0, 6,7850, 1684,1164, 20000, 5000000,0000
2017-06-01 11:29:15:291, 0, 8,5101, 21,2977, 25000, 61728,3951
2017-06-01 11:29:15:460, 0, 10,2138, 10,0808, 30000, 29585,7988
2017-06-01 11:29:15:462, 0, 11,9258, 1711,9637, 35000, 5000000,0000
2017-06-01 11:29:15:463, 0, 13,6348, 1709,0416, 40000, 5000000,0000
2017-06-01 11:29:15:673, 0, 15,3255, 8,0511, 45000, 23809,5238
2017-06-01 11:29:15:962, 0, 17,0417, 5,9384, 50000, 17301,0381
2017-06-01 11:29:15:963, 0, 18,7520, 1710,2280, 55000, 5000000,0000
2017-06-01 11:29:15:963, 0, 20,4568, Infinity, 60000, Infinity
2017-06-01 11:29:16:090, 0, 22,1223, 13,1138, 65000, 39370,0787
2017-06-01 11:29:16:282, 0, 23,8131, 8,8062, 70000, 26041,6667

赞(0）回复(0）举报 2021-06-08

我来回答

未能运行kafka消费者性能测试

1条答案

相关问题

热门标签

最新问答