Kafka消费者测试和报告指标

goucqfw6  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(648)

我想了解Kafka消费者测试是如何工作的,以及如何解释报告的一些数字,
下面是我运行的测试和得到的输出。我的问题是
报告的值 rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec1593109326098, -1593108732333, -0.0003, -0.2800 ; 你能解释一下它怎么会报这么高的负数吗?他们对我没有意义。
所有报告都来自 Metric Name Value 线路报告原因 --print-metrics 旗帜。默认报告的度量与使用此标志报告的度量之间有什么区别?它们是如何计算出来的,我从哪里可以了解到它们的含义?
无论我是扩展并行运行的总使用者还是扩展代理上的网络和io线程, consumer-fetch-manager-metrics:fetch-latency-avg 指标几乎保持不变。你能解释一下吗?随着越来越多的消费者拉数据获取延迟应该更高;同样,对于给定的消耗率,如果我减少代理的io和网络线程参数,那么延迟是否应该更高?
这是我执行的命令

[root@oak-clx17 kafka_2.12-2.5.0]# bin/kafka-consumer-perf-test.sh --topic topic_test8_cons_test1 --threads 1  --broker-list clx20:9092 --messages 500000000 --consumer.config config/consumer.properties --print-metrics

和结果

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms,fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
    WARNING: Exiting before consuming the expected number of messages: timeout (10000 ms) exceeded. You can use the --timeout option to increase the timeout.
    2020-06-25 11:22:05:814, 2020-06-25 11:31:59:579, 435640.7686, 733.6922, 446096147, 751300.8463, 1593109326098, -1593108732333, -0.0003, -0.2800
     
    Metric Name                                                                                                                                    Value
    consumer-coordinator-metrics:assigned-partitions:{client-id=consumer-perf-consumer-25533-1}                                                  : 0.000
    consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-perf-consumer-25533-1}                                                   : 2.700
    consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-perf-consumer-25533-1}                                                   : 4.000
    consumer-coordinator-metrics:commit-rate:{client-id=consumer-perf-consumer-25533-1}                                                          : 0.230
    consumer-coordinator-metrics:commit-total:{client-id=consumer-perf-consumer-25533-1}                                                         : 119.000
    consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=consumer-perf-consumer-25533-1}                                       : 0.000
    consumer-coordinator-metrics:failed-rebalance-total:{client-id=consumer-perf-consumer-25533-1}                                               : 1.000
    consumer-coordinator-metrics:heartbeat-rate:{client-id=consumer-perf-consumer-25533-1}                                                       : 0.337
    consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=consumer-perf-consumer-25533-1}                                          : 6.000
    consumer-coordinator-metrics:heartbeat-total:{client-id=consumer-perf-consumer-25533-1}                                                      : 197.000
    consumer-coordinator-metrics:join-rate:{client-id=consumer-perf-consumer-25533-1}                                                            : 0.000
    consumer-coordinator-metrics:join-time-avg:{client-id=consumer-perf-consumer-25533-1}                                                        : NaN
    consumer-coordinator-metrics:join-time-max:{client-id=consumer-perf-consumer-25533-1}                                                        : NaN
    consumer-coordinator-metrics:join-total:{client-id=consumer-perf-consumer-25533-1}                                                           : 1.000
    consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=consumer-perf-consumer-25533-1}                                           : 2.000
    consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=consumer-perf-consumer-25533-1}                                           : 593.000
    consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=consumer-perf-consumer-25533-1}                                       : NaN
    consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=consumer-perf-consumer-25533-1}                                       : NaN
    consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=consumer-perf-consumer-25533-1}                                           : NaN
    consumer-coordinator-metrics:partition-lost-latency-max:{client-id=consumer-perf-consumer-25533-1}                                           : NaN
    consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=consumer-perf-consumer-25533-1}                                        : 0.000
    consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=consumer-perf-consumer-25533-1}                                        : 0.000
    consumer-coordinator-metrics:rebalance-latency-avg:{client-id=consumer-perf-consumer-25533-1}                                                : NaN
    consumer-coordinator-metrics:rebalance-latency-max:{client-id=consumer-perf-consumer-25533-1}                                                : NaN
    consumer-coordinator-metrics:rebalance-latency-total:{client-id=consumer-perf-consumer-25533-1}                                              : 83.000
    consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=consumer-perf-consumer-25533-1}                                              : 0.000
    consumer-coordinator-metrics:rebalance-total:{client-id=consumer-perf-consumer-25533-1}                                                      : 1.000
    consumer-coordinator-metrics:sync-rate:{client-id=consumer-perf-consumer-25533-1}                                                            : 0.000
    consumer-coordinator-metrics:sync-time-avg:{client-id=consumer-perf-consumer-25533-1}                                                        : NaN
    consumer-coordinator-metrics:sync-time-max:{client-id=consumer-perf-consumer-25533-1}                                                        : NaN
    consumer-coordinator-metrics:sync-total:{client-id=consumer-perf-consumer-25533-1}                                                           : 1.000
    consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=consumer-perf-consumer-25533-1, topic=topic_test8_cons_test1}                  : 434828205.989
    consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=consumer-perf-consumer-25533-1}                                                : 434828205.989
    consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=consumer-perf-consumer-25533-1, topic=topic_test8_cons_test1}                 : 460817319851.000
    consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=consumer-perf-consumer-25533-1}                                               : 460817319851.000
    consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=consumer-perf-consumer-25533-1}                                                  : 58.870
    consumer-fetch-manager-metrics:fetch-latency-max:{client-id=consumer-perf-consumer-25533-1}                                                  : 503.000
    consumer-fetch-manager-metrics:fetch-rate:{client-id=consumer-perf-consumer-25533-1}                                                         : 48.670
    consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-perf-consumer-25533-1, topic=topic_test8_cons_test1}                       : 9543108.526
    consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-perf-consumer-25533-1}                                                     : 9543108.526
    consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-perf-consumer-25533-1, topic=topic_test8_cons_test1}                       : 11412584.000
    consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-perf-consumer-25533-1}                                                     : 11412584.000
    consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=consumer-perf-consumer-25533-1}                                            : 0.000
    consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=consumer-perf-consumer-25533-1}                                            : 0.000
    consumer-fetch-manager-metrics:fetch-total:{client-id=consumer-perf-consumer-25533-1}                                                        : 44889.000
    Exception in thread "main" java.util.IllegalFormatConversionException: f != java.lang.Integer
    at java.base/java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4426)
            at java.base/java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2951)
            at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:2898)
            at java.base/java.util.Formatter.format(Formatter.java:2673)
            at java.base/java.util.Formatter.format(Formatter.java:2609)
            at java.base/java.lang.String.format(String.java:2897)
            at scala.collection.immutable.StringLike.format(StringLike.scala:354)
            at scala.collection.immutable.StringLike.format$(StringLike.scala:353)
            at scala.collection.immutable.StringOps.format(StringOps.scala:33)
            at kafka.utils.ToolsUtils$.$anonfun$printMetrics$3(ToolsUtils.scala:60)
            at kafka.utils.ToolsUtils$.$anonfun$printMetrics$3$adapted(ToolsUtils.scala:58)
            at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
            at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
            at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
            at kafka.utils.ToolsUtils$.printMetrics(ToolsUtils.scala:58)
            at kafka.tools.ConsumerPerformance$.main(ConsumerPerformance.scala:82)
            at kafka.tools.ConsumerPerformance.main(ConsumerPerformance.scala)
pgky5nke

pgky5nke1#

https://medium.com/metrosystemsro/apache-kafka-how-to-test-performance-for-clients-configured-with-ssl-encryption-3356d3a0d52b 这是我写的一篇文章。
根据此kip,rebalance.time.ms和fetch.time.ms的期望值显示消费者组的总再平衡时间和不包括再平衡时间的总提取时间。据我所知,在ApacheKafka版本2.6.0中,这仍然是一项正在进行的工作,目前输出是unix时代的时间戳。fetch.mb.sec和fetch.nmsg.sec用于显示每秒消耗的平均消息量(以mb为单位,并作为计数)
看到了吗https://kafka.apache.org/documentation/#consumer_group_monitoring 对于列出了--print metrics标志的消费者组指标
fetch latency avg(获取请求所用的平均时间)会有所不同,但这在很大程度上取决于测试设置。

相关问题