我有一个
3个docker容器上的1个kafka集群,在ecs上运行,使用efs作为存储服务(为简单起见)。
1个kafka streams应用程序集群,位于3个容器上。
有一个源主题“事件”,有16个分区,复制2。一个papi拓扑处理器“流处理器”产生输出到其他一些主题,并使用3个状态存储。
我可以通过kafka管理器看到数据被消耗,输出被生成到这些其他输出主题。很明显它起作用了(虽然很慢)。
但是通过bin/kafka-consumer-groups.sh查看消费者补偿,我可以看到随着时间的推移,只有一个分区同时被消费。在命令的分离、连续运行中,仅减少一个偏移量。
首次执行:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
events 6 - 4021552 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 13 5030392 5030541 149 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 2 7056462 7056462 0 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 8 671945 6046546 5374601 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 1 164123 3009191 2845068 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 12 1962842 11052506 9089664 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 5 - 4022059 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 0 - 4019992 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 4 - 5032053 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 11 5037439 5037584 145 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 15 1683056 5034689 3351633 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 7 164702 7052434 6887732 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 14 - 3011069 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 3 1927601 6044400 4116799 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 10 5031461 5031612 151 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 9 1686979 8052924 6365945 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
第二次执行:只有分区8提前了其偏移量。在1、5或15分钟后,该分区是唯一消耗的分区。
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
events 6 - 4021552 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 13 5030392 5030541 149 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 2 7056462 7056462 0 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 8 686685 6046546 5359861 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 1 164123 3009191 2845068 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 12 1962842 11052506 9089664 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 5 - 4022059 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 0 - 4019992 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 4 - 5032053 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 11 5037439 5037584 145 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 15 1683056 5034689 3351633 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 7 164702 7052434 6887732 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 14 - 3011069 - stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 3 1927601 6044400 4116799 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 10 5031461 5031612 151 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
events 9 1686979 8052924 6365945 stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here stream-processor-fa36-StreamThread-1-consumer
查看日志,只有一个示例同时打印日志。i、 e.如果一个在工作,其他两个就不工作。
有什么问题吗?
Kafka和Kafka流版本1.1。
暂无答案!
目前还没有任何答案,快来回答吧!