Kafka镜子制造者经常撞车

ehxuflar  于 2021-06-04  发布在  Kafka
关注(0)|答案(0)|浏览(309)

我们在docker容器上运行mirror maker,该容器由kubernetes协调。当输入负载很大时,我们经常会看到镜像机pod崩溃(假设)。
生产者配置

batch.size = 100000
buffer.memory = 50000000
compression.type = gzip
linger.ms = 100
retries=1000
max.request.size = 1000000
acks=1
max.in.flight.requests.per.connection = 5

消费者配置

auto.offset.reset=earliest
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
``` `ERROR` 级别日志:

rg.apache.kafka.common.errors.TimeoutException: Expiring 817 record(s) for xxxTopic-21: 33484 ms has passed since last append
2020-06-16 03:36:28 INFO MirrorMaker$:72 - Closing producer due to send failure.
2020-06-16 03:36:28 INFO KafkaProducer:341 - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 0 ms.
2020-06-16 03:36:28 INFO KafkaProducer:341 - [Producer clientId=producer-1] Proceeding to force close the producer since pending requests could not be completed within timeout 0 ms.
2020-06-16 03:36:28 ERROR ErrorLoggingCallback:52 - Error when sending message to topic xxxTopic with key: null, value: 173 bytes with error:
org.apache.kafka.common.errors.TimeoutException: Expiring 817 record(s) for xxxTopic-21: 33484 ms has passed since last append
2020-06-16 03:36:28 INFO MirrorMaker$:72 - Closing producer due to send failure.
2020-06-16 03:36:28 INFO KafkaProducer:341 - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 0 ms.
2020-06-16 03:36:28 INFO KafkaProducer:341 - [Producer clientId=producer-1] Proceeding to force close the producer since pending requests could not be completed within timeout 0 ms.
2020-06-16 03:36:28 ERROR ErrorLoggingCallback:52 - Error when sending message to topic xxxTopic with key: null, value: 201 bytes with error:
org.apache.kafka.common.errors.TimeoutException: Expiring 817 record(s) for xxxTopic-21: 33484 ms has passed since last append

飞机起飞后吊舱就坠毁了 `FATAL` 液位计 `ERROR` 上述日志:

org.apache.kafka.common.errors.TimeoutException: Expiring 731 record(s) for xxxTopic-22: 30117 ms has passed since batch creation plus linger time
2020-06-16 03:36:44 FATAL MirrorMaker$MirrorMakerThread:120 - [mirrormaker-thread-0] Mirror maker thread failure due to
java.lang.IllegalStateException: Cannot send after the producer is closed.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:198)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:806)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
at kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:711)

at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:431)

2020-06-16 03:36:44 INFO MirrorMaker$:72 - Closing producer due to send failure.
2020-06-16 03:36:44 INFO KafkaProducer:341 - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 0 ms.
2020-06-16 03:36:44 INFO KafkaProducer:341 - [Producer clientId=producer-1] Proceeding to force close the producer since pending requests could not be completed within timeout 0 ms.
2020-06-16 03:36:44 FATAL MirrorMaker$MirrorMakerThread:120 - [mirrormaker-thread-1] Mirror maker thread failure due to
java.lang.IllegalStateException: Cannot send after the producer is closed.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:198)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:806)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
at kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:711)

版本:
1.0.0Kafka客户。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题