我正在尝试使用已装入的nfs卷运行kafka,遇到异常,无法启动kafka:
[2020-03-15 09:36:11,580] ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.KafkaException: Found directory /var/lib/kafka/data/.snapshot, '.snapshot' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
Kafka's log directories (and children) should only contain Kafka topic data. (kafka.log.LogManager)
[2020-03-15 09:36:11,582] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Found directory /var/lib/kafka/data/.snapshot, '.snapshot' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
Kafka's log directories (and children) should only contain Kafka topic data.
at kafka.log.Log$.exception$1(Log.scala:2150)
at kafka.log.Log$.parseTopicPartitionName(Log.scala:2157)
at kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:260)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:345)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
这是我的docker撰写脚本:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.2
environment:
ZOOKEEPER_CLIENT_PORT: 2181
volumes:
- zk-data:/var/lib/zookeeper/data:nocopy
- zk-log:/var/lib/zookeeper/log:nocopy
kafka:
image: confluentinc/cp-kafka:5.3.2
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
volumes:
- kf-data:/var/lib/kafka/data:nocopy
volumes:
zk-data:
driver: local
driver_opts:
type: "nfs"
o: addr=18.0.3.227 #IP of NFS
device: ":/opt/data/zk-data"
zk-log:
driver: local
driver_opts:
type: "nfs"
o: addr=18.0.3.227
device: ":/opt/data/zk-log"
kf-data:
driver: local
driver_opts:
type: "nfs"
o: addr=18.0.3.227
device: ":/opt/data/kf-data"
如果我转到nfs服务器,
ls -la /opt/data/kf-data/.snapshot
total 80
drwxrwxrwx 33 root root 12288 Mar 28 00:10 .
drwx------ 2 root domain^users 4096 Feb 21 19:20 ..
drwx------ 2 root domain^users 4096 Feb 13 11:06 daily.2020-02-14_0010
drwx------ 2 root domain^users 4096 Feb 13 11:06 daily.2020-02-15_0010
drwx------ 2 root domain^users 4096 Feb 13 11:06 daily.2020-02-16_0010
drwx------ 2 root domain^users 4096 Feb 13 11:06 daily.2020-02-17_0010
drwx------ 2 root domain^users 4096 Feb 21 19:20 snapmirror.ka938443-8ea1-22e8-6608-00a067d1a20a_2148891236.2020-02-27_180700
有一个名为.snapshot的隐藏文件夹,此文件夹由nfs自动生成,无法删除。这就是kafka抱怨的原因:found directory/var/lib/kafka/data/.snapshot,“.snapshot”的格式不是topic partition或topic-partition.uniqueid-delete(如果标记为删除)。
这可能是Kafka的一般问题,是否有任何特殊的配置或解决方案让Kafka使用外部nfs卷?
任何想法都将不胜感激!
3条答案
按热度按时间ctrmrzij1#
有一个名为.snapshot的隐藏文件夹,此文件夹由nfs自动生成,无法删除
好吧,如果没有办法的话,Kafka将无法开始,句号。
据我所知,文档中没有提到支持远程连接存储。
jjhzyzn02#
如上所述,由于nfs文件系统的工作方式,kafka on nfs是一个有缺陷的解决方案。您将遇到重新分区和扩展的问题。这与nfs处理删除打开文件的方式有关——愚蠢的重命名行为。你可以在这篇博文(nfs上的Kafka)中看到它。
kqqjbcuj3#
如果将netapp用作nfs平台,此信息可能会有所帮助:禁用netapp中的快照访问是一个全局vfilter函数,而不是每个文件夹或共享的函数。
如果无法关闭对.snapshot的访问,则没有解决方案,除非使用其他nfs平台,否则不会在每个文件夹中生成.snapshot文件夹。