Kafka镜像制作器快速压缩

rkue9o1l  于 2023-11-16  发布在  Apache
关注(0)|答案(2)|浏览(226)

我尝试在两个kafka之间配置镜像制作器，数据镜像对于没有压缩的主题工作正常。当镜像一个有snappy压缩的主题时，我得到以下异常：

java.io.FileNotFoundException: /tmp/snappy-1.1.10-00c148a7-468e-478a-855c-cabd60c7f21d-libsnappyjava.so (Read-only file system)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.xerial.snappy.SnappyLoader.extractLibraryFile(SnappyLoader.java:242)
    at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:358)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:179)
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:157)
    at org.xerial.snappy.Snappy.init(Snappy.java:70)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:446)
    at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:477)
    at java.base/java.io.DataInputStream.readByte(DataInputStream.java:270)
    at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:170)
    at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:205)
    at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:296)
    at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:278)
    at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:617)
    at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:582)
    at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:551)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1556)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1591)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
    at kafka.tools.MirrorMaker$ConsumerWrapper.receive(MirrorMaker.scala:325)
    at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:214)
[2023-11-09 20:30:08,767] ERROR [mirrormaker-thread-1] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2678)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.base/java.lang.System.loadLibrary(System.java:1886)
        at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:185)
        at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:157)
        at org.xerial.snappy.Snappy.init(Snappy.java:70)
        at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
        at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:446)
        at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:477)
        at java.base/java.io.DataInputStream.readByte(DataInputStream.java:270)
        at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:170)
        at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:205)
        at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:296)
        at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:278)
        at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:617)
        at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:582)
        at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:551)
        at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1556)
        at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1591)
        at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432)
        at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684)
        at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635)
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
        at kafka.tools.MirrorMaker$ConsumerWrapper.receive(MirrorMaker.scala:325)
        at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:214)
[2023-11-09 20:30:08,767] ERROR [mirrormaker-thread-2] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:446)
        at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:477)
        at java.base/java.io.DataInputStream.readByte(DataInputStream.java:270)
        at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:170)
        at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:205)
        at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:296)
        at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:278)
        at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:617)
        at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:582)
        at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:551)
        at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1556)
        at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1591)
        at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432)
        at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684)
        at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635)
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
        at kafka.tools.MirrorMaker$ConsumerWrapper.receive(MirrorMaker.scala:325)
        at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:214)

字符串
Docker image：confluentinc/cp-kafka:6.0.15 AND confluentinc/cp-kafka:5.4.1尝试复制jar文件：COPY snappy-java-1.1.10.5.jar /lib/snappy-java-1.1.10.5.jar没有帮助。下面的讨论也没有帮助
UnsatisfiedLinkError：/tmp/snappy-1.1.4-libsnappyjava.so加载共享库ld-linux-x86-64.so.2时出错：没有这样的文件或目录

apache-kafka

2条答案

按热度按时间
zzoitvuj

zzoitvuj1#

这似乎是因为snappy想要写入/tmp，并且它被挂载为只读。
我现在不能测试它，但我想说你可以使用Kafka_OPTS环境变量传递JVM_OPTS。
范例：

KAFKA_OPTS=-Dorg.xerial.snappy.tempdir=/path/to/newtmp

字符串
希望对你有帮助，

赞(0）分享  回复(0）举报  2023-11-16
6qqygrtg

6qqygrtg2#

事实证明，镜像制造商不附带Snappy Lib。我所做的是将libsnappyjava.so复制到docker镜像的/usr/lib/中，并且它可以工作。我已经从github repo here中获取了.so文件

FROM confluentinc/cp-kafka:6.0.15

COPY libsnappyjava.so /usr/lib/libsnappyjava.so

USER root
RUN chmod 755 /usr/lib/libsnappyjava.so

USER appuser

字符串

赞(0）分享  回复(0）举报  2023-11-16
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com