flink读取s3文件导致jackson依赖问题

fruv7luv  于 2021-06-24  发布在  Flink
关注(0)|答案(1)|浏览(665)

我正在我的链接应用程序中读取一个配置yaml文件。我想把这个配置文件保存在s3文件系统上,但是当我在pom中使用awsdk并试图读取时,我得到了这个错误。我知道这是由于Jackson的冲突依赖,但我不能解决它。请帮我解决这个问题。
java.lang.nosuchmethoderror:com.fasterxml.jackson.databind.objectmapper.enable([lcom/fasterxml/jackson/core/jsonparser$feature;)lcom/fasterxml/jackson/databind/objectmapper;在com.amazonaws.partitions.partitionsloader。java:54)在com.amazonaws.regions.regionmetadatafactory.create(regionmetadatafactory。java:30)在com.amazonaws.regions.regionutils.initialize(regionutils。java:65)在com.amazonaws.regions.regionutils.getregionmetadata(regionutils。java:53)在com.amazonaws.regions.regionutils.getregion(regionutils。java:107)在com.amazonaws.services.s3.amazons3client.createsigner(amazons3client。java:4016)在com.amazonaws.services.s3.amazon3client.invoke(amazon3client。java:4913)在com.amazonaws.services.s3.amazon3client.invoke(amazon3client。java:4872)在com.amazonaws.services.s3.amazon3client.getobject(amazon3client。java:1472)在com.bounce.processor.eventprocessor.main(eventprocessor。java:71)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:498)在org.apache.flink.client.program.packagedprogram.callmain方法(packagedprogram。java:576)
这是我用来读取文件的代码

AmazonS3 amazonS3Client = new AmazonS3Client(credentials);

        S3Object object = amazonS3Client.getObject(new GetObjectRequest(S3_PROD_BUCKET, para.get("topology")));
        InputStream awsinputStream = object.getObjectContent();

这是我的pom.xml

<!-- Flink dependencies -->

        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>5.3.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-filesystem_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>commons-httpclient</groupId>
                    <artifactId>commons-httpclient</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.httpcomponents</groupId>
                    <artifactId>httpclient</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.httpcomponents</groupId>
                    <artifactId>httpcore</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-avro</artifactId>
            <version>${flink.format.parquet.version}</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.7</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-parquet_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>2.10.5</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.dataformat</groupId>
            <artifactId>jackson-dataformat-yaml</artifactId>
            <version>2.10.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.esotericsoftware.yamlbeans/yamlbeans -->
        <dependency>
            <groupId>com.esotericsoftware.yamlbeans</groupId>
            <artifactId>yamlbeans</artifactId>
            <version>1.13</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.uber/h3 -->
        <dependency>
            <groupId>com.uber</groupId>
            <artifactId>h3</artifactId>
            <version>3.6.3</version>
        </dependency>

        <dependency>
            <groupId>com.github.davidmoten</groupId>
            <artifactId>geo</artifactId>
            <version>0.7.7</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-elasticsearch6 -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch7_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>tech.allegro.schema.json2avro</groupId>
            <artifactId>converter</artifactId>
            <version>0.2.9</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-statebackend-rocksdb_2.12</artifactId>
            <version>1.10.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro-confluent-registry</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch7_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bundle</artifactId>
            <version>1.11.756</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.10.0</version>
        </dependency>
mum43rcc

mum43rcc1#

仅仅从pom得到冲突的答案是不太可能的。相反,您应该通过调用以下命令来引用maven依赖插件: mvn dependency:tree 这将打印所有依赖项以及这些依赖项的依赖项,这样您就可以找到您要导入的库中哪些库对jackson具有可传递依赖项,并且可以将其标记为已排除。
注意:您在这个依赖关系树中真正寻找的是具有不同版本的jackson依赖关系,因此不需要将它们全部排除。

相关问题