mapreduce mrunit错误

ygya80vv  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(340)

我是hadoop的新手。昨天我按照这本书使用junit作为气象数据的apper单元测试。但也有一些问题。
这是我的pom文件:

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.9.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.9.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.mrunit</groupId>
        <artifactId>mrunit</artifactId>
        <version>1.1.0</version>
        <classifier>hadoop2</classifier>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-minicluster</artifactId>
        <version>2.9.0</version>
        <scope>test</scope>
    </dependency>
  </dependencies>

</project>

这就是问题:
java.lang.CompatibleClassChangeError:找到类org.apache.hadoop.mapreduce.taskinputoutputcontext,但需要接口
期待并感谢您的回答

ffscu2ro

ffscu2ro1#

我刚刚发现在hadoop 2.x中,没有hadoop核心,所以正确的pom是:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.9.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.9.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.9.0</version>
</dependency>
<dependency>
    <groupId>org.apache.mrunit</groupId>
    <artifactId>mrunit</artifactId>
    <version>1.1.0</version>
    <classifier>hadoop2</classifier>
    <scope>test</scope>
</dependency>
ifsvaxew

ifsvaxew2#

尝试添加不同的jar版本。
我从 mrunit-1.1.0.jarmrunit-0.9.0-incubating-hadoop2.jar 旧的jar:- public interface Mapper<K1, V1, K2, V2> extends JobConfigurable, Closeable { } 新建jar:-

public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {}

我刚刚将jar从hadoop1改为hadoop2,然后我的所有单元测试用例都成功运行了。https://issues.apache.org/jira/browse/mrunit-156

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.org</groupId>
    <artifactId>com.org</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <dependencies>
        <!-- JDK Tools -->
        <dependency>
            <groupId>jdk.tools</groupId>
            <artifactId>jdk.tools</artifactId>
            <scope>system</scope>
            <version>1.8.0_73</version>
            <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
        </dependency>
        <!-- Hadoop Core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>
        <!-- Hadoop Mapreduce Client Core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>2.7.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.mrunit</groupId>
            <artifactId>mrunit</artifactId>
            <version>0.9.0-incubating</version>
            <classifier>hadoop1</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.mrunit</groupId>
            <artifactId>mrunit</artifactId>
            <version>0.9.0-incubating</version>
            <classifier>hadoop2</classifier>
        </dependency>

    </dependencies>

</project>

相关问题