自定义输入格式的故障单元测试

ztigrdn8  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(311)

我是hadoop世界的新手。我为我的应用程序开发了一个定制的输入格式和阅读器。我在运行单元测试时遇到问题。我跟随这篇文章开发了我的单元测试:如何对定制的recordreader和inputformat类进行单元测试?。我的单元测试代码与post中的代码完全相同,除了inputformat类:

Configuration conf = new Configuration(false);
    conf.set("fs.default.name", "file:///");
    File testFile = new File("/tmp/testfile");
    Path path = new Path(testFile.getAbsoluteFile().toURI());
    FileSplit split = new FileSplit(path, 0, testFile.length(), null);
    MyFileInputFormat inputFormat =     ReflectionUtils.newInstance(MyFileInputFormat.class, conf);
    TaskAttemptContext context = new TaskAttemptContextImpl(conf, new TaskAttemptID());
    try {
        RecordReader reader = inputFormat.createRecordReader(split, context);
        reader.initialize(split, context);
    } catch (Exception e) {

    }

当我尝试使用“mvn test”运行此测试时,出现以下错误:

java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration$DeprecationDelta
            at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
            at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
            at org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:54)
            at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:42)
            at org.apache.hadoop.mapred.JobConf.(JobConf.java:119)
            at java.lang.Class.forName0(Native Method)
            at java.lang.Class.forName(Class.java:274)
            at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:80)
            at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
            at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

有人能帮我弄明白怎么解决这个问题吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题