如何在mrunit中使用avro参数测试reducer?

waxmsbnn  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(353)

我在试着测试减速器

Reducer<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>

为此,我在测试中使用下一个代码:

CustomReducer reducer = new CustomReducer();
    reduceDriver = new ReduceDriver<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>();

    Job job = new Job();

    reduceDriver = ReduceDriver.newReduceDriver(reducer).withConfiguration(job.getConfiguration());
    reduceDriver.setReducer(reducer);

    job.setInputFormatClass(AvroKeyInputFormat.class);

    job.setMapOutputKeyClass(CompositeKeyWritable.class);
    AvroJob.setMapOutputValueSchema(job, union);

    AvroJob.setOutputKeySchema(job, Class2.SCHEMA$);
    job.setOutputFormatClass(AvroKeyOutputFormat.class);
    job.setOutputValueClass(NullWritable.class);

结果我得到了下一个错误:

java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:86)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:110)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:675)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:167)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:181)
at com.wellcare.caregap.hadoop.reducers.CareGapReducerTest.testReducer(CareGapReducerTest.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

作业配置有什么问题?如果有人能帮我解决这些问题,我会非常感激的。

ao218c7q

ao218c7q1#

调试方法readfields并在class1中编写。当您使用添加输入时,它似乎试图在到达其中一个字段的datainput或dataoutput流的末尾之后读取或写入字段 reduceDriver.withInput(yourkey, yourvalue);

相关问题