hadoop combinefileinputformatinput

gab6jxml  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(224)

在inputformat中使用combinefileinputformatinput时,我得到了错误示例化异常。

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://master:54310");

Job job = new Job(conf, "WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setInputFormatClass(CombineFileInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path("/user/hduser/CombineFileInputFormatInput"));
FileOutputFormat.setOutputPath(job, new Path("/user/hduser/CombineFileInputFormatOutputNew2"));

System.exit(job.waitForCompletion(true) ? 0 : 1);
kyvafyod

kyvafyod1#

CombineFileInputFormat 是抽象类,必须对其进行扩展并实现createrecordreader方法:

/**
 * This is not implemented yet. 
 */
public abstract RecordReader<K, V> createRecordReader(InputSplit split,
  TaskAttemptContext context) throws IOException;

然后更换 job.setInputFormatClass(..) 使用扩展类的名称进行调用。

相关问题