multipleoutputs与aws emr s3

9udxz4iz  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(285)

我的工作是使用多个输出来编写输出。这个程序用普通的hadoop集群给出输出。
但是当我使用aws cluster,并为多个输出提供如下所示的s3n路径时,我在指定的s3n路径中没有得到任何输出。有人能帮我吗?

Configuration config3 = new Configuration();
JobConf conf3 = new JobConf(config3, t1debugJob.class);
conf3.setJobName("PJob3.7 scalability test correct 60r");
conf3.setOutputKeyClass(Text.class);
conf3.setOutputValueClass(Text.class);
conf3.setMapOutputKeyClass(StockKey.class);
conf3.setMapOutputValueClass(Text.class);   
conf3.setPartitionerClass(CustomPartitionerStage3.class);
conf3.setOutputValueGroupingComparator(StockKeyGroupingComparator.class);
conf3.setOutputKeyComparatorClass(StockKeySortComparator.class);
conf3.setReducerClass(dt1Amazon.class);
   //conf3.setNumMapTasks(10);
conf3.setNumReduceTasks(30);
conf3.setInputFormat(TextInputFormat.class);
conf3.setOutputFormat(TextOutputFormat.class);
MultipleInputs.addInputPath(conf3, new  Path(other_args.get(2)),TextInputFormat.class, PMap3aPos.class);
MultipleInputs.addInputPath(conf3, new  Path(other_args.get(1)),TextInputFormat.class, PMap3b.class);
MultipleOutputs.addNamedOutput(conf3,"s3n://gs3test/output/MIDPairspos/pairfile", TextOutputFormat.class, LongWritable.class, Text.class);
MultipleOutputs.addNamedOutput(conf3,"s3n://gs3test/output/MIDpos/idfile", TextOutputFormat.class, LongWritable.class, Text.class);                               
FileOutputFormat.setOutputPath(conf3, new Path(other_args.get(3)));                 
JobClient.runJob(conf3);

谢谢!

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题