为什么context.write不能按预期工作-hadoop map reduce

tyky79it  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(368)

我有一个mr作业,其输出如下所示:

128.187.140.171,11
129.109.6.54,27
129.188.154.200,44
129.193.116.41,5
129.217.186.112,17

在2nd mr job的mapper代码中,我正在这样做;

public void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException {
        // Parse the input string into a nice map
        // System.out.println(value.toString());
        if (value.toString().contains(",")) {
            System.out.println("Inside");
            String[] arr = value.toString().split(",");
            if (arr.length > 1) {
                System.out.println(arr[1]);
                context.write(new Text(arr[1]), new Text(arr[0]));
            }
        }

打印语句的输出正确:

Inside
11
Inside
27

但是context.write始终显示以下输出:

1,slip4068.sirius.com
1,hstar.gsfc.nasa.gov
1,ad11-010.compuserve.com
1,slip85-2.co.us.ibm.net
1,stimpy.actrix.gen.nz
1,j14.ktk1.jaring.my
1,ad08-009.compuserve.com

为什么我的钥匙上总是有1个?这是我的驱动程序代码:

public int run(String[] args) throws Exception {
        // TODO Auto-generated method stub
        Configuration conf = getConf();
        conf.set("mapreduce.output.textoutputformat.separator", ",");

        Job job = new Job(conf, "WL Demo");

        job.setJarByClass(WLDemo.class);

        job.setMapperClass(WLMapper1.class);

    job.setReducerClass(WLReducer1.class);
            job.setInputFormatClass(TextInputFormat.class);

    job.setOutputKeyClass(Text.class);

    job.setOutputValueClass(IntWritable.class);

    Path in = new Path(args[0]);

    Path out = new Path(args[1]);

    Path out2 = new Path(args[2]);

    FileInputFormat.setInputPaths(job, in);

    FileOutputFormat.setOutputPath(job, out);

    boolean succ = job.waitForCompletion(true);
    if (!succ) {
        System.out.println("Job1 failed, exiting");
        return -1;
    }
    Job job2 = new Job(conf, "top-k-pass-2");
    FileInputFormat.setInputPaths(job2, out);
    FileOutputFormat.setOutputPath(job2, out2);
    job2.setJarByClass(WLDemo.class);
    job2.setMapperClass(WLMapper2.class);
    // job2.setReducerClass(Reducer1.class);
    job2.setInputFormatClass(TextInputFormat.class);

    job2.setMapOutputKeyClass(Text.class);
    job2.setMapOutputValueClass(Text.class);
    job2.setNumReduceTasks(1);
    succ = job2.waitForCompletion(true);
    if (!succ) {
        System.out.println("Job2 failed, exiting");
        return -1;
    }
    return 0;
}

如何在第二个mr作业的输出键中获得正确的值?

yx2lnoni

yx2lnoni1#

改变 job2.setNumReduceTasks(1)job2.setNumReduceTasks(0) 。因此,它正在运行一个标识缩减器,将输出键作为1,因此应该将1作为map1 output中某些记录的键。

相关问题