拆分reducer中的文本键(hadoop mapreduce)

3phpmpom  于 2021-06-01  发布在  Hadoop
关注(0)|答案(0)|浏览(214)

我有一个问题:我正在为一个大数据项目做一个map-reduce应用程序。

So, i must split a string in my reduce method. My string's type is: 0006641040-2003.

So, if i split with the method String.split(), i can't take the secondo part of my string after split with index 1. I recived  java.lang.ArrayIndexOutOfBoundsException: 1

但我可以在用索引0分割后得到字符串的一部分。
为什么?
谢谢。

Hi, I have this problem: i'm doing a map-reduce application for a big data project.

So, i must split a string in my reduce method. My string's type is: 0006641040-2003.

So, if i split with the method String.split(), i can't take the secondo part of my string after split with index 1. I recived  java.lang.ArrayIndexOutOfBoundsException: 1

但我可以在用索引0分割后得到字符串的一部分。
为什么?
谢谢。

public class ProductReducerClass extends Reducer<Text, MapWritable, Text, MapWritable> {

MapWritable mw1 = new MapWritable();
String[] out;
public void reduce(Text key, Iterable<MapWritable> values, Context context) 
        throws IOException, InterruptedException {
    out = key.toString().split("-");
    for (MapWritable m : values) {
        context.write(new Text(out[1]),m);

    }

}

}

public class ProductMapperClass extends Mapper<LongWritable, Text, Text, MapWritable> {
private static final IntWritable one = new IntWritable(40);
private static final MapWritable missing = new MapWritable();
MapWritable mw = new MapWritable();
ArrayWritable aw = new ArrayWritable(IntWritable.class);
int [] score;
public void map(LongWritable key, Text value, Context context)
        throws IOException, InterruptedException {
    missing.put(new LongWritable(0), new IntWritable(0));
    String line = value.toString();
    String[] fields = line.split(",");
    try {

        Calendar date=Calendar.getInstance();
        date.setTimeInMillis(Long.parseLong((fields[7]))*1000); 
        long anno=date.get(Calendar.YEAR);
        if(anno>=2003 && anno<=2012) {
            mw.put(new Text("score"), new IntWritable(Integer.parseInt(fields[6])));
            context.write(new Text(fields[1].concat("-").concat(Long.toString(anno))), mw);
        }
    }
    catch (NumberFormatException e) {
        context.write(new Text("perso"),missing);
    }

}

}

public class MainClass {

public static void main (String args[]) throws Exception {

    Job job = new Job(new Configuration(), "MainClass");

    job.setJarByClass(MainClass.class);
    job.setMapperClass(ProductMapperClass.class);
    job.setReducerClass(ProductReducerClass.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(MapWritable.class);

    job.waitForCompletion(true);

}

}

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题