我试图使用mapreduce来查找由标签分隔的tab分隔的输入的总和。数据是这样的

1     5.0    4.0   6.0
2     2.0    1.0   3.0
1     3.0    4.0   8.0

第一列是类标签，所以我希望输出按类标签分类。对于这个示例，输出将是

label 1: 30.0
label 2: 6.0

这是我尝试过的代码，但是我得到了错误的输出
将显示意外的类标签。

public class Total {
 public static class Map extends Mapper<LongWritable, Text, Text, DoubleWritable> {
    private final static DoubleWritable one = new DoubleWritable();
    private Text word = new Text();
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        word.set(tokenizer.nextToken());
        while (tokenizer.hasMoreTokens()) {
            one.set(Double.valueOf(tokenizer.nextToken()));
            context.write(word, one);                                           
        }
    }
 }

public static class Reduce extends Reducer<Text, DoubleWritable, Text, DoubleWritable> {
    private Text Msg = new Text();
    public void reduce(Text key, Iterable<DoubleWritable> values, Context context) 
      throws IOException, InterruptedException {
       firstMsg.set("label " + key+": Total");
       Double sum = 0.0;
         for (DoubleWritable val : values) {
            sum += val.get();
        }
        context.write(Msg, new DoubleWritable(sum));
    }
 }
//void method implementation also exists
}

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
word.set("label " + tokenizer.nextToken());
StringBuilder remainder = new StringBuilder();
while (tokenizer.hasMoreTokens()) {
remainder.append(tokenizer.nextToken()).append(",");
}
String output = remainder.setLength(remainder.getLength() - 1).toString()
context.write(word, new Text(output));
}

1条答案

按热度按时间

oug3syen1#

您的目标是将所有相同的键放入它们自己的减速机中，这样您就可以对这些数字求和。
所以，拿着这个

从本质上说就是创造这个

1     [(5 .0    4.0   6.0), (3.0    4.0   8.0)]
2     [(2.0    1.0   3.0)]

因此，您的Map应该只输出键1和键2，每个键后面都有剩余的值，每个键不一定有很多值。
为此，您可以使用 Mapper<LongWritable, Text, Text, Text> . (将输出数据类型更改为 Text )

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
    String line = value.toString();
    StringTokenizer tokenizer = new StringTokenizer(line);
    word.set("label " + tokenizer.nextToken());
    StringBuilder remainder = new StringBuilder();
    while (tokenizer.hasMoreTokens()) {
        remainder.append(tokenizer.nextToken()).append(",");                                        
    }
    String output = remainder.setLength(remainder.getLength() - 1).toString()
    context.write(word, new Text(output));  
}

然后，在减速器中，使其 Reducer<Text, Text, Text, DoubleWritable> （读入 (Text,Text) 你现在有一个 Iterable<Text> values 它是一个由逗号分隔的字符串组成的iterable，可以将其解析为double，并取累积和。
你真的不需要 firstMsg.set 在减速器中的零件-可以在Map器中完成。

展开查看全部

赞(0）回复(0）举报 2021-05-29

mapreduce计算tab分隔的输入值之和

1条答案

相关问题

热门标签

最新问答