java—仅从减速机中获取一个聚合值

zbwhf8kr  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(405)

我试图打印一个混淆矩阵的j48分类器使用weka。我得到的结果是每个Map器的矩阵数。正在运行的Map程序数设置为2。
这个类是weka分类器输出的一个缩减器,它从Map器中得到一堆交叉验证的数据块,它的工作是将数据聚合到一个解决方案中。

public void reduce(Text key, Iterable<AggregateableEvaluation> values, Context context) throws IOException, InterruptedException {      
        int sum = 0;                    
        // loop through each of the values and "aggregate"
        // which basically means to consolidate the values
        for (AggregateableEvaluation val : values) {
            System.out.println("IN THE REDUCER!");

            // The first time through, give aggEval a value
            if (sum == 0) {
                try {
                    aggEval = val;
                }
                catch (Exception e) {
                    e.printStackTrace();
                }
            }
            else {
                // combine the values
                aggEval.aggregate(val);
            }

            try {
                // This is what is taken from the mapper to be aggregated
                //System.out.println("This is the map result");
                //System.out.println(aggEval.toMatrixString());
            }
            catch (Exception e) {
                e.printStackTrace();
            }                       

            sum += 1;
        }           
        try {
            System.out.println("This is reduce matrix");
            System.out.println(aggEval.toMatrixString());
        }
        catch (Exception e) {
            e.printStackTrace();
        }
yquaqz18

yquaqz181#

我对weka一无所知,但是使用“normal”mapreduce,reduce函数的形式应该是:https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/reducer.html

public class IntSumReducer<Key> extends Reducer<Key,IntWritable,
                                                 Key,IntWritable> {
   private IntWritable result = new IntWritable();

   public void reduce(Key key, Iterable<IntWritable> values,
                      Context context) throws IOException, InterruptedException {
     int sum = 0;
     for (IntWritable val : values) {
       sum += val.get();
     }
     result.set(sum);
     context.write(key, result);
   }
 }

所以基本上,reducer方法对每个键调用一次。您将得到Map到该特定键的所有值,您应该将这些值聚合在一起,然后在完成后执行 context.write(key, aggEval) 从reduce方法发出结果

相关问题