我的mr程序有一些奇怪的问题，不知道为什么会这样。也许你能给我一个提示有什么问题？
我的Map器函数就是这样的：
Integer Click_ID = 0;public void map(LongWritable key, Text value, Context context)throws IOException , InterruptedException{String line = value.toString();String []lineArr = line.split("\t");String nm_uv_id = lineArr[0];String session_id = lineArr[1];String time_stamp = lineArr[2];String click_counter = lineArr[3];String is_robot = lineArr[4];Click_ID++;String full_line = Click_ID + "\t"+ nm_uv_id +"\t"+ session_id+"\t"+time_stamp+"\t"+click_counter+"\t"+ is_robot;context.write(new Text(session_id), new Text(full_line));}
到目前为止，一切正常-当我设置reducer的数量=0时，我的Map器生成预期的输出。
这是我的减速机的样子。我想做的是，对我的每个键迭代两次。为此，我尝试将iterable的每个值缓存在一个单独的arraylist中：
public void reduce(Text key, Iterable<Text> values, Context context)throws IOException, InterruptedException {List<Text> cache = new ArrayList<Text>();// first iterablefor (Text value : values) {cache.add(value); }//second iterablefor (Text entity : cache) {context.write(key, entity); }}
}
我用于mr的输入如下所示：
nm_uv_id_1 session_id_2 1234567891 1 is_robot_nonm_uv_id_1 session_id_2 1234567892 2 is_robot_nonm_uv_id_1 session_id_2 1234567893 3 is_robot_nonm_uv_id_1 session_id_2 1234567894 3 is_robot_nonm_uv_id_1 session_id_1 1234567895 1 is_robot_nonm_uv_id_1 session_id_1 1234567896 2 is_robot_nonm_uv_id_1 session_id_1 1234567897 3 is_robot_nonm_uv_id_1 session_id_1 1234567898 4 is_robot_nonm_uv_id_1 session_id_1 1234567899 5 is_robot_nonm_uv_id_1 session_id_1 1234567888 6 is_robot_nonm_uv_id_1 session_id_1 1234567890 7 is_robot_nonm_uv_id_1 session_id_1 1234567890 8 is_robot_nonm_uv_id_1 session_id_1 1234567890 9 is_robot_nonm_uv_id_1 session_id_1 1234567890 10 is_robot_nonm_uv_id_1 session_id_3 1234567890 1 is_robot_nonm_uv_id_2 session_id_4 1234587890 1 is_robot_nonm_uv_id_2 session_id_4 1234587890 2 is_robot_nonm_uv_id_2 session_id_4 1234587890 3 is_robot_nonm_uv_id_2 session_id_4 1234587890 4 is_robot_nonm_uv_id_2 session_id_4 1234587890 5 is_robot_nonm_uv_id_2 session_id_4 1234587890 6 is_robot_nonm_uv_id_2 session_id_4 1234587890 7 is_robot_nonm_uv_id_2 session_id_4 1234587890 8 is_robot_nonm_uv_id_2 session_id_4 1234587890 9 is_robot_nonm_uv_id_2 session_id_5 1234587890 1 is_robot_nonm_uv_id_2 session_id_5 1234587890 2 is_robot_nonm_uv_id_2 session_id_5 1234587890 3 is_robot_yesnm_uv_id_2 session_id_5 1234587890 4 is_robot_yesnm_uv_id_2 session_id_5 1234587890 5 is_robot_nonm_uv_id_2 session_id_5 123457890 6 is_robot_no
但是，我的输出文件如下所示：
session_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_1 13 nm_uv_id_1 session_id_1 1234567890 9 is_robot_nosession_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_nosession_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_nosession_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_nosession_id_2 2 nm_uv_id_1 session_id_2 1234567892 2 is_robot_nosession_id_3 15 nm_uv_id_1 session_id_3 1234567890 1 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_4 24 nm_uv_id_2 session_id_4 1234587890 9 is_robot_nosession_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_nosession_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_nosession_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_nosession_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_nosession_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_nosession_id_5 30 nm_uv_id_2 session_id_5 123457890 6 is_robot_no
我不明白为什么减速机总是为一个特定的键写相同的键值对。我尝试了几件事，似乎第一个for循环，在那里我做缓存工作良好。当我编写context.write（key，value）时，我得到了预期的输出。然而第二，当我想使用第二个循环中的缓存时，程序为我写了一些奇怪的东西。
有人能帮忙吗？
1条答案
按热度按时间aoyhnmkz1#
它是重复使用相同的
Text作为优化的缓冲区。因此，您需要手动克隆以缓存它。我会改变你的缓存循环：