为什么合并器输入记录的数量多于Map的输出数量？

kq4fsx7k 于 2021-06-03 发布在 Hadoop

关注(0)|答案(1)|浏览(296)

组合器在Map器之后和缩减器之前运行，它将接收给定节点上Map器示例发出的所有数据作为输入。然后它向减速器输出。所以合路器输入的记录应该小于Map输出。

12/08/29 13:38:49 INFO mapred.JobClient:   Map-Reduce Framework

12/08/29 13:38:49 INFO mapred.JobClient:     Reduce input groups=8649

12/08/29 13:38:49 INFO mapred.JobClient:     Map output materialized bytes=306210

12/08/29 13:38:49 INFO mapred.JobClient:     Combine output records=859412

12/08/29 13:38:49 INFO mapred.JobClient:     Map input records=457272

12/08/29 13:38:49 INFO mapred.JobClient:     Reduce shuffle bytes=0

12/08/29 13:38:49 INFO mapred.JobClient:     Reduce output records=8649

12/08/29 13:38:49 INFO mapred.JobClient:     Spilled Records=1632334

12/08/29 13:38:49 INFO mapred.JobClient:     Map output bytes=331837344

12/08/29 13:38:49 INFO mapred.JobClient:   **Combine input records=26154506**

12/08/29 13:38:49 INFO mapred.JobClient:   **Map output records=25312392**

12/08/29 13:38:49 INFO mapred.JobClient:     SPLIT_RAW_BYTES=218

12/08/29 13:38:49 INFO mapred.JobClient:     Reduce input records=17298

hadoop mapreduce combiners

来源：https://stackoverflow.com/questions/12171965/why-is-the-number-of-combiner-input-records-more-than-the-number-of-outputs-of-m