hadoopmapreduce作业只使用了16个reducer中的两个

cwxwcias  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(347)

我是hadoop的新手,我有一个4节点的集群,它有32个map插槽和16个reduce插槽。job使用761个Map和2个还原器处理接近100GB的数据。
我的问题是为什么它只使用2个减速器。请让我如果我错过了任何有关减速器的配置或它是预期的。
我在mapreduce配置中设置了以下属性,但它仍然使用2个reducer。
每个作业的默认减少任务数
mapred.reduce.tasks=8
日志:

  1. 15/12/30 14:58:56 INFO mapred.JobClient: Job complete: job_201512301313_0002
  2. 15/12/30 14:58:56 INFO mapred.JobClient: Counters: 33
  3. 15/12/30 14:58:56 INFO mapred.JobClient: File System Counters
  4. 15/12/30 14:58:56 INFO mapred.JobClient: FILE: Number of bytes read=11711801793
  5. 15/12/30 14:58:56 INFO mapred.JobClient: FILE: Number of bytes written=24324166884
  6. 15/12/30 14:58:56 INFO mapred.JobClient: FILE: Number of read operations=0
  7. 15/12/30 14:58:56 INFO mapred.JobClient: FILE: Number of large read operations=0
  8. 15/12/30 14:58:56 INFO mapred.JobClient: FILE: Number of write operations=0
  9. 15/12/30 14:58:56 INFO mapred.JobClient: HDFS: Number of bytes read=101855418108
  10. 15/12/30 14:58:56 INFO mapred.JobClient: HDFS: Number of bytes written=821001518
  11. 15/12/30 14:58:56 INFO mapred.JobClient: HDFS: Number of read operations=1536
  12. 15/12/30 14:58:56 INFO mapred.JobClient: HDFS: Number of large read operations=0
  13. 15/12/30 14:58:56 INFO mapred.JobClient: HDFS: Number of write operations=2
  14. 15/12/30 14:58:56 INFO mapred.JobClient: Job Counters
  15. 15/12/30 14:58:56 INFO mapred.JobClient: Launched map tasks=761
  16. 15/12/30 14:58:56 INFO mapred.JobClient: Launched reduce tasks=2
  17. 15/12/30 14:58:56 INFO mapred.JobClient: Data-local map tasks=753
  18. 15/12/30 14:58:56 INFO mapred.JobClient: Rack-local map tasks=8
  19. 15/12/30 14:58:56 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=10467348
  20. 15/12/30 14:58:56 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=936182
  21. 15/12/30 14:58:56 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
  22. 15/12/30 14:58:56 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题