这个问题在这里已经有了答案:
hadoop mapreduce控制台输出说明(1个答案)
两年前关门了。
在我的 MapReduce
工作结束后,我得到了很多 Counter
信息:
File System Counters
FILE: Number of bytes read=4386096368
FILE: Number of bytes written=8805370803
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=54583718086
HDFS: Number of bytes written=4382090874
HDFS: Number of read operations=1479
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=369
Launched reduce tasks=1
Data-local map tasks=369
Total time spent by all maps in occupied slots (ms)=34288552
Total time spent by all reduces in occupied slots (ms)=232084
Total time spent by all map tasks (ms)=8572138
Total time spent by all reduce tasks (ms)=58021
Total vcore-seconds taken by all map tasks=8572138
Total vcore-seconds taken by all reduce tasks=58021
Total megabyte-seconds taken by all map tasks=35111477248
Total megabyte-seconds taken by all reduce tasks=237654016
Map-Reduce Framework
Map input records=14753874
Map output records=666776
Map output bytes=4383426830
Map output materialized bytes=4386098552
Input split bytes=47970
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=4386098552
Reduce input records=666776
Reduce output records=666776
Spilled Records=1333552
Shuffled Maps =369
Failed Shuffles=0
Merged Map outputs=369
GC time elapsed (ms)=1121584
CPU time spent (ms)=23707900
Physical memory (bytes) snapshot=152915259392
Virtual memory (bytes) snapshot=2370755190784
Total committed heap usage (bytes)=126644912128
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=49449743227
File Output Format Counters
Bytes Written=4382090874
我在哪里可以找到这些字段含义的解释?其中有些是相当明显的( Number of bytes read
),但其他人则更加模棱两可( Total time spent by all maps in occupied slots
与 Total time spent by all map tasks
).
我找到了所有默认计数器的列表,但似乎找不到它们的解释或描述。
我相当惊讶,我似乎不能很容易地找到有关这个输出的文档。有人能提供一个链接或解释吗?
1条答案
按热度按时间50pmv0ei1#
hadoop的第8章:权威指南(华盛顿州立大学链接中的完整pdf)提供了计数器的详细信息
MapReduce
. 从第225页开始,见表8-1。safari books online上提供了此资源的最新版本(第4版)(您需要先登录)。