在 Reducer
在我的mapreduce程序中,我想读一本 JobCounter
和一个 FileSystemCounter
. 运行命令时 mapred job -status <job id>
,我需要的计数器按其显示名称列出:
...
File System Counters
FILE: Number of bytes read=148874
FILE: Number of bytes written=22010065
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=135823
HDFS: Number of bytes written=44423504133
HDFS: Number of read operations=2185
HDFS: Number of large read operations=0
HDFS: Number of write operations=1316
Job Counters
Launched map tasks=1
Launched reduce tasks=200
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5293
Total time spent by all reduces in occupied slots (ms)=972893
Total time spent by all map tasks (ms)=5293
Total time spent by all reduce tasks (ms)=972893
Total vcore-seconds taken by all map tasks=5293
Total vcore-seconds taken by all reduce tasks=972893
Total megabyte-seconds taken by all map tasks=5420032
Total megabyte-seconds taken by all reduce tasks=996242432
...
如何在运行时从系统内部访问这些计数器 Reducer
的代码?
使用google,我找不到任何关于如何访问这些计数器的有用信息。使用context.getcounter(string groupname,string countername)的直接尝试无法检索 Counter
示例,因此抛出 NullPointerException
一经召唤 getValue()
:
long bytes = context.getCounter(
FileSystemCounter.class.getName(),
FileSystemCounter.BYTES_WRITTEN.name()
).getValue();
long milliseconds = context.getCounter(
JobCounter.class.getName(),
JobCounter.MILLIS_REDUCES.name()
).getValue();
1条答案
按热度按时间snvhrwxg1#
我认为这将有助于打印所有计数器及其值。