我们知道map有两部分“chunk和combine”,reduce有3部分“shuffle、sort和reduce”。在hadoop源代码中,每个部分时间的命令是什么,
xwmevbvl1#
jobtracker有一个api,用于在网络环境中提交和跟踪mr作业。查看此项了解更多详细信息。https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapred/jobtracker.html
TaskReport[] maps = jobtracker.getMapTaskReports("job_id"); for (TaskReport rpt : maps) { System.out.println(rpt.getStartTime()); System.out.println(rpt.getFinishTime()); } TaskReport[] reduces = jobtracker.getReduceTaskReports("job_id"); for (TaskReport rpt : reduces) { System.out.println(rpt.getStartTime());System.out.println(rpt.getFinishTime()); }
TaskReport[] maps = jobtracker.getMapTaskReports("job_id");
for (TaskReport rpt : maps) {
System.out.println(rpt.getStartTime());
System.out.println(rpt.getFinishTime());
}
TaskReport[] reduces = jobtracker.getReduceTaskReports("job_id");
for (TaskReport rpt : reduces) {
或者,如果您使用的是hadoop2.x,那么已经提供了resourcemanagerrestapi。https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/resourcemanagerrest.htmlhttps://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/historyserverrest.html
1条答案
按热度按时间xwmevbvl1#
jobtracker有一个api,用于在网络环境中提交和跟踪mr作业。查看此项了解更多详细信息。https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapred/jobtracker.html
或者,如果您使用的是hadoop2.x,那么已经提供了resourcemanagerrestapi。

https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/resourcemanagerrest.html
https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/historyserverrest.html