java命令在hadoop源代码中用于Map时间和减少时间

zf2sa74q  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(342)

我们知道map有两部分“chunk和combine”,reduce有3部分“shuffle、sort和reduce”。在hadoop源代码中,每个部分时间的命令是什么,

xwmevbvl

xwmevbvl1#

jobtracker有一个api,用于在网络环境中提交和跟踪mr作业。查看此项了解更多详细信息。https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapred/jobtracker.html

  1. TaskReport[] maps = jobtracker.getMapTaskReports("job_id");
  2. for (TaskReport rpt : maps) {
  3. System.out.println(rpt.getStartTime());
  4. System.out.println(rpt.getFinishTime());
  5. }
  6. TaskReport[] reduces = jobtracker.getReduceTaskReports("job_id");
  7. for (TaskReport rpt : reduces) {
  8. System.out.println(rpt.getStartTime());
  9. System.out.println(rpt.getFinishTime());
  10. }

或者,如果您使用的是hadoop2.x,那么已经提供了resourcemanagerrestapi。
https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/resourcemanagerrest.html
https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/historyserverrest.html

展开查看全部

相关问题