以编程方式获取hadoop中的总Map和减少时间

lskq00tm  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(423)

我正在尝试计算mr代码中所有任务的Map、洗牌和减少的总次数。
我需要帮助检索每个mapreduce作业的信息。
有人可以发布任何代码片段来进行计算吗?

zphenhs4

zphenhs41#

您需要使用如下所示的jobclientapi:但是api有一些特殊之处。试试看,让我知道我会帮你的。

JobClient client = null;

         Configuration configuration = new Configuration();

        configuration.set("mapred.job.tracker", jobTrackerURL);
        client = new JobClient(new JobConf(configuration));

        while (true) {

            List<JobStatus> jobEntries = getTrackerEntries(jobName,
                        client);

                    for (JobStatus jobStatus : jobEntries) {

                        JobID jobId = jobStatus.getJobID();
                        String trackerJobName = client.getJob(jobId)
                                .getJobName();
                        TaskReport[] mapReports = client
                                .getMapTaskReports(jobId);
                        TaskReport[] reduceReports = client
                                .getReduceTaskReports(jobId);
                        client.getJob(jobId).getJobStatus().getStartTime();

                        int jobMapper = mapReports.length;
                        mapNumber = +jobMapper;
                        int jobReducers = reduceReports.length;
                        reduceNumber = +jobReducers;

                    }
                }

相关问题