我在跑步 Hadoop 2.7
在我的本地机器上,还有 HBase 1.4
以及 Phoenix 4.15
. 我已经写了一个应用程序,它通过phoenix提交map reduce作业,删除hbase中的数据。每个作业都由 ThreadPoolExecutor
看起来是这样的:
public class MRDeleteTask extends Task {
private final Logger LOGGER = LoggerFactory.getLogger(MRDeleteTask.class);
private String query;
public MRDeleteTask(int id, String q) {
this.setId(id);
this.query = q;
}
@Override
public void run() {
LOGGER.info("Running Task: " + getId());
try {
Configuration configuration = HBaseConfiguration.create();
Job job = Job.getInstance(configuration, "phoenix-mr-job-"+getId());
LOGGER.info("mapper input: " + this.query);
PhoenixMapReduceUtil.setInput(job, DeleteMR.PhoenixDBWritable.class, "Table", QUERY);
job.setMapperClass(DeleteMR.DeleteMapper.class);
job.setJarByClass(DeleteMR.class);
job.setNumReduceTasks(0);
job.setOutputFormatClass(NullOutputFormat.class);
job.setOutputKeyClass(ImmutableBytesWritable.class);
job.setOutputValueClass(Writable.class);
TableMapReduceUtil.addDependencyJars(job);
boolean result = job.waitForCompletion(true);
}
catch (Exception e) {
LOGGER.info(e.getMessage());
}
}
}
如果threadpoolexecutor中只有一个线程,那么一切都很好。如果同时提交多个这样的hadoop作业,则什么也不会发生。根据日志,错误如下所示:
4439 [pool-1-thread-2] INFO MRDeleteTask - java.util.concurrent.ExecutionException: java.io.IOException: Unable to rename file: [/tmp/hadoop-user/mapred/local/1595274269610_tmp/tmp_phoenix-4.15.0-HBase-1.4-client.jar] to [/tmp/hadoop-user/mapred/local/1595274269610_tmp/phoenix-4.15.0-HBase-1.4-client.jar]
4439 [pool-1-thread-1] INFO MRDeleteTask - java.util.concurrent.ExecutionException: ExitCodeException exitCode=1: chmod: /private/tmp/hadoop-user/mapred/local/1595274269610_tmp/phoenix-4.15.0-HBase-1.4-client.jar: No such file or directory
任务是使用 ThreadPoolExecutor.submit()
它们的状态正在使用返回的未来进行检查 future.isDone()
.
1条答案
按热度按时间nom7f22z1#
这些作业没有提交给yarn,而是从intellij本地运行。将以下内容添加到作业配置中解决了此问题: