tez shuffle处理程序安装后失败:map操作符初始化失败

2ekbmq32  于 2021-05-27  发布在  Hadoop
关注(0)|答案(1)|浏览(548)

为tez shuffle处理程序设置后,请按照以下说明进行操作:https://tez.apache.org/shuffle-handler.html 查询执行中出现以下错误:
vertex失败,vertexname=map 2,vertexid=vertex\u 1608273679503\u 0002\u 2\u 01,diagnostics=[任务失败,taskid=task\u 1608273679503\u 0002\u 2\u 01\u000000,diagnostics=[任务尝试0失败,info=[错误:运行任务时出错(失败):尝试\u 1608273679503 \u 0002 \u 2 \u 01 \u000000 \u 0:java.lang.runtimeexception:java.lang.runtimeexception:map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:296)在org.apache.hadoop.hive.ql.exec.tez.tezprocessor.run(tezprocessor。java:250)位于org.apache.tez.runtime.logicalioprocessorruntimetask.run(logicalioprocessorruntimetask)。java:374)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:73)在org.apache.tez.runtime.task.taskrunner2callable$1.run(taskrunner2callable。java:61)在javax.security.auth.subject.doas(主题)中的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1729)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:61)在org.apache.tez.runtime.task.taskrunner2callable.callinternal(taskrunner2callable。java:37)在org.apache.tez.common.callablewithndc.call(callablewithndc。java:36)位于org.apache.hadoop.hive.llap.daemon.impl.statsrecordingthreadpool$wrappedcallable.call(statsrecordingthreadpool)。java:110)在java.util.concurrent.futuretask.run(futuretask。java:266)在java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1149)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:624)在java.lang.thread.run(线程。java:748)原因:java.lang.runtimeexception:Map运算符初始化失败org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor。java:363)位于org.apache.hadoop.hive.ql.exec.tez.tezprocessor.initializeandrunprocessor(tezprocessor)。java:266) ... 另有15个原因:org.apache.hadoop.hive.ql.metadata.hiveexception:异步初始化失败。abortrequested=false,位于org.apache.hadoop.hive.ql.exec.operator.completeinitialization(operator。java:461)在org.apache.hadoop.hive.ql.exec.operator.initialize(operator。java:395)在org.apache.hadoop.hive.ql.exec.operator.initialize(operator。java:568)在org.apache.hadoop.hive.ql.exec.operator.initializechildren(operator。java:520)在org.apache.hadoop.hive.ql.exec.operator.initialize(操作符。java:381)在org.apache.hadoop.hive.ql.exec.tez.maprecordprocessor.init(maprecordprocessor。java:335) ... 还有16个原因:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.hive.ql.metadata.hiveexception:java.lang.nullpointerexception位于org.apache.hadoop.hive.ql.exec.tez.llapobjectcache.retrieve(llapobjectcache)。java:118)在org.apache.hadoop.hive.ql.exec.tez.llapobjectcache$1.call(llapobjectcache。java:143) ... 4更多原因:org.apache.hadoop.hive.ql.metadata.hiveexception:java.lang.nullpointerexception位于org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.vectormapjoinfasthashtableloader.load(vectormapjoinfasthashtableloader)。java:113)在org.apache.hadoop.hive.ql.exec.mapjoinoperator.loadhashtableinternal(mapjoinoperator。java:331)在org.apache.hadoop.hive.ql.exec.mapjoinoperator.loadhashtable(mapjoinoperator。java:400)位于org.apache.hadoop.hive.ql.exec.mapjoinoperator.lambda$initializeop$0(mapjoinoperator)。java:207)在org.apache.hadoop.hive.ql.exec.tez.llapobjectcache.retrieve(llapobjectcache。java:116) ... 还有5个原因:org.apache.tez.runtime.api.impl.teztaskcontextimpl.getserviceconsumermetadata(teztaskcontextimpl)上的java.lang.nullpointerexception。java:190)在org.apache.tez.runtime.library.common.shuffle.impl.shufflemanager.(shufflemanager。java:264)在org.apache.tez.runtime.library.input.unorderedkvinput.start(unorderedkvinput。java:146)在org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.vectormapjoinfasthashtableloader.load(vectormapjoinfasthashtableloader)。java:109) ... 9个以上
有人能帮忙吗?

3zwtqj6y

3zwtqj6y1#

llap在内部使用map reduce shuffle处理程序,因为在tez代码中我们有一个硬编码常量tez\u shuffle\u handler\u service\u id=“mapreduce\u shuffle”。https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/dag/api/tezconstants.java
所以,当我们使用tez\u shuffle和llap时,它会给出“nullpointerexception”。

相关问题