无法通过tez从hive执行mapreduce作业

bq3bfh9z  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(267)

我正面临着一个关于泰兹的问题。
我可以选择一个表存在于配置单元上而没有任何问题 SELECT * FROM Transactions; 但是当尝试在这个表上使用聚合函数或计数(*)时,例如: SELECT COUNT(*) FROM Transactions; 我正对着下面的login hive.log文件
2017-08-13t10:04:27892 info[4a5b6a0c-9edb-45ea-8d49-b2f4b0d2b636 main]conf.hiveconf:使用为日志id传入的默认值:4a5b6a0c-9edb-45ea-8d49-b2f4b0d2b636 2017-08-13t10:04:27,910信息[4a5b6a0c-9edb-45ea-8d49-b2f4b0d2b636 main]session.sessionstate:关闭tez会话时出错java.lang.runtimeexception:java.util.concurrent.executionexception:org.apache.tez.dag.api.sessionnotrunning:tez会话已关闭。应用程序应用程序\u 149057873641 \u 0017失败2次,原因是appattempt的am容器\u 149057873641 \u 0017 \u000002退出,exitcode:-1000失败。诊断:java.io.filenotfoundexception:文件/tmp/hadoop/nm local dir/filecache不存在,无法获得更详细的输出,检查应用程序跟踪页面:http://hadoop-master:8090/cluster/app/application\u 1498057873641\u 0017然后单击指向每次尝试日志的链接。应用程序失败。位于org.apache.hadoop.hive.ql.exec.tez.tezsessionstate.isopen(tezsessionstate)。java:173)~[hive-exec-2.1.1.执行。jar:2.1.1]在org.apache.hadoop.hive.ql.exec.tez.tezsessionstate.tostring(tezsessionstate。java:135)~[hive-exec-2.1.1.执行。jar:2.1.1]在java.lang.string.valueof(string。java:2994)约[?:1.8.0μ131]java.lang.stringbuilder.append(字符串生成器。java:131)~[?:1.8.0_]位于org.apache.hadoop.hive.ql.exec.tez.tezsessionpoolmanager.closeifnotdefault(tezsessionpoolmanager)。java:346)~[hive-exec-2.1.1.执行。jar:2.1.1]在org.apache.hadoop.hive.ql.session.sessionstate.close(sessionstate。java:1524)[配置单元执行-2.1.1。jar:2.1.1]在org.apache.hadoop.hive.cli.clisessionstate.close(clisessionstate。java:66)[配置单元cli-2.1.1。jar:2.1.1]在org.apache.hadoop.hive.cli.clidriver.processcmd(clidriver。java:133)[配置单元cli-2.1.1。jar:2.1.1]在org.apache.hadoop.hive.cli.clidriver.processline(clidriver。java:399)[配置单元cli-2.1.1。jar:2.1.1]在org.apache.hadoop.hive.cli.clidriver.executedriver(clidriver。java:776)[配置单元cli-2.1.1。jar:2.1.1]在org.apache.hadoop.hive.cli.clidriver.run(clidriver。java:714)[配置单元cli-2.1.1。jar:2.1.1]位于org.apache.hadoop.hive.cli.clidriver.main(clidriver。java:641)[配置单元cli-2.1.1。jar:2.1.1]在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)~[?:1.8.0_]在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(method。java:498)~[?:1.8.0_]在org.apache.hadoop.util.runjar.run(runjar。java:234) [hadoop-common-2.8.0.jar:?]位于org.apache.hadoop.util.runjar.main(runjar。java:148)[hadoop-common-2.8.0.jar:?]原因:java.util.concurrent.executionexception:org.apache.tez.dag.api.sessionnotrunning:tezsession已经关闭。应用程序应用程序\u 149057873641 \u 0017失败2次,原因是appattempt的am容器\u 149057873641 \u 0017 \u000002退出,exitcode:-1000失败。诊断:java.io.filenotfoundexception:文件/tmp/hadoop/nm local dir/filecache不存在,无法获得更详细的输出,检查应用程序跟踪页面:http://hadoop-master:8090/cluster/app/application\u 1498057873641\u 0017然后单击指向每次尝试日志的链接。应用程序失败。在java.util.concurrent.futuretask.report(futuretask。java:122)~[?:1.8.0_]位于java.util.concurrent.futuretask.get(futuretask。java:206)~[?:1.8.0_]位于org.apache.hadoop.hive.ql.exec.tez.tezsessionstate.isopen(tezsessionstate)。java:168)~[hive-exec-2.1.1.执行。jar:2.1.1] ... 还有17个原因:org.apache.tez.dag.api.sessionnotrunning:tezsession已经关闭。应用程序应用程序\u 149057873641 \u 0017失败2次,原因是appattempt的am容器\u 149057873641 \u 0017 \u000002退出,exitcode:-1000失败。诊断:java.io.filenotfoundexception:文件/tmp/hadoop/nm local dir/filecache不存在,无法获得更详细的输出,检查应用程序跟踪页面:http://hadoop-master:8090/cluster/app/application\u 1498057873641\u 0017然后单击指向每次尝试日志的链接。应用程序失败。在org.apache.tez.client.tezclient.waitillready(tezclient。java:914)~[tez-api-0.8.4。jar:0.8.4]在org.apache.tez.client.tezclient.waitillready(tezclient。java:883)~[tez-api-0.8.4。jar:0.8.4]位于org.apache.hadoop.hive.ql.exec.tez.tezsessionstate.startsessionandcontainers(tezsessionstate)。java:416)~[hive-exec-2.1.1.执行。jar:2.1.1]在org.apache.hadoop.hive.ql.exec.tez.tezsessionstate.access$000(tezsessionstate)。java:97)~[hive-exec-2.1.1.执行。jar:2.1.1]在org.apache.hadoop.hive.ql.exec.tez.tezsessionstate$1.call(tezsessionstate。java:333)~[hive-exec-2.1.1.执行。jar:2.1.1]在org.apache.hadoop.hive.ql.exec.tez.tezsessionstate$1.call(tezsessionstate。java:329)~[hive-exec-2.1.1.执行。jar:2.1.1] 在java.util.concurrent.futuretask.run(futuretask。java:266)在java.lang.thread.run(thread。java:748) ~[?:1.8.0_131]
我通过在所有集群节点上创建丢失的目录“/tmp/hadoop-hadoop/nm-local-dir/filecache”解决了这个问题。
然后我在hive.log上遇到了一个错误 SELECT COUNT(*) FROM Transactions; ,如下所示:
2017-08-13t10:06:35567 info[main]optimizer.columnprunerprocfactory:rs 3 oldcolexprmap:{value.\u col0=column[\u col0]}2017-08-13t10:06:35568 info[main]optimizer.columnprunerprocfactory:rs 3 newcolexprmap:{value.\u col0=column[\u col0]}2017-08-13t10:06:35,604 info[213ea036-8245-4042-a5a1-ccd686ea2465 main]configuration.deprecation:不推荐使用mapred.input.dir.recursive。相反,请使用mapreduce.input.fileinputformat.input.dir.recursive 2017-08-13t10:06:35658 info[main]annotation.statsrulesprocfactory:stats gby[2]:行数等于0。0行将设置为1 2017-08-13t10:06:35679 info[main]optimizer.setreducerparallelism:确定为1 2017-08-13t10:06:35的还原数,680 info[main]parse.tezcompiler:无周期:true 2017-08-13t10:06:35689 info[213ea036-8245-4042-a5a1-ccd686ea2465 main]configuration.deprecation:mapred.job.name已弃用。相反,请使用mapreduce.job.name 2017-08-13t10:06:35741 info[main]parse.calciteplanner:completed plan generation 2017-08-13t10:06:35742 info[main]ql.driver:semantic analysis completed 2017-08-13t10:06:35742 info[main]ql.driver:returning hive schema:schema(fieldschemas:[fieldschema])(name:c0, type:bigint, comment:null)],properties:null)2017-08-13t10:06:35744 info[main]exec.listsinkoperator:初始化操作符列表\u sink[7]2017-08-13t10:06:35745 info[main]ql.driver:完成编译命令(queryid=hadoop \u 20170813100633 \u 31ca0425-6aca-434c-8039-48bc0e761095);所用时间:2.131秒2017-08-13t10:06:35768 info[main]ql.driver:执行命令(queryid=hadoop\u 20170813100633\u 31ca0425-6aca-434c-8039-48bc0e761095):从事务2017-08-13t10:06:35768 info[main]ql.driver:查询id=hadoop\u 20170813100633\u 31ca0425-6aca-434c-8039-48bc0e761095 2017-08-13t10:06:35,768 info[main]ql.driver:total jobs=1 2017-08-13t10:06:35784 info[main]ql.driver:laung job 1 of 1 2017-08-13t10:06:35784 info[main]ql.driver:starting task[stage-1:mapred]in serial mode 2017-08-13t10:06:35789 info[main]tez.tezsessionpoolmanager:当前用户:hadoop,会话用户:hadoop 2017-08-13t10:06:35,789 info[main]tez.tezsessionpoolmanager:当前队列名称为空传入队列名称为空2017-08-13t10:06:35838 info[213ea036-8245-4042-a5a1-ccd686ea2465 main]configuration.deprecation:mapred.committer.job.setup.cleanup.needed已弃用。相反,请使用mapreduce.job.committer.setup.cleanup.needed 2017-08-13t10:06:35840 info[main]ql.context:新的scratch dir是hdfs://hadoop-master:8020/tmp/hive/hadoop/213ea036-8245-4042-a5a1-ccd686ea2465/hive\ 2017-08-13\ 10-06-33\ 614\ 5648783469307420794-1 2017-08-13t10:06:35845 info[main]exec.task:会话已打开2017-08-13t10:06:35,847 info[main]tez.dagutils:本地化资源,因为它不存在:file:/opt/apache-tez-0.8.4-bin to dest:hdfs://hadoop-master:8020/tmp/hive/hadoop/\u tez\u session\u dir/213ea036-8245-4042-a5a1-ccd686ea2465/apache-tez-0.8.4-bin 2017-08-13t10:06:35,850 info[main]tez.dagutils:看起来另一个线程或进程正在写入同一个文件2017-08-13t10:06:35851 info[main]tez.dagutils:正在等待文件hdfs://hadoop-master:8020/tmp/hive/hadoop/\u tez\u session\u dir/213ea036-8245-4042-a5a1-ccd686ea2465/apache-tez-0.8.4-bin(5次尝试,间隔5000ms)2017-08-13t10:07:00,860错误[main]tez.dagutils:找不到正在上载的jar 2017-08-13t10:07:00861错误[main]exec.task:无法执行tez图。java.io.ioexception:以前的编写器可能无法编写hdfs://hadoop-master:8020/tmp/hive/hadoop/\u tez\u session\u dir/213ea036-8245-4042-a5a1-ccd686ea2465/apache-tez-0.8.4-bin。失败是因为我不太可能写太多。在org.apache.hadoop.hive.ql.exec.tez.dagutils.localizeresource(dagutils。java:1022)在org.apache.hadoop.hive.ql.exec.tez.dagutils.addtempresources(dagutils。java:902)在org.apache.hadoop.hive.ql.exec.tez.dagutils.localizetempfilesfromconf(dagutils。java:845)在org.apache.hadoop.hive.ql.exec.tez.tezsessionstate.refreshlocalresourcesfromconf(tezsessionstate)。java:466)在org.apache.hadoop.hive.ql.exec.tez.teztask.updatesession(teztask)上。java:294)在org.apache.hadoop.hive.ql.exec.tez.teztask.execute(teztask。java:155)在org.apache.hadoop.hive.ql.exec.task.executetask(任务。java:197)在org.apache.hadoop.hive.ql.exec.taskrunner.runsequential(taskrunner。java:100)在org.apache.hadoop.hive.ql.driver.launchtask(driver。java:2073)在org.apache.hadoop.hive.ql.driver.execute(driver。java:1744)在org.apache.hadoop.hive.ql.driver.runinternal(driver。java:1453)在org.apache.hadoop.hive.ql.driver.run(driver。java:1171)在org.apache.hadoop.hive.ql.driver.run(驱动程序。java:1161)在org.apache.hadoop.hive.cli.clidriver.processlocalcmd(clidriver。java:232)在org.apache.hadoop.hive.cli.clidriver.processcmd(clidriver。java:183)在org.apache.hadoop.h

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题