我使用的是hive3.1.2和spark2.4.5(ubuntu18.04上的hadoop3),在运行查询时出现以下错误
0: jdbc:hive2://localhost:10000>从零售订单中选择计数(1);错误:处理语句时出错:失败:执行错误,从org.apache.hadoop.hive.ql.exec.spark.sparktask返回代码30041。未能为spark会话4aa97085-8a23-4ad8-a8c6-b701d41d7110创建spark客户端(状态=42000,代码=30041)
日志:2020-06-19 12:18:28704 info[hiveserver2后台池:thread-72]ql.driver:禁用并发模式,不创建锁管理器2020-06-19 12:18:28,731 error[hiveserver2 background pool:thread-72]operation.operation:运行配置单元查询时出错:org.apache.hive.service.cli.hivesqlexception:处理语句时出错:failed:执行错误,从org.apache.hadoop.hive.ql.exec.spark.sparktask返回代码30041。无法在org.apache.hive.service.cli.operation.operation.tosqlexception(操作)上为spark会话4aa97085-8a23-4ad8-a8c6-b701d41d7110创建spark客户端。java:335)在org.apache.hive.service.cli.operation.sqloperation.runquery(sqloperation。java:226)在org.apache.hive.service.cli.operation.sqloperation.access$700(sqloperation。java:87)在org.apache.hive.service.cli.operation.sqloperation$backgroundwork$1.run(sqloperation。java:316)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1729)在org.apache.hive.service.cli.operation.sqloperation$backgroundwork.run(sqloperation。java:329)在java.util.concurrent.executors$runnableadapter.call(executors。java:511)在java.util.concurrent.futuretask.run(futuretask。java:266)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1149)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:624)在java.lang.thread.run(线程。java:748)原因:org.apache.hadoop.hive.ql.metadata.hiveexception:未能在上为spark会话4aa97085-8a23-4ad8-a8c6-b701d41d7110创建spark客户端org.apache.hadoop.hive.ql.exec.spark.session.sparksessionimpl.gethiveexception(sparksessionimpl。java:221)在org.apache.hadoop.hive.ql.exec.spark.session.sparksessionimpl.open(sparksessionimpl。java:92)在org.apache.hadoop.hive.ql.exec.spark.session.sparksessionmanagerimpl.getsession(sparksessionmanagerimpl。java:115)在org.apache.hadoop.hive.ql.exec.spark.sparkutilities.getsparksession(sparkutilities。java:136)在org.apache.hadoop.hive.ql.exec.spark.sparktask.execute(sparktask。java:115)在org.apache.hadoop.hive.ql.exec.task.executetask(任务。java:205)位于org.apache.hadoop.hive.ql.exec.taskrunner.runsequential(taskrunner)。java:97)在org.apache.hadoop.hive.ql.driver.launchtask(驱动程序。java:2664)在org.apache.hadoop.hive.ql.driver.execute(driver。java:2335)在org.apache.hadoop.hive.ql.driver.runinternal(driver。java:2011)在org.apache.hadoop.hive.ql.driver.run(driver。java:1709)在org.apache.hadoop.hive.ql.driver.run(driver。java:1703)在org.apache.hadoop.hive.ql.reexec.reexecdriver.run(reexecdriver。java:157)在org.apache.hive.service.cli.operation.sqloperation.runquery(sqloperation。java:224) ... 11其他原因:java.lang.noclassdeffounderror:org/apache/spark/sparkconf,位于org.apache.hadoop.hive.ql.exec.spark.hivesparkclientfactory.generatesparkconf(hivesparkclientfactory)。java:263)在org.apache.hadoop.hive.ql.exec.spark.remotehivesparkclient.(remotehivesparkclient。java:98)在org.apache.hadoop.hive.ql.exec.spark.hivesparkclientfactory.createhivesparkclient(hivesparkclientfactory)。java:76)在org.apache.hadoop.hive.ql.exec.spark.session.sparksessionimpl.open(sparksessionimpl。java:87)
1条答案
按热度按时间ryevplcw1#
根据这个错误:
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
配置单元找不到spark类。如果您使用的是cloudera,可以使用以下命令添加它们:
您可以在此处找到更多信息:
https://cwiki.apache.org/confluence/display/hive/hive+on+spark%3a+getting+started
在配置配置单元部分。