无法将spark作为hive执行引擎

y4ekin9u  于 2021-05-29  发布在  Spark
关注(0)|答案(1)|浏览(578)

我使用的是hive3.1.2和spark2.4.5(ubuntu18.04上的hadoop3),在运行查询时出现以下错误
0: jdbc:hive2://localhost:10000>从零售订单中选择计数(1);错误:处理语句时出错:失败:执行错误,从org.apache.hadoop.hive.ql.exec.spark.sparktask返回代码30041。未能为spark会话4aa97085-8a23-4ad8-a8c6-b701d41d7110创建spark客户端(状态=42000,代码=30041)
日志:2020-06-19 12:18:28704 info[hiveserver2后台池:thread-72]ql.driver:禁用并发模式,不创建锁管理器2020-06-19 12:18:28,731 error[hiveserver2 background pool:thread-72]operation.operation:运行配置单元查询时出错:org.apache.hive.service.cli.hivesqlexception:处理语句时出错:failed:执行错误,从org.apache.hadoop.hive.ql.exec.spark.sparktask返回代码30041。无法在org.apache.hive.service.cli.operation.operation.tosqlexception(操作)上为spark会话4aa97085-8a23-4ad8-a8c6-b701d41d7110创建spark客户端。java:335)在org.apache.hive.service.cli.operation.sqloperation.runquery(sqloperation。java:226)在org.apache.hive.service.cli.operation.sqloperation.access$700(sqloperation。java:87)在org.apache.hive.service.cli.operation.sqloperation$backgroundwork$1.run(sqloperation。java:316)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1729)在org.apache.hive.service.cli.operation.sqloperation$backgroundwork.run(sqloperation。java:329)在java.util.concurrent.executors$runnableadapter.call(executors。java:511)在java.util.concurrent.futuretask.run(futuretask。java:266)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1149)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:624)在java.lang.thread.run(线程。java:748)原因:org.apache.hadoop.hive.ql.metadata.hiveexception:未能在上为spark会话4aa97085-8a23-4ad8-a8c6-b701d41d7110创建spark客户端org.apache.hadoop.hive.ql.exec.spark.session.sparksessionimpl.gethiveexception(sparksessionimpl。java:221)在org.apache.hadoop.hive.ql.exec.spark.session.sparksessionimpl.open(sparksessionimpl。java:92)在org.apache.hadoop.hive.ql.exec.spark.session.sparksessionmanagerimpl.getsession(sparksessionmanagerimpl。java:115)在org.apache.hadoop.hive.ql.exec.spark.sparkutilities.getsparksession(sparkutilities。java:136)在org.apache.hadoop.hive.ql.exec.spark.sparktask.execute(sparktask。java:115)在org.apache.hadoop.hive.ql.exec.task.executetask(任务。java:205)位于org.apache.hadoop.hive.ql.exec.taskrunner.runsequential(taskrunner)。java:97)在org.apache.hadoop.hive.ql.driver.launchtask(驱动程序。java:2664)在org.apache.hadoop.hive.ql.driver.execute(driver。java:2335)在org.apache.hadoop.hive.ql.driver.runinternal(driver。java:2011)在org.apache.hadoop.hive.ql.driver.run(driver。java:1709)在org.apache.hadoop.hive.ql.driver.run(driver。java:1703)在org.apache.hadoop.hive.ql.reexec.reexecdriver.run(reexecdriver。java:157)在org.apache.hive.service.cli.operation.sqloperation.runquery(sqloperation。java:224) ... 11其他原因:java.lang.noclassdeffounderror:org/apache/spark/sparkconf,位于org.apache.hadoop.hive.ql.exec.spark.hivesparkclientfactory.generatesparkconf(hivesparkclientfactory)。java:263)在org.apache.hadoop.hive.ql.exec.spark.remotehivesparkclient.(remotehivesparkclient。java:98)在org.apache.hadoop.hive.ql.exec.spark.hivesparkclientfactory.createhivesparkclient(hivesparkclientfactory)。java:76)在org.apache.hadoop.hive.ql.exec.spark.session.sparksessionimpl.open(sparksessionimpl。java:87)

ryevplcw

ryevplcw1#

根据这个错误: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf 配置单元找不到spark类。
如果您使用的是cloudera,可以使用以下命令添加它们:

cd /opt/cloudera/parcels/CDH/lib;
cp  spark/jars/*scala-lib* hive/lib/;
cp  spark/jars/*spark-core* hive/lib/;
cp  spark/jars/*spark-netw*  hive/lib/;
cp  spark/jars/chill-java*.jar  hive/lib/;
cp  spark/jars/jackson-module-*  hive/lib/;
cp  spark/jars/jersey-server-*.jar  hive/lib/;
cp  spark/jars/json4s-ast_*.jar  hive/lib/;
cp  spark/jars/kryo-shaded-*.jar  hive/lib/;
cp  spark/jars/minlog-*.jar  hive/lib/;
cp  spark/jars/scala-xml_*.jar  hive/lib/;
cp  spark/jars/spark-launcher_*.jar  hive/lib/;
cp  spark/jars/spark-network-shuffle_*.jar  hive/lib/;
cp  spark/jars/spark-launcher_*.jar  hive/lib/;
cp  spark/jars/spark-unsafe_*.jar  hive/lib/;
cp  spark/jars/xbean-asm7-shaded-*.jar  hive/lib/;

您可以在此处找到更多信息:
https://cwiki.apache.org/confluence/display/hive/hive+on+spark%3a+getting+started
在配置配置单元部分。

相关问题