查询hiveserver2时,impyla从org.apache.hadoop.hive.ql.exec.mr.mapredtask返回代码1

wh6knrhe  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(424)

我正在使用impyla查询hive的一些结果,但是遇到了以下问题:
来自impyla:

  1. impala.error.OperationalError: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

从hiveserver2:

  1. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  2. Query ID = hduser_20180827031927_fdb148b0-725b-434c-a0f8-98b6843d4348
  3. Total jobs = 1
  4. Launching Job 1 out of 1
  5. Number of reduce tasks not specified. Defaulting to jobconf value of: 1
  6. In order to change the average load for a reducer (in bytes):
  7. set hive.exec.reducers.bytes.per.reducer=<number>
  8. In order to limit the maximum number of reducers:
  9. set hive.exec.reducers.max=<number>
  10. In order to set a constant number of reducers:
  11. set mapreduce.job.reduces=<number>
  12. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

我的源代码是:

  1. from impala.dbapi import connect
  2. import sys
  3. dbName = sys.argv[1:][0]
  4. query = sys.argv[1:][1]
  5. conn = connect(host='192.168.0.10', port=10000, database=dbName , auth_mechanism='NOSASL' , use_ssl=True)
  6. cursor = conn.cursor()
  7. cursor.execute(query, configuration={'hive.exec.reducers.bytes.per.reducer': '100000', 'hive.auto.convert.join.noconditionaltask':'false','mapreduce.job.reduces':'1','hive.auto.convert.join':'false'})
  8. returnData = []
  9. for row in cursor:
  10. returnData.append(row[0])
  11. pprint(returnData)

如您所见,我添加了许多配置,但它不起作用

lstz6jyr

lstz6jyr1#

从你的错误,没有办法知道发生了什么。
我不确定是否在impyla中启用调试日志记录,因此您需要转到yarn ui来查找查询。
如果yarn没有运行,我想您会得到一个更具描述性的错误,比如“无法提交作业”,尽管这个错误可能不是从hiveserver传播的

相关问题