我最近在hdinsight 3.6(以前的3.5)上对hdp集群(2.6.2)进行了改革。从那一刻起,我在旧环境中定义的oozie工作就停止了。有趣的是,它们适用于shell操作,但不适用于hive操作。无论我是从oozie运行它们,还是直接从hdfs运行它们,都没有区别。
以下工作流步骤失败:
<action name="hive_action">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
</configuration>
<script>${CODE_DIR}/hive/sql.hql</script>
<param>WPS_DB=wh</param>
</hive>
我确信参数传递正确。我得到的错误如下:
2018-07-04 14:34:49,721 WARN HiveActionExecutor:523 - SERVER[correct_server_name] USER[correct_user] GROUP[-] TOKEN[] APP[categorization] JOB[correct_job_name] ACTION[correct_action@wh] Launcher exception: HIVETEZHS2USERACCESS
java.lang.NoSuchFieldError: HIVETEZHS2USERACCESS at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.setupSessionAcls(TezSessionState.java:233)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:188)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:336)
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:313)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58)
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
据我所知,似乎该作业所需的字段没有在hive、tez或hive2的任何配置文件中定义。但是在同一类型中,没有hive.tez.hs2.user.access这样的属性来配置hive、tez或hive2配置xmls。我尝试在sql.hql脚本中添加以下行:
set hive.execution.engine=tez;
set hive.tez.hs2.user.access=true;
以及hive-site.xml cond文件的属性。
<property>
<name>hive.tez.hs2.user.access</name>
<value>true</value>
</property>
当然没有效果。我找到的关于此字段的唯一页面是:https://hive.apache.org/javadocs/r2.2.0/api/org/apache/hadoop/hive/conf/hiveconf.confvars.html#hivetezhs2useraccess
但我不知道如何在configuration/hql文件中使用这个变量。谢谢你的帮助。
暂无答案!
目前还没有任何答案,快来回答吧!