在kerberized集群上通过ooziejava操作执行配置单元查询

p8h8hvxi  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(192)

我正在尝试通过安全集群上的oozie java操作执行配置单元查询，但是我得到以下异常：

>>> Invoking Main class now >>>

Launch time = 1458542514082
Job launch time = 1458542514082 mapreduce.job.tags = oozie-2e8d7ed9fc7551a353667830e09bef2b
Main class        : com.citiustech.main.Test
Arguments         :

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception,  
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException:  
Could not open client transport with JDBC Uri: jdbc:hive2://10.60.2.26:10000/default;principal=hive/example-qa1- 
dn2@EXAMPLE.com;auth=NOSASL: GSS initiate failed
org.apache.oozie.action.hadoop.JavaMainException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC  
Connection; nested exception is java.sql.SQLException: Could not open client transport with JDBC Uri:  
jdbc:hive2://10.60.2.26:10000/default;principal=hive/example-qa1-dn2@EXAMPLE.com;auth=NOSASL: GSS initiate failed
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
    at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is  
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://10.60.2.26:10000/default;principal=hive/example- 
qa1-dn2@EXAMPLE.com;auth=NOSASL: GSS initiate failed
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:391)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:471)
    at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:481)
    at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:491)
    at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
    at com.citiustech.main.Test.main(Test.java:26)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:55)
    ... 15 more
Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri:  
jdbc:hive2://10.60.2.26:10000/default;principal=hive/example-qa1-dn2@EXAMPLE.com;auth=NOSASL: GSS initiate failed
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:210)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:156)
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
    at java.sql.DriverManager.getConnection(DriverManager.java:571)
    at java.sql.DriverManager.getConnection(DriverManager.java:187)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager 
(DriverManagerDataSource.java:153)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:144)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver 
(AbstractDriverBasedDataSource.java:155)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120)
    at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
    at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
    ... 26 more
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
    at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:185)
    ... 36 more

Oozie Launcher failed, finishing Hadoop job gracefully

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://example-qa1-nn:8020/user/admin/oozie-oozi/0000000- 
160317141545276-oozie-oozi-W/javaAction--java/action-data.seq

Oozie Launcher ends

请查看以下工作流xml，其中指定了用于执行配置单元查询的oozie java操作和java程序：
工作流.xml：

<workflow-app name="WorkFlowForJavaActionToExecuteHiveQuery" xmlns="uri:oozie:workflow:0.2.5">
    <credentials>                      
  <credential name='hive_credentials' type='hcat'>
                        <property>
                            <name>hcat.metastore.uri</name>
                            <value>thrift://example-qa1-dn2:9083</value>
                        </property>
                        <property>
                            <name>hcat.metastore.principal</name>
                            <value>hive/_HOST@EXAMPLE.COM</value>
                        </property>
                </credential>
 </credentials>
    <start to="javaAction"/>
<action name="javaAction" cred="hive_credentials">
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>

    <main-class>com.demo.main.Test</main-class>
        </java>
        <ok to="end"/>
        <error to="fail"/>
    </action>

     <kill name="fail">
        <message>Job failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

java程序：

package com.demo.main;

import java.io.IOException;

import org.apache.hadoop.security.UserGroupInformation;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;

public class Test {
    public static void main(String[] args) throws IOException {
        UserGroupInformation.loginUserFromKeytab("admin@EXAMPLE.COM", "/etc/security/keytabs/admin.keytab");

        org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
        conf.set("hadoop.security.authentication", "Kerberos");
        UserGroupInformation.setConfiguration(conf);
        UserGroupInformation.loginUserFromKeytab("admin@EXAMPLE.COM", "/etc/security/keytabs/admin.keytab");

        DriverManagerDataSource dataSource = new DriverManagerDataSource 
("jdbc:hive2://10.60.2.26:10001/default;principal=hive/example-qa1-dn2@EXAMPLE.com", "hive", "");
        dataSource.setDriverClassName("org.apache.hive.jdbc.HiveDriver");
        JdbcTemplate jdbcTemplate = new JdbcTemplate();
        jdbcTemplate.setDataSource(dataSource);

        long count = 0;
        count = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM provider",Long.class);
        System.out.println("Count is*************************"+count);
    }
}

我已提交工作流作业使用管理用户谁有权访问配置单元。
我也用usergroupinformationapi为管理员用户编程实现了kinit，但我还是遇到了上述异常。我只在oozie hdfs缓存中保留了springcore、springjdbc和hivejdbcjar。
我也尝试了hortonworks，stack overflow和其他论坛上提到的所有可能的解决方案，但是我仍然面临同样的问题。
有人能提出解决这个问题的办法吗？

JavaHiveooziekerberoshortonworks-data-platform

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com