如何使用apachespark查询kerberos配置单元表?

ijnw1ujt  于 2021-06-25  发布在  Hive
关注(0)|答案(1)|浏览(559)

我尝试在本地使用scala和apachespark来查询由kerberos保护的配置单元表。我没有问题连接和查询数据编程没有Spark。但是,当我尝试在spark中连接和查询时,问题就出现了。
在没有spark的情况下本地运行时的代码:

Class.forName("org.apache.hive.jdbc.HiveDriver")

    System.setProperty("kerberos.keytab", keytab)
    System.setProperty("kerberos.principal", keytab)
    System.setProperty("java.security.krb5.conf", krb5.conf)
    System.setProperty("java.security.auth.login.config", jaas.conf)

    val conf = new Configuration
    conf.set("hadoop.security.authentication", "Kerberos")

    UserGroupInformation.setConfiguration(conf)
    UserGroupInformation.createProxyUser("user", UserGroupInformation.getLoginUser)
    UserGroupInformation.loginUserFromKeytab(user, keytab)
    UserGroupInformation.getLoginUser.checkTGTAndReloginFromKeytab()

    if (UserGroupInformation.isLoginKeytabBased) {
      UserGroupInformation.getLoginUser.reloginFromKeytab()
    }
    else if (UserGroupInformation.isLoginTicketBased) UserGroupInformation.getLoginUser.reloginFromTicketCache()

    val con = DriverManager.getConnection("jdbc:hive://hdpe-hive.company.com:10000", user, password)
    val ps = con.prepareStatement("select * from table limit 5").executeQuery();

有人知道我如何将keytab、krb5.conf和jaas.conf包含到spark初始化函数中,以便我能够通过kerberos进行身份验证以获得tgt吗?
我的spark初始化功能:

conf = new SparkConf().setAppName("mediumData")
      .setMaster(numCores)
      .set("spark.driver.host", "localhost")
      .set("spark.ui.enabled","true") //enable spark UI
      .set("spark.sql.shuffle.partitions",defaultPartitions)
    sparkSession = SparkSession.builder.config(conf).enableHiveSupport().getOrCreate()

我没有hive-site.xml、core-site.xml等文件。
谢谢您!

daolsyd0

daolsyd01#

查看您的代码,您需要在终端上的spark submit命令中设置以下属性。

spark-submit ---master yarn \
--principal YOUR_PRINCIPAL_HERE \
--keytab YOUR_KEYTAB_HERE \
--conf spark.driver.extraJavaOptions="-Djava.security.auth.login.config=JAAS_CONF_PATH"
--conf spark.driver.extraJavaOptions="-Djava.security.krb5.conf=KRB5_PATH" \
--conf spark.executor.extraJavaOptions="-Djava.security.auth.login.config=JAAS_CONF_PATH" \
--conf spark.executor.extraJavaOptions="-Djava.security.krb5.conf=KRB5_PATH" \
--class YOUR_MAIN_CLASS_NAME_HERE code.jar

相关问题