spark java api、kerberos和配置单元出现问题

vwoqyblh  于 2021-06-26  发布在  Hive
关注(0)|答案(1)|浏览(382)

我正在尝试使用sparkjavaapi对配置单元表运行sparksql测试。我遇到的问题是kerberos。每当我试图运行程序时,都会收到以下错误消息:

Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS];
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:69)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
    at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:79)
    at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:79)
    at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
    at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
    at tester.SparkSample.lambda$0(SparkSample.java:62)
    ... 5 more

在这行代码上:

ss.sql("select count(*) from entps_pma.baraccount").show();

现在,当我运行代码时,我很好地登录到kerberos并得到以下消息:

18/05/01 11:21:03 INFO security.UserGroupInformation: Login successful for user <kerberos user> using keytab file /root/hdfs.keytab

我甚至连接到Hive元存储:

18/05/01 11:21:06 INFO hive.metastore: Trying to connect to metastore with URI thrift://<hiveserver>:9083
18/05/01 11:21:06 INFO hive.metastore: Connected to metastore.

但就在那之后我得到了错误。很感谢你给我指路。这是我的密码:

public static void runSample(String fullPrincipal) throws IOException {

    System.setProperty("hive.metastore.sasl.enabled", "true");
    System.setProperty("hive.security.authorization.enabled", "true");
    System.setProperty("hive.metastore.kerberos.principal", fullPrincipal);
    System.setProperty("hive.metastore.execute.setugi", "true");
    System.setProperty("hadoop.security.authentication", "kerberos");

    Configuration conf = setSecurity(fullPrincipal);

    loginUser = UserGroupInformation.getLoginUser();
    loginUser.doAs((PrivilegedAction<Void>) () -> {

    SparkConf sparkConf = new SparkConf().setMaster("local");
        sparkConf.set("spark.sql.warehouse.dir", "hdfs:///user/hive/warehouse");
        sparkConf.set("hive.metastore.uris", "thrift://<hive server>:9083");
        sparkConf.set("hadoop.security.authentication", "kerberos");
        sparkConf.set("hadoop.rpc.protection", "privacy");
        sparkConf.set("spark.driver.extraClassPath",
                "/opt/cloudera/parcels/CDH/jars/*.jar:/opt/cloudera/parcels/CDH/lib/hive/conf:/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar");
        sparkConf.set("spark.executor.extraClassPath",
                "/opt/cloudera/parcels/CDH/jars/*.jar:/opt/cloudera/parcels/CDH/lib/hive/conf:/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar");
        sparkConf.set("spark.eventLog.enabled", "false");

        SparkSession ss = SparkSession
              .builder()
              .enableHiveSupport()
              .config(sparkConf)
              .appName("Jim Test Spark App")
              .getOrCreate();

        ss.sparkContext()
            .hadoopConfiguration()
            .addResource(conf);

        ss.sql("select count(*) from entps_pma.baraccount").show();
        return null;
    });
}
acruukt9

acruukt91#

我猜你是在吹毛求疵。您需要指定spark.yarn.principal和spark.yarn.keytab参数。请检查Yarn文档上的运行Spark

相关问题