msck无法通过spark sql工作

ih99xse1 于 2021-06-26 发布在 Hive

关注(0)|答案(2)|浏览(435)

我已经创建了hive context对象并尝试执行msck命令，该命令将分区添加到hive表中，但它给出了以下异常

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
Operation not allowed: msck repair table(line 1, pos 0)

== SQL ==
msck repair table table_name
^^^

        at org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:43)
        at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitFailNativeCommand$1.apply(SparkSqlParser.scala:837)
        at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitFailNativeCommand$1.apply(SparkSqlParser.scala:828)
        at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
        at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitFailNativeCommand(SparkSqlParser.scala:828)
        at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitFailNativeCommand(SparkSqlParser.scala:53)
        at org.apache.spark.sql.catalyst.parser.SqlBaseParser$FailNativeCommandContext.accept(SqlBaseParser.java:900)
        at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
        at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
        at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
        at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
        at org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:63)
        at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:54)
        at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
        at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:82)
        at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:46)
        at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
        at com.mcd.spark.driver.R2D2Driver$.main(R2D2Driver.scala:321)
        at com.mcd.spark.driver.R2D2Driver.main(R2D2Driver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

创建了如下所述的spark上下文和hive上下文。

val conf = new SparkConf().setAppName(appName).setMaster(master)
    var sc: SparkContext = null
    sc = new SparkContext(conf)
     val hqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

hqlContext.sql("msck repair table table_name")

Can some one help me to solve how to add partitions into hive table?

Regards,
Aswin

Hive apache-spark apache-spark-sql

来源：https://stackoverflow.com/questions/42760800/msck-not-working-through-spark-sql

2条答案

按热度按时间

cyvaqqii1#

我想我以前也碰到过这个。最近spark增加了对msck的支持，支持fast stats。你能试试其他的语法对你有用吗？ ALTER TABLE {table_name} RECOVER PARTITIONS; 另外，你可能想看看https://issues.apache.org/jira/browse/spark-20697 防止副作用。

赞(0）回复(0）举报 2021-06-26

lp0sw83n2#

用它试试 "runSqlHive" 比如：

hqlContext.runSqlHive("msck repair table table_name")

或

try {
      Class.forName("org.apache.hive.jdbc.HiveDriver");
      Connecton connection = DriverManager.getConnection("jdbc:hive2://<hostname>:<port>/<db_name>" , <user_name> , "");

      Statement stmt = connection.createStatement();
      stmt.execute("msck repair table table_name");
      } catch (final ClassNotFoundException | SQLException e {
            throw new RuntimeException(e);
      }

赞(0）回复(0）举报 2021-06-26

我来回答

msck无法通过spark sql工作

2条答案

相关问题

热门标签

最新问答