使用spark在数据库名称中插入“.”的配置单元数据库

mxg2im7a  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(324)

我有以下代码从配置单元表中选择一些数据,进行一些简单的转换,然后将它们插入到同一个表中:

sqlContext.sql("USE `database.with.dot`");
val ds = sqlContext.sql("select * from Table where column = 'SomeValue'");
ds.withColumn("anotherColumn", lit("SomeOtherValue"))
  .write.insertInto("Table")

然而,我得到:

org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table database.with.dot.Table
  at org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2350)
  at org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2336)
  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1070)
  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1058)
  at org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1581)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.spark.sql.hive.client.Shim_v1_2.loadDynamicPartitions(HiveShim.scala:823)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(HiveClientImpl.scala:689)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply(HiveClientImpl.scala:687)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply(HiveClientImpl.scala:687)
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:230)
  at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:229)
  at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:272)
  at org.apache.spark.sql.hive.client.HiveClientImpl.loadDynamicPartitions(HiveClientImpl.scala:687)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(HiveExternalCatalog.scala:796)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply(HiveExternalCatalog.scala:784)
  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply(HiveExternalCatalog.scala:784)
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:95)
  at org.apache.spark.sql.hive.HiveExternalCatalog.loadDynamicPartitions(HiveExternalCatalog.scala:784)
  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:268)
  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:170)
  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:347)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
  at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:263)
  at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:243)
  ... 46 elided

有没有什么方法可以使用spark插入到数据库名中有点的配置单元表中?
edit:我可以使用hiveshell或sparksql从表中进行查询,而不是插入回表中。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题