spark 2.0.0抛出alreadyexistsexception(message:database default 已存在)与hive 1.0.0交互时

jchrr9hc  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(641)

我正在尝试用spark java连接到hive。当我通过spark在hive中运行任何查询时，它会返回异常，如：

16/10/06 09:37:56 ERROR metastore.RetryingHMSHandler: AlreadyExistsException(message:Database default already exists)

我的版本是：
Spark2.0.0
Hive1.0.0
这是我的全部资料：

16/10/06 09:37:56 ERROR metastore.RetryingHMSHandler: AlreadyExistsException(message:Database default already exists)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
    at com.sun.proxy.$Proxy14.create_database(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:644)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
    at com.sun.proxy.$Proxy15.createDatabase(Unknown Source)
    at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:306)
    at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createDatabase$1.apply$mcV$sp(HiveClientImpl.scala:291)
    at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createDatabase$1.apply(HiveClientImpl.scala:291)
    at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createDatabase$1.apply(HiveClientImpl.scala:291)
    at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:262)
    at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:209)
    at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:208)
    at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:251)
    at org.apache.spark.sql.hive.client.HiveClientImpl.createDatabase(HiveClientImpl.scala:290)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createDatabase$1.apply$mcV$sp(HiveExternalCatalog.scala:99)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createDatabase$1.apply(HiveExternalCatalog.scala:99)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createDatabase$1.apply(HiveExternalCatalog.scala:99)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:72)
    at org.apache.spark.sql.hive.HiveExternalCatalog.createDatabase(HiveExternalCatalog.scala:98)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:147)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89)
    at org.apache.spark.sql.hive.HiveSessionCatalog.<init>(HiveSessionCatalog.scala:51)
    at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:49)
    at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48)
    at org.apache.spark.sql.hive.HiveSessionState$$anon$1.<init>(HiveSessionState.scala:63)
    at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63)
    at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
    at in.inndata.sparkjoinsexamples.SparkJoinExample.main(SparkJoinExample.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

16/10/06 09:37:57 INFO metastore.HiveMetaStore: 0: get_database: default
16/10/06 09:37:57 INFO HiveMetaStore.audit: ugi=karuturi    ip=unknown-ip-addr  cmd=get_database: default   
16/10/06 09:37:57 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=aaa
16/10/06 09:37:57 INFO HiveMetaStore.audit: ugi=karuturi    ip=unknown-ip-addr  cmd=get_table : db=default tbl=aaa  
Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `default`.`aaa`; line 1 pos 14
    at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:71)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:67)
    at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:125)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:125)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:125)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:67)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:58)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
    at in.inndata.sparkjoinsexamples.SparkJoinExample.main(SparkJoinExample.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/10/06 09:37:57 INFO spark.SparkContext: Invoking stop() from shutdown hook
16/10/06 09:37:57 INFO server.ServerConnector: Stopped ServerConnector@3f20e4fa{HTTP/1.1}{0.0.0.0:4040}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1bdf8190{/stages/stage/kill,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4f8969b0{/api,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6fefce9e{/,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@74cec793{/static,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@f9b7332{/executors/threadDump/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@18e7143f{/executors/threadDump,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@209775a9{/executors/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5db4c359{/executors,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2c177f9e{/environment/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@33617539{/environment,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@47874b25{/storage/rdd/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@290b1b2e{/storage/rdd,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1fc0053e{/storage/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@77307458{/storage,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@389adf1d{/stages/pool/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7bf9b098{/stages/pool,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@72e34f77{/stages/stage/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6e9319f{/stages/stage,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6fa590ba{/stages/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2416a51{/stages,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@293bb8a5{/jobs/job/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@37ebc9d8{/jobs/job,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5217f3d0{/jobs/json,null,UNAVAILABLE}
16/10/06 09:37:57 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@32232e55{/jobs,null,UNAVAILABLE}
16/10/06 09:37:57 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.131:4040
16/10/06 09:37:57 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/10/06 09:37:57 INFO memory.MemoryStore: MemoryStore cleared
16/10/06 09:37:57 INFO storage.BlockManager: BlockManager stopped
16/10/06 09:37:57 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/10/06 09:37:57 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/10/06 09:37:57 INFO spark.SparkContext: Successfully stopped SparkContext
16/10/06 09:37:57 INFO util.ShutdownHookManager: Shutdown hook called
16/10/06 09:37:57 INFO util.ShutdownHookManager: Deleting directory /private/var/folders/6n/nrvn14r50tvdvcfdds6jxyx40000gn/T/spark-b0f5733d-a475-4289-956d-c2650d9792d0

这是我的Spark代码：

public static void main(String[] arr){
        SparkSession session = new SparkSession.Builder().appName("SparkJoinExample").master("local").enableHiveSupport().getOrCreate();
        Dataset<Row> dset = session.sql("select * from test.results");

    }
JavasqlhadoopHiveapache-spark-sql

1条答案

按热度按时间
omhiaaxx

omhiaaxx1#

它将在$home文件夹中创建一个metastore，在metastore中只需删除dbex.lck

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com