我正在使用spring框架创建一个api来查询hadoop中的一些表。我使用的命令:
println("-----------------------------------------------------------------before )
val spark = SparkSession
.builder()
.appName("API")
.master("local[*])
.enableHiveSupport()
.getOrCreate()
println("--------------------------------------------------------------------Session was created")
我使用的是spark 2.11.6和scala v2.2.0。当我使用spark shell时,我连接到远程集群。
在日志中,我没有发现任何错误,但我看到创建了一个本地配置单元存储库:
[ main] o.a.h.hive.metastore.MetaStoreDirectSql : Using direct SQL, underlying DB is DERBY
main] o.a.hadoop.hive.ql.session.SessionState : Created local directory: C:/Users/..../.../Local/Temp/..._resources
2018-05-10 16:32:32.556 INFO 16148 --- [ main] o.a.hadoop.hive.ql.session.SessionState : Created HDFS directory: /tmp/hive/myuser/....
我正在尝试连接到一个远程cloudera群集。我将xml文件(hive site、hdfs site、core stire、yarn site)复制到我的项目中的conf目录中,$spark\u conf dir。我将spark\u home路径添加到path变量,并将haddop\u home变量指定为指向winutils位置。
我还能做什么?
日志很长,我看到了一些信息,可能对您有任何暗示:
-----------------------------------------------------------------ENV=local[*]
2018-05-10 16:32:16.930 WARN 16148 --- [ main] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ main] org.apache.spark.util.Utils : Successfully started service 'SparkUI' on port 4040.
main] o.s.jetty.server.handler.ContextHandler : Started o.s.j.s.ServletContextHandler@13ee97af{/stages/pool/json,null,AVAILABLE,@Spark}
[ main] org.apache.spark.ui.SparkUI : Bound SparkUI to 0.0.0.0, and started at http://192.168.56.1:4040
[ main] o.apache.spark.sql.internal.SharedState : URL.setURLStreamHandlerFactory failed to set FsUrlStreamHandlerFactory
[ main] DataNucleus.Persistence : Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
[ main] DataNucleus.Datastore : The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
[ main] DataNucleus.Query : Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
[ main] o.a.h.hive.metastore.MetaStoreDirectSql : Using direct SQL, underlying DB is DERBY
[ main] o.a.hadoop.hive.metastore.ObjectStore : Failed to
get database global_temp, returning NoSuchObjectException
[ main] o.a.hadoop.hive.ql.session.SessionState : Created local directory: C:/Users/myuser/AppData/Local/Temp/1fa7a82b-fe17-4795-8973-212010634cd1_resources
[ main] o.a.hadoop.hive.ql.session.SessionState : Created HDFS directory: /tmp/hive/myuser/1fa7a82b-fe17-4795-8973-212010634cd1
[ main] o.a.hadoop.hive.ql.session.SessionState : Created local directory: C:/Users/myuser/AppData/Local/Temp/myuser/fileasdasdsa
[ main] o.a.hadoop.hive.ql.session.SessionState : Created HDFS directory: /tmp/hive/myuser/asdsadsa/_tmp_space.db
[ main] o.a.s.sql.hive.client.HiveClientImpl : Warehouse location for Hive client (version 1.2.1) is file:/C:/Users/myuser/SpringScalaAPI/spark-warehouse
[ main] o.a.s.s.e.s.s.StateStoreCoordinatorRef : Registered StateStoreCoordinator endpoint
--------------------------------------------------------------------Session was created
老实说,这不是我第一次处理这种错误。上次我用游戏框架。在这种情况下需要采取哪些具体步骤?哪些变量应该真正配置,哪些变量不重要?
1条答案
按热度按时间bnlyeluc1#
有了spark 2你可以试试这个,
注意
hive.metastore.uris
属性,将localhost更改为指向您的沙盒或集群。1
ss
初始化后,您可以读取如下表,希望这有帮助。干杯。