我尝试使用sparkr-sql库创建查询,但是尽管我使用了相同类型的代码,但我发现在macos中它可以工作并返回正确的查询结果,而在windows中它返回以下错误。
下面我报告查询和macos/windows输出
来自r-studio的查询
# SETTING ENVIRONMENT VARIABLES#######################################################################
# MacOs
Sys.setenv(SPARK_HOME = "/Users/work/Spark/spark-2.4.5-bin-hadoop2.7")
# Windows
Sys.setenv(SPARK_HOME = "/Users/Andrea/Desktop/spark-2.4.5-bin-hadoop2.7")
Sys.setenv(JAVA_HOME = "/Users/Andrea/Desktop/jre1.8.0_251")
Sys.setenv(HADOOP_HOME = "/Users/Andrea/Desktop/spark-2.4.5-bin-hadoop2.7/tmp/hadoop/bin/winuitls.exe")
# Same code for both Windows and MacOS############################################################
library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sparkR.session()
df <- as.DataFrame(iris)
createOrReplaceTempView(df, 'iris2')
collect(sql("SELECT * FROM iris2 LIMIT 5"))
macos输出
Sepal_Length Sepal_Width Petal_Length Petal_Width Species
1 5.1 3.5 1.4 0.2 setosa
2 4.9 3.0 1.4 0.2 setosa
3 4.7 3.2 1.3 0.2 setosa
4 4.6 3.1 1.5 0.2 setosa
5 5.0 3.6 1.4 0.2 setosa
windows输出
Error in handleErrors(returnStatus, conn) :
org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141)
at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136)
at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
at org.apache.spark.sql.ca
如何解决此问题以获得正确的结果?
暂无答案!
目前还没有任何答案,快来回答吧!