sparkr sql-function windows vs macos输出:与windows上的配置单元相关的错误

nxowjjhe  于 2021-06-25  发布在  Hive
关注(0)|答案(0)|浏览(181)

我尝试使用sparkr-sql库创建查询,但是尽管我使用了相同类型的代码,但我发现在macos中它可以工作并返回正确的查询结果,而在windows中它返回以下错误。
下面我报告查询和macos/windows输出
来自r-studio的查询


# SETTING ENVIRONMENT VARIABLES#######################################################################

# MacOs

Sys.setenv(SPARK_HOME = "/Users/work/Spark/spark-2.4.5-bin-hadoop2.7")

# Windows

Sys.setenv(SPARK_HOME = "/Users/Andrea/Desktop/spark-2.4.5-bin-hadoop2.7")
Sys.setenv(JAVA_HOME = "/Users/Andrea/Desktop/jre1.8.0_251")
Sys.setenv(HADOOP_HOME = "/Users/Andrea/Desktop/spark-2.4.5-bin-hadoop2.7/tmp/hadoop/bin/winuitls.exe")

# Same code for both Windows and MacOS############################################################

library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"))) 
sparkR.session()  
df <- as.DataFrame(iris)
createOrReplaceTempView(df, 'iris2')
collect(sql("SELECT * FROM iris2 LIMIT 5"))

macos输出

Sepal_Length Sepal_Width Petal_Length Petal_Width Species                     
1          5.1         3.5          1.4         0.2  setosa
2          4.9         3.0          1.4         0.2  setosa
3          4.7         3.2          1.3         0.2  setosa
4          4.6         3.1          1.5         0.2  setosa
5          5.0         3.6          1.4         0.2  setosa

windows输出

Error in handleErrors(returnStatus, conn) : 
  org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141)
    at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
    at org.apache.spark.sql.ca

如何解决此问题以获得正确的结果?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题