我正在使用Windows7 professional中为Hadoop2.6预先构建的程序，在PySpark1.6.2中运行这段代码
在我定义自定义项之前，一切都正常。你能给点建议吗。我需要用hive编译spark吗？那么hadoop2.6的预构建有什么用呢。我不能更改c:\tmp\hive权限，因为我不是系统管理员。这可能是此错误消息的原因吗？
from pyspark import SparkContext
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
rdd = sc.parallelize([('u1', 1, [1 ,2, 3]), ('u1', 4, [1, 2, 3])])
df = rdd.toDF(['user', 'item', 'fav_items'])
# Print dataFrame
df.show()
from pyspark.sql.functions import *
from pyspark.sql.types import IntegerType
function = udf(lambda item, items: 1 if item in items else 0, IntegerType())
df.select('user', 'item', 'fav_items', function(col('item'), col('fav_items')).alias('result')).show()然后我得到这个错误：
You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\pyspark\sql\functions.py", line 1597, in udf
return UserDefinedFunction(f, returnType)
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\pyspark\sql\functions.py", line 1558, in __init__
self._judf = self._create_judf(name)
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\pyspark\sql\functions.py", line 1569, in _create_judf
jdt = ctx._ssql_ctx.parseDataType(self.returnType.json())
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\pyspark\sql\context.py", line 683, in _ssql_ctx
self._scala_HiveContext = self._get_hive_ctx()
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\pyspark\sql\context.py", line 692, in _get_hive_ctx
return self._jvm.HiveContext(self._jsc.sc())
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\java_gateway.py", line 1064,
l__
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\pyspark\sql\utils.py", line 45, in deco
return f(*a,**kw)
File "C:\Users\yrxt028\Downloads\spark-1.6.2-bin-hadoop2.6\spark-1.6.2-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\protocol.py", line 308, in g
_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.
: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:214)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 21 more
2条答案
按热度按时间flvtvl501#
在我的hive-site.xml中，我只有以下代码：
这篇文章也帮助了我：
$spark\u home中包含的hive-site.xml是什么样子的？
因此，我创建了另一个文件夹/tmp1/hive，而不是/tmp/hive，我无法更改它的权限。非常感谢@bigdatalearnner。还有另一种方法可以使用spark-env.sh覆盖目录——我试过了，但没有成功
spark-env.sh代码：
5vf7fwbs2#
你需要创造
hive-site.xml文件输入$SPARK_HOME/conf位置。在该文件中，可以覆盖scratch dir路径。这些是您应该包含在中的重要配置hive-site.xml但如果出现其他错误，可以检查此链接以了解其他设置：