我有一个简单的python文件,它使用
SparkSession.builder.getOrCreate()
看到以下py4j错误:
Traceback (most recent call last):
File "/home/centos/pysubmit-test.py", line 11, in <module>
spark = (SparkSession.builder.config(conf=conf).getOrCreate())
File "/home/centos/pyspark/sql/session.py", line 169, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/home/centos/pyspark/context.py", line 310, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/home/centos/pyspark/context.py", line 115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/home/centos/pyspark/context.py", line 259, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/home/centos/pyspark/java_gateway.py", line 117, in launch_gateway
java_import(gateway.jvm, "org.apache.spark.SparkConf")
File "/home/centos/py4j/java_gateway.py", line 180, in java_import
return_value = get_return_value(answer, gateway_client, None, None)
File "/home/centos/py4j/protocol.py", line 332, in get_return_value
format(target_id, ".", name, value))
py4j.protocol.Py4JError: An error occurred while calling None.None. Trace:
Authentication error: unexpected command.
spark版本:2.1.3
py4j版本:0.10.7
下面是我的bashèu简介。
export PATH=$PATH:$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*:/opt/jdk8u232-b09/bin:/home/centos/maven/bin:/home/centos/spark/bin
export HADOOP_CLASSPATH=$HADOOP_HOME/share/hadoop/tools/lib/*
export JAVA_HOME=/opt/jdk8u232-b09
export SPARK_HOME=/home/centos/spark
知道缺少什么配置吗?
暂无答案!
目前还没有任何答案,快来回答吧!