我正在aws emr集群中运行geospark。代码是:
# coding=utf-8
from pyspark.sql import SparkSession
import pyspark.sql.functions as f
import pyspark.sql.types as t
from geospark.register import GeoSparkRegistrator
from geospark.utils import GeoSparkKryoRegistrator
from geospark.register import upload_jars
import config as cf
import yaml
if __name__ == "__main__":
# Read files
with open("/tmp/param.yml", 'r') as ymlfile:
param = yaml.load(ymlfile, Loader=yaml.SafeLoader)
# Register jars
upload_jars()
# Creation of spark session
print("Creating Spark session")
spark = SparkSession \
.builder \
.getOrCreate()
GeoSparkRegistrator.registerAll(spark)
我得到以下错误 upload_jars()
功能:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/findspark.py", line 143, in init
py4j = glob(os.path.join(spark_python, "lib", "py4j-*.zip"))[0]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "geo_processing.py", line 21, in <module>
upload_jars()
File "/usr/local/lib/python3.7/site-packages/geospark/register/uploading.py", line 39, in upload_jars
findspark.init()
File "/usr/local/lib/python3.7/site-packages/findspark.py", line 146, in init
"Unable to find py4j, your SPARK_HOME may not be configured correctly"
Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly
如何解决此错误?
暂无答案!
目前还没有任何答案,快来回答吧!