使用upload\u jars函数获取geospark错误

dsekswqp  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(301)

我正在aws emr集群中运行geospark。代码是:


# coding=utf-8

from pyspark.sql import SparkSession
import pyspark.sql.functions as f
import pyspark.sql.types as t
from geospark.register import GeoSparkRegistrator
from geospark.utils import GeoSparkKryoRegistrator
from geospark.register import upload_jars

import config as cf

import yaml

if __name__ == "__main__":
    # Read files
    with open("/tmp/param.yml", 'r') as ymlfile:
        param = yaml.load(ymlfile, Loader=yaml.SafeLoader)

    # Register jars
    upload_jars()

    # Creation of spark session
    print("Creating Spark session")
    spark = SparkSession \
        .builder \
        .getOrCreate()

    GeoSparkRegistrator.registerAll(spark)

我得到以下错误 upload_jars() 功能:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/findspark.py", line 143, in init
    py4j = glob(os.path.join(spark_python, "lib", "py4j-*.zip"))[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "geo_processing.py", line 21, in <module>
    upload_jars()
  File "/usr/local/lib/python3.7/site-packages/geospark/register/uploading.py", line 39, in upload_jars
    findspark.init()
  File "/usr/local/lib/python3.7/site-packages/findspark.py", line 146, in init
    "Unable to find py4j, your SPARK_HOME may not be configured correctly"
Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly

如何解决此错误?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题