aws glue pyspark作业失败,返回java.lang.illegalstateexception:用户未初始化spark上下文

iovurdzv  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(336)

我的代码spark2.4和python-3声明如下,带有spark上下文,但总是失败,出现以下错误。
是什么,我都错过了申报。
错误: java.lang.IllegalStateException :用户未初始化spark上下文!。


# lib imported

from pyspark.sql import SparkSession
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
import pyspark.sql.functions as F
from datetime import datetime
from pyspark.sql.utils import AnalysisException

main():
{  

# Initialize start of the Glue job

sc = SparkContext()
glueContext = GlueContext(sc.getOrCreate())
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'],args)

}

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题