Apache Spark Azure数据块增量删除的广播超时

9udxz4iz  于 2023-01-31  发布在  Apache
关注(0)|答案(1)|浏览(147)

您好,我正在尝试从增量表中删除记录。它会不时地导致广播超时错误。有人能帮助解决这个问题吗

spark.sql(s"""DELETE FROM stg.bl  WHERE concat(key,':',revision) in 
   (Select distinct concat(bl.key,':',bl.revision) from stg.bl bl left semi join
    tgt.bl tgt ON bl.key = tgt.key and bl.revision = tgt.revision)""")
org.apache.spark.SparkException: Could not execute broadcast in 300 secs. 
You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or 
disable broadcast join 
by setting spark.sql.autoBroadcastJoinThreshold to -1

tyg4sfes

tyg4sfes1#

这可能是一个有点晚,但你有没有尝试设置广播超时限制从300秒到一个更大的数字?

spark.conf.set("spark.sql.broadcastTimeout", "300")

spark.conf.set("spark.sql.broadcastTimeout", "3000")

相关问题