您好,我正在尝试从增量表中删除记录。它会不时地导致广播超时错误。有人能帮助解决这个问题吗
spark.sql(s"""DELETE FROM stg.bl WHERE concat(key,':',revision) in
(Select distinct concat(bl.key,':',bl.revision) from stg.bl bl left semi join
tgt.bl tgt ON bl.key = tgt.key and bl.revision = tgt.revision)""")
org.apache.spark.SparkException: Could not execute broadcast in 300 secs.
You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or
disable broadcast join
by setting spark.sql.autoBroadcastJoinThreshold to -1
1条答案
按热度按时间tyg4sfes1#
这可能是一个有点晚,但你有没有尝试设置广播超时限制从300秒到一个更大的数字?