pyspark-java.lang.outofmemoryerror:写入csv文件时的java堆空间

2lpgd968 于 2021-05-27 发布在 Spark

关注(0)|答案(2)|浏览(369)

尝试使用以下代码写入csv文件时

DF.coalesce(1).write.option("header","false").option("sep",",").option("escape",'"').option("ignoreTrailingWhiteSpace","false").option("ignoreLeadingWhiteSpace","false").mode("overwrite").csv(filename)

我得到下面的错误

ileFormatWriter.scala:169)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space

有人能建议一个解决办法吗？

apache-spark pyspark python-3.x

来源：https://stackoverflow.com/questions/63379819/pyspark-java-lang-outofmemoryerror-java-heap-space-while-writing-to-csv-file

2条答案

按热度按时间

dwthyt8l1#

对我来说，添加下面的spark配置修复了这个问题

spark = SparkSession.builder.master('local[*]').config("spark.driver.memory", "15g").appName('sl-app').getOrCreate()

赞(0）回复(0）举报 2021-05-27

6yjfywim2#

尝试增加 executor.memory 在你的spark提交申请
像这样的

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

赞(0）回复(0）举报 2021-05-27

我来回答

pyspark-java.lang.outofmemoryerror:写入csv文件时的java堆空间

2条答案

相关问题

热门标签

最新问答