有趣的是，我正在研究一个类似的问题。我可以看到Spark中的一些代码实际上调用了System.gc()。
可能值得在Spark打开一个JIRA来讨论这个问题。
我知道使用System.gc()进行调用不是最佳实践，主要是因为它将停止所有其他线程，这对性能有很大影响。但是，我可以在Java Oracle文档中看到，从Java 1.6开始引入了一个额外的JVM参数，以便并发地运行System.gc()（-XX：+ExplicitGCInvokesConcurrent）：
http://docs.oracle.com/javase/6/docs/technotes/guides/vm/cms-6.html
您也许可以尝试将其设置为附加参数：
"-XX：+显式GC调用并发”
根据您设置参数的方式，您可以将其放在Spark的配置文件中，或者在spark命令（spark-submit、spark-shell等）中使用--conf行参数传递它。
更新：
在Spark 2.x的ContextCleaner.scala文件中找到以下注解：

/**

* How often to trigger a garbage collection in this JVM.
* 
* This context cleaner triggers cleanups only when weak references are  garbage collected.
* In long-running applications with large driver JVMs, where there is little memory pressure
* on the driver, this may happen very occasionally or not at all. Not  cleaning at all may
* lead to executors running out of disk space after a while.
* /

https://github.com/apache/spark/blob/branch-3.3/core/src/main/scala/org/apache/spark/ContextCleaner.scala#L92

/**
   * How often to trigger a garbage collection in this JVM.
   *
   * This context cleaner triggers cleanups only when weak references are garbage collected.
   * In long-running applications with large driver JVMs, where there is little memory pressure
   * on the driver, this may happen very occasionally or not at all. Not cleaning at all may
   * lead to executors running out of disk space after a while.
   */
  private val periodicGCInterval = sc.conf.get(CLEANER_PERIODIC_GC_INTERVAL)

和文档https://spark.apache.org/docs/latest/configuration.html#memory-management

spark.cleaner.periodicGC.interval   30min   Controls how often to trigger a garbage collection.

This context cleaner triggers cleanups only when weak references are garbage collected. In long-running applications with large driver JVMs, where there is little memory pressure on the driver, this may happen very occasionally or not at all. Not cleaning at all may lead to executors running out of disk space after a while.