我正在将spark从2.3.1版升级到2.4.5版。我正在使用dataproc image 1.4.27-debian9在google云平台的dataproc上重新培训spark2.4.5模型。当我在本地机器上加载dataproc生成的模型时，使用spark2.4.5验证模型。不幸的是，我得到了以下例外：

20/05/27 08:36:35 INFO HadoopRDD: Input split: file:/Users/.../target/classes/model.ml/stages/1_gbtc_961a6ef213b2/metadata/part-00000:0+657
20/05/27 08:36:35 INFO HadoopRDD: Input split: file:/Users/.../target/classes/model.ml/stages/1_gbtc_961a6ef213b2/metadata/part-00000:0+657
Exception in thread "main" java.lang.IllegalArgumentException: gbtc_961a6ef213b2 parameter impurity given invalid value variance.

加载模型的代码非常简单：

import org.apache.spark.ml.PipelineModel

object ModelLoad {
  def main(args: Array[String]): Unit = {
    val modelInputPath = getClass.getResource("/model.ml").getPath
    val model = PipelineModel.load(modelInputPath)
  }
}

我沿着烟囱的轨迹去检查 1_gbtc_961a6ef213b2/metadata/part-00000 模型元数据文件并找到以下内容：

{
    "class": "org.apache.spark.ml.classification.GBTClassificationModel",
    "timestamp": 1590593177604,
    "sparkVersion": "2.4.5",
    "uid": "gbtc_961a6ef213b2",
    "paramMap": {
        "maxIter": 50
    },
    "defaultParamMap": {
        ...
        "impurity": "variance",
        ...
    },
    "numFeatures": 1,
    "numTrees": 50
}

杂质设置为 variance 但我的本地spark 2.4.5预计 gini . 为了进行合理性检查，我在本地spark2.4.5上重新训练了模型。这个 impurity 模型中的元数据文件设置为 gini .
所以，我检查了gbtjavadoc中的spark2.4.5setinclusion方法。上面写着 The impurity setting is ignored for GBT models. Individual trees are built using impurity "Variance." . dataproc使用的spark2.4.5似乎与apachespark文档一致。但是，我从maven central使用的spark 2.4.5设置了 impurity 价值 gini .
有人知道为什么dataproc中的spark2.4.5和maven central之间会有这样的不一致吗？
我创建了一个简单的训练代码来在本地重现结果：

import java.nio.file.Paths

import org.apache.spark.ml.classification.GBTClassifier
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.sql.{DataFrame, SparkSession}

object SimpleModelTraining {
  def main(args: Array[String]) {

    val currentRelativePath = Paths.get("")
    val save_file_location = currentRelativePath.toAbsolutePath.toString

    val spark = SparkSession.builder()
      .config("spark.driver.host", "127.0.0.1")
      .master("local")
      .appName("spark-test")
      .getOrCreate()

    val df: DataFrame = spark.createDataFrame(Seq(
      (0, 0),
      (1, 0),
      (1, 0),
      (0, 1),
      (0, 1),
      (0, 1),
      (0, 2),
      (0, 2),
      (0, 2),
      (0, 3),
      (0, 3),
      (0, 3),
      (1, 4),
      (1, 4),
      (1, 4)
    )).toDF("label", "category")

    val pipeline: Pipeline = new Pipeline().setStages(Array(
      new VectorAssembler().setInputCols(Array("category")).setOutputCol("features"),
      new GBTClassifier().setMaxIter(30)
    ))

    val pipelineModel: PipelineModel = pipeline.fit(df)
    pipelineModel.write.overwrite().save(s"$save_file_location/test_model.ml")
  }
}

谢谢您！