我刚刚尝试使用apachesparkml库进行logistic回归,但每次尝试时都会出现一条错误消息,例如
“错误:失败!重置历史记录:breeze.optimize.nanhistory:“
logistic回归的数据集示例如下:
+-----+---------+---------+---------+--------+-------------+
|state|dayOfWeek|hourOfDay|minOfHour|secOfMin| features|
+-----+---------+---------+---------+--------+-------------+
| 1.0| 7.0| 0.0| 0.0| 0.0|(4,[0],[7.0])|
逻辑回归的代码如下:
//Data Set
StructType schema = new StructType(
new StructField[]{
new StructField("state", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("dayOfWeek", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("hourOfDay", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("minOfHour", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("secOfMin", DataTypes.DoubleType, false, Metadata.empty())
});
List<Row> dataFromRDD = bucketsForMLs.map(p -> {
return RowFactory.create(p.label(), p.features().apply(0), p.features().apply(1), p.features().apply(2), p.features().apply(3));
}).collect();
Dataset<Row> stateDF = sparkSession.createDataFrame(dataFromRDD, schema);
String[] featureCols = new String[]{"dayOfWeek", "hourOfDay", "minOfHour", "secOfMin"};
VectorAssembler vectorAssembler = new VectorAssembler().setInputCols(featureCols).setOutputCol("features");
Dataset<Row> stateDFWithFeatures = vectorAssembler.transform(stateDF);
StringIndexer labelIndexer = new StringIndexer().setInputCol("state").setOutputCol("label");
Dataset<Row> stateDFWithLabelAndFeatures = labelIndexer.fit(stateDFWithFeatures).transform(stateDFWithFeatures);
MLRExecutionForDF mlrExe = new MLRExecutionForDF(javaSparkContext);
mlrExe.execute(stateDFWithLabelAndFeatures);
// Logistic Regression part
LogisticRegressionModel lrModel = new LogisticRegression().setMaxIter(maxItr).setRegParam(regParam).setElasticNetParam(elasticNetParam)
// This part would occur error
.fit(stateDFWithLabelAndFeatures);
1条答案
按热度按时间68bkxrlz1#
我也遇到了同样的错误。它来自breeze scalanlp软件包,spark刚刚进口。它说一些关于衍生产品的东西是不能产生的。
我不确定这到底意味着什么,但在我的数据集中,我可以说我使用的数据越少,抛出错误的频率就越高。这意味着,对于要训练的类来说,缺失特征的比例越高,错误发生的频率就越高。我认为这与由于缺少类的信息而无法正确优化有关。
否则,该错误似乎不会阻止代码运行。