spark2.0:spark信息论特性选择java.lang.nosuchmethoderror:breeze.linalg.densematrix

vc6uscn9  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(392)

我正在尝试使用第三方infogain的mrmr特性(https://github.com/sramirez/spark-infotheoretic-feature-selection)Spark包。但是我的集群是2.0,我得到了这个例外。即使我将所有必需的jar文件添加到spark类路径中。但它仍然不起作用。虽然它在本地机器上正常工作,但在集群上却不正常。
例外情况:

  1. 18/03/29 01:16:43 WARN TaskSetManager: Lost task 3.0 in stage 14.0 (TID 47, EUREDWORKER3): java.lang.NoSuchMethodError: breeze.linalg.DenseMatrix$.canMapValues(Lscala/reflect/ClassTag;)Lbreeze/generic/UFunc$UImpl2;
  2. at org.apache.spark.mllib.feature.InfoTheorySparse$$anonfun$15.apply(InfoTheory.scala:172)
  3. at org.apache.spark.mllib.feature.InfoTheorySparse$$anonfun$15.apply(InfoTheory.scala:172)
  4. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$43$$anonfun$apply$44.apply(PairRDDFunctions.scala:759)
  5. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$43$$anonfun$apply$44.apply(PairRDDFunctions.scala:759)
  6. at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
  7. at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:214)
  8. at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)
  9. at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)
  10. at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
  11. at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)
  12. at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)
  13. at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
  14. at org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
  15. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  16. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
  17. at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:332)
  18. at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:330)
  19. at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)
  20. at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:926)
  21. at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
  22. at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:926)
  23. at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:670)
  24. at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
  25. at org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
  26. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  27. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
  28. at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
  29. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  30. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
  31. at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
  32. at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
  33. at org.apache.spark.scheduler.Task.run(Task.scala:86)
  34. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
  35. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  36. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  37. at java.lang.Thread.run(Thread.java:745)

spark类路径的参考

thigvfpy

thigvfpy1#

这是微风版本的问题。我添加了一个旧版本breeze_2.11_0.11,通过将其更改为breeze_2.11-0.13.2.jar解决了这个问题。

相关问题