不兼容的jackson版本:关于尝试在spark中创建行的rdd的2.10.2

7gyucuyw  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(488)

我尝试使用sc.parallelize创建rdd。它给出了一个例外:

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
    at org.apache.spark.SparkContext.parallelize(SparkContext.scala:718)
    at df_avro.SampleDf$.main(SampleDf.scala:25)
    at df_avro.SampleDf.main(SampleDf.scala)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.10.2
    at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:808)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
    ... 4 more

这是代码:

val conf = new SparkConf()
  .setMaster("local[2]")
  .setAppName("TheApp")
val spark = SparkSession.builder()
  .config(conf)
  .getOrCreate()

val sc = spark.sparkContext

val rowArray: Array[Row] = Array(
  Row(1,"hello", true),
  Row(2,"goodbye", false)
)

val rows: RDD[Row] = sc.parallelize(rowArray)

println(rows.count())

为什么这会导致异常。我错过什么了吗?
使用的依赖项:

val spark2Version ="2.2.1"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % spark2Version,
  "org.apache.spark" %% "spark-sql" % spark2Version,
  "org.apache.spark" %% "spark-streaming" % spark2Version )
zqdjd7g9

zqdjd7g91#

可以在build.sbt中添加以下依赖项覆盖,请重试。

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.6.5"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.6.5"

相关问题