sbt与spark core 3.0.1异常noclassdeffound for org/apache/log4j/logger

ecr0jaav  于 2021-05-18  发布在  Spark
关注(0)|答案(0)|浏览(258)

我正在使用 Log4jScala 2.12.12 以及 Spark-Core 3.0.1 但是,当我将库依赖项更改为不将spark core打包到胖jar中时,在尝试运行它时会出现以下错误:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Logger
    at com.some.package.name.Utils$.setup(Utils.scala:207)
    at com.some.package.name.Main$.main(Main.scala:9)
    at com.some.package.name.Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 3 more

编译成功,如果我删除 provided 从dependencies行中删除子句一切正常。我的 build.sbt 具体如下:

scalaVersion := "2.12.12"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "3.0.1" % "provided",
"org.apache.logging.log4j" % "log4j-api" % "2.13.3",
"org.apache.logging.log4j" % "log4j-core" % "2.13.3",
"org.scalatest" %% "scalatest" % "3.2.0" % "test",
"com.holdenkarau" %% "spark-testing-base" % "3.0.1_1.0.0" % Test)

如果我删除写入记录器的代码 SparkContext 表示为产生错误的行。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题