spark streaming kafka java.lang.classnotfoundexception:org.apache.kafka.common.serialization.stringdeserializer

ryoqjall  于 2021-06-08  发布在  Kafka
关注(0)|答案(1)|浏览(514)

我使用的是带有kafka集成的spark流媒体,当我在本地模式下从ide运行流媒体应用程序时,一切都很有魅力。但是,一旦我将其提交到集群,我就会出现以下错误:
java.lang.classnotfoundexception:org.apache.kafka.common.serialization.stringdeserializer
我正在使用sbt汇编来构建我的项目。
我的sbt如下:

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" % Provided,
  "org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided,
  "org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided,
  "org.marc4j" % "marc4j" % "2.8.2",
  "net.sf.saxon" % "Saxon-HE" % "9.7.0-20"
)

run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated

mainClass in assembly := Some("EstimatorStreamingApp")

我还尝试使用--package选项
尝试1

--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0

尝试2

--packages org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:2.2.0

都没有成功。有人有什么建议吗

ltskdhd1

ltskdhd11#

您需要从kafka依赖项中删除“provided”标志,因为它是一个不提供spark的ootb依赖项:

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0",
  "org.apache.spark" % "spark-core_2.11" % "2.2.0" % Provided,
  "org.apache.spark" % "spark-streaming_2.11" % "2.2.0" % Provided,
  "org.marc4j" % "marc4j" % "2.8.2",
  "net.sf.saxon" % "Saxon-HE" % "9.7.0-20"
)

相关问题