我有一个Spark流+Kafka的例子。它在ide中运行良好。但是当我试图从控制台用sbt编译它时,就像sbt compile一样。有个错误。
主要类别:
val conf = new SparkConf().setMaster("local[*]").setAppName("KafkaReceiver")
val ssc = new StreamingContext(conf, Seconds(5))
val kafkaStream1 = KafkaUtils.createStream(ssc, "localhost:2181", "spark-streaming-consumer-group", Map("t1" -> 5))
//val kafkaStream2 = KafkaUtils.createStream(ssc, "localhost:2181", "spark-streaming-consumer-group", Map("topic2" -> 5))
//kafkaStream.fla
kafkaStream1.print()
ssc.start()
ssc.awaitTermination()
错误消息:
[error] bad symbolic reference. A signature in package.class refers to type compileTimeOnly
[error] in package scala.annotation which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling package.class.
Reference to method any2ArrowAssoc in object Predef should not have survived past type checking,
[error] it should have been processed and eliminated during expansion of an enclosing macro.
[error] val kafkaStream1 = KafkaUtils.createStream(ssc, "localhost:2181", "spark-streaming-consumer-group", Map("t1" -> 5))
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
sbt公司:
name := "test"
val sparkVersion = "2.0.0"
lazy val commonSettings = Seq(
organization := "com.test",
version := "1.0",
scalaVersion := "2.11.8",
test in assembly := {}
)
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-streaming_2.11" % sparkVersion,
"org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVersion
)
你有办法修理它吗?
1条答案
按热度按时间dfuffjeb1#
你能分享你的build.sbt吗。可能导致“错误符号引用”问题的原因之一是scala版本不匹配。有关这个问题的更多细节,请看这个主题。另外,请确保您使用的scala版本与spark预期的版本相同,请看这个博客文章以获取更多细节