我正在eclipse中编写一个简单的kafka-spark流式代码,使用spark流式处理来自kafka代理的消息。下面是代码,我在尝试从eclipse运行代码时收到错误。
我还确保了依赖jar的位置,请帮助摆脱这个错误
对象SparkKafka流{
def main(args: Array[String]) {
val conf = new SparkConf()
.setAppName("The swankiest Spark app ever")
.setMaster("local[*]")
val ssc = new StreamingContext(conf, Seconds(60))
ssc.checkpoint("C:\\keerthi\\software\\eclipse-jee-mars-2-win32- x86_64\\eclipse")
println("Parameters:" + "zkorum:" + "group:" + "topicMap:"+"number of threads:")
val zk = "xxxxxxxx:2181"
val group = "test-consumer-group"
val topics = "my-replicated-topic"
val numThreads = 2
val topicMap = topics.split(",").map((_,numThreads.toInt)).toMap
val lines = KafkaUtils.createStream(ssc,zk,group,topicMap).map(_._2)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x,1L)).count()
println("wordCounts:"+wordCounts)
//wordCounts.print
}
}
例外情况:
线程“main”java.lang.noclassdeffounderror中出现异常:org/apache/spark/streaming/kafka/kafkautils$位于org.firststream.spark\u kakfa.spark\u kafka\u streaming$.main(spark\u kafka\u streaming)。scala:30)在org.firststream.spark\u kakfa.spark\u kafka\u streaming.main(spark\u kafka\u streaming.scala)上,由以下原因引起:java.lang.classnotfoundexception:org.apache.spark.streaming.kafka.kafkautils$位于java.net.urlclassloader.findclass(未知源)位于java.lang.classloader.loadclass(未知源)位于sun.misc.launcher$appclassloader.loadclass(未知源)位于java.lang.classloader.loadclass(未知源)。。。2个以上
依赖项:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.1.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>jmxri</artifactId>
<groupId>com.sun.jmx</groupId>
</exclusion>
<exclusion>
<artifactId>jms</artifactId>
<groupId>javax.jms</groupId>
</exclusion>
<exclusion>
<artifactId>jmxtools</artifactId>
<groupId>com.sun.jdmk</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.8.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.2.0</version>
</dependency>
1条答案
按热度按时间am46iovg1#
我评论了下面的依赖项。通过点击buildpath->configure build path->external jars,直接在eclpise中的引用库中添加了spark-streaming-kafka\u 2.10和kafka\u 2.10-0.8.1.1 jar。这就解决了问题。