spark结构化流媒体与kafka

hfyxw5xn  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(410)

连接Kafka主题时 imageinstringspark.readStream 我遇到了这个错误:

Py4JJavaError: An error occurred while calling o49.load.
: java.lang.NoClassDefFoundError:    
org/apache/spark/sql/sources/v2/reader/SupportsScanUnsafeRow

我正在使用:
Kafka2.11-0.8.2.2
spark-2.4.5-bin-hadoop2.7版本
具有以下依赖关系:

org.apache.spark:spark-streaming-kafka-0-8_2.11:2.4.5
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.5

这是我的密码:

import findspark
findspark.init()

# Spark

from pyspark.sql import  SparkSession

# Spark Streaming

from pyspark.streaming import StreamingContext

# Kafka

from pyspark.streaming.kafka import KafkaUtils

# json parsing

import json

spark = SparkSession.builder.appName("StructuredNetwork").getOrCreate() 
df=spark.readStream.format("kafka")\
.option("kafka.bootstrap.servers","localhost:9092")\
.option("subscribe","imageinstring").load()
query = df.writeStream.outputMode("complete").format("console").start()
query.awaitTermination()

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题