连接Kafka主题时 imageinstring
与 spark.readStream
我遇到了这个错误:
Py4JJavaError: An error occurred while calling o49.load.
: java.lang.NoClassDefFoundError:
org/apache/spark/sql/sources/v2/reader/SupportsScanUnsafeRow
我正在使用:
Kafka2.11-0.8.2.2
spark-2.4.5-bin-hadoop2.7版本
具有以下依赖关系:
org.apache.spark:spark-streaming-kafka-0-8_2.11:2.4.5
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.5
这是我的密码:
import findspark
findspark.init()
# Spark
from pyspark.sql import SparkSession
# Spark Streaming
from pyspark.streaming import StreamingContext
# Kafka
from pyspark.streaming.kafka import KafkaUtils
# json parsing
import json
spark = SparkSession.builder.appName("StructuredNetwork").getOrCreate()
df=spark.readStream.format("kafka")\
.option("kafka.bootstrap.servers","localhost:9092")\
.option("subscribe","imageinstring").load()
query = df.writeStream.outputMode("complete").format("console").start()
query.awaitTermination()
暂无答案!
目前还没有任何答案,快来回答吧!