我们把Kafka的版本从0.9升级到了2.0
需要帮助为spark结构化流媒体找到合适的客户端库吗
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"不起作用。
这是它抛出的错误： 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Completed connection to node -1. Fetching API versions. 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Initiating API versions fetch from node -1. 11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected java.io.EOFException: null at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:560)
1条答案
按热度按时间bnlyeluc1#
请提供更多信息（错误情况如何？！）。kafka代理被设计成向后兼容的，这有助于大大落后于最新api（比如spark）的项目。
你引用的artifactid
spark-streaming-kafka-0-10对于spark流媒体，为了使用spark结构化流媒体，您将需要使用spark-sql-kafka-0-10_2.11