每当我在dockercontainer获得spark工作时,我都会得到引导代理localhost:9092 (id:-1机架:空)已断开

9bfwbjaz  于 2021-05-18  发布在  Spark
关注(0)|答案(1)|浏览(772)

我得到的错误是:- 20/11/02 13:34:51 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-366ac503-c5a4-4338-869c-84786983aab3--188679505-driver-0-1, groupId=spark-kafka-source-366ac503-c5a4-4338-869c-84786983aab3--188679505-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected 我的docker-compose.yml文件

version: "3"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
    - 2181:2181

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
    - 9092:9092

    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "kafkatutorial:1:1"
      ALLOW_PLAINTEXT_LISTENER: "yes"
      SSL: localhost:9092
    expose:
     - 9092

  spark:
   build:
    dockerfile: DockerFileSpark
    context: .

   environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no

   ports:
      - '8080:8080'
   links:
    - kafka

   depends_on:
    - zookeeper
    - kafka
8qgya5xd

8qgya5xd1#

SSL: localhost:9092 不是kafka容器的有效环境变量;您需要在两个侦听器变量中设置ssl,并向映像中添加ssl证书
不清楚你的Spark代码在哪里运行,但是在一个容器里, localhost 无法连接到外部服务。拆下 links 在您的yaml中并连接到 kafka:9092 在代码中

相关问题