docker中的kafka服务器问题

quhf5bfb 于 2021-07-12 发布在 Spark

关注(0)|答案(2)|浏览(703)

我正在使用docker在windows机器上执行我的示例spark+kafka项目。我正面临

WARN ClientUtils: Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
[error] (run-main-0) org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
[error] org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
------
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers

下面是我的docker-compose.yml

version: '2'
services:
    test1:
        build: test1service/.
        depends_on:
            - kafka
    test2:
        build: test2/.
        depends_on:
            - kafka
            - test1
    zookeeper:
        image: confluentinc/cp-zookeeper:5.1.0
        ports:
         - 2181:2181
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        extra_hosts:
          - "localhost: 127.0.0.1"
    kafka:
        image: confluentinc/cp-kafka:5.1.0
        ports:
         - 9092:9092
        depends_on:
          - zookeeper
        environment:
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          #KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
          KAFKA_DELETE_TOPIC_ENABLE: "true"
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        extra_hosts:
          - "localhost: 127.0.0.1"

下面是我在test2服务中的示例代码

val inputStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9092")
                           .option("subscribe", "test1")
                           .option("startingOffsets", "earliest")
                           .load()

docker ps命令输出为

CONTAINER ID   IMAGE                             COMMAND                  CREATED             STATUS          PORTS                                        NAMES
4c8feb49e12b   test1                             "/usr/bin/start.sh"      About an hour ago   Up 50 minutes                                                test1
4535ce246541   test2                             "/usr/bin/myservice-…"   About an hour ago   Up 50 minutes                                                test2
733766f72adb   confluentinc/cp-kafka:5.1.0       "/etc/confluent/dock…"   About an hour ago   Up 51 minutes   0.0.0.0:9092->9092/tcp       kafka_1
d915e25cb226   confluentinc/cp-zookeeper:5.1.0   "/etc/confluent/dock…"   About an hour ago   Up 51 minutes   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zookeeper_1

有没有人遇到过类似的问题，你是如何解决的？

apache-kafka apache-spark docker

来源：https://stackoverflow.com/questions/66590997/kafka-server-issue-in-docker

2条答案

按热度按时间

rhfm7lfc1#

你试着去接触kafka:9092，但docker compose生成了容器kafka_1，这就是没有名称解析的原因。
docker为容器提供内部ip，并使用容器名称在此网络上创建内部dns（使用embbed dns服务器）
docker compose不会更改您的环境变量，以使其适合容器的名称。
您应该在容器描述中使用“容器名称：kafka”来获取静态容器名称。

赞(0）回复(0）举报 2021-07-12

8gsdolmq2#

你唯一需要的就是网络。之后，所有服务都将连接起来，您可以使用kafka:9092.

version: '2'
services:
    test1:
        build: test1service/.
        depends_on:
            - kafka
        networks:
          - app-network
    test2:
        build: test2/.
        depends_on:
            - kafka
            - test1
        networks:
          - app-network
    zookeeper:
        image: confluentinc/cp-zookeeper:5.1.0
        ports:
         - 2181:2181
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        extra_hosts:
          - "localhost: 127.0.0.1"
        networks:
          - app-network
    kafka:
        image: confluentinc/cp-kafka:5.1.0
        ports:
         - 9092:9092
        depends_on:
          - zookeeper
        environment:
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          #KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
          KAFKA_DELETE_TOPIC_ENABLE: "true"
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        extra_hosts:
          - "localhost: 127.0.0.1"
        networks:
          - app-network
networks:
  app-network:
    driver: bridge

赞(0）回复(0）举报 2021-07-12

我来回答

docker中的kafka服务器问题

2条答案

相关问题

热门标签

最新问答