如何在docker中设置zookeeper datadir(图yml)

mmvthczy  于 2021-06-08  发布在  Kafka
关注(0)|答案(2)|浏览(601)

我已经在docker的fig.yml文件中配置了zookeeper和kafka容器。两个容器都很好。但是在发送了大量消息之后,我的应用程序/zk客户端挂起。在检查zookeeper日志时,我看到错误: Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers 我的fig.yml如下:

  1. zookeeper:
  2. image: wurstmeister/zookeeper
  3. ports:
  4. - "2181:2181"
  5. environment:
  6. ZK_ADVERTISED_HOST_NAME: xx.xx.x.xxx
  7. ZK_CONNECTION_TIMEOUT_MS: 6000
  8. ZK_SYNC_TIME_MS: 2000
  9. ZK_DATADIR: /path/to/data/zk/data/dir
  10. kafka:
  11. image: wurstmeister/kafka:0.8.2.0
  12. ports:
  13. - "xx.xx.x.xxx:9092:9092"
  14. links:
  15. - zookeeper:zk
  16. environment:
  17. KAFKA_ADVERTISED_HOST_NAME: xx.xx.x.xxx
  18. KAFKA_LOG_DIRS: /home/svc_cis4/dl
  19. volumes:
  20. - /var/run/docker.sock:/var/run/docker.sock

我已经找了很长时间了,但还没有找到解决办法。我也尝试过使用zk\u datadir在fig.yml中设置数据目录:'/path/to/zk/data/dir',但似乎没有帮助。如有任何帮助,我们将不胜感激。
更新
的内容 /opt/kafka_2.10-0.8.2.0/config/server.properties :

  1. broker.id=0
  2. port=9092
  3. num.network.threads=3
  4. num.io.threads=8
  5. socket.send.buffer.bytes=102400
  6. socket.receive.buffer.bytes=102400
  7. socket.request.max.bytes=104857600
  8. num.partitions=1
  9. num.recovery.threads.per.data.dir=1
  10. log.retention.hours=168
  11. log.segment.bytes=1073741824
  12. log.retention.check.interval.ms=300000
  13. log.cleaner.enable=false
  14. zookeeper.connect=localhost:2181
  15. zookeeper.connection.timeout.ms=6000
o3imoua4

o3imoua41#

在过去的两天里,我一直在使用的配置没有任何问题,包括为zookeeper和kafka指定主机地址。我的 fig.yml 内容是:

  1. zookeeper:
  2. image: wurstmeister/zookeeper
  3. ports:
  4. - "xx.xx.x.xxx:2181:2181"
  5. kafka:
  6. image: wurstmeister/kafka:0.8.2.0
  7. ports:
  8. - "9092:9092"
  9. links:
  10. - zookeeper:zk
  11. environment:
  12. KAFKA_ADVERTISED_HOST_NAME: xx.xx.x.xxx
  13. KAFKA_NUM_REPLICA_FETCHERS: 4
  14. ...other env variables...
  15. volumes:
  16. - /var/run/docker.sock:/var/run/docker.sock
  17. validator:
  18. build: .
  19. volumes:
  20. - .:/host
  21. entrypoint: /bin/bash
  22. command: -c 'java -jar /host/app1.jar'
  23. links:
  24. - zookeeper:zk
  25. - kafka
  26. analytics:
  27. build: .
  28. volumes:
  29. - .:/host
  30. entrypoint: /bin/bash
  31. command: -c 'java -jar /host/app2.jar'
  32. links:
  33. - zookeeper:zk
  34. - kafka
  35. loader:
  36. build: .
  37. volumes:
  38. - .:/host
  39. entrypoint: /bin/bash
  40. command: -c 'java -jar /host/app3.jar'
  41. links:
  42. - zookeeper:zk
  43. - kafka

以及随行人员 Dockerfile 内容:

  1. FROM ubuntu:trusty
  2. MAINTAINER Wurstmeister
  3. RUN apt-get update; apt-get install -y unzip openjdk-7-jdk wget git docker.io
  4. RUN wget -q http://apache.mirrors.lucidnetworks.net/kafka/0.8.2.0/kafka_2.10-0.8.2.0.tgz -O /tmp/kafka_2.10-0.8.2.0.tgz
  5. RUN tar xfz /tmp/kafka_2.10-0.8.2.0.tgz -C /opt
  6. VOLUME ["/kafka"]
  7. ENV KAFKA_HOME /opt/kafka_2.10-0.8.2.0
  8. ADD start-kafka.sh /usr/bin/start-kafka.sh
  9. ADD broker-list.sh /usr/bin/broker-list.sh
  10. CMD start-kafka.sh
展开查看全部
uemypmqf

uemypmqf2#

您遇到的问题与zookeeper的数据目录无关。错误 Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers 是因为您的应用程序在zookeeper的数据中找不到任何代理znode。发生这种情况可能是因为kafka容器没有正确连接zookeeper,而查看wurstmeister的图片,我认为问题可能与变量有关 KAFKA_ADVERTISED_HOST_NAME 可能是错的。我不知道是否有理由通过必须传递的env变量来分配该变量,但从我的观点来看,这不是一个好方法。有多种方法可以配置Kafka(实际上不需要设置 advertised.host.name 您可以对其进行注解,kafka将采用默认主机名(可以使用docker设置),但使用此选项的快速解决方案是编辑 start-kafka.sh 重建图像:

  1. # !/bin/bash
  2. if [[ -z "$KAFKA_ADVERTISED_PORT" ]]; then
  3. export KAFKA_ADVERTISED_PORT=$(docker port `hostname` 9092 | sed -r "s/.*:(.*)/\1/g")
  4. fi
  5. if [[ -z "$KAFKA_BROKER_ID" ]]; then
  6. export KAFKA_BROKER_ID=$KAFKA_ADVERTISED_PORT
  7. fi
  8. if [[ -z "$KAFKA_LOG_DIRS" ]]; then
  9. export KAFKA_LOG_DIRS="/kafka/kafka-logs-$KAFKA_BROKER_ID"
  10. fi
  11. if [[ -z "$KAFKA_ZOOKEEPER_CONNECT" ]]; then
  12. export KAFKA_ZOOKEEPER_CONNECT=$(env | grep ZK.*PORT_2181_TCP= | sed -e 's|.*tcp://||' | paste -sd ,)
  13. fi
  14. if [[ -n "$KAFKA_HEAP_OPTS" ]]; then
  15. sed -r -i "s/^(export KAFKA_HEAP_OPTS)=\"(.*)\"/\1=\"$KAFKA_HEAP_OPTS\"/g" $KAFKA_HOME/bin/kafka-server-start.sh
  16. unset KAFKA_HEAP_OPTS
  17. fi
  18. for VAR in `env`
  19. do
  20. if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then
  21. kafka_name=`echo "$VAR" | sed -r "s/KAFKA_(.*)=.*/\1/g" | tr '[:upper:]' '[:lower:]' | tr _ .`
  22. env_var=`echo "$VAR" | sed -r "s/(.*)=.*/\1/g"`
  23. if egrep -q "(^|^#)$kafka_name=" $KAFKA_HOME/config/server.properties; then
  24. sed -r -i "s@(^|^#)($kafka_name)=(.*)@\2=${!env_var}@g" $KAFKA_HOME/config/server.properties #note that no config values may contain an '@' char
  25. else
  26. echo "$kafka_name=${!env_var}" >> $KAFKA_HOME/config/server.properties
  27. fi
  28. fi
  29. done
  30. ### NEW###
  31. IP=$(hostname --ip-address)
  32. sed -i -e "s/^advertised.host.name.*/advertised.host.name=$IP/" $KAFKA_HOME/config/server.properties
  33. ### END###
  34. $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties

如果这不能解决您的问题,您可以在容器内启动会话(即: docker exec -it kafkadocker_kafka_1 /bin/bash 为了Kafka和 docker exec -it kafkadocker_zookeeper_1 /bin/bash 对于zookeeper的),并检查Kafka日志,或zookeeper控制台( /opt/zookeeper-3.4.6/bin/zkCli.sh )

展开查看全部

相关问题