如何在docker compose产生的kafka容器上安装kafkacat

ukxgm1gy  于 2021-06-05  发布在  Kafka
关注(0)|答案(2)|浏览(706)

我的上下文是我正在尝试创建一个docker compose,它将启动几个容器来运行elk+filebeats和3个kafka容器。整个docker compose及其yml配置文件可以在github docker compose中找到
简单地说,这是我的docker compose部件下载和启动我的kafka容器之一:

  1. kafka1:
  2. image: wurstmeister/kafka
  3. command: [start-kafka.sh]
  4. depends_on:
  5. - zoo1
  6. - zoo2
  7. - zoo3
  8. links:
  9. - zoo1
  10. - zoo2
  11. - zoo3
  12. ports:
  13. - "9092:9092"
  14. environment:
  15. KAFKA_LISTENERS: PLAINTEXT://:9092
  16. KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
  17. KAFKA_BROKER_ID: 1
  18. KAFKA_ADVERTISED_PORT: 9092
  19. KAFKA_LOG_RETENTION_HOURS: "168"
  20. KAFKA_LOG_RETENTION_BYTES: "100000000"
  21. KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
  22. KAFKA_CREATE_TOPICS: "log:3:3"
  23. KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

我想用Kafka调查Kafka

  1. C:\Dockers\megalog-try-1>docker exec -it megalog-try-1_kafka1_1 bash
  2. bash-4.4# kafka-topics.sh --list --bootstrap-server localhost:9092
  3. __consumer_offsets
  4. log
  5. bash-4.4# kafkacat -b megalog-try-1_kafka1_1:9092 L
  6. bash: kafkacat: command not found
  7. bash-4.4# apt-get install kafkacat
  8. bash: apt-get: command not found
  9. bash-4.4# exit
  10. exit

编辑
安装kafkacat后的日志

  1. / # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
  2. Offset: %o\n--\n'
  3. /bin/sh: kafkacat: not found
  4. / # apk --no-cache add ca-certificates wget
  5. fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
  6. fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
  7. (1/1) Installing wget (1.20.3-r0)
  8. Executing busybox-1.29.3-r10.trigger
  9. OK: 359 MiB in 76 packages
  10. / # wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
  11. / # wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
  12. --2020-02-05 22:56:46-- https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
  13. Resolving github.com... 18.228.52.138
  14. Connecting to github.com|18.228.52.138|:443... connected.
  15. HTTP request sent, awaiting response... 302 Found
  16. Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive [following]
  17. --2020-02-05 22:56:46-- https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive
  18. Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.236.203
  19. Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.236.203|:443... connected.
  20. HTTP request sent, awaiting response... 200 OK
  21. Length: 22602 (22K) [application/vnd.android.package-archive]
  22. Saving to: 'kafkacat-1.5.0-r0.apk'
  23. kafkacat-1.5.0-r0.apk 100%[=================================================>] 22.07K --.-KB/s in 0.1s
  24. 2020-02-05 22:56:48 (167 KB/s) - 'kafkacat-1.5.0-r0.apk' saved [22602/22602]
  25. / # apk add --no-cache kafkacat-1.5.0-r0.apk
  26. fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
  27. fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
  28. (1/6) Installing lz4-libs (1.8.3-r2)
  29. (2/6) Installing db (5.3.28-r1)
  30. (3/6) Installing libsasl (2.1.27-r2)
  31. (4/6) Installing librdkafka (0.11.6-r1)
  32. (5/6) Installing yajl (2.1.0-r0)
  33. (6/6) Installing kafkacat (1.5.0-r0)
  34. Executing busybox-1.29.3-r10.trigger
  35. OK: 361 MiB in 82 packages
  36. / # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
  37. Offset: %o\n--\n'
  38. Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
  39. / # kafkacat -b megalog-try-1_kafka1_1:9092 L
  40. Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
  41. / # kafka-topics.sh --list --bootstrap-server localhost:9092
  42. __consumer_offsets
  43. log
  44. / #

根据cricket的建议编辑
docker-compose.yml(仅编辑行)

  1. ...
  2. image: confluentinc/cp-kafka:latest
  3. ...
  4. image: confluentinc/cp-zookeeper:latest
  5. ...

相关日志
Kafka:

  1. ===> Configuring ...
  2. advertised.port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.

Zookeeper:

  1. ZOOKEEPER_CLIENT_PORT is required.
  2. Command [/usr/local/bin/dub ensure ZOOKEEPER_CLIENT_PORT] FAILED !
sirbozc5

sirbozc51#

您必须使用apk在这个kafka图像容器上安装任何包。
此软件包的当前安装方法是使用wget或curl将其拉入并使用apk安装本地文件:

  1. apk --no-cache add ca-certificates wget
  2. wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
  3. wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
  4. apk add --no-cache kafkacat-1.5.0-r0.apk
ippsafx7

ippsafx72#

请改用Kafka听众
只需删除端口变量。你已经有这个了 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092 需要zookeeper\客户端\端口
添加它 ZOOKEEPER_CLIENT_PORT: 2181 你可以混合和匹配容器。。。我只是建议您在现有设置中使用现有的kafkacat容器,它可以连接到任何远程代理。
或者在本地系统上安装kafkacat

相关问题