on-rails-kafka连接问题

a0zr77ik  于 2021-06-06  发布在  Kafka
关注(0)|答案(1)|浏览(371)

我在docker swarm上安装了Kafka，但我一直面临以下问题。
有人能解释一下问题是什么吗？我在这里错过了什么？

fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    | bundler: failed to load command: bin/subscriber_kafka (bin/subscriber_kafka)
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | I, [2018-10-21T07:37:41.204851 #1]  INFO -- : Leaving group `accounts_service.account`
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.204879 #1] DEBUG -- : Getting group coordinator for `accounts_service.account`
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/cluster.rb:121:in `get_group_coordinator'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.204927 #1] DEBUG -- : Opening connection to ip-10-0-1-109.ap-southeast-1.compute.internal:9094 with client id ruby-kafka...
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:41.204555 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:190:in `coordinator'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206519 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:41.206599 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:131:in `block in join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | I, [2018-10-21T07:37:41.206633 #1]  INFO -- : Leaving group `accounts_service.customer`
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206660 #1] DEBUG -- : Getting group coordinator for `accounts_service.customer`
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206717 #1] DEBUG -- : Opening connection to ip-10-0-1-109.ap-southeast-1.compute.internal:9094 with client id ruby-kafka...
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222250 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:21:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222428 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:35:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:130:in `join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222955 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208336 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:47:in `join'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.208412 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208545 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:443:in `join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208265 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:506:in `fetch_batches'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.208815 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208867 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:206:in `block in each_message'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:394:in `block in consumer_loop'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.209073 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.209110 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:21:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.209170 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:35:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | bundler: failed to load command: bin/subscriber_kafka (bin/subscriber_kafka)
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:393:in `consumer_loop'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:205:in `each_message'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools/consumer.rb:65:in `listen_messages'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools/consumer.rb:20:in `run'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools.rb:26:in `block (2 levels) in consume'

我的 Docker

services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    networks:
      - fastlane
    deploy:
      mode: global

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: ${KAFKA_HOSTNAME_COMMAND}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      BROKER_ID_COMMAND: ${BROKER_ID_COMMAND}
    networks:
      - fastlane
    depends_on:
      - zookeeper

  kafka_manager:
    image: deric/kafka-manager:1.3.3.18
    networks:
      - fastlane
    environment:
      ZK_HOSTS: zookeeper:2181
      APPLICATION_SECRET: letmein
    deploy:
      labels:
        - traefik.backend.loadbalancer.swarm=true
        - traefik.backend=kafka-manager
        - traefik.frontend.rule=Host:kafka-manager.${DOMAIN}
        - traefik.port=9000
        - traefik.docker.network=fastlane
    depends_on:
      - zookeeper
KAFKA_HOSTNAME_COMMAND = curl http://169.254.169.254/latest/meta-data/local-hostname
BROKER_ID_COMMAND="curl http://169.254.169.254/latest/meta-data/local-ipv4 | awk -F '.' '{print \$4}'"
apache-kafkaruby-on-railsamazon-web-servicesamazon-ec2

1条答案

按热度按时间
8ulbf1ek

8ulbf1ek1#

你看过自述了吗？它特别提到了在aws中运行需要做什么
如果 KAFKA_ADVERTISED_HOST_NAME 指定时，它优先于 HOSTNAME_COMMAND 对于aws部署，可以使用元数据服务获取容器主机的ip: HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4 参考文献：http://docs.aws.amazon.com/awsec2/latest/userguide/ec2-instance-metadata.html
所以，首先，先尝试一下文档中所说的，然后去掉等号周围的空格 KAFKA_HOSTNAME_COMMAND = 如果您在本地运行ruby代码，而不是在aws网络中运行，则需要在aws控制台的vpc设置中打开kafka端口后，获取上述命令以返回外部地址。
我想提到的是，如果您的ruby应用程序（a）也是compose文件的一部分，那么连接到kafka示例会更容易。例如，使用覆盖网络或træ菲克，你可以进入 kafka:9094 直接服务。

赞(0）分享  回复(0）举报  2021-06-06
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com