从我的本地计算机连接在ec2计算机上运行的kafka

agxfikkp  于 2021-06-07  发布在  Kafka
关注(0)|答案(2)|浏览(556)

我是新来Kafka和搜索不同的职位在论坛上,但找不到解决办法。我已经在一个ec2示例上安装了kafka,并试图从我的ubuntu本地机器上连接它。我的目标是让python-kafka客户机(生产者和消费者)在我的本地机器上运行,并通过ec2-kafka示例发送/接收数据。有可能吗?
在server.properties配置文件中设置的属性:

listeners=PLAINTEXT://0.0.0.0:9092 
advertised.listeners=PLAINTEXT://<ec2-public-DNS>:9092

Kafkaec2示例:

netstat -an | grep LISTEN 
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 
tcp6 0 0 :::9092 :::* LISTEN

在kafka ec2示例上的zookeeper cli上:

get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://<ec2-public-DNS>:9092"],"jmx_port":-1,"host":"<ec2-public-DNS>","timestamp":"1492900361516","port":9092,"version":4}
cZxid = 0xed
ctime = Sat Apr 22 22:32:41 UTC 2017
mZxid = 0xed
mtime = Sat Apr 22 22:32:41 UTC 2017
pZxid = 0xed
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15b97cb9d060000
dataLength = 250
numChildren = 0

本地计算机上的python客户端(生产者):

from kafka import KafkaProducer
import time
import json

producer = KafkaProducer(bootstrap_servers="<ec2-public-DNS>:9092")
for i in range(100):
    dict = {}
    dict['name_'+str(i)] = 'FILE_' + str(i)
    dict['size_'+str(i)] = '23.' + str(i)
    dict['host_'+str(i)] = '10.0.0.0' + str(i)
    jd = json.dumps(dict)
    producer.send('console-test-topic', jd)
    time.sleep(2)

本地计算机上的python客户端(使用者):

from kafka import KafkaConsumer

consumer = KafkaConsumer('console-test-topic', bootstrap_servers="<ec2-public-DNS>:9092")
for msg in consumer:
    print (msg)

但是,生产者无法连接到kafka ec2示例,失败,错误如下:


**kafka.errors.NoBrokersAvailable: NoBrokersAvailable**

请参阅我的安全组规则的链接:
goo.gl/zuvknv
在本地计算机上以调试模式运行producer:

DEBUG:kafka.producer.kafka:Starting the Kafka producer
DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
INFO:kafka.client:Bootstrapping cluster metadata from [('ec2-54-91-87-14.compute-1.amazonaws.com', 9092, 0)]
DEBUG:kafka.client:Attempting to bootstrap via node at ec2-54-91-87-14.compute-1.amazonaws.com:9092
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.latency
DEBUG:kafka.client:Node bootstrap connected
DEBUG:kafka.cluster:Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 2, groups: 0)
INFO:kafka.client:Bootstrap succeeded: found 1 brokers and 2 topics.
DEBUG:kafka.client:Initiating connection to node 0 at ec2-54-91-87-14.compute-1.amazonaws.com:9092
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.latency
INFO:kafka.producer.kafka:Kafka producer closed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/kafka/producer/kafka.py", line 335, in __init__
  **self.config)
  File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 210, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 828, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

我尝试在另一个ec2示例(与kafka示例在同一个vpn中)中运行producer client,效果很好。但是,当producer在我的本地机器上运行时,它就不工作了。“adverted.listeners”属性是否在同一个(awsvpn)网络中为kafka代理做广告?或者我也可以从我的本地机器连接它?如果有人能给我指出正确的方向,请告诉我。

0s0u357o

0s0u357o1#

你有没有疏忽开入境港 9092 在您的aws安全组中连接到本地网络的外部ip?把它打开 0.0.0.0/0 如果您选择向所有人开放(但要注意安全性)。

ia2d9nvy

ia2d9nvy2#

几个月前,我经历了一个类似的例子,基本上我在n。virginia和我在本地机器上配置了topbeat,以便向那个ec2示例发送度量。我通过添加

advertised.host.name=public-ip

作为kafka的server.properties中的配置,但根据文档,此属性已被弃用。
阅读进一步的文档,据说如果您在iaas环境中,则必须配置与代理绑定的接口不同的advised.listeners。

相关问题