我是新来Kafka和搜索不同的职位在论坛上,但找不到解决办法。我已经在一个ec2示例上安装了kafka,并试图从我的ubuntu本地机器上连接它。我的目标是让python-kafka客户机(生产者和消费者)在我的本地机器上运行,并通过ec2-kafka示例发送/接收数据。有可能吗?
在server.properties配置文件中设置的属性:
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://<ec2-public-DNS>:9092
Kafkaec2示例:
netstat -an | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::9092 :::* LISTEN
在kafka ec2示例上的zookeeper cli上:
get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://<ec2-public-DNS>:9092"],"jmx_port":-1,"host":"<ec2-public-DNS>","timestamp":"1492900361516","port":9092,"version":4}
cZxid = 0xed
ctime = Sat Apr 22 22:32:41 UTC 2017
mZxid = 0xed
mtime = Sat Apr 22 22:32:41 UTC 2017
pZxid = 0xed
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15b97cb9d060000
dataLength = 250
numChildren = 0
本地计算机上的python客户端(生产者):
from kafka import KafkaProducer
import time
import json
producer = KafkaProducer(bootstrap_servers="<ec2-public-DNS>:9092")
for i in range(100):
dict = {}
dict['name_'+str(i)] = 'FILE_' + str(i)
dict['size_'+str(i)] = '23.' + str(i)
dict['host_'+str(i)] = '10.0.0.0' + str(i)
jd = json.dumps(dict)
producer.send('console-test-topic', jd)
time.sleep(2)
本地计算机上的python客户端(使用者):
from kafka import KafkaConsumer
consumer = KafkaConsumer('console-test-topic', bootstrap_servers="<ec2-public-DNS>:9092")
for msg in consumer:
print (msg)
但是,生产者无法连接到kafka ec2示例,失败,错误如下:
**kafka.errors.NoBrokersAvailable: NoBrokersAvailable**
请参阅我的安全组规则的链接:
goo.gl/zuvknv
在本地计算机上以调试模式运行producer:
DEBUG:kafka.producer.kafka:Starting the Kafka producer
DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
INFO:kafka.client:Bootstrapping cluster metadata from [('ec2-54-91-87-14.compute-1.amazonaws.com', 9092, 0)]
DEBUG:kafka.client:Attempting to bootstrap via node at ec2-54-91-87-14.compute-1.amazonaws.com:9092
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap.latency
DEBUG:kafka.client:Node bootstrap connected
DEBUG:kafka.cluster:Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 2, groups: 0)
INFO:kafka.client:Bootstrap succeeded: found 1 brokers and 2 topics.
DEBUG:kafka.client:Initiating connection to node 0 at ec2-54-91-87-14.compute-1.amazonaws.com:9092
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-0.latency
INFO:kafka.producer.kafka:Kafka producer closed
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/kafka/producer/kafka.py", line 335, in __init__
**self.config)
File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 210, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 828, in check_version
raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
我尝试在另一个ec2示例(与kafka示例在同一个vpn中)中运行producer client,效果很好。但是,当producer在我的本地机器上运行时,它就不工作了。“adverted.listeners”属性是否在同一个(awsvpn)网络中为kafka代理做广告?或者我也可以从我的本地机器连接它?如果有人能给我指出正确的方向,请告诉我。
2条答案
按热度按时间0s0u357o1#
你有没有疏忽开入境港
9092
在您的aws安全组中连接到本地网络的外部ip?把它打开0.0.0.0/0
如果您选择向所有人开放(但要注意安全性)。ia2d9nvy2#
几个月前,我经历了一个类似的例子,基本上我在n。virginia和我在本地机器上配置了topbeat,以便向那个ec2示例发送度量。我通过添加
作为kafka的server.properties中的配置,但根据文档,此属性已被弃用。
阅读进一步的文档,据说如果您在iaas环境中,则必须配置与代理绑定的接口不同的advised.listeners。